-1

I have a third party software that does deduplication of files, but I want it to think it copied the files to disk but without pulling it from the remote server because I already have the files on disk and want to just symlink to it.

Is there a way to make folder /x being some kind of fake write filesystem? Perhaps with fuse or something?

I want to turn a arbitrary folder where you can run software/scripts that writes to it, and it will return a successful write but don't actually write anything.

Is that possible? Example running touch file should run fine but nothing is written.

Freedo
  • 1,255
  • ptrace it, use a fuse or kernel module to implement a fake filesystem, seccomp it. The key point is make the write syscall return succeed but actually do nothing. – 炸鱼薯条德里克 Aug 01 '19 at 05:18
  • @炸鱼薯条德里克 can you expand on that? I'm not that good at linux, I'd love if you could answer this question <3 – Freedo Aug 01 '19 at 05:50
  • You might also need to "suppress" things like creating directories, doing chmod on the file and so. Are you sure your software does not have a "dummy run" mode that will provide output without trying to copy files? Alternatively, if you provide it a read-only directory will it just produce error messages but continue nonetheless so you can get the output. – meuh Aug 01 '19 at 06:05
  • If it can't write it'll just error out, I need it to "write" the files before I can use symlinks – Freedo Aug 01 '19 at 06:19
  • See also https://unix.stackexchange.com/questions/9332/how-can-i-create-a-dev-null-like-blackhole-directory The main answer is to something that only really supports creating files, it doesn't support creating directories if you need that. (If you click in the link, it has a variant that supports creating directories, but it says it crashes). – sourcejedi Aug 01 '19 at 20:14
  • The answer didn't work for, it failed with a syntax error for some reason...and the nullfs I tried it, but my software says it failed to copy the file - maybe it's because it makes the folder root and the software is not running as root or something else who knows....it doesn't seem to be possible to run nullfs as non-user or allow non-root users to write to it – Freedo Aug 02 '19 at 05:08
  • Just use tmpfs? Your requirements are not entirely clear to me. – frostschutz Aug 02 '19 at 08:10

1 Answers1

1

Using a FUSE filesystem (Filesystem in Userspace), you can write a program using the libfuse library to implement most file operations. You effectively mount a program on a directory, and any operations you do in the directory are passed via the kernel to the program which provides a reply. There are several Perl and Python packages with example programs that can be fairly simple to modify to your requirements.

In particular, for my Fedora 25 there is a fuse-python rpm that provides an example program xmp.py that "simply" duplicates every operation done under the mount point, to a real directory you specify at the start.

For example, if you run (not as root, but as an ordinary user)

mkdir -p /tmp/myfs/under /tmp/myfs/write
xmp.py /tmp/myfs/write -o root=/tmp/myfs/under

then everything you do to a file under the mount point /tmp/myfs/write, will also be seen in the real duplicate "root" directory /tmp/myfs/under. Eg

echo abc >/tmp/myfs/write/file1

creates file1 in the real directory. Of course, if you ls /tmp/myfs/*/file1 the file appears in both, since xmp.py shows you the real directory too:

-rw-r--r-- 1  4 Aug  1 18:43 /tmp/myfs/under/file1
-rw-r--r-- 1  4 Aug  1 18:43 /tmp/myfs/write/file1

If you look at the Python example code, it is quite short. You can see the implentation of the system call write() here:

    def write(self, buf, offset):
        self.file.seek(offset)
        self.file.write(buf)
        return len(buf)

To suppress the actual write to the file you only need to comment out the third line:

    def write(self, buf, offset):
        self.file.seek(offset)
        # self.file.write(buf)
        return len(buf)

Unmount the fuse filesystem with

fusermount -u /tmp/myfs/write

then run the new xmp.py and now when you write to a file, it appears to succeed, but no data is written to the underlying real directory. It is probably wise to not try stubbing out other file operations, such as creating directories and so on, as your program may see spurious problems.

Note, I used the xmp.py provided with the package I installed on my system. The links to the github code are just for browsing. You should have no problems repeating this experiment if you find and use the appropriate package for your system. Note, there are several Python libraries for FUSE. Make sure you install the one with the above example code.

Your login must be in group fuse to be able to use the program. You should see this listed in the output of command id. If the group isn't there add it with sudo usermod -a -G fuse $USER (where $USER is your login), and login again.

On Ubuntu the package is python-fuse. My tests in 18.04.2 LTS however did not work. The example xmp.py, unchanged, would create a file but not write into it, saying Invalid argument. It seems this is due to an incompatibility in the versions of this and libfuse. When I ran xmp.py with the extra debug option -o debug I saw the errors

AttributeError: 'XmpFile' object has no attribute 'direct_io'
AttributeError: 'XmpFile' object has no attribute 'keep_cache'

You may want to try other means to fix this, but I simply added the missing attributes to xmp.py by adding to class XmpFile, function def __init__, after the line self.fd = self.file.fileno() the following 2 lines:

self.direct_io = None
self.keep_cache = None

Make sure you keep exactly the same indentation for these lines, using spaces only, not tabs.

meuh
  • 51,383
  • Thanks so much <3 I'd never be able to do this, i didn't even know where to start...thanks so much – Freedo Aug 01 '19 at 18:29
  • For Ubuntu 18.04 I installed python-fuse package – Freedo Aug 02 '19 at 04:16
  • I'm seeing the error File "./xmp.py", line 268 print("can't enter root of underlying filesystem", file=sys.stderr) and yes I'm running as root in my terminal – Freedo Aug 02 '19 at 04:24
  • I found an Ubuntu 18.04 and tried the xmp.py program. Don't run this as root! I ran it without any changes, exactly as I did on my Fedora, but it didn't work. It created file1 but wouldl not write into it, giving the error Invalid argument. Note: your login must be in group fuse to be able to use the program. You should see this listed in the output of command id. If the group isnt there add it with sudo usermod -a -G fuse $USER (where $USER is your login), and login again. I'll try to see why it doesnt work. – meuh Aug 02 '19 at 07:56
  • Why I shouldn't run this as root? And ubuntu is the only OS supported :/ – Freedo Aug 02 '19 at 08:04
  • I said do not run this as user root. There is no need. I hope there is no confusion over the option -o root=... you must give to the program xmp.py. – meuh Aug 02 '19 at 08:06
  • I've updated my answer with the 2 lines of Python code you must add to xmp.py to get it to work on my Ubuntu 18.04 at least. I ran this as an ordinary user (which has group fuse in its list of groups). – meuh Aug 02 '19 at 10:09