ipfs doesn't fully support fuse api

[wpfnihao]

Version information:

go-ipfs version: 0.4.10-
Repo version: 5
System version: amd64/linux
Golang version: go1.8.3

Type:

Medium

Severity:

Medium

Description:

We have encountered a weird situation while mounting and using ipfs as a local filesystem with FUSE.

1. Using ls –al, ipfs seems to know whether the hash linked file is a directory or a file, though the output is not exactly the same as the native filesystem;
2. While walking through the dir using python, the os.walk() function fails to differ file from directory when ipfs is used. However, the python script works successfully on native filesystem.

We have also tried to save linux roofs on ipfs. The results show that ipfs doesn't support symbol link and executable attribute.

So here is the question: does the partial support of fuse on purpose? do you have any further plan on fully support ipfs with fuse?

# the python script
 
 import os, time, sys
 import random
 
  def getFileList(path):
     fileList = []
     for parent, dirs, files in os.walk(path):
         for oneFile in files:
             fileList.append(os.path.join(parent, oneFile))
 
      return fileList
 
  
  def readFilesRandomly(path, readTimes):
     fileList = getFileList(path)
     for i in range(readTimes):
         fileName = random.choice(fileList)
         #print(fileName)
         with open(fileName, 'rb') as f:
             byte = f.read(1)
             #while byte != "":
             #    # Do stuff with byte.
             #    byte = f.read(1)
 
  
  def main():
     t = time.time()
 
     path = sys.argv[1]
     readTimes = int(sys.argv[2])
     readFilesRandomly(path, readTimes)
 
      t = time.time() - t
 
      print('Read files in the folder: ' + path + ' ' + sys.argv[2] + ' times, cost: ' + str(t))
   
  if __name__ == '__main__':
       ##main()

[whyrusleeping]

@wpfnihao We plan on improving fuse support, having a clear list of issues is definitely very helpful. From your post I see:

• potential issue with directory vs file in fd mode
• Symlinks not supported in fuse (but we have general symlink support in unixfs)
• Executable bit not preserved

Any other things that should be added?

[wpfnihao]

@whyrusleeping Thanks for your instant response. You have well summarized the issues we encountered, thank you.

[whyrusleeping]

@wpfnihao I'm putting together a PR (#4194) that should address the first issue listed above. And I verified that symlinks work (at least in my limited test case):

why@manwe /t/foobar> echo "beep boop" > file1
why@manwe /t/foobar> ln -s file1 file2
why@manwe /t/foobar> ls
file1  file2@
why@manwe /t/foobar> ipfs add -r .
added QmfMBdbXMmdjUEUc7cunTZxDVk3uVmtMakkCbdgqP71nn2 foobar/file1
added QmZXp6b49zUqAA4Z1rAKA62JitP3MYKTQxraXT7n9DTbaM foobar/file2
added Qmb4gs22XGa4K8GihpLtHCSnrjg4Prm18dpdZSLKGRx6sn foobar
why@manwe /t/foobar> ipfs mount
IPFS mounted at: /ipfs
IPNS mounted at: /ipns
why@manwe /t/foobar> cd /ipfs/Qmb4gs22XGa4K8GihpLtHCSnrjg4Prm18dpdZSLKGRx6sn
why@manwe /i/Qmb4gs22XGa4K8GihpLtHCSnrjg4Prm18dpdZSLKGRx6sn> ls
file1  file2@
why@manwe /i/Qmb4gs22XGa4K8GihpLtHCSnrjg4Prm18dpdZSLKGRx6sn> cat file2
beep boop

Could you provide a way to reproduce your symbolic link issue?

[whyrusleeping]

@wpfnihao we just merged some fuse fixes. Mind trying your script again with a build from latest master?

[wpfnihao]

@whyrusleeping sorry for the late reply. I have tested the script trying to randomly read files from a folder containing 1,000 sub-folders. IPFS works great now on the test folder. Thank you!

After that, I have also done the following experiment using the latest master branch:

1. add one 5GB file on node a
2. mount ipfs on node b
3. run cp <ipfs-mount-point>/<multi-hash> <path> on node b

I was expecting the behavior of this scenario might be similar to ipfs get. Unfortunately, IPFS+FUSE ran extremely slow. I had to abort the task after 10mins of waiting. Please see the figures below:

Performance of the Sender (node a)

Performance of the Receiver (node b)

[whyrusleeping]

@wpfnihao Thanks for that! we will take a look at those performance issues. Do you mind opening a new issue for that specfically?

[wpfnihao]

@whyrusleeping I have opened a new issue #4228.