I can open an S3 file in root, but not in a subdirectory

[davidparks21]

Originally posted here:
https://stackoverflow.com/questions/57227152/opening-a-file-in-pyfilesystems-s3-filesystem-fails

I'm able to open a file in the root of an S3 bucket without a problem, however I cannot open a file with a standard path prefix:

>>> s3fs.listdir('dfparks/test')
['test.txt']
>>> s3fs.open('dfparks/test/test.txt')

I get the error:

fs.errors.ResourceNotFound: resource 'dfparks/test/test.txt' not found

I've tried every variant of the path / s3 url I can think of but I always get the not found error.

[mrk-its]

Could you try to reproduce this using code from #60? (you can simply install fs-s3fs-ng package - it is the same as fs-s3fs==1.1.1 but with #60 and #61 applied)

[willmcgugan]

@davidparks21 Did you create the test directory with S3FS?

S3FS has some limitations, which are documented here.

[davidparks21]

I understand now, that does appear to be the issue, I hadn't caught that detail in the docs. This might be worth a big red flag in the docs.

@mrk-its your PR is brilliant, it solved my next problem too. I was doing a test to see if I could open a large file (3+ GB) and only read the first few bytes of its header. With fs_s3fs_ng it worked no problem, but with fs_s3fs with strict=False I was able to open the file it appears, but it started reading the entire file in.

I'll watch your PRs, you've got my vote for sure! :)