Idea: filesystem-like API on top of sequential-storage

[diondokter]

I think a nice filesystem api could be built on top of the existing map.

It would have APIs for creating and deleting files. A file would then have APIs for for reading, writing and seeking.
It would be challenging to create, but it should be possible.

Obviously it wouldn't really be compatible with anything. But it'd save on using e.g. littlefs2.

If you've got interest in this or have ideas, then make sure to post here!

[paulyoung]

I don’t know much about this project but I saw your post on Mastodon and thought you might be interested in the approach I took in https://github.com/codebase-labs/icfs

• icfs provides implementations of std::io::{Read, Write, Seek} backed by stable memory to enable the use of existing Rust code that requires implementations of these traits.
• icfs-fatfs uses icfs to leverage the fatfs crate in providing a FAT file system. All this currently does is implement a fatfs::TimeProvider backed by ic_cdk::api::time()

Basically, by only implementing a few traits I was able to leverage some existing crates that had already done most of the work.

I provided a Unix-style API and had plans to work on one similar to std::fs so that it felt familiar.

Perhaps if rust-lang/libs-team#293 got some traction, that would be a nice approach.

[avsaase]

I would be interested in an API like this. Do you have any ideas for the design of a file system?

I looked into littlefs2 for an embedded-friendly file system but unfortunately it doesn't support async.

[diondokter]

I don't have a design ready for it. But it could use the map of s-s. The key would be a u32 of u64 or something like that.

Then at key 0 you've got the root folder. Its value contains the keys with the subfolders and files in that folder.
A file would then be a file descriptor with its name and the keys to the items that contain the data of the file.

So all of the block management is handled by s-s and only really the various types of items need to be defined.
This can definitely work! My biggest question though is whether this is performant enough. But there's only one real way to find out.

[avsaase]

Interesting. I wonder how that will work when appending data to a file. With this design you need to add the key of the newly written bytes to the file descriptor but this list can grown unbounded. Maybe a linked list would work better? Then the file descriptor could just hold the key of the first data item and some metadata which would have a fixed size.

[diondokter]

Maybe! That might be better for erase cycles. But probably worse for seek operations. You know, the standard linked list pros and cons :P

Say an item can be max 1kb and the keys used are u32.
Then the file descriptor (without name) could contain 256 data pointers or 128 data pointers + length. So then files up to either 256kB or 128kB would be supported.

So it depends on the usecase I guess. The linked list idea would support arbitrary sizes though.

But like I said, I don't have a concrete design. Just some ideas. I you want to hack on this, then be my guest!
In theory it can be a separate crate, but I'd love to have it in here as a module.