Fast LZMA2 Support?

[Phantop]

So I was checking out fast-lzma2, testing it out and whatnot, and realized that the gains in speed in comparison to the loss in compression ratio are pretty substantial.

Of course, the typical usecase for dwarfs is not quickly making images, but rather quickly accessing them, however I do think it could be useful to quickly make well compressed images if necessary. Not sure if the library is easily integrateable, or even possible to do so, but I think it would be a nice addition (assuming it actually helps).

[mhx]

This certainly looks interesting! I reckon plugging it into DwarFS should be pretty easy, as there's already an abstraction layer for compression algorithms and this would merely be an additional implementation.

Not sure when I'll get around to taking a closer look, I'd much rather try and fix some of the open issues first, but they're mostly non-trivial so I'll need a couple of days of spare time. :)

[Phantop]

So I have found that simply substituting liblzma from xz with libflzma from fxz does build without issue. Currently doing some performance comparisons. A small question regarding building: is there a performance difference in building with system zstd/xxhash/gtest? I did that for faster compilation while testing but my normal build of dwarfs is built with the default options.

[mhx]

is there a performance difference in building with system zstd/xxhash/gtest?

Possibly, as it might depend on the compile flags used to build zstd/xxhash, and those might be different for the system version compared to the bundled ones. But I'd be surprised if it was significant.

[Phantop]

I asked because it there seemed to be no performance difference but then I realized that, according to ldd, the binaries are still just linking to liblzma unfortunately. Very strange...

[mhx]

Does it only link against liblzma or against both liblzma and libflzma? folly also links against liblzma and this might be why you're still seeing it.

[Phantop]

No it's linking only against liblzma. libflzma looks to be a drop in replacement but dwarfs isn't even giving errors when building related to it so I dunno the issue.

I actually did try editing all the dependencies to also use flzma which built but still links against lzma for some reason?

[M-Gonzalo]

I made a detailed comparison for precomp a while ago. Mind you, it's not entirely transferable because of the reduced block size in dwarfs, but it could be useful nonetheless.

[Phantop]

Yeah, totally! I do the same. I ran the tests because precomp's author intends to keep going in that direction and I wanted to support the project, but I never use the built-in compression. I usually do something like wimlib to duplicate at the file level (or tar if there aren't duplicates) >> rep for a fast local deduplication >> srep for global dedupe >> fxz for the final stage. Waaay faster than any comparable solution.

I was doing dwarfs to srep but I've found that using srep on a tar is very comparable. Using lrz over srep has gotten me smaller files before, though, but sometimes that file is actually less compressable with something like xz.

[M-Gonzalo]

Yes, I never could find a case where lrzip performed better than srep as a preprocessor (it can generate smaller files but after lzma or similar, the final result always ends up being bigger). It's also an order of magnitude slower if I remember correctly. So I quickly dropped it long ago.

[mhx]

I usually do something like wimlib to duplicate at the file level (or tar if there aren't duplicates) >> rep for a fast local deduplication >> srep for global dedupe >> fxz for the final stage.

That's roughly what mkdwarfs is doing:

• dedupe on file/link level (hardlinks are directly deduped, everything else based on checksums)
• similarity ordering (this is only useful for diverse sets of files)
• dedupe locally on segment level, with locality being configurable (through lookback size)
• compression algorithm of choice

The limitation, if you want to call it that, is that you don't really get "full global" segment deduplication, simply because that's unsuitable for a file system. So in principle, it's always possible for a pure archiving solution to outperform DwarFS.

[M-Gonzalo]

OTOH, I believe it's also possible for dwarfs to surpass most archivers. You just have to configure it with a big enough deduplication block so it can find all duplicates, and a big enough block size so it can be considered "solid". Of course that is a horror show to use as a file system but it can be done nonetheless AFAICT

[mhx]

OTOH, I believe it's also possible for dwarfs to surpass most archivers. You just have to configure it with a big enough deduplication block so it can find all duplicates, and a big enough block size so it can be considered "solid". Of course that is a horror show to use as a file system but it can be done nonetheless AFAICT

If you had asked me a few hours earlier I would probably have agreed. However, the deduplication (segmenting) as done by srep has potential advantages over the segmenting done in DwarFS. See my comments here.