zfs send -I - sent over network data in RAW format

[Neptunet-GitHub]

Hi,

I created environment base on 2 servers (S)ource and (T)arget, both with zfs-2.2.7-1~bpo12+1 (Debian).
On (S) and (T) I created: zpool create -O dedup=on -O compression=lz4 -O canmount=on -f vd raidz /dev/sdb /dev/sdc

on (S) zpool is mounted, and I write data on it, on (T) zpool is umount.

Next I want to use zfs send-I snap-on-S-1 snap-on-S-2 | ssh (T) "zfs receive -Fs vd" data from (S) to (T). Everything works, but I observed that the amount of data transferred over LAN is huge.

Please look:
zpool list

I have 2 snapshots:

Size:

difference between them is 300GB.

It suggests, that zfs send will send over network rededuplication and decompression data, next on (T) again deduplicat and compress it.

And I see, that transfer over network is real 300GB - over 1Gbs LAN it needs about 1 hour.

I expected transfer over network between two zpool with dedup and compression data "after" dedup and compression, not it RAW format - if it needs transfer RAW format, it's not performance optimisation :(

Maybe I set/configure something wrong, or zfs send "that's how it works" - is it true?

[IvanVolosyuk]

It look like it is no longer possible to send deduplicated stream anymore. See 'man zfs-send', 'man zstreamdump'.
You can send compressed stream though (-c) or compress/decompress before/after ssh, e.g.:

zfs send ... | zstd -T0 --adapt=min=5 | ssh 'zstd -d | zfs recv'

I would also suggest to send with '-L' to keep large blocks >128k, but they may be already broken on destination, so don't do that on incremental send if original send wasn't using that.

Btw, 'raw send' is reserved for sending encrypted zfs stream without decryption.

I don't know why sending deduplicated stream was deprecated. Also, the same issue will happen if sending zfs stream with ref-links AFAIK.

[Neptunet-GitHub]

Thanks @IvanVolosyuk - I saw that information about --dedup - and it's strange for me. This way we lost a full potential of ZFS + dedup + compression.

About reflink I didn't hear - if it also has input to replication, it's huge not good performance information :( and using zfs send is unuseful :(

[Neptunet-GitHub]

What is the scale of problem You can see on next screens.

zpool is configured with deduplication, symlink and compression.

I create first snapshot -> vprotect_data@bzfs-01
I wrote on it 60GB data (incremental backups)
and create second snapshot -> vprotect_data@rta-snap-2025-01-06_09:36:59

Now I want to run replication, but look what report zfs:

zfs send with incremental transfer, need to transfer to second zpool 4,37 TB (!!!!!)

When whole zpool has allocated 329GB space:

Enable compression gzip, zstd isn't effective, it consumes plenty of CPU time and overall transfer efficiency drops even further due to compression overhead.

Guys, is there any solution to allow run replication between zpool ONLY new data not in RAW format? openZFS get amazing deduplication ratio, and it's very satisfaction for my solution, but problem with amount of data which needs to be replicated isn't allowing use this solution :(
Why did you disable --dedup switch in zfs send?

[amotin]

I've never used zdb to predict replication size, so can't speak about it. zfs send has --dryrun argument which should with -v report expected stream size.

The use of "RAW" in this issue is confusing to me. zfs send with raw means as stored on disk -- compressed, encrypted, etc. What "run replication between zpool ONLY new data not in RAW format" means I can't even guess. ZFS incremental send for data written after the previous snapshot can be send in any way: compressed, encrypted, or plain.

If you use zfs send with -Lce, or even with -w, you should get the stream already compressed as it is stored on disk. Deduplication though, same as newer block cloning, can not be replicated, since it works below the dataset level and covers block from the whole pool. From the dataset perspective deduplicated blocks are still an individual blocks, and if they were written after the last snapshot created, they will be replicated individually. If the receive side has dedup enabled, then they may get deduplicated there too, but it will be later. I was not familiar with the deduplicated send before it was removed, but I guess it was a separate implementation that was difficult to manage. It is generally impossible to say which dataset blocks are sharing the same deduplicated pool block, unless you accumulate the list during the scan, which must be expensive and complicated.

You haven't described how you do your incremental backups. If you just create a new directory each time, write there everything and and rely on dedup to remove duplicates, then it is just not the way replicaiton can live with. Would you write the new data over the old one and just creating shapshots after each replication, then with strong checksums enabled ZFS could use nop-write feature, not updating the blocks that were not changed. This way you would nto even depend on deduplication and replication would easily know which blocks are really new and need to be send. I.e. if possible, just do not create ZFS extra problems to fight later.

[Neptunet-GitHub]

@amotin use zdb or zfs send --dryrun -v -I return the same value:
zdb

= 0,3 TB
zfs send --dryrun -v -I

= 0,341 TB

Because my zpool allocated only 367G

it suggest, that incremental replication is in RAW format - it needs transfer almost the whole data allocated on zpool on source after deduplication and compression, when difference amount of data between these 2 snapshots after dedup and compress is about 60 GB

These above data are with zfs send with parameters:
--props --compressed --large-block --embed --replicate

I afraid. because dedup ratio now is 62:1, so now, on zpool I need 367G space, but run first replication between new zpool, I need to transfer 21,2TB, RAW blockdevice for that zpool has disc 1TB, so the better option will be transfer whole block device, than use zfs send :(

[amotin]

Because my zpool allocated only 367G ... it suggest, that incremental replication is in RAW format

It does not. It is merely a coincidence. Zpool ALLOC value reports space used on pool after dedup, but for all data, while zfs send reports size modified after the snapshot specified, but before dedup. Those two values are just incomparable in this case, like green and soft. Any of them can be bigger depending on dedup ratio and when snapshot was created.

I afraid. because dedup ratio now is 62:1, so now, on zpool I need 367G space, but run first replication between new zpool, I need to transfer 21,2TB, RAW blockdevice for that zpool has disc 1TB, so the better option will be transfer whole block device, than use zfs send :(

True. But I would instead reconsider how this backup works, since looking on the huge 62:1 ratio, either you back up dozens of identical hosts or the backup is not even close to being incremental.