EDIT: SOLUTION:
Nevermind, I am an idiot. As @ClickyMcTicker pointed out, it’s the client side that is causing the trouble. His comment gave me thought so I checked my testing procedure again. Turns out that, completely by accident, everytime I copied files to the LVM-based NAS, I used the SSD on my PC as the source. In contrast, everytime I copied to the ZFS-based NAS, I used my hard drive as the source. I did that about 10 times. Everything is fine now. Maybe this can help some other dumbass like me in the futere. Thanks everyone!
Hello there.
I’m trying to setup a NAS on Proxmox. For storage, I’m using a single Samsung Evo 870 with 2TB (backups will be done anyway, no need for RAID). In order to do this, I setup a Debian 12 container, installed Cockpit and the tools needed to share via SMB. I set everything up and transfered some files: about 150mb/s with huge fluctuations. Not great, not terrible. Iperf reaches around 2.25Gbit/s, so something is off. Let’s do some testing. I started with the filesystem. This whole setup is for testing anyway.
- Storage via creating a directory with EXT4, then adding a mount point to the container. This is what gave me those speeds mentioned above. Okay, not good. –> 150mb/s, speed fluctuates
- a Let’s do ZFS, which I want to use anyway. I created a ZFS pool with ashift=12, atime=off, compression=lz4, xattr=sa and 1MB record size. I did “some” research and this is what I came up with, please correct me. Mount to container, and go. –> 170mb/s, stable speed
- b Tried OpenMediaVault and used EXT4 with ZFS as base for the VM-Drive. –> around 200mb/s
- LVM-Thin using Proxmox GUI, then mount to container. –> 270mb/s, which is pretty much what I’m reaching with Iperf.
So where is my mistake when using ZFS? Disable compression? A different record size? Any help would be appreciated.
Good point. I used fio with different block sizes:
fio --ioengine=libaio --direct=1 --sync=1 --rw=read --bs=4K --numjobs=1 --iodepth=1 --runtime=60 --time_based --name seq_read --filename=/dev/sda 4K = IOPS=41.7k, BW=163MiB/s (171MB/s) 8K = IOPS=31.1k, BW=243MiB/s (254MB/s) IOPS=13.2k, BW=411MiB/s (431MB/s) 512K = IOPS=809, BW=405MiB/s (424MB/s) 1M = IOPS=454, BW=455MiB/s (477MB/s)I’m gonna be honest though, I have no idea what to make of these values. Seemingly, the drive is capable of maxing out my network. The CPU shouldn’t be the problem, it’s a i7 10700.
Basically you’re getting 477MB/s for a sequential read, which is spot on for a SATA SSD.
What size are the files you were transferring when you only got 150Mbps? Also did you mean Mb/s or MB/s? There’s an 8x difference between the two.
I meant mega byte (I hope that’s correct I always mix them up). I transferred large videos files, both when the file system was zfs or lvm, yet different transfer speeds. The files were between 500mb to 1.5gb in size
@Pete90 @MangoPenguin Bytes (B) are used for storage, bits (b) are used for network. 1B=8b.
2.5Gbps equals 312.5MBps.
With that in mind, there are a lot of moving parts to diagnose, assuming you want to reach that speed for a transfer. Can the storage of both machines reach that speed? I believe I saw the NAS’s disk tested and clocked at 470ish MBps, but can the client side keep up? I saw the iPerf test, but what was the exact command used? Did you multithread it?
Both machines are easily capable of reaching around 2.2Gbps. I can’t reach full 2.5Gbps speed even with Iperf. I tried some tuning but that didn’t help, so its fine for now. I used
iperf3 -c xxx.xxx.xxx.xxx, nothing else.The slowdown MUST be related to ZFS, since LVM as a storage base can reach the “full” 2.2Gbps when used as a smb share.
Nevermind, I am an idiot. You’re comment gave me thought and so I checked my testing procedure again. Turns out that, completly by accident, everytime I copied files to the LVM-based NAS, I used the SSD on my PC as the source. In contrast, everytime I copied to the ZFS-based NAS, I used my hard driver as the source. I did that about 10 times. Everything is fine now. THANKS!