Replies: 7 comments 23 replies
-
|
I have the same issue with an icy box DAS (same model as described by the author) using a Zen3 CPU (Ryzen 9 5950X). The icy box is connected with USB3 gen 2 (10Gbit/s), connecting it with USB3 gen 1 (5Gbit/s) does not solve the issue. Using an Intel based workstation or Intel based laptop at either USB speed has no issues. The problem only occurs when applying heavy write I/O to the DAS, for example moving several GB of files to a filesystem on the DAS. After the errors occur I can resilver on the AMD platform. Large read workloads have no issue. The DAS contains 4x6TB WD red disks. I run on Ubuntu 20.04.2, kernel 5.4.0-73-generic, but had the problem with earlier kernels as well. |
Beta Was this translation helpful? Give feedback.
-
|
I had a free x4 slot. The card arrived today. When I hook up the DAS to the separate controller everything works fine. When I put it back on an MB USB port (both 10Gbit and 5 Gbit) the problem comes back. I'd say this is final proof something is wrong with the chipset or its firmware. Although I do have a good workaround now, it did increase the price of my MB by 30% and it requires my only x4 slot. It works but AMD should really fix the X570 USB controllers/firmware instead. The good news is of course I don't need to buy another DAS. |
Beta Was this translation helpful? Give feedback.
-
|
Tried the same on Windows 10 and it gives the same picture. ASMedia USB 3.1 controller works fine, AMD USB 3.1 controller gives a ton of errors. That excludes OS, drivers, DAS, disks and cables and leaves only AMD hardware and firmware as possible sources of the problem. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for sharing this valuable information @hugovdb1965. I've made one more post at /r/Amd here: https://www.reddit.com/r/Amd/comments/nuawhz/amd_uasuasp_confirmed_broken_on_zen3/ I will leave it at that, because we (at the office) and I (at home) replaced AMD hardware with Intel, at least were UAS devices were concerned. |
Beta Was this translation helpful? Give feedback.
-
Offtopic: I was able to claim the Far Cry 5 code that came with my Ryzen 7 5800X (sold as part of an upgrade kit with the CPU, a B550 Tomahawk and a rudimentary CPU cooler). On their website instead of receiving a .exe I received a .sh file that worked perfectly fine. |
Beta Was this translation helpful? Give feedback.
-
This must be new! When I contacted customer support they explicitly told me I needed Windows to claim the game. |
Beta Was this translation helpful? Give feedback.
-
|
I thought I might be helpful to share my hardware knowledge: All Ryzen processors (desktop and laptop) are System on a Chip and can work without a chipset. Laptops and mini pc's very often don't have a chipset and depend on the USB- and other controllers integrated in the processor. So your 2700u probably doesn't have a chipset. |
Beta Was this translation helpful? Give feedback.
-
|
I can confirm exactly the same problem with two AMD-based ThinkPad laptops - T14s Gen 1 (Zen 2) and T16 Gen 1 (Zen 3+). This is with the USB enclosure Icy Box IB-3780-C31, with 8 drives (2 mirrors with 2 disks each, 1 raidz2 with 4 disks). ZFS 2.1.6 on Arch Linux (kernel 6.0.10). As a workaround, I am forcing UAS(P) to be disabled with the following modprobe configuration:
With this, I get the following logs when connecting the enclosure: Note the "UAS is ignored for this device" line. With those quirks enabled, drives appear to work fine and ZFS is completely functional. The only problem is that I get the impression that read performance is somewhat affected, at least when I'm moving something between the datasets that are on those disks, I am not able to max out the SSDs. The enclosure itself is supposed to be USB 3.1 Gen 2 (10GBps) so I think it should be able to handle the traffic. When I copy some stuff to the local laptop drive and then to the SSDs in the enclosure, the speed is "correct" - so it's definitely "something" with the USB throughput. However, overall this workaround has been satisfactory for my use case. Without it, the symptoms are exactly the same as the original poster mentioned, and while ZFS is fighting the errors bravely, it is simply unusable. |
Beta Was this translation helpful? Give feedback.
-
Good find @ppawel! I wish I knew about that quirk option sooner, so I'm quoting it here for clarity. For me it's too late. I was so intensely frustrated with AMD, their ignorant customer support which I have contacted multiple times, and their toxic community of victim blaming fans telling me I'm an idiot for buying an UAS device in the first place, that I have sold/replaced my AMD devices with Intel devices. I'm all for supporting the underdog, but not when they are so ignorant. All they do is cater to gamers and support some video cards, but they don't really gaf about Linux or big data analysts and supporting their preferred data storage peripherals. |
Beta Was this translation helpful? Give feedback.
-
|
Just a small note that I have now reconnected the aforementioned USB enclosure to my new Intel-based home server system and all the issues have disappeared - no quirks needed, everything works smoothly and performance has improved as well. |
Beta Was this translation helpful? Give feedback.
-
|
I have the same issue with an Intel-based server system.
Disks always disappeared pairwise during high load situations (copy a large amount of files) with errors like As workaround I had to shut down the server, turn off and on the Icy Box and restart the server. (Just restarting the server did not help). Interestingly some controllers where using uas, some usb-storage: Since setting |
Beta Was this translation helpful? Give feedback.
-
|
I am using the 4-drive variant and am seeing the problem on multiple machines (Intel and AMD). The only one where it seems stable is a intel 13700k on a z690 motherboard. Currently trying the quirks (for my device it seems to be |
Beta Was this translation helpful? Give feedback.
-
|
Sadly, with my 5600G in a B550 mobo with AGESA updated to 1.2.0.7, I still can't use my DAS neither using UAS or regular usb-storage mode. It "works" in in both modes until I start writing data and I get tons of write failures, data corruption, and the whole connection hanging for minutes at a time. The 3.1 support seems abysmal. The only way I got my DAS to work consistently was by using a USB 2 cable with 480M throughput, so I'm currently waiting on a PCIE card with a chipset that hopefully actually works.. So, for others coming across this: your mileage may wary, but nothing has worked for me. |
Beta Was this translation helpful? Give feedback.
-
|
@Pectojin I've just seen this topic, however my problems went away after disabling USB-suspend (both with a service like TLP and also the port itself). See https://gist.github.com/francoism90/7dd103249c5ffe3dc36be44a8b291ef1 when using Thunderbolt over USB. It has been a while, and I'm not using it anymore, but I got the same issues on my Intel NUC. |
Beta Was this translation helpful? Give feedback.
-
|
I got my PCIE card (an icybox one) and it's been working without issue for ~2 months. It's a bit disappointing, but spending ~40 euro on an external card has been well worth it for me. |
Beta Was this translation helpful? Give feedback.
-
|
I had SATA device error, (IO error) or switching to read-only state. Error appears on the disks on the heavy load process, backup for example. iommu configuration solved my issue.
In ProxMox VM backup process log For AMD Epic series processor I solved the issue with adding boot parameter
Check if iommu is supported
Set IOMMU
add parameters to the string GRUB_CMDLINE_LINUX_DEFAULT reboot the server after rebook check for boot errors I have got a bit slower IO to the SATA disks, but more stable work with SATA devices. Now SATA devices do not dissappear or switches to read-only state.
|
Beta Was this translation helpful? Give feedback.
-
|
Thanks for sharing your findings. I would have loved to test this with multi-disk UAS hardware on AMD, but I have replaced all AMD hardware with Intel machines to solve my problem. If anyone can confirm this also fixes the high load USB Attached SCSI Device problem, it will be beneficial for the community, and frankly for AMD as well. With AMD tech support being very similar to /dev/null in helping us solve this issue, I'd need to see peer testimony before choosing AMD again. |
Beta Was this translation helpful? Give feedback.
-
|
I have the same problem with UAS resets, but on Ceph instead of ZFS. |
Beta Was this translation helpful? Give feedback.
-
|
I noticed this topic started almost 4 years ago - but what the hell is wrong with you people? don'T use storage connected via usb for anything serious!
well - it should be clear now that the issue is UAS - not ZFS |
Beta Was this translation helpful? Give feedback.
-
|
@Pectojin It's not only AMD. This was on an Intel NUC with a closure that had external power. USB/TB has become better, but it requires users to have the latest firmware, and good vendor support (e.g. controller). |
Beta Was this translation helpful? Give feedback.
-
|
I think it is just a fact of life that USB is not considered by vendors (due to pricing) as supposed to have the same level of reliability and compatibility as even SATA, not mentioning SAS or enterprise grade NVMe. Even with the last ones we regularly hit hardware/firmware bugs during qualification process, but there we at least have a vendor to complain to have it fixed. I don't believe this and similar threads has anything to do with ZFS or any other file system, considering that properly implemented device should not fail miserably even if mistreated somehow, there are just too many protocol stack layers in between for that to matter. But if people need a place to share experience, so be it. |
Beta Was this translation helpful? Give feedback.
-
I reported the original issue almost 4 years ago, both to Github and to AMD. I ran UAS without problem for many years, until 4 years ago I started replacing my Intel machines with AMD machines. They wouldn't stay online without errors. I found they have this issue across at least 4 AMD chipsets. AMD just kept telling me to flash the lastest BIOS and install Windows. I flashed many BIOSes. Finally I moved back to Intel and I'm running UAS again without a single problem for 4 years on 3 Intel systems. If you have quality cables, the problem is with AMD. Not with USB. I guess the real advise is: Don'T use AMD for anything serious! |
Beta Was this translation helpful? Give feedback.
-
@amotin I agree; this likely has nothing to do with ZFS, although I didn't know it yet at the time of reporting. Anyone else moving to AMD experiencing the same problem will initially think it's a ZFS problem though. On the other hand, if AMD requires certain quirks to be activated and suddenly UAS works properly, I wouldn't mind ZFS being able to detect the problem and inform the user in a |
Beta Was this translation helpful? Give feedback.
-
|
I think it is not USB issue. I had an issue with SATA HDD drives. |
Beta Was this translation helpful? Give feedback.
-
I'm using OpenZFS on a UAS DAS and keep running into I/O errors on AMD hardware only. This happens on both ZEN and ZEN2 architecture. I haven't tested ZEN+ or ZEN3. The same DAS works fine thoroughly tested on multiple Intel devices. I discourage people from purchasing AMD hardware if they plan to use OpenZFS on a UAS DAS.
My USB Attached SCSI Device, a 4x 8TB ZFS Direct Attached Storage (Icy Box IB-3740-C31), gets many read/write and checksum errors when attached to an AMD computer. It does not give these errors when attached to Intel computers.
Originally observed on a Gigabyte Aorus X570 I PRO WIFI + AMD 3700x (ZEN2) desktop connected for testing the DAS, reported to r/Amd and AMD email tech support; I got the impression this was related to the 500 series chipset problem and I could safely purchase another AMD mini PC with a non-500-series chipset. The OpenZFS DAS works fine and successfully completes a scrub on Intel hardware. I've tried three machines, among which an Intel Core i7-7600U and an Intel Xeon CPU D-1518.
Below the steps to reproduce this on Ubuntu 20.04 with zfsutils-linux installed. Assuming the USB ZFS pool is attached.
Although this is very likely not related to the DAS and cables due to the succesful function on three Intel machines, I have also tested different cables. I have used both the USB type C port and the USB 3.1 port on the back. I have tried flashing the latest bios for this mainboard (33c). I have tested a different DAS (Fantec QB-35U31). The results are the same. Resilvers fine on Intel hardware, breaks on AMD 500 series chipset.
AMD has since announced a fix for 500 series chipset USB problems.
But here is the problem: I'm experiencing the same issue on a MinisForum UM270 with AMD Ryzen 7 2700u (ZEN), purchased for the NAS build. This is a 2017 CPU, so I am assuming a 300/400 series chipset, although I'm unsure where to find exact details about these mini PC's.
Syslog
At some point the UAS simply disappears. Again, I can attach the DAS to my Intel laptop or desktop and it resilvers fine. No errors.
I'm uncertain how to proceed. Given that all USB devices, external storage and single disk non-ZFS UAS devices seem to work fine on both AMD computers tested, the only combination that structurally causes these issues for me is AMD + UAS + OpenZFS.
Can anyone else reproduce this specifically on AMD with UAS hardware? Did anyone experience this? Are there some known factors that I can disable on a system (Ubuntu 20.04) or BIOS (American Megatrends 5.13 1/21/2021) level?
Until this is resolved, I discourage anyone from buying AMD hardware released on ZEN platform or later when planning to use OpenZFS on a DAS.
Beta Was this translation helpful? Give feedback.
All reactions