After some time, ALSA fails with cannot allocate memory

[sreich]

I'm currently tracing a system crash periodically that...as I dig more into this, I am becoming more suspect it is this app/node-midi, in particular

I'm currently connecting my raspberry pi as a client to this desktop machine's server of rtpmidid. It could very well be rtpmidid, but more digging on the error pointed me towards this issue below

Log issue I receive is:

Aug 24 20:57:42 sreich-archlinux-desktop rtpmidid[674]: [rtppeer.cpp:351] Send CK0 to raspberrypi
Aug 24 20:57:42 sreich-archlinux-desktop rtpmidid[674]: [rtppeer.cpp:292] Latency raspberrypi: 65.60 ms (client / 2)
Aug 24 20:57:42 sreich-archlinux-desktop rtpmidid[674]: [rtppeer.cpp:300] Latency raspberrypi/rtpmidi raspberrypi-sreich-archlinux-desktop: 3.50 ms (server / 3)
Aug 24 20:57:42 sreich-archlinux-desktop rtpmidid[674]: [mdns_rtpmidi.cpp:297] (Browser) REMOVE: service 'raspberrypi' of type '_apple-midi._udp' in domain 'local'
Aug 24 20:57:42 sreich-archlinux-desktop rtpmidid[674]: [rtpmidid.cpp:284] Removing rtp midi client raspberrypi
Aug 24 20:57:42 sreich-archlinux-desktop rtpmidid[674]: [rtpmidid.cpp:290] Found client to delete: alsa port 3. Deletes all known addreses.
Aug 24 20:57:42 sreich-archlinux-desktop rtpmidid[674]: [rtpmidid.cpp:647] Removing peer from known peers list. Port 3
Aug 24 20:57:42 sreich-archlinux-desktop rtpmidid[674]: [rtpmidid.cpp:340] Disconnect aseq port 3, signal: disconnect(4)
Aug 24 20:57:42 sreich-archlinux-desktop rtpmidid[674]: [rtppeer.cpp:53] ~rtppeer 'sreich-archlinux-desktop/PipeWire-RT-Event-input' (local) <-> 'raspberrypi' (remote)
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: ALSA lib seq_hw.c:466:(snd_seq_hw_open) open /dev/snd/seq failed: Cannot allocate memory
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: MidiOutAlsa::initialize: error creating ALSA sequencer client object.
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: ALSA lib seq_hw.c:466:(snd_seq_hw_open) open /dev/snd/seq failed: Cannot allocate memory
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: MidiOutAlsa::initialize: error creating ALSA sequencer client object.
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: ALSA lib seq_hw.c:466:(snd_seq_hw_open) open /dev/snd/seq failed: Cannot allocate memory
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: MidiOutAlsa::initialize: error creating ALSA sequencer client object.
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: ALSA lib seq_hw.c:466:(snd_seq_hw_open) open /dev/snd/seq failed: Cannot allocate memory
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: MidiOutAlsa::initialize: error creating ALSA sequencer client object.
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: ALSA lib seq_hw.c:466:(snd_seq_hw_open) open /dev/snd/seq failed: Cannot allocate memory
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: MidiOutAlsa::initialize: error creating ALSA sequencer client object.
Aug 24 20:57:42 sreich-archlinux-desktop MIDI-Jar-1.1.0_b94c6fccb5821201111ab46ec635f799.appimage[1413]: ALSA lib seq_hw.c:466:(snd_seq_hw_open) open /dev/snd/seq failed: Cannot allocate memory

when I run aconnect -l I get this

ALSA lib seq_hw.c:466:(snd_seq_hw_open) open /dev/snd/seq failed: Cannot allocate memory

My system gets into a state of this, looks like it is resource locked until I reboot. Mind you, this is now for all clients. But I think it might be a resource leak caused by midi-jar which of course causes issues for everyone.

justinlatimer/node-midi#118

Mind you these are just bits of info I've been scavenging, i'm trying to trace it through my system. Do you happen to know if the architecture could cause something like this, and the node-midi that you are using? I don't want to waste your time if it is not this software, but perhaps you will have a "ah-ha!" moment by me asking this

I will further investigate(gotta wait for it to reproduce itself) to see if I can blame it on the other MIDI client software that I am using (rtpmidid). But for now it is pointing this way. I will keep you updated.

• MIDI Jar version:1.1.0
• Operating System: Archlinux
• Hardware Devices: Roland FP-30

[ArTiSTiX]

@sreich no "ah-ah" moment sorry.

Cannot allocate memory on windows is the classical error when a software tries to connect to a device that is already being used.

But according to the logs you provided, it seems that something is wrong.

My first thought is that when a client is connected, it is created in ALSA, and when it is disconnected in rtpmidid, the associated MIDI device is removed from ALSA.
MIDI Jar should then be able to detect a change of how many devices are connected, and will try to reestablish its routing (disconnecting all outputs and inputs, and reconnecting it).
But i had some issue with node-midi (on windows) not being able to fully know that a device is disconnected, and reconnecting it will certainly fail.
I think it would not be an issue, as failing to connect to a device is not supposed to have side effect - just MIDI Jar not being able to use this device (in this case, the device in the routing settings page should have an error state visible).
But crashing your system is very embarassing...

I'll try to reproduce it with rtpmidid, see if i can understand why there is this "MidiOutAlsa::initialize: error creating ALSA sequencer client object.".

[sreich]

Okay thank you

I would wait if I were you, I should know in a few days if it's rtpmidid or midi-jar, then I'll update you and we can go from there. Could be rtpmidid not freeing properly and that breaking any app which happens to include yours

Thanks!

[sreich]

@ArTiSTiX i am fairly confident it is midi-jar causing this issue somehow. i don't know if it is a combination of it and rtpmidid.

But, basically I started midi-jar last night after my rtpmidid and piano led visualizer working just fine for weeks. but now it's giving me that out of memory error, because of resource exhaustion, after having your app run overnight

i wish i had more information to help you, but idk where to proceed from here with this

[ArTiSTiX]

@sreich Huh.

I'm very sorry that midi-jar is causing an issue with ALSA, and may be conflicting with rtpmidid.
I will try my best to debug this issue.

A few general questions though:

• Does it cause a crash of your computer ?
• Does it overload your system memory when running ?
• If i understand right, it causes ALSA to no longer being able to do anything with midi - until you restarts your system ?
• I suppose restarting MIDI Jar does nothing ?
• Does the devices connected to rtpmidid are regularly connected/disconnected (like network issues happening causing up/down of MIDI devices over night) ?

And a few environment variables i would need:

• which version of rtpmidid are you running ?
• which version of ALSA are you running ?

I can't promise you to find a solution, particularly if it's the node-midi implementation midi jar depend on that is faulty, but i can investigate this issue and try to reproduce it.

I hope you will still be able to use midi jar on some occasions, maybe letting it always run is not possible with your issue, but i don't recall if MIDI devices are exclusive on ALSA, so maybe running MIDI jar only when you play would fit your needs, without using routing (other than for chord-display).

[sreich]

@ArTiSTiX

yeah, midi devices are not exclusive on linux, that's a windows limitation with their midi driver. here's the thing, i'm not actually using routing in midi-jar, i don't need it because pipewire already does that, to be honest. i don't know if that helps narrow anything down

i'm at a weird spot with it, i haven't had issues in weeks with midi-jar not running. i wish i were more help here in trying to track it down

sounds like guys who know more than me have tracked it down to node-midi, perhaps start with investigating this: colinbdclark/flocking-midi-router#4 there is a fork of node-midi he has that works around it. perhaps we can start there? depending on its complexity, it might be easier for you to switch to that, make me a build and i will run it overnight

then we will know

[ArTiSTiX]

@sreich thanks for the resources, i might try something.

First solution could be to add a setting to disable polling the midi device every 100ms (my only solution since RtMidi does not provide observability on connect/disconnect) - and have a longer delay could maybe help a bit. Thus you will need to manually refresh devices after a midi connect/disconnect device you use.
Second would be to rewrite my midi logic to ensure i destroy the node-midi objects, i seem to rely on the garbage collector, which might be an issue.

I'll try to reproduce the issue mentioned here.

Thanks for the investigation !

[ArTiSTiX]

@sreich I'm sorry for the long delay for sorting this issue out.
I was able to test a bit this weekend the behaviour with ALSA.

For what it seems, when opening a port with RtMidi, it automatically creates an additional virtual port.
My first test reproduced the issue you mentioned with ALSA (i tried to connect all available devices in the same refresh loop as MIDI Jar), and in a fraction of a second, i created more than 32 ports.

For some reason, when closing the port, the resource is not immediately released, so this might be the issue here. When letting more delay for the garbage collector to free those resources, some ports are removed (and it raises a new kind of issue).
I'll dig more into it but fixing it will be a bit more complicated than i anticipated.

Currently, MIDI Jar closes and opens new ports each time there is a different port count detected. Works fine on Windows & MacOS CoreAudio, but can be an issue with ALSA (each connection triggers a new refresh, which can at some point, lead to some feedback refresh loop).

Now i would have to rewrite the entire logic to avoid closing ports that did not change, ignore those internal ports, and I might have to create a better abstraction for MIDI Devices.

Sorry again for taking my time on the subject.

[sreich]

Yeah I was afraid it would be tricky to fix, at least you have a better understanding of it and can reproduce, that's good news!

No problem. I'm sure it isn't the most exciting of things to work on, but I am looking forward to being able to use this software when this is fixed

[sreich]

FYI, this does not affect me now. Basically I bypassed RTP MIDI. I don't run it on my computer anymore, I bought a hardware solution that multiplexes MIDI. And that seems to work fine. It's basically a dongle that I connect to my raspberry pi and then the USB from that goes out into my computer to complete the second connection that I need

So, not as high priority given it's able to work just fine. Now that I have the solution. You can feel free to reprioritize as you choose, there's another bug on filing now that affects me more ;)

[ArTiSTiX]

Ok i see.
As said, this issue is specific to how RTMidi interacts with ALSA, and it would require a full rewrite of the "refresh loop" of midi devices - i took the lazy path here that seems to work fine on Windows & MacOS.

Since linux was unfortunately not on my prio list, i am not planning to rewrite this right now (but this might be useful at some point for future features).

Just out of curiosity, what do you use the Raspberry PI for ?
Is this for something like a RGB lightstrip to highlights your notes, à la Rousseau on Youtube ?

This is the kind of features i would like to explore in the future of MIDI Jar (interactions with Philips Hue & Nanoleaf maybe) and i am wondering if you are using other open source projects to interact with a raspberry for such usecases.

[sreich]

Yep I understand

Precisely, I have a light strip hooked up to my piano. I use this project for it, it works really well. Lots of customization for the strip

https://github.com/onlaj/Piano-LED-Visualizer

So I have that running on my pi. I haven't used it for the learning feature, only the lights. I'd like to get the actual light strip setup to look more pro. Right now it's just a plain strip and tape haha. Such is the way I am, polishing the projects is not my forte lol. Just ask if you've got further questions on it