This isn't how these work.
You shouldn't put an aria-label attribute on a non-focusable element.
This is exactly why we have <span>aria label</span> elements, visually hidden with the .aria-label class, such that they read in place as normal text.

<span class="aria-label">hidden text</span> is fine as is as we use this elsewhere in Adapt. I think we typically have the two use cases:
1, injecting additional context for on screen text (see Slider example).
2, to provide a label for a parent element which we want to do here.

I can replicate the issue reported on Mac running voiceOver in FF. Looking into this, Drawer accessibility is handled similar to Notify. Comparing the two, I've noticed Notify sets focus to the first accessible element on opening but Drawer doesn't. Applying the same to Drawer resolves the issue for me. @oliverfoster, do you see any issues with doing this?
Setting focus on showDrawer:

    if (!this._isVisible) {
      a11y.popupOpened(this.$el);
      a11y.scrollDisable('body');
      this._isVisible = true;
      // Set focus to first accessible element
      a11y.focusFirst(this.$('.drawer__inner'), { defer: false });
    }

It should be. Maybe the defer needs changing to false?