[Request]: When describing status item for screen reader users include the text content

[pvagner]

Describe the request

Hello @GeopJr

Huge thanks for taking screen reader accessibility seriously. It's working fine for me.

When using up and down arrow keys to navigate in the list of toots the item's accessibility text is reported.
However in order to reach and read the actual post content I need to use tab to navigate to the label with the post text.
I think it would be easier if that accessible text also contained the text content of the toots.

I am not a real coder, but my quick attempt at doing it is here

diff --git a/src/Widgets/Status.vala b/src/Widgets/Status.vala
index 47f1b28..2168641 100644
--- a/src/Widgets/Status.vala
+++ b/src/Widgets/Status.vala
@@ -857,12 +857,24 @@
 		string aria_translation = "";
 		if (translation_label != null) aria_translation = translation_label.get_text ();
 
+		string content_aria = "";
+
+		var w = this.content.get_first_child ();
+		while (w != null) {
+			var temp_label = w as RichLabel;
+			if (temp_label != null) {
+				content_aria += @"\n$(temp_label.accessible_text)";
+			}
+			w = w.get_next_sibling ();
+		};
+
 		this.update_property (
 			Gtk.AccessibleProperty.LABEL,
-			"%s %s %s %s %s %s %s %s %s %s %s %s.".printf (
+			"%s %s %s %s %s %s %s %s %s %s %s %s %s.".printf (
 				header_label.accessible_text,
 				post,
 				aria_date_prefixed,
+				content_aria,
 				aria_pinned,
 				aria_quote,
 				aria_attachments,

Implementation Details

• This follows the GNOME HIG.

[GeopJr]

Thanks for the suggestion and code!

There's a few things to consider:

• Post content can be of any length. Sometimes it can be blog post / 5k+ word long. Having it in between of other info might end with people skipping it altogether and never hearing the other metadata.
• It lacks consent. The current approach won't read the post content until the user reveals the spoiler. This approach not only announces the post body even if there's a spoiler (content warning) but also announces it before the content warning. This should be easily fixable (I'll do it if we move forward with this change) but it needs further testing to make sure everything is announced correctly after revealing the spoiler. Which raises the question of, if the spoiler gets revealed, do we announce everything again or just the content? (probably the gtk default)
• It needs testing in regards of custom widgets. 'RichLabels' are not technically labels, but containers that hold labels and custom widgets (like custom emoji images). Those widgets need to be announced at the position they are placed and they need to only be announced then, so the screen reader needs to ignore them and never announce them outside of their 'label'. This is a tricky one.
• It needs to be audibly distinct. The other elements of the aria label follow a conversation-like announcement. Like 'Contains 5 attachments.'. If the post body is just announced between the other elements, it can be confusing. E.g. 'Contains 5 attachments. Look at this picture of my cat!!! #catsOfMastodon This post is pinned. The post has a spoiler.'.
• Lastly but not that important currently, the aria label should be generated in the responsible widget and not in the status. That's the case already, the accessible_label is already a custom aria label deep down the widget hierarchy. I'll fetch it from there if we move forward with it.

[GeopJr]

Sorry for the wall of text. There's just so much to consider, so much info to announce and so many edge cases to take care of that it has to be done as perfectly as possible.

[pvagner]

Hello,
Yes you are right I have missed most of these points.
I am sure you do already know this but I'll repeat it so you can better try to adapt to my thinking. When screen readers send some text for text synthesis it's usually an async operation i.e. the text is sent, text to speech is handling it. If user presses more keys or uses the mouse the UI might change and the new text is sent for synthesis effectivelly interupting reading of the previous one. Visually disabled people are used to this behaviour. For example when using up and down arrow keys to navigate in the list and I see that this is a message I have read previously or I have recognized I am not interested in that message, I can continue the navigation without listening to the generated voice prompt to the end.
As for the spoiler it should definatelly be respected i.e. the corresponding RichLabel and thus the part of the accessibility text we are discussing here should only contain the brief content.
As for the distinct announcements I think this is where our opinions may diverge slightly. I think saying "published: 2 hours ago" is much more verbose than just saying "2 hours ago" and the latter preserves the meaning. I would wote for less verbosity if you would be willing to reconsider that. The same goes for post stats, I think it's enough to say "2 replies and 3 boosts".
The most tricky thing is how and where to report the content. What do you do if it's a long post such as blog post, are you elipsizing the label or is it displayed on the time line in its entirety? If it's displayed as a whole then I would do the same and include the whole content in the accessibility text. If it's elipsized I would do the same cutting off the end.
My reasoning behind putting the post content into the middle of accessible text is that there are properties such as whether the status is a reply, its visibility, who and when has posted it. These details are expected to be short and are almost as important as the actual post content. However the rest such as how many replies and boosts are there, what app has been used to post it and others are just supplementary. And I think if you find out you are not interested in the post you will most likelly also don't mind skipping those complementary details.
Also another little thing I have realized. Now we are overriding presentation of these list items by providing our own accessible text instead of the text that might be calculated out of its children that is a sane default. When there are children that are not interactive i.e. can't receive focus such as non clickable images or not focusable labels and information from these is already contained in the accessible text of their parent I would consider setting their accessible role to presentation. I think at least the status image and the time label can be tweaked this way.
Also when clicking the avatar and the poster display name it opens their profile. Is it always the case or might there be different actions? I am not sure but if it's always equal would there be a way to rethink the user experience so it will only consume single tab stop when using tab or shift+tab to navigate?

Thanks for listening.

Greetings

Peter

[GeopJr]

As for the distinct announcements I think this is where our opinions may diverge slightly. I think saying "published: 2 hours ago" is much more verbose than just saying "2 hours ago" and the latter preserves the meaning. I would wote for less verbosity if you would be willing to reconsider that. The same goes for post stats, I think it's enough to say "2 replies and 3 boosts".

I'd love to be less verbose but at the same time be extremely clear on the post's details to avoid any misunderstandings. For example, the date. If we announce just '2 hours ago', couldn't this lead to a post with the content "15 hours ago" that was posted 2 hours ago be announced as "2 hours ago 15 hours ago"? Also, there are actually two dates that can exist at the same time, one for when a post was published and one for when a post was edited. Announcing it as 'Edited 2 hours ago', the screen reader users also understands that the post has been edited.

Similarly for spoilers. If we don't separate when the spoiler ends and when the content starts it can be messy.

Same thing for post stats. If the post ends with "15 boosts" and the screen reader announces the posts stats without a prefix, it can end up as "15 boosts. 15 boosts. 2 favourites.".

My point is that fedi has all sorts of things, from long blog posts to trolls who might interfere with our aria labels and unless we account for everything the screen reader users will be the ones most affected from it. You are both familiar with fedi and linux' accessibility quirks but for someone new to both or either of these, this could lead to confusion.

What do you do if it's a long post such as blog post, are you elipsizing the label or is it displayed on the time line in its entirety? If it's displayed as a whole then I would do the same and include the whole content in the accessibility text. If it's elipsized I would do the same cutting off the end.

Currently all posts are displayed in full, so I wouldn't worry about this.

My reasoning behind putting the post content into the middle of accessible text is that there are properties such as whether the status is a reply, its visibility, who and when has posted it. These details are expected to be short and are almost as important as the actual post content. However the rest such as how many replies and boosts are there, what app has been used to post it and others are just supplementary. And I think if you find out you are not interested in the post you will most likelly also don't mind skipping those complementary details.

I agree and this separation of label and description somewhat worked towards that. Post metadata are currently in the aria LABEL, while the post content is in the aria DESCRIPTION. My thought process was that you can decide if you want to skip the LABEL and read the DESCRIPTION or if you want to skip the whole post altogether. I don't think skipping just the LABEL is possible however...

Okay, how about we remove metadata that can be focused / announced by tabbing? Attachments and their alt text, post stats, quote, poll. But then again, a screen reader user might not want to continue if they knew that the post had 100 attachments etc.

Also another little thing I have realized. Now we are overriding presentation of these list items by providing our own accessible text instead of the text that might be calculated out of its children that is a sane default.

This is being done when possible (e.g. the label with custom emojis, custom emojis are marked as presentation). But I'm a bit hesitant to do this on everything that cannot be focused because they still have tooltip texts that may influence the alt text. E.g. the date tooltip text shows the full date time string in YYYY-MM-DD HH-MM-SS and the visibility icon tooltip text shows the visibility name.

Also when clicking the avatar and the poster display name it opens their profile. Is it always the case or might there be different actions? I am not sure but if it's always equal would there be a way to rethink the user experience so it will only consume single tab stop when using tab or shift+tab to navigate?

In most cases, clicking the avatar in a post will open a 'mini-profile' popup (which is basically the user's profile for quick-view). While the display name will open the full profile every time.

[pvagner]

Hello,
I think you have thought of all the possible use cases in a great detail that I am not able to improve on. Frankly I was not targetting such a well polished experience.
So if you are fine with that let's take one thing at a time instead trying to fix only the part that needs a bit of improvement.
My number one issue with the app as it is today is that it requires an extra effort to get into the post content with the screen reader. I either need to use tab or shift+tab to navigate as an addition to navigating with up and down arrow keys or I have to listen to all the meta data and interactive buttons before the content. In fact post stats are effectivelly reported twice now. The first occurence is in the aria label of the item, the second is the text of the buttons and the buttons are reported even before the description which holds the post content currently.

Seperating to aria label and aria description may not really work well in our case I am afraid as orca presents the descendants if their content is not already a part of the parent's name or accessible text. So we will either become more aggressive with role=presentation or we'll move everything to the accessible label I think. And of course using role=presentation on focusable elements is not suitable as these need to be reported when focused.

[tyrylu]

I would almost think that if the item has its own a11y label, Orca should not use the descendant speech or maybe braille generator logic, but I did not ask Joanie yet whether it is a bug or not. Anyway, personally, I'd put the post content before the stats in the a11y label.

[pvagner]

@tyrylu The thing is that post content is not yet part of the accessible label at all and @GeopJr is not leaning towards adding it as it may be a long form post such as a blog article in extreme cases. We screen reader users would definatelly like to hear the post content while navigating in the list. So I'm not sure what the consensus might become here.

[GeopJr]

My opinion doesn't have much value in this matter, but I still want to point out extreme cases on the fediverse. From troll posts to blog posts, from memes to code blocks; the post content has to be audibly separated from the post metadata. Repeating myself from a previous comment:

You are both familiar with fedi and linux' accessibility quirks but for someone new to both or either of these, this could lead to confusion.

Either way, I'm close to a feature freeze so I can start focusing on strings, including aria labels, soon.

[tyrylu]

Yes, the character limits are wild in the Fediverse, and I am aware of that, however, in the a11y case, I'd either be okay with it, or I'd include the text if it is not longer than, say, 280 or so characters, and include a truncated version if it is longer than that.

[GeopJr]

Okay so, going back to the original request of adding the content on the LABEL; removing the aria relation of 'the post is described by the widget that shows the post content', causes GTK to try and figure out the description from the children no matter what I do.

So the screen reader ends up announcing posts like this:

• long verbose summary of the post that now includes the post content
• tooltips of the widgets including the post content again

So it ends up appending useless unrelated tooltips and then the post content again 🤷 I tried a lot but nothing seems to prevent GTK from doing this. This requires more research from my side.

A similar situation is the post stat buttons that get announced as "5 toggle button not pressed Favorite". The suggestion was to announce them as "Favorite (5)" instead, but no matter what I do I cannot prevent them from being labelled by the button content which is a label with the text '5' in this example.

---

With that said, I will proceed with merging the rest of the accessibility fixes in the #1226 PR which includes anything from incorrect aria roles to a post aria verbosity setting.

That setting will allow you to set how verbose you want the post aria labels to be from 0-3. It's not meant to be a permanent solution but rather as a way to see what's the optimal situation.

0 = no custom aria label
1 = basics
2-3 = amount of extra details

3 being what's the default

You'll be able to set it either with dconf-editor or with the following command outside of flatpak:

$ gsettings set dev.geopjr.Tuba status-aria-verbosity 3

in flatpak:

$ flatpak run --command=gsettings dev.geopjr.Tuba get status-aria-verbosity 3

(replace 3 with the desired verbosity level. Anything outside the 0-3 range will get normalized)

[tyrylu]

I suspect that the Orca behavior is a bug, e. g. I'd expect the a11y label to take precedence over the heuristic. Maybe report it upstream? Reproducing this should not be so hard in a test case.

[GeopJr]

I'm pretty sure it's on GTK's side and not Orca's because the GTK inspector's 'Accessibility' tab reports "5" as the final name - despite also listing the aria label as "Favorite (5)"

[tyrylu]

Well, yes, GTK can do a descendant deep dive for some roles as well, and excluding some controls from it is not possible, you'd have to set a11y labels, maybe labeled by relations, but for labeled by relations, they're not supposed to be recursive, so a control having a labeled by relation would ignore a labeled by relation on the intermediate widget for its own computation.

[joanmarie]

Hi. Orca maintainer here. @GeopJr: A number of significant changes were made in Orca related to presentation of GTK list items. Some of these are in Orca v47; others are only in the main branch. I just built Tuba from source and tried it with Orca from its main branch. It would be extremely helpful if you could build Orca and tell me what isn't right. Then we can work together to find the best solution. Thanks in advance!

[GeopJr]

Hello 👋

I really don't think Orca is responsible for the button aria label, here's what GTK inspector shows:

While the LABEL is set to "Favorite (7)" and the DESCRIPTION to an empty string (done mostly in the hopes of stopping GTK from going further down the tree), the final Name is still "7". Not even "Favorite (7) 7", the LABEL is being completely ignored.

---

I'll test them both with the main orca branch in a bit because we've been dealing with Orca freezing our machines lately and I can't get it to read the widgets due to that. bgtlover on Matrix has also been dealing with it and recently got a journalctl and has a reproduce, hopefully they'll open an issue on Orca's repo soon!

[joanmarie]

That's interesting. I use the Accerciser accessibility inspector -- without Orca running -- and can sometimes crash Tuba just by looking at Tuba's accessibility tree. Something bad is clearly happening somewhere. Knowing what it is would be helpful.