Adding Gujarati Language support

[sarjil77]

Hey @felixdittrich92,

i have added the gujarati language and made changes in datasets.srt, please review my PR.

[felixdittrich92]

Hi @sarjil77 👋🏼,

One small issue left here: (See the failing CI job)

AssertionError: Duplicate characters in gujarati_vowels vocab: ['અ']

Please remove all duplicates and afterwards run make style for formatting :)

[sarjil77]

the 'અ' is not repeating multiple times, it may seems like there is repeating but it is not, the error we are getting is becuase of there is '.' and ':' with the 'અ'. and it is considering it as duplication but in real they are distinct characters.

Solution: i will remove 'અ' containing other full stop and colon. and keeping only character and will use formatting as well.

[sarjil77]

done sir, thanks :)

[felixdittrich92]

Formatting looks good now 👍🏼 But still duplicates:

AssertionError: Duplicate characters in gujarati_consonants vocab: ['ક', 'જ', 'ઞ', 'ષ', '્']

:)

[sarjil77]

shit, this duplication wont let me sleep, let me fix it now.

[sarjil77]

and yes there is not a single duplicated character, but there is some hidden character exists within the char and thats why it is flagging it as an duplicate,

such as: because of 'જ્ઞ' we were getting 'જ', 'ઞ', '્' as duplicate. 'ક' and 'ષ' because it contains subcharcter within the char.

i hope you are getting me

i am fixing it as of now :) :)

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 96.56%. Comparing base (48873e0) to head (85876b1).
Report is 11 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1845      +/-   ##
==========================================
- Coverage   96.60%   96.56%   -0.05%     
==========================================
  Files         165      165              
  Lines        7929     7941      +12     
==========================================
+ Hits         7660     7668       +8     
- Misses        269      273       +4     

Flag	Coverage Δ
unittests	96.56% <100.00%> (-0.05%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[sarjil77]

@felixdittrich92, i think it is fine now, please let me knwo if further error arises, going to sleep now :)

thanks in advance

[felixdittrich92]

Looks good now 👍🏼 thanks 😊

[sarjil77]

Thanks @felixdittrich92, i would also like to hop on to the #1131 which is good first issue, would you please guide me over there, i have made comments over there.

[felixdittrich92]

Thanks @felixdittrich92, i would also like to hop on to the #1131 which is good first issue, would you please guide me over there, i have made comments over there.

👍🏼 Sure will write something done if online in a few minutes 😅