-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
5.38 outputs locale errors with UTF-8 LANG environment variable set #150
Comments
These are the same symptoms as in #97 The workaround for that (not documented in that issue) was to ensure LC_ALL was set. See also Perl/perl5#21567 |
Also, are you running under docker? |
setting LC_ALL doesn't seem to work for me?
I'm not running in docker (yet..), just a standard MSYS2 UCRT64 terminal |
What happens if you run using a Cmd shell? Run the |
No changes with or without portableshell.bat Without:
With:
|
Thanks for testing. I suspect this is an issue in the perl core and might have been addressed in the 5.39 development series. Just as a final check, what happens if you change the code page in the Cmd prompt before calling perl? For example for "plain" UTF-8: |
No change |
I think this is just part of the general poor support for UTF-8 locales with mingw-w64 msvcrt runtimes.
Same system, perl-5.38.0 built by Visual Studio 2022:
Same system, perl-5.39.3, built mingw-w64 port of gcc-13.2.0 ( "ucrt" build, not "msvcrt" build):
It's tempting to therefore think that this issue has been fixed in perl-5.39.3. I haven't tried building perl-5.38.0 with that same UCRT mingw-w64 compiler. Cheers, |
I think another batch of locale changes landed in time for perl 5.39.4, which was released yesterday. Would be worth testing. |
I'm game. ...how do I do that? :) |
For a basic vanilla perl, which is all that is needed here, you can adapt the instructions in #39 (comment) Alternately you could run with the strawberry perl infrastructure but delete any of the CPAN entries to save time from the config file. The one for 5.38 is at https://github.com/StrawberryPerl/Perl-Dist-Strawberry/blob/master/share/64bit-5.38.0.1.pp . Clone this repo, then build the Perl::Dist::Strawberry module using whichever perl you like (edit - whichever strawberry perl). Then adapt the script at https://github.com/StrawberryPerl/Perl-Dist-Strawberry/blob/master/devel.utils/_build-5.38.bat to refer to the correct locations. |
You'll need a UCRT compiler - eg from https://winlibs.com. On a perl-5.39.4 built using
But the output of that one-liner reverts to:
when perl-5.39.4 is built using Cheers, |
Well that answers my suggestion that recent perl changes might fix the issue for MSVCRT compilers. So the next question is whether the MSVCRT limitations can be worked around? |
I don't know. The latest gcc-13.2.0 compilers have boosted the runtime version from, 11.0.0 to 11.0.1. Perhaps (unlikely ?) they address this issue. (I haven't tested.) There was a request for submissions re the backporting of stuff into 5.38.1. For 5.40 onwards, I think that using a UCRT compiler will be the thing to do. BTW, I have just checked that 5.38.0 doesn't build with a UCRT compiler - and it doesn't. Cheers, |
This discussion looks to also be relevant to #18 |
Are you suggesting that modifying perlexe.manifest as @ikegami suggested (https://learn.microsoft.com/en-us/windows/apps/design/globalizing/use-utf8-code-page) might fix @LordAro's issue ? I take it that it should be fairly simple to test. I'm not entirely sure what that "appropriate form" looks like.
Even simpler:
Place that file in the directory that contains perl.exe, cd to that directory and run:
Win32::GetACP() then returns 65001. What else do we want to test ? NOTE 1: We should probably also bring perl5.38.0.exe into line with perl.exe:
NOTE 2: It might be a good idea to do all of this in a separate installation of portable SP-5.38.0. I used mt.exe from my VisualStudio-2022 installation. Cheers, |
I was more thinking the UTF-8 issues raised here would also impact that request, but if there is a solution without needing a new build then that's also welcome. |
Heh ... I see. @LordAro earlier mentioned the "workaround" of setting the PERL_BADLANG environment variable to a true value. @LordAro, are you sure that the requested locale was actually being set in perl-5.32 and 5.36 ? Cheers, |
The changes need to build with UCRT are a lot simpler than I expected, See Perl/perl5@26b63da
You can test whether the locale supported with something like:
If you have the perl source unpacked, the loc_tools.pl test library can list the libraries it finds on the system:
Perl is limited to the locales available from the CRT whether on Windows or POSIX systems. |
Perl/perl5#21567 is unrelated to LC_ALL being set in the environment. |
This seems.. possible? (Reverted to en_US to make sure that my output matches yours, also couldn't work out what the "proper" full name of en_GB.UTF-8 was)
(with codepage 850) |
When, on a UCRT build of perl-5.39.4 I run the script that @tonycoz provided, I see the following listings for "English":
(That's a bit odd - my locale is apparently English_Australia.1252, and that's not even mentioned.) Oddly, with SP-5.38.0, when I try to set LC_ALL to
I can't explain that difference. Cheers, |
The warnings are based on what's in the environment. As to the Australia locale not being listed, the list on Windows is synthesized from a list at the end of |
That list in |
Maybe the powershell code in this link is a good place to start? |
Did anybody solve this? |
If you're asking whether there's a known solution that will fix this problem on current Strawberry Perl builds, then AFAIK the answer is "no". If you're asking whether there's a known solution that will avoid this problem on future releases of Strawberry Perl then I think the answer is "yes". Based on the behaviour of my current perl builds that were built using UCRT compilers (both Visual Studio 2022 and mingw-w64 ports of gcc-13.2.0), it seems that will fix many locale-related issues. |
UCRT aside, this is something that needs to be handled in Perl itself. See Perl/perl5#21562 |
Yes, thanks for pointing that out. That issue has been labelled as a "Release Blocker", so there's obviously something there that needs to be addressed by perl itself, in addition to what UCRT takes care of. UPDATE: Just to make it a little clearer, it was incorrect of me to say that UCRT will help with this particular issue. |
I just ran the original test case on blead, and got the same result as reported. Further investigation showed that the message displayed is actually correct. There is no en_GB.UTF-8 locale on my system. If I run
I get no message. Similarly with The release blocker is more about the default ACP. It comes up not because the locale doesn't exist on the system, but because using the locale could crash perl. These are multi-byte locales that aren't UTF-8, and it would be a lot of work to support them. But just because the machine's default locale is unsupported by perl doesn't mean that things will actually proceed to have perl use that locale. XS code certainly could and run into trouble, and some POSIX:: functions also could, but pure perl requires a 'use locale' to enable any use of said unsupported locale, and the locale may have been changed to a legal one in the meantime. So the message is potentially bogus. The case in this ticket is that the environment is explicitly set up to use a locale unknown to the system. And perl has long warned about this; it just didn't catch these instances before. |
Perl 5.39.10 contains code to address this issue (see Perl/perl5#22160). A strawberry perl build for testing can be accessed via https://github.com/StrawberryPerl/Perl-Dist-Strawberry/releases/tag/SP_5.39.10 |
SP 5.40.0.1 has now been released. This uses UCRT so this problem should be resolved now. A 5.38.x UCRT build might be released but it can be done under a separate issue. I'll close this in a few days, or when next I do a sweep of issues. Edit: release is at https://github.com/StrawberryPerl/Perl-Dist-Strawberry/releases/tag/SP_54001_64bit_UCRT |
Closing as noted. |
Any execution of perl 5.38 with a UTF-8 language set results in
it's not quite clear what languages are supported and installed on my system, and why this is only an issue now.
There's a few locale related things in 5.38, which it appears to be related to:
https://perldoc.perl.org/perl5380delta#Locale-category-LC_NAME-now-supported-on-participating-platforms
https://perldoc.perl.org/perl5380delta#Changes-to-%22thread-safe%22-locales
though I've no idea which.
Codepage is 850, but there's no difference if I set it to 65001 either.
Happens in both powershell and msys, and with either of
LANG
&LC_NAME
- which would definitely suggest that it's perl at fault rather than MSYS / powershell.There is a workaround, but it's hacky:
No issues with 5.32 or 5.36.
This feels related to, but distinct from, #119 . I have tried 5.38.0.2, it had no effect.
The text was updated successfully, but these errors were encountered: