Repository navigation
Unicode conversion issue #1466
Description
Environment
- Pythonnet version: 3.0.0-preview2021-06-04
- Python version: 3.8.8
- Operating System: ubuntu 21.04
- .NET Runtime: 5.0.300
Details
Unicode characters get lost / mangled during conversion:
scope.Exec("testStr = 'Nom 🍗';");
scope.Exec("""print("python:", testStr);""")
Console.WriteLine($"""dotnet: {scope.Get("testStr").ToString()}""");writes
python: Nom 🍗
dotnet: Nom �
Reacted by Victor
Activity
pkese commentedon Jun 7, 2021
Presumably the problem is surrogate pairs.
It seems that two-pair unicode sequences get incorrect string length - e.g. in actual letters rather than the amount of unicode characters that are required to properly encode these letters.
results in
^^^ Notice how one dot at the end of string is missing in dotnet version
pkese commentedon Jun 7, 2021
Analogously, putting two unicode surrogates into string will eat two dots at the end
pkese commentedon Jun 8, 2021
It looks this problem goes in both directions:
"foo🐼"to be 4 characters long, but we somehow create a Python object with Length 5.I found some hints of how Python treats unicode conversion internally https://stackoverflow.com/questions/36098984/python-3-3-c-api-and-utf-8-strings
Using
PyUnicode_DecodeUTF16orPyUnicode_DecodeUTF16Statefulinstead ofPyUnicode_FromKindAndDatawould maybe alleviate this problem. But getting these functions imported from .dlls and getting delegates in place is probably beyond my capacity.filmor commentedon Jun 8, 2021
Thank you for the detailed evaluation, I'll have a look at this this weekend. We indeed treat all strings coming from .NET as UCS2 and that breaks as soon as surrogate pairs are involved.
pkese commentedon Jun 8, 2021
I managed to fix the Python to dotnet path in runtime.cs: 1b6c6c0
This fixes one side of the problem - i.e. the initial bug report.
However the dotnet -> Python conversion issue still remains.
filmor commentedon Jun 8, 2021
Very nice, I'll work on the other direction. The underlying issue (I think) is that C#
"some str".Lengththat we use returns the number of Unicode code points, which may be 16 or 32 bit in UTF-16. The safest solution is likely the one you suggested, to use the UTF16-specific functions of the Python API.filmor commentedon Jun 11, 2021
Fixed by #1467, thank you very much :)
pkese commentedon Jun 11, 2021
And thank You, @filmor, for fixing all the rest.