Repository navigation
In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores. #81932
Description
Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.
Show more details
GitHub fields:
assignee = None
closed_at = <Date 2021-04-24.16:55:03.898>
created_at = <Date 2019-08-03.11:34:13.329>
labels = ['type-bug', '3.9', 'expert-unicode']
title = "In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores."
updated_at = <Date 2022-01-25.00:29:03.446>
user = 'https://github.com/qigangxu'bugs.python.org fields:
activity = <Date 2022-01-25.00:29:03.446>
actor = 'gregory.p.smith'
assignee = 'none'
closed = True
closed_date = <Date 2021-04-24.16:55:03.898>
closer = 'shihai1991'
components = ['Unicode']
creation = <Date 2019-08-03.11:34:13.329>
creator = 'qigangxu'
dependencies = []
files = []
hgrepos = []
issue_num = 37751
keywords = ['patch']
message_count = 31.0
messages = ['348953', '348954', '348956', '348959', '349448', '350086', '350087', '350155', '359970', '359971', '359972', '359973', '359974', '360005', '391648', '391652', '391653', '391654', '391664', '391666', '391671', '391785', '392077', '392159', '392161', '392178', '395730', '395764', '411536', '411537', '411538']
nosy_count = 11.0
nosy_names = ['lemburg', 'gregory.p.smith', 'vstinner', 'mark', 'ezio.melotti', 'methane', 'miss-islington', 'shihai1991', 'qigangxu', 'akdor1154', 'bodograumann']
pr_nums = ['15092', '17997', '23096', '25643', '25659', '25677']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue37751'
versions = ['Python 3.9']
Activity
qigangxu commentedon Aug 3, 2019
In codecs.c, when _PyCodec_Lookup() call normalizestring(), both spaces and hyphens should be convered to underscores. Not convert spaces to hyphens.
see:https://github.com/python/peps/blob/master/pep-0100.txt, Codecs (Coder/Decoders) Lookup
qigangxu commentedon Aug 3, 2019
and I will try to fix it.
shihai1991 commentedon Aug 3, 2019
Hm, there is a bit misleading between desc(https://github.com/python/cpython/blob/master/Python/codecs.c#L53) and the code (https://github.com/python/cpython/blob/master/Python/codecs.c#L74).
qigangxu commentedon Aug 3, 2019
The design and code of the following four places need to be consistent,
No.1 https://github.com/python/peps/blob/master/pep-0100.txt#L292
No.2 https://github.com/python/cpython/blob/master/Python/codecs.c#L113
No.3 https://github.com/python/cpython/blob/master/Python/codecs.c#L53
No.4 https://github.com/python/cpython/blob/master/Python/codecs.c#74
malemburg commentedon Aug 12, 2019
Jordon is right. Conversion has to be to underscores, not hyphens. I guess this bug was introduced when the normalization function was converted to C.
vstinner commentedon Aug 21, 2019
New changeset 20f59fe by Victor Stinner (Jordon Xu) in branch 'master':
bpo-37751: Fix codecs.lookup() normalization (GH-15092)
20f59fe
vstinner commentedon Aug 21, 2019
Thanks for the fix Jordon Xu.
IMHO this change is not strictly a bugfix, but more like an enhancement. I close the issue.
If you consider that a backport to Python 3.7 and 3.8 is needed, please say so.
qigangxu commentedon Aug 22, 2019
Thanks vstinner. I also don't think it's necessary to backport to the old version. Close this issue is fine.
hroncok commentedon Jan 14, 2020
The change is backwards incompatible and a backport would break things. See for example how it breaks latexcodec:
https://bugzilla.redhat.com/show_bug.cgi?id=1789613#c2
vstinner commentedon Jan 14, 2020
I reopen the issue. I proposed PR 17997 to *document* the incompatible change in What's New in Python 3.8. IMO it's a deliberate change and it's correct.
I rely on Marc-Andre Lemburg who implemented codecs and encodings modules. He wrote: "Jordon is right. Conversion has to be to underscores, not hyphens.".
9 remaining items
[-]In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.[/-][+]In codecs, function 'normalizestring' should convert both spaces and hyphens to underscores.[/+]malemburg commentedon Apr 23, 2021
On 23.04.2021 07:47, Inada Naoki wrote:
I guess just an oversight on my part.
PEP-100 is certainly what I meant and implemented. I should have also
made it clear in PEP-100 that I meant lower case ASCII letters.
vstinner commentedon Apr 23, 2021
New changeset 32980fb by Hai Shi in branch 'master':
bpo-37751: Document codecs.lookup() change in What's New in Python 3.9 (GH-23096)
32980fb
shihai1991 commentedon Apr 24, 2021
Thanks Marc-Andre for your supplement of PEP-100.
Thanks Inada, victor for your review and merge.
After PR-23096 merged, I suggest to close this bpo.
If there have any other questions, we can reopen it again.
vstinner commentedon Apr 27, 2021
New changeset 531c810 by Miss Islington (bot) in branch '3.9':
bpo-37751: Document codecs.lookup() change in What's New in Python 3.9 (GH-23096) (GH-25659)
531c810
methane commentedon Apr 28, 2021
New changeset 5c84bb5 by Inada Naoki in branch 'master':
bpo-37751: Update
codecs.register()doc. (GH-25643)5c84bb5
methane commentedon Apr 28, 2021
New changeset cf9d65c by Miss Islington (bot) in branch '3.9':
bpo-37751: Update
codecs.register()doc. (GH-25643)cf9d65c
vstinner commentedon Apr 28, 2021
Thanks Inada-san for documenting the change in codecs.register() doc!
bodograumann commentedon Jun 13, 2021
Unfortunately this is not quite finished yet.
First of all, the change is bigger than what is documented: “Changed in version 3.9: Hyphens and spaces are converted to underscore.“
In reality, now
| Normalization works as follows: all non-alphanumeric
| characters except the dot used for Python package names are
| collapsed and replaced with a single underscore, e.g. ' -;#'
| becomes '_'. Leading and trailing underscores are removed.”
Cf. encodings/init.py
Secondly, this change breaks lots of iconv codecs with the python-iconv binding. E.g.
ASCII//TRANSLITis now normalized toascii_translit, which iconv does not understand. Codec names which use hyphens also break and iinm not all of them have aliases in iconv without hyphens.Cf. python-iconv #4
The codecs api feels extremely well-fitting for integrating iconv in python and any alternative I can think of seems unsatisfactory.
Please advise.
vstinner commentedon Jun 13, 2021
This issue is now closed, would you mind to open a new issue?
gpshead commentedon Jan 25, 2022
https://bugs.python.org/issue46508 filed to track fixing the acceptance and use of garbage codec values regression that this caused.
gpshead commentedon Jan 25, 2022
(note: this might not be the true cause of that issue; though it sounds potentially related - I haven't investigated far enough yet)
gpshead commentedon Jan 25, 2022
note that Bodo's own followup issue about the breaking change for python-iconv was filed as https://bugs.python.org/issue44723