object.__new__ argument calling autodetection faulty

@doko42

BPO	5322
Nosy	@doko42, @scoder, @vstinner, @larryhastings, @benjaminp, @ned-deily, @mitsuhiko, @Trundle, @serhiy-storchaka, @jdemeyer, @pppery, @iritkatriel
Files	update_one_slot.patch update_one_slot2-3.x.patch update_one_slot2-2.7.patch UnsupportedOperation-bases-order.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = None
created_at = <Date 2009-02-19.20:34:23.266>
labels = ['interpreter-core', 'type-bug', '3.9', '3.10', '3.11']
title = 'object.__new__ argument calling autodetection faulty'
updated_at = <Date 2022-01-25.17:58:17.453>
user = 'https://github.com/mitsuhiko'

bugs.python.org fields:

activity = <Date 2022-01-25.17:58:17.453>
actor = 'vstinner'
assignee = 'serhiy.storchaka'
closed = False
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2009-02-19.20:34:23.266>
creator = 'aronacher'
dependencies = []
files = ['13408', '45525', '45526', '45784']
hgrepos = []
issue_num = 5322
keywords = []
message_count = 31.0
messages = ['82497', '82505', '84112', '85828', '86111', '86702', '145289', '281064', '282224', '282225', '282605', '282612', '282615', '282617', '282852', '282942', '282969', '282974', '282976', '282979', '282999', '283170', '283178', '283209', '283243', '283245', '284637', '411478', '411479', '411480', '411647']
nosy_count = 17.0
nosy_names = ['doko', 'scoder', 'vstinner', 'larry', 'sebastinas', 'benjamin.peterson', 'ned.deily', 'aronacher', 'prologic', 'Trundle', 'Ringding', 'python-dev', 'serhiy.storchaka', 'jdemeyer', 'ppperry', 'thansen', 'iritkatriel']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue5322'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

In 2.6 a deprecation warning was added if object.__new__ was called
with arguments. Per se this is fine, but the detection seems to be faulty.

The following code shows the problem:

>>> class A(object):
...     def __new__(self):
...         raise TypeError('i do not exist')
... 
>>> class B(A):
...     __new__ = object.__new__
...     def __init__(self, x):
...         self.x = x
... 
>>> B(1)
__main__:1: DeprecationWarning: object.__new__() takes no parameters
<__main__.B object at 0x88dd0>

In the B case __new__ is not overridden (in the sense that it
differs from object.new) but __init__ is. Which is the default
behaviour. Nonetheless a warning is raised.

I used the pattern with the "__new__ switch" to achieve a
cStringIO.StringIO behavior that supports typechecks: IterIO() returns
either a IterI or IterO object, both instances of IterIO so that
typechecks can be performed.

Real-world use case here:
http://dev.pocoo.org/projects/werkzeug/browser/werkzeug/contrib/iterio.py

The problem seems to be caused by tp_new being slot_tp_new which then
invokes whatever __new__ in the class dict is.

I'm not so sure what would be the solution to this. One could of course
check if tp_new is either object_new or slot_tp_new and in the latter
case check if the class dict's __new__ item is object_new...

I think the real problem here is update_one_slot and not object_new. It
is impossible to set "new" to a PyCFunction inside Python code, which
may be a feature, but is in fact very irritating.

For example the following snippet:

>>> class Dict(dict): __new__ = object.__new__
...
>>> Dict.__new__ is object.__new__
True
>>> Dict()
{}

I would rather expect this behaviour (or at least that Dict.__new__ is not
object.__new__):

>>> Dict.__new__ is object.__new__
True
>>> Dict()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object.__new__(Dict) is not safe, use dict.__new__()

The attached patch leads to that behaviour, which also fixes the argument
calling autodetection of object.__new__.

I'm sorry, I don't have any opinion on this.

The problem is that type_setattro() sets the new "new" attribute
in the type's dict (through PyObject_GenericSetAttr()), but the
corresponding slot will never be updated if the new "new" is a
PyCFunction.

The affected code in update_one_slot() was added by Guido van Rossum
in r28090, so maybe he would like to comment on that.

See also issue bpo-1694663.

I think it needs tests.

Here are updated patches with tests for 3.x and 2.7.

New changeset a37cc3d926ec by Serhiy Storchaka in branch '2.7':
Issue bpo-5322: Fixed setting __new__ to a PyCFunction inside Python code.
https://hg.python.org/cpython/rev/a37cc3d926ec

New changeset f89ef18f9824 by Serhiy Storchaka in branch '2.7':
Issue bpo-5322: Restored tests for __new__.
https://hg.python.org/cpython/rev/f89ef18f9824

New changeset 06e4b9f2e4b0 by Serhiy Storchaka in branch '3.5':
Revert changeset 1f31bf3f76f5 (bpo-5322) except tests.
https://hg.python.org/cpython/rev/06e4b9f2e4b0

BTW, at least for bpo-25731, I think the right approach in the MI case is to synthesize a __new__ on the subclass that calls the solid base __new__.

Yes, it was what the patch did by setting tp_new to slot_tp_new. The problem is that the same code is used for inherited __new__ and assigned in class body. It is hard to distinguish between these cases.

In any case I think that Cython shouldn't generate trivial __new__. This will help to change the order of __new__ resolution at least in 3.7.

It worries me that nothing in the Python docs nor in any PEP describes how tp_new is inherited. In my opinion, this patch makes a significant change which should be subject to a PEP. However, neither the old nor new behaviour is described anywhere. This also makes it harder to argue which behaviour is correct.

Can we close this now?

>>> class A(object):
...     def __new__(self):
...          raise TypeError('i do not exist')
... 
>>> class B(A):
...     __new__ = object.__new__
...     def __init__(self, x):
...         self.x = x
... 
>>> B(1)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: object.__new__() takes exactly one argument (the type to instantiate)
>>>

The bug is still there, just that it's now not just a warning but an error. The auto detection is incorrect here. It should allow the instantiation of the object with arguments.

Right, I see now.

Python has the same behavior since Python 2.6. While it annoys a few persons, the majority doesn't care. I suggest to close the issue.

It's easy to workaround limitation that object.__new__() which doesn't accept arguments.

Navigation Menu

Repository navigation

object.new argument calling autodetection faulty #49572

15 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

object.__new__ argument calling autodetection faulty #49572

Description

Activity

mitsuhiko commented on Feb 19, 2009

mitsuhiko commented on Feb 19, 2009

Trundle commented on Mar 24, 2009

pitrou commented on Apr 9, 2009

Trundle commented on Apr 18, 2009

Trundle commented on Apr 27, 2009

benjaminp commented on Oct 10, 2011

serhiy-storchaka commented on Nov 17, 2016

python-dev commented on Dec 2, 2016

15 remaining items

python-dev commented on Dec 14, 2016

benjaminp commented on Dec 15, 2016

serhiy-storchaka commented on Dec 15, 2016

jdemeyer commented on Jan 4, 2017

iritkatriel commented on Jan 24, 2022

mitsuhiko commented on Jan 24, 2022

iritkatriel commented on Jan 24, 2022

vstinner commented on Jan 25, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

object.new argument calling autodetection faulty #49572