gh-95756: Lazily created cached co_* attrs #97791

Fidget-Spinner · 2022-10-03T20:31:21Z

As long as these aren't accessed, they don't actually consume additional space over 3.11.

Issue: Better performance for code getters in C API #95756

Fidget-Spinner · 2022-10-03T20:40:11Z

Note that these were all O(1) in 3.10. This restores their behavior. They also previously returned just incref of a cache.
https://github.com/python/cpython/blame/08eb754d840696914928355014c2d424131f8835/Objects/codeobject.c

markshannon · 2022-10-03T21:44:20Z

I'd like to reduce the size of code objects for a few reasons. PEP 649 will create lots of small code objects and we want multiple interpreters to share as much data as possible.

markshannon · 2022-10-07T01:05:18Z

I'm merging this, as we can add the line array, and any other tables needed in unusual cases, to the _co_cached structs.
In other words, this will save space.

Doc/c-api/code.rst

Objects/codeobject.c

bedevere-bot · 2022-10-07T01:10:35Z

When you're done making the requested changes, leave the comment: I have made the requested changes; please review again.

…trs_cached

markshannon · 2022-10-10T14:40:57Z

Objects/codeobject.c

+    if (*cached_field != NULL) {
+        return Py_NewRef(*cached_field);
+    }
+    assert(*cached_field == NULL);


No need to assert this. The test two lines above ensures that it is true.

markshannon

One assert that could be removed, but it's harmless.

Fidget-Spinner · 2022-10-11T03:25:53Z

I'll leave the assert in since it communicates some info should the code change in the future. And like you mentioned it's harmless anyways.

bedevere-bot · 2022-10-11T03:38:28Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM64 macOS 3.x has failed when building commit b399115.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/all/#builders/725/builds/2786) and take a look at the build logs.
Check if the failure is related to this commit (b399115) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/all/#builders/725/builds/2786

Failed tests:

test_syslog

Summary of the results of the build (if available):

==

Click to see traceback logs

remote: Enumerating objects: 30, done.        
remote: Counting objects:   3% (1/30)        
remote: Counting objects:   6% (2/30)        
remote: Counting objects:  10% (3/30)        
remote: Counting objects:  13% (4/30)        
remote: Counting objects:  16% (5/30)        
remote: Counting objects:  20% (6/30)        
remote: Counting objects:  23% (7/30)        
remote: Counting objects:  26% (8/30)        
remote: Counting objects:  30% (9/30)        
remote: Counting objects:  33% (10/30)        
remote: Counting objects:  36% (11/30)        
remote: Counting objects:  40% (12/30)        
remote: Counting objects:  43% (13/30)        
remote: Counting objects:  46% (14/30)        
remote: Counting objects:  50% (15/30)        
remote: Counting objects:  53% (16/30)        
remote: Counting objects:  56% (17/30)        
remote: Counting objects:  60% (18/30)        
remote: Counting objects:  63% (19/30)        
remote: Counting objects:  66% (20/30)        
remote: Counting objects:  70% (21/30)        
remote: Counting objects:  73% (22/30)        
remote: Counting objects:  76% (23/30)        
remote: Counting objects:  80% (24/30)        
remote: Counting objects:  83% (25/30)        
remote: Counting objects:  86% (26/30)        
remote: Counting objects:  90% (27/30)        
remote: Counting objects:  93% (28/30)        
remote: Counting objects:  96% (29/30)        
remote: Counting objects: 100% (30/30)        
remote: Counting objects: 100% (30/30), done.        
remote: Compressing objects:   6% (1/15)        
remote: Compressing objects:  13% (2/15)        
remote: Compressing objects:  20% (3/15)        
remote: Compressing objects:  26% (4/15)        
remote: Compressing objects:  33% (5/15)        
remote: Compressing objects:  40% (6/15)        
remote: Compressing objects:  46% (7/15)        
remote: Compressing objects:  53% (8/15)        
remote: Compressing objects:  60% (9/15)        
remote: Compressing objects:  66% (10/15)        
remote: Compressing objects:  73% (11/15)        
remote: Compressing objects:  80% (12/15)        
remote: Compressing objects:  86% (13/15)        
remote: Compressing objects:  93% (14/15)        
remote: Compressing objects: 100% (15/15)        
remote: Compressing objects: 100% (15/15), done.        
remote: Total 16 (delta 14), reused 2 (delta 1), pack-reused 0        
From https://github.com/python/cpython
 * branch                  main       -> FETCH_HEAD
Note: switching to 'b399115ef18945f7526492225d72a512048ad20d'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at b399115ef1 gh-95756: Lazily created cached co_* attrs (GH-97791)
Switched to and reset branch 'main'

Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include/c++/4.2.1
configure: WARNING: ffi.h: accepted by the compiler, rejected by the preprocessor!
configure: WARNING: ffi.h: proceeding with the compiler's result
configure: WARNING: pkg-config is missing. Some dependencies may not be detected correctly.

Fatal Python error: Fatal Python error: Segmentation faultSegmentation fault



Thread 0x000000028301f000 (most recent call first):
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/test/test_syslog.py", line 67 in logger
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 986 in run
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 1049 in _bootstrap_inner
  File "/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/Lib/threading.py", line 1006 in _bootstrap

Thread 0x0000000282013000make: *** [buildbottest] Segmentation fault: 11

Cannot open file '/Users/buildbot/buildarea/3.x.pablogsal-macos-m1.macos-with-brew/build/test-results.xml' for upload

Fidget-Spinner · 2022-10-11T05:15:45Z

I'm confused. Why is this the only buildbot failing? In any case I might revert this within 24 hours.

vstinner · 2022-10-11T07:37:11Z

I'm confused. Why is this the only buildbot failing? In any case I might revert this within 24 hours.

I created #98178 about this crash. It seems to be specific to macOS and unrelated to your change. If your change would introduce a bug, more tests would crash on more platforms.

Cached co_* attrs

2e28fb7

bedevere-bot added the awaiting core review label Oct 3, 2022

📜🤖 Added by blurb_it.

4e85e31

Fidget-Spinner marked this pull request as ready for review October 3, 2022 20:33

Fidget-Spinner requested a review from markshannon as a code owner October 3, 2022 20:33

Fidget-Spinner requested a review from brandtbucher October 3, 2022 20:33

Fidget-Spinner added the sprint label Oct 3, 2022

markshannon requested changes Oct 7, 2022

View reviewed changes

Doc/c-api/code.rst Outdated Show resolved Hide resolved

Doc/c-api/code.rst Outdated Show resolved Hide resolved

Doc/c-api/code.rst Outdated Show resolved Hide resolved

Objects/codeobject.c Outdated Show resolved Hide resolved

Objects/codeobject.c Outdated Show resolved Hide resolved

bedevere-bot removed the awaiting core review label Oct 7, 2022

bedevere-bot added the awaiting changes label Oct 7, 2022

Fidget-Spinner added 5 commits October 7, 2022 12:17

apply suggestions

6cfa7ae

fix segfault

ce5f0e8

yes

d998aa1

Update 2022-10-03-20-33-24.gh-issue-95756.SSmXlG.rst

d321cfb

Merge branch 'main' of https://github.com/python/cpython into code_at…

3e227fa

…trs_cached

markshannon reviewed Oct 10, 2022

View reviewed changes

markshannon approved these changes Oct 10, 2022

View reviewed changes

bedevere-bot added awaiting merge and removed awaiting changes labels Oct 10, 2022

Fidget-Spinner merged commit b399115 into python:main Oct 11, 2022

bedevere-bot removed the awaiting merge label Oct 11, 2022

Fidget-Spinner deleted the code_attrs_cached branch October 11, 2022 03:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

gh-95756: Lazily created cached co_* attrs #97791

gh-95756: Lazily created cached co_* attrs #97791

Fidget-Spinner commented Oct 3, 2022 •

edited

Loading

Fidget-Spinner commented Oct 3, 2022 •

edited

Loading

markshannon commented Oct 3, 2022

markshannon commented Oct 7, 2022

bedevere-bot commented Oct 7, 2022

markshannon Oct 10, 2022

markshannon left a comment

Fidget-Spinner commented Oct 11, 2022

bedevere-bot commented Oct 11, 2022

Fidget-Spinner commented Oct 11, 2022 •

edited

Loading

vstinner commented Oct 11, 2022

gh-95756: Lazily created cached co_* attrs #97791

gh-95756: Lazily created cached co_* attrs #97791

Conversation

Fidget-Spinner commented Oct 3, 2022 • edited Loading

Fidget-Spinner commented Oct 3, 2022 • edited Loading

markshannon commented Oct 3, 2022

markshannon commented Oct 7, 2022

bedevere-bot commented Oct 7, 2022

markshannon Oct 10, 2022

Choose a reason for hiding this comment

markshannon left a comment

Choose a reason for hiding this comment

Fidget-Spinner commented Oct 11, 2022

bedevere-bot commented Oct 11, 2022

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Fidget-Spinner commented Oct 11, 2022 • edited Loading

vstinner commented Oct 11, 2022

Fidget-Spinner commented Oct 3, 2022 •

edited

Loading

Fidget-Spinner commented Oct 3, 2022 •

edited

Loading

Fidget-Spinner commented Oct 11, 2022 •

edited

Loading