Skip to content

gh-128213: fast path for bytes creation from list and tuple #132590

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
19 commits
Select commit Hold shift + click to select a range
6f699b5
gh-128213: fast path for bytes creation from list and tuple
blhsing Dec 24, 2024
18c8e4a
coerce long to char after validation of integer in byte range
blhsing Dec 24, 2024
406fbdb
📜🤖 Added by blurb_it.
blurb-it[bot] Dec 24, 2024
4912a05
Update 2024-12-24-08-44-49.gh-issue-128213.Y71jDi.rst
blhsing Dec 24, 2024
56f802e
updated for thread-safety, style choices, function choices and benchm…
blhsing Dec 25, 2024
4e1e3e6
revert to PyLong_AsLongAndOverflow for easier overflow handling
blhsing Dec 25, 2024
bf96d06
fixed issue of a label at the end of a compound statment; revert to u…
blhsing Dec 26, 2024
f3a9423
Add FT test for bytes from list
eendebakpt Mar 26, 2025
8bbc021
Merge branch 'main' into fast-bytes-creation-from-list-tuple-2
eendebakpt Apr 16, 2025
8260c7a
Merge branch 'test_bytes_from_list' into fast-bytes-creation-from-lis…
eendebakpt Apr 16, 2025
bc6f8f2
refactor
eendebakpt Apr 16, 2025
970c10b
refactor
eendebakpt Apr 16, 2025
cb664fe
refactor
eendebakpt Apr 16, 2025
c357217
Update Misc/NEWS.d/next/Core_and_Builtins/2024-12-24-08-44-49.gh-issu…
eendebakpt Apr 16, 2025
5d53346
Merge branch 'main' into fast-bytes-creation-from-list-tuple-2
eendebakpt Apr 16, 2025
739987e
Merge branch 'main' into fast-bytes-creation-from-list-tuple-2
eendebakpt May 23, 2025
8b7c5e6
Merge branch 'main' into fast-bytes-creation-from-list-tuple-2
eendebakpt Jun 18, 2025
ab82e24
lint
eendebakpt Jul 1, 2025
2e5c3c1
Merge branch 'main' into fast-bytes-creation-from-list-tuple-2
eendebakpt Jul 1, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
updated for thread-safety, style choices, function choices and benchm…
…ark notes
  • Loading branch information
blhsing committed Dec 25, 2024
commit 56f802e86688dcf134f6d2adfd50ea62d5c4cf50
Original file line number Diff line number Diff line change
@@ -1,2 +1,3 @@
Speed up :class:`bytes` creation from :class:`list` and :class:`tuple` by around 81%.
Speed up :class:`bytes` creation from :class:`list` and :class:`tuple` of integers. Benchmarks show that from a list with 1000000 random numbers the time to create a bytes object is reduced by around 31%, or 30% with 10000 numbers, or 27% with 100 numbers.

Patch by Ben Hsing
31 changes: 20 additions & 11 deletions Objects/bytesobject.c
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
#include "pycore_bytesobject.h" // _PyBytes_Find(), _PyBytes_Repeat()
#include "pycore_call.h" // _PyObject_CallNoArgs()
#include "pycore_ceval.h" // _PyEval_GetBuiltin()
#include "pycore_critical_section.h" // Py_BEGIN_CRITICAL_SECTION_SEQUENCE_FAST()
#include "pycore_format.h" // F_LJUST
#include "pycore_global_objects.h"// _Py_GET_GLOBAL_OBJECT()
#include "pycore_initconfig.h" // _PyStatus_OK()
Expand Down Expand Up @@ -2813,18 +2814,21 @@
_PyBytes_FromSequence(PyObject *x)
{
Py_ssize_t size = PySequence_Fast_GET_SIZE(x);
PyObject *bytes = PyBytes_FromStringAndSize(NULL, size);
if (bytes == NULL)
PyObject *bytes = _PyBytes_FromSize(size, 0);
if (bytes == NULL) {
return NULL;
char *s = PyBytes_AS_STRING(bytes);
PyObject **items = PySequence_Fast_ITEMS(x);
}
char *str = PyBytes_AS_STRING(bytes);
PyObject *const *items = PySequence_Fast_ITEMS(x);
Py_BEGIN_CRITICAL_SECTION_SEQUENCE_FAST(x);
for (Py_ssize_t i = 0; i < size; i++) {
if (!PyLong_CheckExact(items[i])) {
if (!PyLong_Check(items[i])) {
Py_DECREF(bytes);
return Py_None; // None as fallback sentinel to the slow path
/* Py_None as a fallback sentinel to the slow path */
bytes = Py_None;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
/* Py_None as a fallback sentinel to the slow path */
bytes = Py_None;
/* Py_None as a fallback sentinel to the slow path */
Py_INCREF(Py_None);
bytes = Py_None;

Is needed, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Py_None is immortal, so not needed

goto done;
}
int overflow;
long value = PyLong_AsLongAndOverflow(items[i], &overflow);
int value = PyLong_AsInt(items[i]);
if (value == -1 && PyErr_Occurred()) {
goto error;
}
Expand All @@ -2834,12 +2838,16 @@
"bytes must be in range(0, 256)");
goto error;
}
s[i] = (char)value;
*str++ = (char) value;
}
return bytes;
goto done;
error:
Py_DECREF(bytes);
return NULL;
bytes = NULL;
done:

Check failure on line 2847 in Objects/bytesobject.c

View workflow job for this annotation

GitHub Actions / Address sanitizer (ubuntu-24.04)

label at end of compound statement
/* both success and failure need to end the critical section */
Py_END_CRITICAL_SECTION_SEQUENCE_FAST();

Check failure on line 2849 in Objects/bytesobject.c

View workflow job for this annotation

GitHub Actions / Windows / build and test (arm64)

syntax error: missing ';' before '}' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check failure on line 2849 in Objects/bytesobject.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build and test (arm64)

syntax error: missing ';' before '}' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check failure on line 2849 in Objects/bytesobject.c

View workflow job for this annotation

GitHub Actions / Windows / build and test (x64)

syntax error: missing ';' before '}' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]

Check failure on line 2849 in Objects/bytesobject.c

View workflow job for this annotation

GitHub Actions / Windows (free-threading) / build and test (x64)

syntax error: missing ';' before '}' [D:\a\cpython\cpython\PCbuild\_freeze_module.vcxproj]
return bytes;
}

static PyObject *
Expand Down Expand Up @@ -2924,6 +2932,7 @@

if (PyList_CheckExact(x) || PyTuple_CheckExact(x)) {
PyObject *bytes = _PyBytes_FromSequence(x);
/* Py_None as a fallback sentinel to the slow path */
if (bytes != Py_None) {
return bytes;
}
Expand Down
Loading