ext/standard: Add php_next_utf8_char_fast() by ArtUkrainskiy · Pull Request #24283 · php/php-src

ArtUkrainskiy · 2026-10-12T21:44:09Z

json_encode(), utf8_decode() and the xml extension decode one character at a time through the exported php_next_utf8_char(). This adds php_next_utf8_char_fast() in html.h to decode valid UTF-8 at the call site. Invalid or truncated input falls back to the existing exported function, preserving its error handling and ABI.

get_next_char() is already always inlined into the exported decoder. This still avoids the exported call for valid input. The measurements compare the change with master 97335aa86f7, including #24145.

Fresh release builds use the official Docker-image compiler and linker flags, plus -g and --enable-werror. No explicit alignment flags are used. Five alternating passes per case on Intel and fifteen on AMD. Unescaped Unicode rows use JSON_UNESCAPED_UNICODE. Negative = fewer cycles (faster).

call	master, CPU cycles per call (GCC 13.3)	GCC 13.3	GCC 14.2	clang 18.1	clang 20.1	GCC 15.2 (AMD)	clang 21.1 (AMD)
Cyrillic, 4 KiB	35,063	−19%	−28%	+12%	−22%	−22%	−14%
Cyrillic, unescaped Unicode, 4 KiB	56,511	−15%	−29%	−21%	−25%	−17%	−22%
CJK, 4 KiB	22,791	−7%	−11%	−20%	−19%	−24%	−21%
CJK, unescaped Unicode, 4 KiB	25,056	−11%	−11%	−13%	−15%	−18%	−20%
Emoji, 4 KiB	25,041	−10%	−9%	−14%	−9%	−7%	−11%
Emoji, unescaped Unicode, 4 KiB	18,933	−17%	−16%	−14%	−14%	−22%	−18%
French, 4 KiB	24,467	−4%	−3%	−1%	−17%	−15%	−8%
ASCII with random-position e acute, 4 KiB	23,056	−5%	−7%	−3%	−10%	−5%	−4%
ASCII with periodic e acute, unescaped Unicode, 4 KiB	21,786	−17%	+24%	−6%	−4%	−6%	−3%
ASCII, 16 bytes	135	+7%	−4%	0%	+1%	+1%	+1%
ASCII, 4 KiB	8,114	+6%	0%	0%	0%	+6%	+1%
ASCII, 64 KiB	134,588	+3%	0%	0%	0%	+5%	0%
HTML, 4 KiB	28,450	+5%	−1%	−1%	0%	+4%	0%
URLs, 4 KiB	30,128	+8%	+1%	−1%	0%	−1%	−1%
Quotes only, 4 KiB	25,683	0%	0%	−1%	+1%	+12%	0%
A newline every 40 bytes, 4 KiB	22,313	−33%	−21%	0%	0%	+7%	0%
1,000 arrays of 10 fields	888,668	+16%	−6%	+3%	−4%	−1%	0%
1,000 objects of 10 properties	832,933	+13%	+2%	+1%	+4%	0%	0%

The full report includes instructions, all 26 inputs, pass ranges and regressions. Maximum absolute A/A median deviation: Intel 1.31%, AMD 0.23%. Differences comparable to the controls should not be treated as established gains or regressions. The earlier alignment-profile figures are archived as diagnostics.

The existing exhaustive byte-sequence checks and fuzz generators match master on all six Docker-profile builds. JSON, XML and string tests pass on GCC 13, clang 20 and GCC 15. I haven't rerun sanitizers or tested aarch64, 32-bit or MSVC builds.

Full results, exact flags and reproduction: https://github.com/ArtUkrainskiy/php-src-bench/tree/main/reports/json-encode-utf8-inline

adapik

I went through the implementation and I find the approach reasonable: the inline path only takes sequences that are valid by construction, and everything else falls back to the unchanged php_next_utf8_char(), so the error semantics can't diverge. I also ran it through my own benchmarks and the results are solid.

Regarding overlap with my other PRs (24283 and 24282):

The dirty table is a small, self-contained change to the clean/dirty classification of a byte. I think it stands on its own and combines well with this PR, since this PR changes what happens after a byte is classified as dirty (non-ASCII), not the classification itself.
I don't see any overlap with the SIMD work either. The two optimizations operate on different branches of php_json_escape_string() (the SIMD one on runs of clean ASCII, this one on the non-ASCII decode), so they complement each other.

Depending on which one is merged first, I'm happy to rebase mine, that's not a problem.

adapik · 2026-10-13T16:53:29Z

@@ -0,0 +1,217 @@
+--TEST--
+json_encode() with valid and invalid UTF-8 sequences around the two-, three- and four-byte boundaries


The new test only covers json_encode(). Since the validation logic is now duplicated between html.c and html.h, a small phpt exercising the same boundary sequences through utf8_decode() would stop the two copies from drifting apart silently.

ArtUkrainskiy · 2026-10-13T18:36:10Z

@adapik Thanks for reviewing it and for running it through your benchmarks.

Test: I added ext/standard/tests/strings/utf8_decode_sequences.phpt with the same boundary sequences through utf8_decode(), plus overlong encodings of é, which utf8_decode() makes visible as é instead of "?". The expected output comes from master. To check that the two tests catch drift between the copies, I mutated the inline decoder one condition at a time:

accepting C0/C1 leads, three- and four-byte overlong forms, skipping the third-byte check, advancing the cursor by 2 instead of 3: both tests fail;
accepting surrogates or code points above U+10FFFF: only the json_encode() test fails, because utf8_decode() turns both a valid character above U+00FF and an invalid sequence into a single "?".

Header: I put the function in html.h next to php_next_utf8_char() because of the earlier review of #18126, where the suggestion was to optimize the existing decoder rather than add a new one elsewhere (#18126 (comment)). A small dedicated header, with php_next_utf8_char() staying exported from html.c, is a change of a few lines here.

@bukka, as the owner of ext/standard and ext/json: do you prefer the inline decoder in html.h, as it is now, or in a separate header (for example ext/standard/php_utf8.h)? I'll update the PR either way.

ArtUkrainskiy · 2026-10-26T08:02:19Z

@bukka @iluuu1994 could you take a look when you get a chance? The only open point is the function name, happy to rename. It adds php_next_utf8_char_inline() in html.h and uses it in json_encode. Stacks cleanly with #24666.

bukka

I think this look reasonable. Just naming could be better as noted. I merged #24145 so get_next_char is now always inlined which slightly reduced the initial motivation for this. It should still improve things but it would be good to re-measure it. It should also update the commit message which is no longer completely true...

bukka · 2026-10-31T18:12:47Z


+/* Same contract as php_next_utf8_char(). Decodes valid UTF-8 inline; invalid input goes
+ * to php_next_utf8_char(). */
+static zend_always_inline unsigned int php_next_utf8_char_inline(


I don't like this name - Can you rename it to php_next_utf8_char_fast or something more consistent.

Renamed to php_next_utf8_char_fast(), remeasured and updated description

The remaining ASCII/escaping regressions may be addressed by the scan changes in #23675. I'll check the combined results once this PR lands.

Decode valid UTF-8 at the call site in json_encode(), utf8_decode() and the xml extension. Invalid or truncated input falls back to the unchanged exported php_next_utf8_char(), preserving its error handling and ABI. get_next_char() is already always inlined into php_next_utf8_char(), but callers in other translation units still pay for the exported call.

github-actions Bot added Extension: json Extension: standard Extension: xml ABI break labels Oct 12, 2026

ArtUkrainskiy mentioned this pull request Oct 12, 2026

ext/json: Add SSE2/NEON fast path for clean-string runs in php_json_escape_string() #23675

Open

ArtUkrainskiy force-pushed the json-utf8-2byte branch from ca35c07 to 63792a2 Compare October 13, 2026 00:28

ArtUkrainskiy mentioned this pull request Oct 13, 2026

ext/json: replace packed charmap bitmap with a byte-indexed lookup table #23682

Open

ArtUkrainskiy marked this pull request as ready for review October 13, 2026 02:15

ArtUkrainskiy requested review from bukka and devnexen as code owners October 13, 2026 02:15

adapik reviewed Oct 13, 2026

View reviewed changes

ArtUkrainskiy force-pushed the json-utf8-2byte branch from 63792a2 to f331165 Compare October 13, 2026 18:09

ArtUkrainskiy mentioned this pull request Oct 14, 2026

ext/standard: Check for UTF-8 first when resolving the charset of html functions #23961

Merged

ArtUkrainskiy mentioned this pull request Oct 26, 2026

ext/json: Speed up scanning and escaping of strings #24666

Open

bukka requested changes Oct 31, 2026

View reviewed changes

ArtUkrainskiy force-pushed the json-utf8-2byte branch from f331165 to df2b554 Compare October 31, 2026 20:05

ArtUkrainskiy changed the title ~~ext/standard: Add php_next_utf8_char_inline()~~ ext/standard: Add php_next_utf8_char_fast() Oct 31, 2026

Navigation Menu

Repository navigation

ext/standard: Add php_next_utf8_char_fast() - #24283

ext/standard: Add php_next_utf8_char_fast()#24283
ArtUkrainskiy wants to merge 1 commit into
php:masterfrom
ArtUkrainskiy:json-utf8-2byte

ArtUkrainskiy commented Oct 12, 2026 •

edited

Loading

Uh oh!

adapik left a comment

Uh oh!

adapik Oct 13, 2026

Uh oh!

Uh oh!

ArtUkrainskiy commented Oct 13, 2026

Uh oh!

ArtUkrainskiy commented Oct 26, 2026

Uh oh!

bukka left a comment

Uh oh!

bukka Oct 31, 2026

Uh oh!

ArtUkrainskiy Oct 31, 2026

Uh oh!

ArtUkrainskiy Oct 31, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		@@ -0,0 +1,217 @@
		--TEST--
		json_encode() with valid and invalid UTF-8 sequences around the two-, three- and four-byte boundaries

Conversation

ArtUkrainskiy commented Oct 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

adapik left a comment

Choose a reason for hiding this comment

Uh oh!

adapik Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ArtUkrainskiy commented Oct 13, 2026

Uh oh!

ArtUkrainskiy commented Oct 26, 2026

Uh oh!

bukka left a comment

Choose a reason for hiding this comment

Uh oh!

bukka Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

ArtUkrainskiy Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

ArtUkrainskiy Oct 31, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ArtUkrainskiy commented Oct 12, 2026 •

edited

Loading