ERROR: scoreboard: failed to lock (already locked)

Description

After the recent upgrade of application containers to PHP 8.2.14 I see plenty of the error messages in the log:

ERROR: scoreboard: failed to lock (already locked)
ERROR: status: unable to find or access status shared

Is there something I can do to prevent it from happening? End users seem not to be affected as far as I can tell. These log messages are written randomly and only for the PHP-FPM containers. Long running PHP-CLI commands operating under the Supervisor do not have these issues.

opcode: enabled
jit: disabled

Edit: worth to mention that errors randomly occurs even on the container which is barely used during of waiting period.

PHP Version

PHP 8.2.14

Operating System

Debian GNU/Linux 12 (bookworm)

I just checked and this happens when scoreboard is already used by other process. This could happen due to holding lock longer (there was potentially one such change) but a bit before I think. What version was your previous version that you updated from?

Another thing that could play role is high number of children configured. Could you provide your fpm config?

I have upgraded from 8.1.18-fpm-bullseye to 8.2.14-fpm-bookworm

FPM config:

[global]
daemonize = no

log_level = notice

emergency_restart_threshold = 3
emergency_restart_interval = 1m
process_control_timeout = 10s

[www]
access.log = /dev/null

listen = 9000

request_terminate_timeout = 180s

pm = static
pm.max_children = 10
pm.max_requests = 200
pm.process_idle_timeout = 10s

pm.status_path = /status

Could you try to increase (significantly) or remove pm.max_requests and see if that helps?

You should generally not need this and it seems it didn't work correctly before 8.2 (we had couple of reports regarding to that) - it means it didn't restart correctly which was somehow fixed in 8.2

I have removed pm.max_requests option, but error is still present even on the newest 8.2.17-fpm-bookworm.
EDIT: could it have something to do with OPCACHE enabled? but still it happens only now and then.

Used memory | 83587760
Free memory | 84184400

I have been thinking about this and looking through the code / changes. I don't see any specific FPM change that should impact locking. Last fix in that area was also merged to 8.1.

What has got most likely biggest impact on scoreboard locking is probably status collection. Do you have some external collector for status? If so, how often does it run? If often, try to decrease it or temporarily disable it just to see if it helps.

Otherwise there are some other factors that can be the cause. Could you also try it with minimal config just to see if it helps and then try to potentially increase pm.max_children (up to 100) if that still happening:

[global]
daemonize = no
log_level = notice

[www]
listen = 9000

pm = static
pm.max_children = 10

pm.status_path = /status

If it disappears, try to add back the removed options one by one and see when it happens again

@bukka

@bukka thank you for looking into the issue.

I use healthcheck which periodically calls /status endpoint:
https://raw.githubusercontent.com/renatomefi/php-fpm-healthcheck/master/php-fpm-healthcheck

It's set to 10s. Let me test the settings and I will let you know if it is of any help.

@jkabat

@jkabat I'm using the same healthcheck also with an interval set to 10s, getting the same messages in log. Have you been able to resolve this?

If you're running this in a k8s pod this is probably caused by your liveness and readyness probes being executed at the same exact moment.

@realcnbs

@realcnbs yes I have in k8s pods - what would be the solution? Do you have any?

So I finally managed to reliably recreate issue with my new tool called WST funded by STF (Soverign tech fund) that allows me to spin multiple services and test various things in its pipeline - specifically allowing to run load test and checking the logs. The actual test for this can be seen here.

After that I looked properly into the scoreboard locking and found couple of issues. I have got an initial fix in #15805 after which I no longer see the locking error but it might be changing a bit too much so not sure if I should maybe go for lighter version for PHP 8.2 and do the rest in master as it might be a bit risky.

The same with php 8.3.10 and using php-fpm-healthcheck with k8s

Fix phpGH-13437: FPM: ERROR: scoreboard: failed to lock (already locked)

Fix GH-13437: FPM: ERROR: scoreboard: failed to lock (already locked)

@bukka

@bukka i see this as patched which is great to start with. Do you know how far this is backported?

@pimjansen

@pimjansen It will be part of 8.3.16 and 8.4.3 . 8.2 branch is already closed for normal bug fixes and only accepts security fixes.

@pimjansen

@pimjansen It will be part of 8.3.16 and 8.4.3 . 8.2 branch is already closed for normal bug fixes and only accepts security fixes.

Fair! Thanks for quick reply

@bukka

@bukka thanks for investing so much time to resolve the issue!!!

Navigation Menu

Repository navigation

ERROR: scoreboard: failed to lock (already locked) #13437

Description

PHP Version

Operating System

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ERROR: scoreboard: failed to lock (already locked) #13437

Description

Description

PHP Version

Operating System

Activity

bukka commented on Feb 22, 2024

jkabat commented on Feb 22, 2024

bukka commented on Mar 8, 2024

bukka commented on Mar 8, 2024

jkabat commented on Mar 18, 2024

bukka commented on Mar 21, 2024

jkabat commented on Mar 26, 2024

ToshY commented on Apr 18, 2024

realcnbs commented on Jun 12, 2024

jkabat commented on Jul 11, 2024

15 remaining items

bukka commented on Sep 8, 2024

goffyara commented on Sep 11, 2024

pimjansen commented on Dec 14, 2024

bukka commented on Dec 14, 2024

pimjansen commented on Dec 14, 2024

jkabat commented on Dec 19, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions