Repository navigation
Unknown memory leak in PHP 8.2 FPM #13775
Description
Description
This is an follow up on the issue hi-jacking I did earlier (sorry!). I haven't been able to replicate the issue in development or on our test servers during this time. So I tried a memory profiler on the production server, without any luck. Now I need some new advice from some experts!
#8646 (comment)
After running our application for about an hour we start to see an unhealthy amount of RAM usage. Fast forward a couple of hours and our sever will run out of RAM and start writing to disk instead.
PHP versions tested: 8.2.17, 8.2.14, 8.2.12, 8.2.9, 8.2.8, 8.2.7, 8.2.5, 8.2.4
Configuration: #8646 (comment)
sudo add-apt-repository ppa:ondrej/php
sudo apt update -y
sudo apt install php8.1-common php8.1-fpm php8.1-gd php8.1-mysql php8.1-mbstring php8.1-intl php8.1-xml php8.1-redis php8.1-curl
12 days with PHP 8.1.27 and we utilize 1.3 GB of RAM. Same utilization can be seen using PHP 5.6 - 8.1.
Around 3 hours with PHP 8.2.17 and we utilize 4.5 GB of RAM. This will build up to use all available memory.
I tried to use valgrind massif tool to locate the issue, but it's stable for 7 hours.
I have made two runs, using these settings, they give us the same behavior.
sudo valgrind --tool=massif /usr/sbin/php-fpm8.2
sudo valgrind --tool=massif /usr/sbin/php-fpm8.2 --nodaemonize --fpm-config /etc/php/8.2/fpm/php-fpm.conf
MB
4.480^ #
| # :::::::::::@:::::::::::::::::@:@:
| #::::: :: : ::: @:::: ::: :::: :::@:@::
| #: :: :: : ::: @:::: ::: :::: :::@:@::
| #: :: :: : ::: @:::: ::: :::: :::@:@::
| @#: :: :: : ::: @:::: ::: :::: :::@:@::
| @#: :: :: : ::: @:::: ::: :::: :::@:@::
| @#: :: :: : ::: @:::: ::: :::: :::@:@::
| @#: :: :: : ::: @:::: ::: :::: :::@:@::
| @@#: :: :: : ::: @:::: ::: :::: :::@:@::
| @@#: :: :: : ::: @:::: ::: :::: :::@:@::
| @@#: :: :: : ::: @:::: ::: :::: :::@:@:::
| @@@#: :: :: : ::: @:::: ::: :::: :::@:@:::
| ::::@@@#: :: :: : ::: @:::: ::: :::: :::@:@:::
| :::::: @@@#: :: :: : ::: @:::: ::: :::: :::@:@:::
| ::::::::::::::: @@@#: :: :: : ::: @:::: ::: :::: :::@:@:::
| :@:: :::::: @@@#: :: :: : ::: @:::: ::: :::: :::@:@:::
| ::::@:: :::::: @@@#: :: :: : ::: @:::: ::: :::: :::@:@:::
| : ::@:: :::::: @@@#: :: :: : ::: @:::: ::: :::: :::@:@:::
| ::: ::@:: :::::: @@@#: :: :: : ::: @:::: ::: :::: :::@:@:::
0 +----------------------------------------------------------------------->Mi
0 121.6
Number of snapshots: 52
Detailed snapshots: [6, 16, 17, 18, 19 (peak), 30, 45, 47]
@nielsdos You helped me out before and stated there was another way of using a memory profiler. I didn't see any memory leak using Valgrid thought. But these servers fail 100% of time not using Valgrid.
Did you observe memory leaking behaviour of this run? If yes, that means the problem is somehow not originating from a persistent allocation, and we can give instructions on how to include request memory inside the graphs.
- Is it still worth tracking request?
If not, I will start testing our application with PHP 8.2.0 to see if it's compatible before going the git-bisect route.
PHP Version
8.2.17
Operating System
Ubuntu 22.04.4
Activity
ndossche commentedon Mar 21, 2024
Do you use any third party extensions as well? If yes, I wonder if there may be a leak there too.
Given the information you gave and the tests you did with Valgrind and massif, it seems indeed persistent allocations are not the problem.
I'd say it is worth trying to track request allocations too. To do so, you need to set the environment variable USE_ZEND_ALLOC to 0. This disables Zend's allocator and uses system malloc instead. This will make request allocations visible to Valgrind and massif.
Beware that this may reduce the performance of your code by a few percentage points though.
Edit: although Zend's memory manager should clean up unfreed memory after request shutdown...
jreklund commentedon Mar 22, 2024
We are using php-redis, other than that I would say it's vanilla/core extensions.
I have now made a 6,5 hour Valgrind test with
USE_ZEND_ALLOCdisabled, and according to "htop" I can see a larger memory consumption, but nothing obvious to my eyes in the massif logs. It was a slow build-up and nothing like we use it normally, but over time I guess it would fill up.massif.out.zip <-- download logs here
I read in Generating a valgrind log that you can enable
ZEND_DONT_UNLOAD_MODULES, is that for e.g. tracking third party extensions like redis?ndossche commentedon Mar 22, 2024
I don't really notice anything weird, but that data . I'm sorry, but I don't really have an idea what might be going on.
It's going to be very hard also without a reproducer. Bisecting is an option but might be painful to go through. The advantage is that we might get an exact commit though where it broke.
The allocations will be tracked in any case, but the advantage of using this environment variable is that modules will not be unloaded and therefore symbol names will be visible in stack traces of third party modules.
jreklund commentedon Mar 22, 2024
Thanks for taking a look! I haven't been able to find a reproducer, so next step will be set up a test environment to learn how to make our production environment co-exist with a manually built PHP.
It looks like it's only 10 steps to find the correct commit, but we will need to run the good one for a while, so will take some time for me to set it up.
AFAIK, this is the correct way to start testing it:
ndossche commentedon Mar 22, 2024
That looks right. It's possible you may need to re-run buildconf and configure too when building PHP, during bisection, because internal APIs and build files changed between PHP 8.1 and PHP 8.2.
25 remaining items
jreklund commentedon Jul 4, 2024
Hi! Your latest patch seem stable. Have been running it for 8 hours now. Will leave it over night.
/proc/$pid/mapsfrom a sub-process reaching 80 MB:php.3107394.maps.txt
arnaud-lb commentedon Jul 4, 2024
Great! I will see if we can merge this, then.
jreklund commentedon Jul 5, 2024
No leaks after around 24 hours, so no long time surprises either. 🎉
Will keep it running for around 8 hours more before I switch back. Will be going on vacation and don't want any surprises.
Thank you everyone for your help with my issue and development of PHP in general!
kanevbg commentedon Jul 5, 2024
Great job. I know that PHP 8.1 is out of active support, but is there any chance that being merged for 8.1 too as being fundamental memory issue? Will be great to get it merged at least in 8.1 too :)
jreklund commentedon Jul 5, 2024
@kanevbg This specific problem don't exists in PHP 8.1 or earlier. The code that caused the problem (for us) was introduced in PHP 8.2.
DPlachkov1991 commentedon Jul 5, 2024
Was this released with PHP Version 8.2.21 04 Jul 2024(yesterday) or will it be on a later stage?
ndossche commentedon Jul 5, 2024
No, it will be part of a future release of 8.2.x.
Hint the opcache shm mapping location only when JIT is enabled
arnaud-lb commentedon Jul 22, 2024
#14793 has been merged.
This prevents the opcache SHM from being allocated too close to the heap in most cases, which is deemed to trigger the memory leak.
Non-PIE builds may still allocate the opcache SHM just after the heap when JIT is enabled, but it may be possible to fix that as well if we get more insights regarding the root cause.
jreklund commentedon Jul 22, 2024
Thanks! Will try out the official patch in 3 weeks, when I get back to work. Hopefully I or someone else will find the real cause of the problem in the future.
cpernot-hf commentedon Aug 2, 2024
@arnaud-lb #14793 does not seem to have been merged, but only closed.
The code does not appear in the PHP-8.2.2 tag : https://github.com/php/php-src/blob/PHP-8.2.22/ext/opcache/shared_alloc_mmap.c#L48
Do you have more information ?
ndossche commentedon Aug 2, 2024
@cpernot-hf The 8.2.22 version was tagged before the fix was merged in to the 8.2.x branch. The fix will be included in 8.2.23 which will be released in september. See
php-src/NEWS
Lines 32 to 33 in dc670cb
jreklund commentedon Aug 9, 2024
I'm back from vacation and have been evaluated the new fix available in PHP 8.2 branch and it works without issues. Have been running it for over 3 days.