Gabriel Torres Posted May 16, 2019 Share Posted May 16, 2019 Hello, I am passing a fine-tooth comb here and found the errors below in our php-fpm.log. It seems we have scripts giving segmentation fault. Should I worry about this or should I investigate this further to discover what is causing the issue? [16-May-2019 13:20:45] WARNING: [pool www] child 29596 exited on signal 11 (SIGSEGV) after 209.615147 seconds from start [16-May-2019 13:20:45] NOTICE: [pool www] child 29978 started [16-May-2019 13:21:04] WARNING: [pool www] child 29598 exited on signal 11 (SIGSEGV) after 228.275454 seconds from start [16-May-2019 13:21:04] NOTICE: [pool www] child 29992 started [16-May-2019 13:21:15] WARNING: [pool www] child 29599 exited on signal 11 (SIGSEGV) after 239.868879 seconds from start [16-May-2019 13:21:15] NOTICE: [pool www] child 29998 started [16-May-2019 13:21:16] WARNING: [pool www] child 29672 exited on signal 11 (SIGSEGV) after 170.376334 seconds from start [16-May-2019 13:21:16] NOTICE: [pool www] child 29999 started [16-May-2019 13:21:17] WARNING: [pool www] child 29601 exited on signal 11 (SIGSEGV) after 241.358841 seconds from start [16-May-2019 13:21:17] NOTICE: [pool www] child 30001 started [16-May-2019 13:21:40] WARNING: [pool www] child 29578 exited on signal 11 (SIGSEGV) after 264.793022 seconds from start [16-May-2019 13:21:40] NOTICE: [pool www] child 30023 started [16-May-2019 13:21:42] WARNING: [pool www] child 30001 exited on signal 11 (SIGSEGV) after 25.237446 seconds from start Thanks. Link to comment Share on other sites More sharing options...
The Old Man Posted May 16, 2019 Share Posted May 16, 2019 Hi, Yes, I'm seeing similar in my PHP-FPM logs for all of my IPS community sites and have done for a while. I also get occasional random 503 errors too, I refresh the page and all is fine. There's another topic here where it was suspected to be PHP7.3 related. Im running latest Apache 2.4, CentOS7.6, PHP7.3, MariaDB 10.3, Opcache, Redis. 16-May-2019 10:52:20] NOTICE: ready to handle connections [16-May-2019 10:52:20] NOTICE: systemd monitor interval set to 10000ms [16-May-2019 10:52:32] WARNING: [pool mysite_net] child 8183 exited on signal 11 (SIGSEGV) af$[16-May-2019 10:52:32] NOTICE: [pool mysite_net] child 8615 started [16-May-2019 10:52:38] NOTICE: Terminating ... [16-May-2019 10:52:38] NOTICE: exiting, bye-bye! [16-May-2019 10:52:39] NOTICE: fpm is running, pid 9519 [16-May-2019 10:52:39] NOTICE: ready to handle connections [16-May-2019 10:52:39] NOTICE: systemd monitor interval set to 10000ms [16-May-2019 12:54:20] NOTICE: [pool my_second_site_co_uk] child 16435 exited with code 0 after 35.679686 s$[16-May-2019 12:54:20] NOTICE: [pool my_second_site_co_uk] child 17718 started [16-May-2019 13:45:18] NOTICE: [pool my_second_site_co_uk] child 28848 exited with code 0 after 9.778749 se$[16-May-2019 13:45:18] NOTICE: [pool my_second_site_co_uk] child 29135 started [16-May-2019 16:03:57] WARNING: [pool my_second_site_co_uk] child 3031 exited on signal 11 (SIGSEGV) after $[16-May-2019 16:03:57] NOTICE: [pool my_second_site_co_uk] child 3032 started [16-May-2019 16:42:42] NOTICE: [pool mysite_net] child 17740 exited with code 0 after 79.9165$[16-May-2019 16:42:42] NOTICE: [pool mysite_net] child 20543 started [16-May-2019 17:41:30] NOTICE: [pool my_third_site_info] child 15840 exited with code 0 after 19.563616 second$[16-May-2019 17:41:30] NOTICE: [pool my_third_site_info] child 16497 started [16-May-2019 17:46:28] NOTICE: [pool mysite_net] child 27588 exited with code 0 after 7.71593$[16-May-2019 17:46:28] NOTICE: [pool mysite_net] child 27784 started [16-May-2019 17:48:04] WARNING: [pool mysite_net] child 30617 said into stderr: "^@" [16-May-2019 17:51:15] NOTICE: [pool my_second_site_co_uk] child 5030 exited with code 0 after 9.689596 sec$[16-May-2019 17:51:15] NOTICE: [pool my_second_site_co_uk] child 5318 started [16-May-2019 18:00:37] NOTICE: [pool mysite_net] child 23294 exited with code 0 after 60.7625$[16-May-2019 18:00:37] NOTICE: [pool mysite_net] child 25460 started [16-May-2019 18:05:36] NOTICE: [pool my_third_site_info] child 3312 exited with code 0 after 12.748291 seconds$[16-May-2019 18:05:36] NOTICE: [pool my_third_site_info] child 3720 started [16-May-2019 18:28:29] NOTICE: [pool mysite_net] child 20135 exited with code 0 after 15.9234$[16-May-2019 18:28:29] NOTICE: [pool mysite_net] child 20667 started [16-May-2019 19:23:33] NOTICE: [pool mysite_net] child 6325 exited with code 0 after 145.0255$[16-May-2019 19:23:33] NOTICE: [pool mysite_net] child 11422 started [16-May-2019 19:50:40] NOTICE: [pool mysite_net] child 2143 exited with code 0 after 66.94470$[16-May-2019 19:50:40] NOTICE: [pool mysite_net] child 4398 started [16-May-2019 19:55:12] NOTICE: [pool mysite_net] child 11711 exited with code 0 after 76.2264$[16-May-2019 19:55:12] NOTICE: [pool mysite_net] child 14559 started [16-May-2019 19:57:18] NOTICE: [pool mysite_net] child 15225 exited with code 0 after 105.117$[16-May-2019 19:57:18] NOTICE: [pool mysite_net] child 18917 started [16-May-2019 20:13:29] WARNING: [pool mysite_net] server reached max_children setting (10), c$[16-May-2019 20:25:41] NOTICE: [pool mysite_net] child 14231 exited with code 0 after 37.0384$[16-May-2019 20:25:41] NOTICE: [pool mysite_net] child 15398 started [16-May-2019 20:47:29] WARNING: [pool mysite_net] child 28629 exited on signal 11 (SIGSEGV) a$[16-May-2019 20:47:29] NOTICE: [pool mysite_net] child 29011 started [16-May-2019 21:36:54] WARNING: [pool mysite_net] child 6925 exited on signal 11 (SIGSEGV) af$[16-May-2019 21:36:54] NOTICE: [pool mysite_net] child 6986 started I increased the number of workers to match FPM workers as read earlier this week to avoid a bottleneck, contrary to WHM advice on not needing to increase from the default 5, but it's still happening. May be 2 or 3 separate things. Here's the link if it's of interest. https://confluence1.cpanel.net/plugins/servlet/mobile?contentId=17183046#content/view/17183046 Link to comment Share on other sites More sharing options...
Gabriel Torres Posted May 17, 2019 Author Share Posted May 17, 2019 Hi @The Old Man I will investigate this further and let you know. This was also happening with PHP 7.2. I'm also getting random 502 errors here, I am discussing this in a separate topic: https://invisioncommunity.com/forums/topic/452388-excessive-502-errors/ Link to comment Share on other sites More sharing options...
Gabriel Torres Posted May 17, 2019 Author Share Posted May 17, 2019 Hello! After a little studying, I discovered that we need to enable to generate a "core dump" so we can see the details of what is going on. I followed the tutorial available at https://ma.ttias.be/generate-php-core-dumps-segfaults-php-fpm/ To make it work correctly, I had to execute a few steps that are not in this tutorial: 1. Needed to recompile PHP with --enable-debug and change the path of the external modules (opcache.so and redis.so in my case) to debug-non-zts-20180731 2. Needed to run: ulimit -c unlimited echo 0 > /proc/sys/kernel/core_uses_pid echo 2 > /proc/sys/fs/suid_dumpable Otherwise php-fpm wouldn't generate the dump files. With php-fpm creating the dumps correctly at /tmp, I was able to see that all segmentation faults were related to the mysqlnd module. See below a full backtrace: Program terminated with signal SIGSEGV, Segmentation fault. #0 0x00007f1e98de2931 in php_mysqlnd_free_field_metadata (meta=0x7f1e7f001018) at /downloads/php-7.3.5/ext/mysqlnd/mysqlnd_result_meta.c:36 36 meta->root = NULL; #0 0x00007f1e98de2931 in php_mysqlnd_free_field_metadata (meta=0x7f1e7f001018) at /downloads/php-7.3.5/ext/mysqlnd/mysqlnd_result_meta.c:36 #1 0x00007f1e98de341c in mysqlnd_mysqlnd_res_meta_free_pub ( meta=0x7f1e7f3d6b78) at /downloads/php-7.3.5/ext/mysqlnd/mysqlnd_result_meta.c:106 #2 0x00007f1e98dd84f2 in mysqlnd_mysqlnd_res_free_result_contents_internal_pub (result=0x7f1e7f3d6048) at /downloads/php-7.3.5/ext/mysqlnd/mysqlnd_result.c:300 #3 0x00007f1e98dd8691 in mysqlnd_mysqlnd_res_free_result_internal_pub ( result=0x7f1e7f3d6048) at /downloads/php-7.3.5/ext/mysqlnd/mysqlnd_result.c:316 #4 0x00007f1e98ddfd5a in mysqlnd_mysqlnd_res_free_result_pub ( result=0x7f1e7f3d6048, implicit=0 '\000') at /downloads/php-7.3.5/ext/mysqlnd/mysqlnd_result.c:1507 #5 0x00007f1e98bc36e0 in zif_mysqli_free_result (execute_data=0x7f1e8e8231d0, return_value=0x7ffcc39a86c0) at /downloads/php-7.3.5/ext/mysqli/mysqli_api.c:1373 #6 0x00007f1e98f0026d in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /downloads/php-7.3.5/Zend/zend_vm_execute.h:980 #7 0x00007f1e98f67a86 in execute_ex (ex=0x7f1e8e823060) at /downloads/php-7.3.5/Zend/zend_vm_execute.h:55485 #8 0x00007f1e98e84ced in zend_call_function (fci=0x7ffcc39a8a40, fci_cache=0x7ffcc39a89f0) #9 0x00007f1e98ec5444 in zend_call_method (object=0x7f1e7e017178, obj_ce=0x7f1e7faac800, fn_proxy=0x7f1e7faac9e8, function_name=0x7f1e9977a195 "rewind", function_name_len=6, retval_ptr=0x0, param_count=0, arg1=0x0, arg2=0x0) at /downloads/php-7.3.5/Zend/zend_interfaces.c:103 #10 0x00007f1e98ec59d2 in zend_user_it_rewind (_iter=0x7f1e7e017140) at /downloads/php-7.3.5/Zend/zend_interfaces.c:224 #11 0x00007f1e98c6f11d in spl_dual_it_rewind (intern=0x7f1e7e34b000) at /downloads/php-7.3.5/ext/spl/spl_iterators.c:1671 #12 0x00007f1e98c6f3f1 in zim_spl_dual_it_rewind (execute_data=0x7f1e8e823010, return_value=0x7ffcc39a8d30) at /downloads/php-7.3.5/ext/spl/spl_iterators.c:1735 #13 0x00007f1e98e84dc5 in zend_call_function (fci=0x7ffcc39a8d40, fci_cache=0x7ffcc39a8cf0) at /downloads/php-7.3.5/Zend/zend_execute_API.c:770 #14 0x00007f1e98ec5444 in zend_call_method (object=0x7f1e7e107358, obj_ce=0x7f1e80f8a6c0, fn_proxy=0x7f1e80f8a8a8, function_name=0x7f1e9977a195 "rewind", function_name_len=6, retval_ptr=0x0, param_count=0, arg1=0x0, arg2=0x0) at /downloads/php-7.3.5/Zend/zend_interfaces.c:103 #15 0x00007f1e98ec59d2 in zend_user_it_rewind (_iter=0x7f1e7e107320) at /downloads/php-7.3.5/Zend/zend_interfaces.c:224 #16 0x00007f1e98c7468a in spl_iterator_apply (obj=0x7f1e8e823000, apply_func=0x7f1e98c7477f <spl_iterator_to_array_apply>, puser=0x7f1e8e822f70) at /downloads/php-7.3.5/ext/spl/spl_iterators.c:3516 #17 0x00007f1e98c749bc in zif_iterator_to_array (execute_data=0x7f1e8e822fb0, return_value=0x7f1e8e822f70) at /downloads/php-7.3.5/ext/spl/spl_iterators.c:3601 #18 0x00007f1e98effe6f in ZEND_DO_FCALL_BY_NAME_SPEC_RETVAL_USED_HANDLER () at /downloads/php-7.3.5/Zend/zend_vm_execute.h:892 #19 0x00007f1e98f67a79 in execute_ex (ex=0x7f1e8e822b40) at /downloads/php-7.3.5/Zend/zend_vm_execute.h:55481 #20 0x00007f1e98e84ced in zend_call_function (fci=0x7ffcc39a9240, fci_cache=0x7ffcc39a91f0) at /downloads/php-7.3.5/Zend/zend_execute_API.c:756 #21 0x00007f1e98ec5444 in zend_call_method (object=0x7f1e8e8220f0, obj_ce=0x7f1e81308398, fn_proxy=0x7f1e813084c8, function_name=0x7f1e9977da63 "__tostring", function_name_len=10, retval_ptr=0x7ffcc39a9350, param_count=0, arg1=0x0, arg2=0x0) at /downloads/php-7.3.5/Zend/zend_interfaces.c:103 #22 0x00007f1e98eebeb2 in zend_std_cast_object_tostring ( readobj=0x7f1e8e8220f0, writeobj=0x7ffcc39a93f0, type=6) at /downloads/php-7.3.5/Zend/zend_object_handlers.c:1699 #23 0x00007f1e98e8dec1 in zval_get_string_func (op=0x7f1e8e8220f0) at /downloads/php-7.3.5/Zend/zend_operators.c:878 #24 0x00007f1e98ef518a in zval_get_string (op=0x7f1e8e8220f0) at /downloads/php-7.3.5/Zend/zend_operators.h:272 #25 0x00007f1e98f49ebf in ZEND_CAST_SPEC_CV_HANDLER () at /downloads/php-7.3.5/Zend/zend_vm_execute.h:37586 #26 0x00007f1e98f6b8b3 in execute_ex (ex=0x7f1e8e81e6c0) at /downloads/php-7.3.5/Zend/zend_vm_execute.h:59377 #27 0x00007f1e98e84ced in zend_call_function (fci=0x7ffcc39a9820, fci_cache=0x7ffcc39a9800) at /downloads/php-7.3.5/Zend/zend_execute_API.c:756 #28 0x00007f1e98cca899 in zif_call_user_func_array ( execute_data=0x7f1e8e81e650, return_value=0x7f1e8e81e600) at /downloads/php-7.3.5/ext/standard/basic_functions.c:4942 #29 0x00007f1e98effe6f in ZEND_DO_FCALL_BY_NAME_SPEC_RETVAL_USED_HANDLER () at /downloads/php-7.3.5/Zend/zend_vm_execute.h:892 #30 0x00007f1e98f67a79 in execute_ex (ex=0x7f1e8e81e030) at /downloads/php-7.3.5/Zend/zend_vm_execute.h:55481 #31 0x00007f1e98f6d080 in zend_execute (op_array=0x7f1e8e87e000, return_value=0x0) at /downloads/php-7.3.5/Zend/zend_vm_execute.h:60881 #32 0x00007f1e98e9bcfe in zend_execute_scripts (type=8, retval=0x0, file_count=3) at /downloads/php-7.3.5/Zend/zend.c:1568 #33 0x00007f1e98e0a989 in php_execute_script (primary_file=0x7ffcc39ac270) at /downloads/php-7.3.5/main/main.c:2630 #34 0x00007f1e98f7f026 in main (argc=6, argv=0x7ffcc39ac4a8) This is all Martian to me. I need now to investigate what this really means. Any help would be highly appreciated. @The Old Man I suggest you to go through the same process to see what is causing the segmentation faults in your server. I am really curious about that. Cheers, Gabriel. Link to comment Share on other sites More sharing options...
The Old Man Posted May 17, 2019 Share Posted May 17, 2019 Thanks Gabriel, fascinating stuff! I need to work through this and the guide. Coredumps I actually turned off years again as they completely filled my hosting in almost every folder when I was with a previous host. It seems you can limit them from what a quick read of that site. Link to comment Share on other sites More sharing options...
Gabriel Torres Posted May 17, 2019 Author Share Posted May 17, 2019 @The Old Man Yep, that is what I did here. Enabled only to get a few coredumps from php-fpm and then disabled it. Cheers. Link to comment Share on other sites More sharing options...
bfarber Posted May 17, 2019 Share Posted May 17, 2019 https://bugs.php.net/bug.php?id=77955 https://bugs.php.net/bug.php?id=77734 Looks like a PHP 7.3-specific bug with php-fpm that is currently open (the first link...the second was a duplicate report). Link to comment Share on other sites More sharing options...
The Old Man Posted May 17, 2019 Share Posted May 17, 2019 Awesome, thanks for that. Hopefully they'll be able to nail it soon! 😀 Link to comment Share on other sites More sharing options...
Gabriel Torres Posted May 17, 2019 Author Share Posted May 17, 2019 Thanks, @bfarber! It seems to be a bug indeed! The backtrace posted there seems identical to mine! 🙂 So, no other solution than waiting for PHP 7.3.6 and hope they fix this. Link to comment Share on other sites More sharing options...
Gabriel Torres Posted May 19, 2019 Author Share Posted May 19, 2019 @The Old Man @bfarber Downgraded to PHP 7.2 and the segmentation faults are now gone. So it is really a PHP 7.3 issue. Will keep monitoring to see if we got rid of the 502 errors we were getting as well. Will keep you posted in the other topic. Link to comment Share on other sites More sharing options...
The Old Man Posted May 20, 2019 Share Posted May 20, 2019 Thanks for that, I might downgrade to 7.2 also. Link to comment Share on other sites More sharing options...
The Old Man Posted May 20, 2019 Share Posted May 20, 2019 I downgraded to 7.2 and the errors stopped. I even remembered to edit my Cron jobs from php73 to php72, and after wondering why all of my sites went down, I remembered to re-install the Redis and imagick PHP extensions for 7.2! Link to comment Share on other sites More sharing options...
Gabriel Torres Posted May 20, 2019 Author Share Posted May 20, 2019 After downgrading to 7.2, we stopped seeing the 502 errors as well. So it seems these 502 errors are in fact related to these segmentation faults. Cheers. Link to comment Share on other sites More sharing options...
ASTRAPI Posted May 20, 2019 Share Posted May 20, 2019 It will be good if we vote on that bug to get a bit higher attention maybe... https://bugs.php.net/bug.php?id=77955 Link to comment Share on other sites More sharing options...
RevengeFNF Posted May 20, 2019 Share Posted May 20, 2019 21 hours ago, Gabriel Torres said: @The Old Man @bfarber Downgraded to PHP 7.2 and the segmentation faults are now gone. So it is really a PHP 7.3 issue. Will keep monitoring to see if we got rid of the 502 errors we were getting as well. Will keep you posted in the other topic. >I also see the segmentation faults with php 7.3. Link to comment Share on other sites More sharing options...
Gabriel Torres Posted May 20, 2019 Author Share Posted May 20, 2019 1 minute ago, RevengeFNF said: >I also see the segmentation faults with php 7.3. Downgrade to 7.2 4 minutes ago, ASTRAPI said: It will be good if we vote on that bug to get a bit higher attention maybe... https://bugs.php.net/bug.php?id=77955 Already done it. In fact, I was the first one to vote... LOL 🙂 Link to comment Share on other sites More sharing options...
ASTRAPI Posted May 23, 2019 Share Posted May 23, 2019 The bug is fixed ! Link to comment Share on other sites More sharing options...
The Old Man Posted May 23, 2019 Share Posted May 23, 2019 Great news, thank you! Link to comment Share on other sites More sharing options...
ASTRAPI Posted May 27, 2019 Share Posted May 27, 2019 Hello As the patch will not be available for the new php version 7.3.6 (coming soon) and we must wait more than a month for the 7.3.7 to get that patch i decide to do a custom back port to check if that patch will fix the issue. After testing it for a day the problem is fixed and all are working as expected ! Will push that fix to all my clients tomorrow. So the version that will have that fix will be 7.3.7 and it will be released probably at July. Link to comment Share on other sites More sharing options...
The Old Man Posted May 27, 2019 Share Posted May 27, 2019 Thanks Astrapi, much appreciated. Shame it's not in the next 7.3 release. Link to comment Share on other sites More sharing options...
Gabriel Torres Posted May 28, 2019 Author Share Posted May 28, 2019 Thanks @ASTRAPI! I thought in doing that myself (patching manually), however, we can run PHP 7.2 here in the meantime. Cheers. Link to comment Share on other sites More sharing options...
ASTRAPI Posted May 28, 2019 Share Posted May 28, 2019 Php 7.3.6 (that doesn't have the fix) will be released tomorrow .... Link to comment Share on other sites More sharing options...
AlexJ Posted June 16, 2019 Share Posted June 16, 2019 Thank God for this topic! Seriously, with my php 7.3 on Debian 9 it's same issue. php -v PHP 7.3.6-1+0~20190531112735.39+stretch~1.gbp6131b7 (cli) (built: May 31 2019 11:27:35) ( NTS ) Copyright (c) 1997-2018 The PHP Group Zend Engine v3.3.6, Copyright (c) 1998-2018 Zend Technologies with Zend OPcache v7.3.6-1+0~20190531112735.39+stretch~1.gbp6131b7, Copyright (c) 1999-2018, by Zend Technologies [Sun Jun 16 00:58:05.805852 2019] [core:notice] [pid 13773] AH00052: child pid 27190 exit signal Segmentation fault (11) [Sun Jun 16 00:58:52.922712 2019] [core:notice] [pid 13773] AH00052: child pid 27346 exit signal Segmentation fault (11) [Sun Jun 16 01:06:54.573596 2019] [core:notice] [pid 13773] AH00052: child pid 29066 exit signal Segmentation fault (11) [Sun Jun 16 01:07:08.606385 2019] [core:notice] [pid 13773] AH00052: child pid 29067 exit signal Segmentation fault (11) [Sun Jun 16 01:10:08.839122 2019] [core:notice] [pid 13773] AH00052: child pid 29836 exit signal Segmentation fault (11) [Sun Jun 16 01:18:13.551029 2019] [core:notice] [pid 13773] AH00052: child pid 31447 exit signal Segmentation fault (11) [Sun Jun 16 01:30:27.724824 2019] [core:notice] [pid 13773] AH00052: child pid 1119 exit signal Segmentation fault (11) [Sun Jun 16 02:12:59.378401 2019] [core:notice] [pid 13773] AH00052: child pid 9652 exit signal Segmentation fault (11) [Sun Jun 16 04:28:56.832962 2019] [core:notice] [pid 13773] AH00052: child pid 5102 exit signal Segmentation fault (11) [Sun Jun 16 04:40:28.667607 2019] [core:notice] [pid 13773] AH00052: child pid 8707 exit signal Segmentation fault (11) Link to comment Share on other sites More sharing options...
ASTRAPI Posted June 16, 2019 Share Posted June 16, 2019 Just do a custom patch on existing version or wait for the new release that will have that fix 🙂 Link to comment Share on other sites More sharing options...
AlexJ Posted June 17, 2019 Share Posted June 17, 2019 On 6/16/2019 at 9:03 AM, ASTRAPI said: Just do a custom patch on existing version or wait for the new release that will have that fix 🙂 Q - would that fix work for mod php as well? I don't have php fpm and yet i have same issue. Thanks bud! Link to comment Share on other sites More sharing options...
Recommended Posts
Archived
This topic is now archived and is closed to further replies.