apache2 saturated by www-data process not autokilled
short description
apache2 is stuck because lot of processes owned by www-data aren't killed properly apache2 reload, does nothing apache2 restart, solves the issue
problem occured multiple times, no special errors in logs
correct behaviour
processes should stop, disapeared & respawn
environment
Server OS: debian 9.5 ISPConfig version: 3.1.12
Server version: Apache/2.4.25 (Debian) Server built: 2018-06-02T08:01:13
PHP 7.2.8-1+020180725124257.2+stretch1.gbp571e56 (cli) (built: Jul 25 2018 12:43:00) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
with Zend OPcache v7.2.8-1+020180725124257.2+stretch1.gbp571e56, Copyright (c) 1999-2018, by Zend Technologies
apache2 status
Apache Server Status for localhost (via ::1)
Server Version: Apache/2.4.25 (Debian) mod_fastcgi/mod_fastcgi-SNAP-0910052141 mod_fcgid/2.3.9 Phusion_Passenger/5.0.30 mod_python/3.3.1 Python/2.7.13 OpenSSL/1.0.2l Server MPM: prefork Server Built: 2018-06-02T08:01:13
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Current Time: Wednesday, 01-Aug-2018 08:49:53 CEST Restart Time: Monday, 30-Jul-2018 09:20:03 CEST Parent Server Config. Generation: 5 Parent Server MPM Generation: 4 Server uptime: 1 day 23 hours 29 minutes 50 seconds Server load: 0.05 0.07 0.02 Total accesses: 223720 - Total Traffic: 23.7 GB CPU Usage: u46.26 s2.38 cu0 cs0 - .0284% CPU load 1.31 requests/sec - 145.1 kB/second - 110.9 kB/request 150 requests currently being processed, 0 idle workers
GGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGWGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGGG GGGGGGGGGGGGGGGGGGGGGG
Scoreboard Key: "_" Waiting for Connection, "S" Starting up, "R" Reading Request, "W" Sending Reply, "K" Keepalive (read), "D" DNS Lookup, "C" Closing connection, "L" Logging, "G" Gracefully finishing, "I" Idle cleanup of worker, "." Open slot with no current process
apache2 error log
Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 108968. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 109121. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 109531. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 109532. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 109533. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 109534. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 109542. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 109543. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 109544. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 109545. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 111069. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 111070. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 111080. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 111081. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 111082. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 111187. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 112745. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 113926. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 114493. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 114494. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 114495. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 114496. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 114497. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 114498. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 115115. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 115153. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 118078. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 118260. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 121719. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 121720. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 122362. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 123031. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 123237. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 123715. [Wed Aug 01 02:31:59.778688 2018] [mpm_prefork:error] [pid 17170] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 125177. Argument ""-"" isn't numeric in numeric gt (>) at /usr/local/ispconfig/server/scripts/vlogger line 514, line 126123. [Wed Aug 01 06:25:05.223776 2018] [mpm_prefork:notice] [pid 17170] AH00171: Graceful restart requested, doing restart
[ 2018-08-01 06:25:05.8596 11700/7f2c4b646700 age/Cor/CoreMain.cpp:532 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown) [ 2018-08-01 06:25:05.8601 11700/7f2c51e68780 age/Cor/CoreMain.cpp:901 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected... [ 2018-08-01 06:25:05.9956 11700/7f2c4943e700 Ser/Server.h:817 ]: [ApiServer] Freed 0 spare client objects [ 2018-08-01 06:25:06.0125 11700/7f2c4943e700 Ser/Server.h:464 ]: [ApiServer] Shutdown finished [ 2018-08-01 06:25:06.0212 11700/7f2c4a542700 Ser/Server.h:817 ]: [ServerThr.3] Freed 128 spare client objects [ 2018-08-01 06:25:06.0202 11700/7f2c4b646700 Ser/Server.h:817 ]: [ServerThr.1] Freed 128 spare client objects [ 2018-08-01 06:25:06.0208 11700/7f2c49cc0700 Ser/Server.h:817 ]: [ServerThr.4] Freed 128 spare client objects [ 2018-08-01 06:25:06.0471 11700/7f2c4a542700 Ser/Server.h:464 ]: [ServerThr.3] Shutdown finished [ 2018-08-01 06:25:06.0471 11700/7f2c4b646700 Ser/Server.h:464 ]: [ServerThr.1] Shutdown finished [ 2018-08-01 06:25:06.0471 11700/7f2c49cc0700 Ser/Server.h:464 ]: [ServerThr.4] Shutdown finished [ 2018-08-01 06:25:06.0211 11700/7f2c4adc4700 Ser/Server.h:817 ]: [ServerThr.2] Freed 128 spare client objects [ 2018-08-01 06:25:06.0486 11700/7f2c4adc4700 Ser/Server.h:464 ]: [ServerThr.2] Shutdown finished
[ 2018-08-01 06:25:06.0782 11709/7f8396ca4700 age/Ust/UstRouterMain.cpp:422 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown) [ 2018-08-01 06:25:06.0787 11709/7f839d2eb780 age/Ust/UstRouterMain.cpp:492 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected... [ 2018-08-01 06:25:06.1235 11709/7f8396ca4700 Ser/Server.h:464 ]: [UstRouter] Shutdown finished [ 2018-08-01 06:25:06.1235 11709/7f83964a3700 Ser/Server.h:817 ]: [UstRouterApiServer] Freed 0 spare client objects [ 2018-08-01 06:25:06.1236 11709/7f83964a3700 Ser/Server.h:464 ]: [UstRouterApiServer] Shutdown finished [ 2018-08-01 06:25:06.1621 11709/7f839d2eb780 age/Ust/UstRouterMain.cpp:523 ]: Passenger UstRouter shutdown finished [ 2018-08-01 06:25:06.2544 11700/7f2c51e68780 age/Cor/CoreMain.cpp:967 ]: Passenger core shutdown finished