记一次docker mysql被oom排查

409次阅读
共计 13802 个字符，预计需要花费 35 分钟才能阅读完成。
最近博客老是出现无法连接数据库故障！
排查腾讯云监控的时候看到内存使用 700 多 M, 问题就出在这里，当时没有使用 top 命令查看，
腾讯云监控的内存使用的计算方式是取 /proc/meminfo 的
Memtotal-MemFree-Buffers-Cached-SReclaimable（SReclaimable 代表可回收的 slab 内存大小）
所有与 top 命令的出入。本服务器没有开启交换空间 sawp
所以是因为内存不足导致的 oom 通过系统的计算算出 docker mysql 的内存占用最大所以直接 oom 了！
排查出现 oom 的过程可以查看 /var/log/messages 日志情况
实际上是我的 php-fpm 进程占用内存过多导致的，优化 fpf-fpm.conf 参数即可，优化需要根据自己的业务场景优化！
就不贴出我的配置了！有需要的可以找我！
日志如下
    Dec 26 10:46:02 docker_web systemd: Started Session 370294 of user root.
    Dec 26 10:46:22 docker_web kernel: daotunnel invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
    Dec 26 10:46:22 docker_web kernel: daotunnel cpuset=/ mems_allowed=0
    Dec 26 10:46:22 docker_web kernel: CPU: 0 PID: 30670 Comm: daotunnel Tainted: G               ------------ T 3.10.0-957.10.1.el7.x86_64 #1
    Dec 26 10:46:22 docker_web kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014
    Dec 26 10:46:22 docker_web kernel: Call Trace:
    Dec 26 10:46:22 docker_web kernel: [<ffffffff95762e41>] dump_stack+0x19/0x1b
    Dec 26 10:46:22 docker_web kernel: [<ffffffff9575d86a>] dump_header+0x90/0x229
    Dec 26 10:46:22 docker_web kernel: [<ffffffff95101092>] ? ktime_get_ts64+0x52/0xf0
    Dec 26 10:46:22 docker_web kernel: [<ffffffff951582af>] ? delayacct_end+0x8f/0xb0
    Dec 26 10:46:22 docker_web kernel: [<ffffffffc060d71a>] ? virtballoon_oom_notify+0x2a/0x70 [virtio_balloon]
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951ba604>] oom_kill_process+0x254/0x3d0
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951ba0ad>] ? oom_unkillable_task+0xcd/0x120
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951ba156>] ? find_lock_task_mm+0x56/0xc0
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951bae46>] out_of_memory+0x4b6/0x4f0
    Dec 26 10:46:23 docker_web kernel: [<ffffffff9575e36e>] __alloc_pages_slowpath+0x5d6/0x724
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951c1224>] __alloc_pages_nodemask+0x404/0x420
    Dec 26 10:46:23 docker_web kernel: [<ffffffff9520e188>] alloc_pages_current+0x98/0x110
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951b6467>] __page_cache_alloc+0x97/0xb0
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951b90c8>] filemap_fault+0x298/0x490
    Dec 26 10:46:23 docker_web kernel: [<ffffffffc0576186>] ext4_filemap_fault+0x36/0x50 [ext4]
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951e466a>] __do_fault.isra.59+0x8a/0x100
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951e4c1c>] do_read_fault.isra.61+0x4c/0x1b0
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951e95c4>] handle_pte_fault+0x2f4/0xd10
    Dec 26 10:46:23 docker_web kernel: [<ffffffff9525836d>] ? core_sys_select+0x26d/0x340
    Dec 26 10:46:23 docker_web kernel: [<ffffffff950d6530>] ? try_to_wake_up+0x190/0x390
    Dec 26 10:46:23 docker_web kernel: [<ffffffff951ec0fd>] handle_mm_fault+0x39d/0x9b0
    Dec 26 10:46:23 docker_web kernel: [<ffffffff957705e3>] __do_page_fault+0x203/0x500
    Dec 26 10:46:23 docker_web kernel: [<ffffffff95257683>] ? poll_select_copy_remaining+0x113/0x180
    Dec 26 10:46:23 docker_web kernel: [<ffffffff957709c6>] trace_do_page_fault+0x56/0x150
    Dec 26 10:46:23 docker_web kernel: [<ffffffff9576ff42>] do_async_page_fault+0x22/0xf0
    Dec 26 10:46:23 docker_web kernel: [<ffffffff9576c788>] async_page_fault+0x28/0x30
    Dec 26 10:46:23 docker_web kernel: Mem-Info:
    Dec 26 10:46:23 docker_web kernel: active_anon:186895 inactive_anon:4166 isolated_anon:0#012 active_file:926 inactive_file:8714 isolated_file:0#012 unevictable:0 dirty:0 writeback
    :0 unstable:0#012 slab_reclaimable:17707 slab_unreclaimable:10069#012 mapped:4819 shmem:4583 pagetables:5739 bounce:0#012 free:12245 free_pcp:0 free_cma:0
    Dec 26 10:46:23 docker_web kernel: Node 0 DMA free:4588kB min:708kB low:884kB high:1060kB active_anon:8348kB inactive_anon:100kB active_file:12kB inactive_file:1388kB unevictable:
    0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:116kB shmem:140kB slab_reclaimable:752kB slab_unreclaimable:29
    6kB kernel_stack:0kB pagetables:92kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7575 all_unreclaimable? yes
    Dec 26 10:46:23 docker_web kernel: lowmem_reserve[]: 0 972 972 972
    Dec 26 10:46:23 docker_web kernel: Node 0 DMA32 free:44392kB min:44344kB low:55428kB high:66516kB active_anon:739232kB inactive_anon:16564kB active_file:3692kB inactive_file:33468
    kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1032048kB managed:998980kB mlocked:0kB dirty:0kB writeback:0kB mapped:19160kB shmem:18192kB slab_reclaimable:70076
    kB slab_unreclaimable:39980kB kernel_stack:4016kB pagetables:22864kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclai
    mable? no
    Dec 26 10:46:23 docker_web kernel: lowmem_reserve[]: 0 0 0 0
    Dec 26 10:46:23 docker_web kernel: Node 0 DMA: 25*4kB (UEM) 14*8kB (UE) 8*16kB (UE) 3*32kB (EM) 5*64kB (UEM) 6*128kB (UEM) 0*256kB 2*512kB (UE) 0*1024kB 1*2048kB (M) 0*4096kB = 45
    96kB
    Dec 26 10:46:24 docker_web kernel: Node 0 DMA32: 764*4kB (UE) 403*8kB (UEM) 318*16kB (UEM) 252*32kB (UEM) 166*64kB (UEM) 66*128kB (UEM) 19*256kB (UE) 2*512kB (UE) 0*1024kB 0*2048k
    B 0*4096kB = 44392kB
    Dec 26 10:46:24 docker_web kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
    Dec 26 10:46:24 docker_web kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
    Dec 26 10:46:24 docker_web kernel: 14228 total pagecache pages
    Dec 26 10:46:24 docker_web kernel: 0 pages in swap cache
    Dec 26 10:46:24 docker_web kernel: Swap cache stats: add 0, delete 0, find 0/0
    Dec 26 10:46:24 docker_web kernel: Free swap  = 0kB
    Dec 26 10:46:24 docker_web kernel: Total swap = 0kB
    Dec 26 10:46:24 docker_web kernel: 262010 pages RAM
    Dec 26 10:46:24 docker_web kernel: 0 pages HighMem/MovableOnly
    Dec 26 10:46:24 docker_web kernel: 8288 pages reserved
    Dec 26 10:46:24 docker_web kernel: [pid]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
    Dec 26 10:46:24 docker_web kernel: [1298]     0  1298    35445      174      76        0             0 systemd-journal
    Dec 26 10:46:24 docker_web kernel: [1318]     0  1318    29157       76      26        0             0 lvmetad
    Dec 26 10:46:24 docker_web kernel: [1329]     0  1329    11261      307      23        0         -1000 systemd-udevd
    Dec 26 10:46:24 docker_web kernel: [2047]     0  2047    13880      161      27        0         -1000 auditd
    Dec 26 10:46:24 docker_web kernel: [2162]   999  2162   153634     1459      63        0             0 polkitd
    Dec 26 10:46:24 docker_web kernel: [2166]     0  2166     6653      165      18        0             0 systemd-logind
    Dec 26 10:46:24 docker_web kernel: [2167]   998  2167     2144       38      10        0             0 lsmd
    Dec 26 10:46:24 docker_web kernel: [2169]    81  2169    15052      167      35        0          -900 dbus-daemon
    Dec 26 10:46:24 docker_web kernel: [2192]    38  2192    12339      227      30        0             0 ntpd
    Dec 26 10:46:24 docker_web kernel: [2264]     0  2264     1096       35       7        0             0 acpid
    Dec 26 10:46:24 docker_web kernel: [2767]     0  2767     6476       53      18        0             0 atd
    Dec 26 10:46:24 docker_web kernel: [2768]     0  2768    31579      219      17        0             0 crond
    Dec 26 10:46:24 docker_web kernel: [2819]     0  2819    26866      497      51        0             0 dhclient
    Dec 26 10:46:24 docker_web kernel: [2895]     0  2895    28216      256      57        0         -1000 sshd
    Dec 26 10:46:24 docker_web kernel: [2898]     0  2898   144002     2798      96        0             0 tuned
    Dec 26 10:46:24 docker_web kernel: [2900]     0  2900    74400     3188      30        0             0 daomonit
    Dec 26 10:46:24 docker_web kernel: [2904]     0  2904   182656     3964     187        0             0 rsyslogd
    Dec 26 10:46:24 docker_web kernel: [2933]     0  2933   119544     4818      56        0             0 containerd
    Dec 26 10:46:24 docker_web kernel: [2935]     0  2935   131843    10077     106        0          -500 dockerd
    Dec 26 10:46:24 docker_web kernel: [3007]     0  3007    27526       32      10        0             0 agetty
    Dec 26 10:46:24 docker_web kernel: [3008]     0  3008    27526       33      10        0             0 agetty
    Dec 26 10:46:24 docker_web kernel: [3426]     0  3426    24364      170      18        0             0 sgagent
    Dec 26 10:46:24 docker_web kernel: [4706]     0  4706    57223     3392      67        0             0 python
    Dec 26 10:46:24 docker_web kernel: [4711]     0  4711    26998       28      10        0             0 tail
    Dec 26 10:46:24 docker_web kernel: [22308]    99 22308    28438     1647      19        0             0 frps
    Dec 26 10:46:24 docker_web kernel: [30669]     0 30669   117389      795      30        0             0 daotunnel
    Dec 26 10:46:24 docker_web kernel: [3871]     0  3871    23820     1001      40        0             0 nginx
    Dec 26 10:46:24 docker_web kernel: [7638]   995  7638    29672     9686      61        0             0 nginx
    Dec 26 10:46:24 docker_web kernel: [11183]     0 11183   114532     1489     113        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [11184]    48 11184   138272     4760     128        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [11185]    48 11185   138236     4703     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [11186]    48 11186   138067     4476     126        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [11188]    48 11188   138239     4698     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [11189]    48 11189   138235     4703     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [11210]    48 11210   138236     4679     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [11240]    48 11240   138265     4664     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [1517]    48  1517   138215     4623     129        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [1519]    48  1519   138064     4018     126        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [1520]    48  1520   138258     4574     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [3953]    48  3953   138215     4339     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [5261]    48  5261   138229     4265     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [5282]    48  5282   138571     4223     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [4554]    48  4554   138228     4565     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [6115]    48  6115   138228     4260     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [6121]    48  6121   135800     4579     121        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [6122]    48  6122   138039     4369     125        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [9013]    48  9013   138566     4186     126        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [9016]    48  9016   138231     4250     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [9017]    48  9017   138213     4257     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [11215]    48 11215   138253     4258     127        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [9344]    48  9344   138075     4020     126        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [14702]    48 14702   138252     4574     130        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [14707]    48 14707   138044     4042     125        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [14708]    48 14708   135784     4270     119        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [14694]     0 14694    28825       65      13        0             0 sh
    Dec 26 10:46:24 docker_web kernel: [14695]     0 14695     5139       97      16        0             0 luajit
    Dec 26 10:46:24 docker_web kernel: [14959]     0 14959    28825       65      13        0             0 sh
    Dec 26 10:46:24 docker_web kernel: [14960]     0 14960     5139      100      15        0             0 luajit
    Dec 26 10:46:24 docker_web kernel: [19278]     0 19278    38810     1645      29        0             0 barad_agent
    Dec 26 10:46:24 docker_web kernel: [19285]     0 19285    39548     1966      32        0             0 barad_agent
    Dec 26 10:46:24 docker_web kernel: [19286]     0 19286   169263     3062      51        0             0 barad_agent
    Dec 26 10:46:24 docker_web kernel: [12748]     0 12748    25598      175      21        0             0 YDLive
    Dec 26 10:46:24 docker_web kernel: [30302]     0 30302    28825       64      13        0             0 sh
    Dec 26 10:46:24 docker_web kernel: [30303]     0 30303     5139      140      17        0             0 luajit
    Dec 26 10:46:24 docker_web kernel: [933]     0   933    28825       64      13        0             0 sh
    Dec 26 10:46:24 docker_web kernel: [936]     0   936     5139      125      16        0             0 luajit
    Dec 26 10:46:24 docker_web kernel: [4338]     0  4338    28825       64      13        0             0 sh
    Dec 26 10:46:24 docker_web kernel: [4340]     0  4340     5139       96      17        0             0 luajit
    Dec 26 10:46:24 docker_web kernel: [14513]    48 14513   116384     3419     115        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [27451]    48 27451   116126     3168     115        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [27460]    48 27460   116383     3399     115        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [18774]    48 18774   116766     3390     116        0             0 php-fpm
    Dec 26 10:46:24 docker_web kernel: [13083]     0 13083    27241      288      17        0          -500 docker-proxy
    Dec 26 10:46:24 docker_web kernel: [13091]     0 13091     2680      300      10        0          -999 containerd-shim
    Dec 26 10:46:24 docker_web kernel: [13108]   999 13108   240779    27745     132        0             0 mysqld
    Dec 26 10:46:24 docker_web kernel: [2697]     0  2697    49045    13859      63        0             0 YDService
    Dec 26 10:46:24 docker_web kernel: [7302]     0  7302    39185      383      79        0             0 sshd
    Dec 26 10:46:24 docker_web kernel: [7315]     0  7315    29196      517      13        0             0 bash
    Dec 26 10:46:24 docker_web kernel: [16556]     0 16556   251102    23921     186        0             0 yum
    Dec 26 10:46:24 docker_web kernel: Out of memory: Kill process 13108 (mysqld) score 109 or sacrifice child
    Dec 26 10:46:24 docker_web kernel: Killed process 13108 (mysqld) total-vm:963116kB, anon-rss:110980kB, file-rss:0kB, shmem-rss:0kB
    Dec 26 10:46:25 docker_web containerd: time="2019-12-26T10:46:25.042577073+08:00" level=info msg="shim reaped" id=08f9f376ee7297c89c5ccf377ee4ffdce64a3b1d04505f37a1399ea3f903e2c3
    Dec 26 10:46:25 docker_web dockerd: time="2019-12-26T10:46:25.081242626+08:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*event
    s.TaskDelete"
    Dec 26 10:46:25 docker_web kernel: docker0: port 1(veth85673a7) entered disabled state
    Dec 26 10:46:25 docker_web kernel: docker0: port 1(veth85673a7) entered disabled state
    Dec 26 10:46:25 docker_web kernel: device veth85673a7 left promiscuous mode
    Dec 26 10:46:25 docker_web kernel: docker0: port 1(veth85673a7) entered disabled state
    Dec 26 10:46:36 docker_web yum[16556]: Updated: ntpdate-4.2.6p5-29.el7.centos.x86_64
    Dec 26 10:46:37 docker_web yum[16556]: Updated: ntp-4.2.6p5-29.el7.centos.x86_64
    Dec 26 10:46:37 docker_web systemd: Reloading.
    Dec 26 10:46:37 docker_web systemd: Cannot add dependency job for unit rpcbind.socket, ignoring: Unit not found.
    Dec 26 10:46:37 docker_web systemd: Stopping Network Time Service...
    Dec 26 10:46:38 docker_web ntpd[2192]: ntpd exiting on signal 15
    Dec 26 10:46:38 docker_web systemd: Stopped Network Time Service.
    Dec 26 10:46:38 docker_web systemd: Starting Network Time Service...
    Dec 26 10:46:38 docker_web ntpd[16820]: ntpd 4.2.6p5@1.2349-o Thu Aug  8 11:47:59 UTC 2019 (1)
    Dec 26 10:46:38 docker_web systemd: Started Network Time Service.
    Dec 26 10:46:38 docker_web ntpd[16821]: proto: precision = 0.050 usec
    Dec 26 10:46:38 docker_web ntpd[16821]: 0.0.0.0 c01d 0d kern kernel time sync enabled
    Dec 26 10:46:38 docker_web ntpd[16821]: Listen normally on 0 lo 127.0.0.1 UDP 123
    Dec 26 10:46:38 docker_web ntpd[16821]: Listen normally on 1 eth0 172.30.0.5 UDP 123
    Dec 26 10:46:38 docker_web ntpd[16821]: Listening on routing socket on fd #18 for interface updates
    Dec 26 10:46:38 docker_web ntpd[16821]: 0.0.0.0 c016 06 restart
    Dec 26 10:46:38 docker_web ntpd[16821]: 0.0.0.0 c012 02 freq_set kernel -22.321 PPM
    Dec 26 10:46:38 docker_web systemd: Reloading.
    Dec 26 10:46:45 docker_web ntpd[16821]: 0.0.0.0 c615 05 clock_sync
    Dec 26 10:47:01 docker_web systemd: Started Session 370295 of user root.
    Dec 26 10:48:01 docker_web systemd: Started Session 370296 of user root.
    Dec 26 10:49:01 docker_web systemd: Started Session 370297 of user root.
    Dec 26 10:50:01 docker_web systemd: Started Session 370298 of user root.
    Dec 26 10:50:01 docker_web systemd: Started Session 370299 of user root.
    Dec 26 10:51:01 docker_web systemd: Started Session 370300 of user root.
    Dec 26 10:51:04 docker_web systemd: Started Session 370301 of user root.
    Dec 26 10:51:04 docker_web systemd-logind: New session 370301 of user root.
    Dec 26 10:51:04 docker_web dbus[2169]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)

Dec 26 10:51:04 docker_web dbus[2169]: [system] Successfully activated service 'org.freedesktop.problems'
Dec 26 10:52:01 docker_web systemd: Started Session 370302 of user root.
Dec 26 10:53:01 docker_web systemd: Started Session 370303 of user root.
Dec 26 10:54:01 docker_web systemd: Started Session 370304 of user root.
Dec 26 10:54:40 docker_web systemd: Started Session 370305 of user root.
Dec 26 10:54:40 docker_web systemd-logind: New session 370305 of user root.
Dec 26 10:54:40 docker_web dbus[2169]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Dec 26 10:54:40 docker_web dbus[2169]: [system] Successfully activated service 'org.freedesktop.problems'
Dec 26 10:55:01 docker_web systemd: Started Session 370306 of user root.
Dec 26 10:56:01 docker_web systemd: Started Session 370307 of user root.
Dec 26 10:56:23 docker_web systemd-logind: Removed session 370305.
正文完