记一次docker mysql被oom排查 - kilig-linux mint

kilig-linux mintkilig-linux mint

分享运维中出现的问题,并分享运维中的心得!

记一次docker mysql被oom排查

最近博客老是出现无法连接数据库故障!
排查腾讯云监控的时候看到内存使用700多M,问题就出在这里,当时没有使用top命令查看,
腾讯云监控的内存使用的计算方式是取 /proc/meminfo 的
Memtotal-MemFree-Buffers-Cached-SReclaimable(SReclaimable代表可回收的 slab 内存大小)
所有与top命令的出入。本服务器没有开启交换空间sawp
所以是因为内存不足导致的oom 通过系统的计算算出docker mysql 的内存占用最大所以直接oom了!
排查出现oom的过程可以查看/var/log/messages 日志情况
实际上是我的php-fpm进程占用内存过多导致的,优化fpf-fpm.conf参数即可,优化需要根据自己的业务场景优化!
就不贴出我的配置了!有需要的可以找我!
日志如下
Dec 26 10:46:02 docker_web systemd: Started Session 370294 of user root.
Dec 26 10:46:22 docker_web kernel: daotunnel invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
Dec 26 10:46:22 docker_web kernel: daotunnel cpuset=/ mems_allowed=0
Dec 26 10:46:22 docker_web kernel: CPU: 0 PID: 30670 Comm: daotunnel Tainted: G ------------ T 3.10.0-957.10.1.el7.x86_64 #1
Dec 26 10:46:22 docker_web kernel: Hardware name: Smdbmds KVM, BIOS seabios-1.9.1-qemu-project.org 04/01/2014
Dec 26 10:46:22 docker_web kernel: Call Trace:
Dec 26 10:46:22 docker_web kernel: [<ffffffff95762e41>] dump_stack+0x19/0x1b
Dec 26 10:46:22 docker_web kernel: [<ffffffff9575d86a>] dump_header+0x90/0x229
Dec 26 10:46:22 docker_web kernel: [<ffffffff95101092>] ? ktime_get_ts64+0x52/0xf0
Dec 26 10:46:22 docker_web kernel: [<ffffffff951582af>] ? delayacct_end+0x8f/0xb0
Dec 26 10:46:22 docker_web kernel: [<ffffffffc060d71a>] ? virtballoon_oom_notify+0x2a/0x70 [virtio_balloon]
Dec 26 10:46:23 docker_web kernel: [<ffffffff951ba604>] oom_kill_process+0x254/0x3d0
Dec 26 10:46:23 docker_web kernel: [<ffffffff951ba0ad>] ? oom_unkillable_task+0xcd/0x120
Dec 26 10:46:23 docker_web kernel: [<ffffffff951ba156>] ? find_lock_task_mm+0x56/0xc0
Dec 26 10:46:23 docker_web kernel: [<ffffffff951bae46>] out_of_memory+0x4b6/0x4f0
Dec 26 10:46:23 docker_web kernel: [<ffffffff9575e36e>] __alloc_pages_slowpath+0x5d6/0x724
Dec 26 10:46:23 docker_web kernel: [<ffffffff951c1224>] __alloc_pages_nodemask+0x404/0x420
Dec 26 10:46:23 docker_web kernel: [<ffffffff9520e188>] alloc_pages_current+0x98/0x110
Dec 26 10:46:23 docker_web kernel: [<ffffffff951b6467>] __page_cache_alloc+0x97/0xb0
Dec 26 10:46:23 docker_web kernel: [<ffffffff951b90c8>] filemap_fault+0x298/0x490
Dec 26 10:46:23 docker_web kernel: [<ffffffffc0576186>] ext4_filemap_fault+0x36/0x50 [ext4]
Dec 26 10:46:23 docker_web kernel: [<ffffffff951e466a>] __do_fault.isra.59+0x8a/0x100
Dec 26 10:46:23 docker_web kernel: [<ffffffff951e4c1c>] do_read_fault.isra.61+0x4c/0x1b0
Dec 26 10:46:23 docker_web kernel: [<ffffffff951e95c4>] handle_pte_fault+0x2f4/0xd10
Dec 26 10:46:23 docker_web kernel: [<ffffffff9525836d>] ? core_sys_select+0x26d/0x340
Dec 26 10:46:23 docker_web kernel: [<ffffffff950d6530>] ? try_to_wake_up+0x190/0x390
Dec 26 10:46:23 docker_web kernel: [<ffffffff951ec0fd>] handle_mm_fault+0x39d/0x9b0
Dec 26 10:46:23 docker_web kernel: [<ffffffff957705e3>] __do_page_fault+0x203/0x500
Dec 26 10:46:23 docker_web kernel: [<ffffffff95257683>] ? poll_select_copy_remaining+0x113/0x180
Dec 26 10:46:23 docker_web kernel: [<ffffffff957709c6>] trace_do_page_fault+0x56/0x150
Dec 26 10:46:23 docker_web kernel: [<ffffffff9576ff42>] do_async_page_fault+0x22/0xf0
Dec 26 10:46:23 docker_web kernel: [<ffffffff9576c788>] async_page_fault+0x28/0x30
Dec 26 10:46:23 docker_web kernel: Mem-Info:
Dec 26 10:46:23 docker_web kernel: active_anon:186895 inactive_anon:4166 isolated_anon:0#012 active_file:926 inactive_file:8714 isolated_file:0#012 unevictable:0 dirty:0 writeback
:0 unstable:0#012 slab_reclaimable:17707 slab_unreclaimable:10069#012 mapped:4819 shmem:4583 pagetables:5739 bounce:0#012 free:12245 free_pcp:0 free_cma:0
Dec 26 10:46:23 docker_web kernel: Node 0 DMA free:4588kB min:708kB low:884kB high:1060kB active_anon:8348kB inactive_anon:100kB active_file:12kB inactive_file:1388kB unevictable:
0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:116kB shmem:140kB slab_reclaimable:752kB slab_unreclaimable:29
6kB kernel_stack:0kB pagetables:92kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:7575 all_unreclaimable? yes
Dec 26 10:46:23 docker_web kernel: lowmem_reserve[]: 0 972 972 972
Dec 26 10:46:23 docker_web kernel: Node 0 DMA32 free:44392kB min:44344kB low:55428kB high:66516kB active_anon:739232kB inactive_anon:16564kB active_file:3692kB inactive_file:33468
kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:1032048kB managed:998980kB mlocked:0kB dirty:0kB writeback:0kB mapped:19160kB shmem:18192kB slab_reclaimable:70076
kB slab_unreclaimable:39980kB kernel_stack:4016kB pagetables:22864kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclai
mable? no
Dec 26 10:46:23 docker_web kernel: lowmem_reserve[]: 0 0 0 0
Dec 26 10:46:23 docker_web kernel: Node 0 DMA: 254kB (UEM) 148kB (UE) 816kB (UE) 332kB (EM) 564kB (UEM) 6128kB (UEM) 0256kB 2512kB (UE) 01024kB 12048kB (M) 0*4096kB = 45
96kB
Dec 26 10:46:24 docker_web kernel: Node 0 DMA32: 7644kB (UE) 4038kB (UEM) 31816kB (UEM) 25232kB (UEM) 16664kB (UEM) 66128kB (UEM) 19256kB (UE) 2512kB (UE) 01024kB 02048k
B 0*4096kB = 44392kB
Dec 26 10:46:24 docker_web kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Dec 26 10:46:24 docker_web kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
Dec 26 10:46:24 docker_web kernel: 14228 total pagecache pages
Dec 26 10:46:24 docker_web kernel: 0 pages in swap cache
Dec 26 10:46:24 docker_web kernel: Swap cache stats: add 0, delete 0, find 0/0
Dec 26 10:46:24 docker_web kernel: Free swap = 0kB
Dec 26 10:46:24 docker_web kernel: Total swap = 0kB
Dec 26 10:46:24 docker_web kernel: 262010 pages RAM
Dec 26 10:46:24 docker_web kernel: 0 pages HighMem/MovableOnly
Dec 26 10:46:24 docker_web kernel: 8288 pages reserved
Dec 26 10:46:24 docker_web kernel: [ pid ] uid tgid total_vm rss nr_ptes swapents oom_score_adj name
Dec 26 10:46:24 docker_web kernel: [ 1298] 0 1298 35445 174 76 0 0 systemd-journal
Dec 26 10:46:24 docker_web kernel: [ 1318] 0 1318 29157 76 26 0 0 lvmetad
Dec 26 10:46:24 docker_web kernel: [ 1329] 0 1329 11261 307 23 0 -1000 systemd-udevd
Dec 26 10:46:24 docker_web kernel: [ 2047] 0 2047 13880 161 27 0 -1000 auditd
Dec 26 10:46:24 docker_web kernel: [ 2162] 999 2162 153634 1459 63 0 0 polkitd
Dec 26 10:46:24 docker_web kernel: [ 2166] 0 2166 6653 165 18 0 0 systemd-logind
Dec 26 10:46:24 docker_web kernel: [ 2167] 998 2167 2144 38 10 0 0 lsmd
Dec 26 10:46:24 docker_web kernel: [ 2169] 81 2169 15052 167 35 0 -900 dbus-daemon
Dec 26 10:46:24 docker_web kernel: [ 2192] 38 2192 12339 227 30 0 0 ntpd
Dec 26 10:46:24 docker_web kernel: [ 2264] 0 2264 1096 35 7 0 0 acpid
Dec 26 10:46:24 docker_web kernel: [ 2767] 0 2767 6476 53 18 0 0 atd
Dec 26 10:46:24 docker_web kernel: [ 2768] 0 2768 31579 219 17 0 0 crond
Dec 26 10:46:24 docker_web kernel: [ 2819] 0 2819 26866 497 51 0 0 dhclient
Dec 26 10:46:24 docker_web kernel: [ 2895] 0 2895 28216 256 57 0 -1000 sshd
Dec 26 10:46:24 docker_web kernel: [ 2898] 0 2898 144002 2798 96 0 0 tuned
Dec 26 10:46:24 docker_web kernel: [ 2900] 0 2900 74400 3188 30 0 0 daomonit
Dec 26 10:46:24 docker_web kernel: [ 2904] 0 2904 182656 3964 187 0 0 rsyslogd
Dec 26 10:46:24 docker_web kernel: [ 2933] 0 2933 119544 4818 56 0 0 containerd
Dec 26 10:46:24 docker_web kernel: [ 2935] 0 2935 131843 10077 106 0 -500 dockerd
Dec 26 10:46:24 docker_web kernel: [ 3007] 0 3007 27526 32 10 0 0 agetty
Dec 26 10:46:24 docker_web kernel: [ 3008] 0 3008 27526 33 10 0 0 agetty
Dec 26 10:46:24 docker_web kernel: [ 3426] 0 3426 24364 170 18 0 0 sgagent
Dec 26 10:46:24 docker_web kernel: [ 4706] 0 4706 57223 3392 67 0 0 python
Dec 26 10:46:24 docker_web kernel: [ 4711] 0 4711 26998 28 10 0 0 tail
Dec 26 10:46:24 docker_web kernel: [22308] 99 22308 28438 1647 19 0 0 frps
Dec 26 10:46:24 docker_web kernel: [30669] 0 30669 117389 795 30 0 0 daotunnel
Dec 26 10:46:24 docker_web kernel: [ 3871] 0 3871 23820 1001 40 0 0 nginx
Dec 26 10:46:24 docker_web kernel: [ 7638] 995 7638 29672 9686 61 0 0 nginx
Dec 26 10:46:24 docker_web kernel: [11183] 0 11183 114532 1489 113 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [11184] 48 11184 138272 4760 128 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [11185] 48 11185 138236 4703 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [11186] 48 11186 138067 4476 126 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [11188] 48 11188 138239 4698 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [11189] 48 11189 138235 4703 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [11210] 48 11210 138236 4679 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [11240] 48 11240 138265 4664 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 1517] 48 1517 138215 4623 129 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 1519] 48 1519 138064 4018 126 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 1520] 48 1520 138258 4574 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 3953] 48 3953 138215 4339 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 5261] 48 5261 138229 4265 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 5282] 48 5282 138571 4223 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 4554] 48 4554 138228 4565 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 6115] 48 6115 138228 4260 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 6121] 48 6121 135800 4579 121 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 6122] 48 6122 138039 4369 125 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 9013] 48 9013 138566 4186 126 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 9016] 48 9016 138231 4250 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 9017] 48 9017 138213 4257 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [11215] 48 11215 138253 4258 127 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [ 9344] 48 9344 138075 4020 126 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [14702] 48 14702 138252 4574 130 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [14707] 48 14707 138044 4042 125 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [14708] 48 14708 135784 4270 119 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [14694] 0 14694 28825 65 13 0 0 sh
Dec 26 10:46:24 docker_web kernel: [14695] 0 14695 5139 97 16 0 0 luajit
Dec 26 10:46:24 docker_web kernel: [14959] 0 14959 28825 65 13 0 0 sh
Dec 26 10:46:24 docker_web kernel: [14960] 0 14960 5139 100 15 0 0 luajit
Dec 26 10:46:24 docker_web kernel: [19278] 0 19278 38810 1645 29 0 0 barad_agent
Dec 26 10:46:24 docker_web kernel: [19285] 0 19285 39548 1966 32 0 0 barad_agent
Dec 26 10:46:24 docker_web kernel: [19286] 0 19286 169263 3062 51 0 0 barad_agent
Dec 26 10:46:24 docker_web kernel: [12748] 0 12748 25598 175 21 0 0 YDLive
Dec 26 10:46:24 docker_web kernel: [30302] 0 30302 28825 64 13 0 0 sh
Dec 26 10:46:24 docker_web kernel: [30303] 0 30303 5139 140 17 0 0 luajit
Dec 26 10:46:24 docker_web kernel: [ 933] 0 933 28825 64 13 0 0 sh
Dec 26 10:46:24 docker_web kernel: [ 936] 0 936 5139 125 16 0 0 luajit
Dec 26 10:46:24 docker_web kernel: [ 4338] 0 4338 28825 64 13 0 0 sh
Dec 26 10:46:24 docker_web kernel: [ 4340] 0 4340 5139 96 17 0 0 luajit
Dec 26 10:46:24 docker_web kernel: [14513] 48 14513 116384 3419 115 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [27451] 48 27451 116126 3168 115 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [27460] 48 27460 116383 3399 115 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [18774] 48 18774 116766 3390 116 0 0 php-fpm
Dec 26 10:46:24 docker_web kernel: [13083] 0 13083 27241 288 17 0 -500 docker-proxy
Dec 26 10:46:24 docker_web kernel: [13091] 0 13091 2680 300 10 0 -999 containerd-shim
Dec 26 10:46:24 docker_web kernel: [13108] 999 13108 240779 27745 132 0 0 mysqld
Dec 26 10:46:24 docker_web kernel: [ 2697] 0 2697 49045 13859 63 0 0 YDService
Dec 26 10:46:24 docker_web kernel: [ 7302] 0 7302 39185 383 79 0 0 sshd
Dec 26 10:46:24 docker_web kernel: [ 7315] 0 7315 29196 517 13 0 0 bash
Dec 26 10:46:24 docker_web kernel: [16556] 0 16556 251102 23921 186 0 0 yum
Dec 26 10:46:24 docker_web kernel: Out of memory: Kill process 13108 (mysqld) score 109 or sacrifice child
Dec 26 10:46:24 docker_web kernel: Killed process 13108 (mysqld) total-vm:963116kB, anon-rss:110980kB, file-rss:0kB, shmem-rss:0kB
Dec 26 10:46:25 docker_web containerd: time="2019-12-26T10:46:25.042577073+08:00" level=info msg="shim reaped" id=08f9f376ee7297c89c5ccf377ee4ffdce64a3b1d04505f37a1399ea3f903e2c3
Dec 26 10:46:25 docker_web dockerd: time="2019-12-26T10:46:25.081242626+08:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*event
s.TaskDelete"
Dec 26 10:46:25 docker_web kernel: docker0: port 1(veth85673a7) entered disabled state
Dec 26 10:46:25 docker_web kernel: docker0: port 1(veth85673a7) entered disabled state
Dec 26 10:46:25 docker_web kernel: device veth85673a7 left promiscuous mode
Dec 26 10:46:25 docker_web kernel: docker0: port 1(veth85673a7) entered disabled state
Dec 26 10:46:36 docker_web yum[16556]: Updated: ntpdate-4.2.6p5-29.el7.centos.x86_64
Dec 26 10:46:37 docker_web yum[16556]: Updated: ntp-4.2.6p5-29.el7.centos.x86_64
Dec 26 10:46:37 docker_web systemd: Reloading.
Dec 26 10:46:37 docker_web systemd: Cannot add dependency job for unit rpcbind.socket, ignoring: Unit not found.
Dec 26 10:46:37 docker_web systemd: Stopping Network Time Service...
Dec 26 10:46:38 docker_web ntpd[2192]: ntpd exiting on signal 15
Dec 26 10:46:38 docker_web systemd: Stopped Network Time Service.
Dec 26 10:46:38 docker_web systemd: Starting Network Time Service...
Dec 26 10:46:38 docker_web ntpd[16820]: ntpd 4.2.6p5@1.2349-o Thu Aug 8 11:47:59 UTC 2019 (1)
Dec 26 10:46:38 docker_web systemd: Started Network Time Service.
Dec 26 10:46:38 docker_web ntpd[16821]: proto: precision = 0.050 usec
Dec 26 10:46:38 docker_web ntpd[16821]: 0.0.0.0 c01d 0d kern kernel time sync enabled
Dec 26 10:46:38 docker_web ntpd[16821]: Listen normally on 0 lo 127.0.0.1 UDP 123
Dec 26 10:46:38 docker_web ntpd[16821]: Listen normally on 1 eth0 172.30.0.5 UDP 123
Dec 26 10:46:38 docker_web ntpd[16821]: Listening on routing socket on fd #18 for interface updates
Dec 26 10:46:38 docker_web ntpd[16821]: 0.0.0.0 c016 06 restart
Dec 26 10:46:38 docker_web ntpd[16821]: 0.0.0.0 c012 02 freq_set kernel -22.321 PPM
Dec 26 10:46:38 docker_web systemd: Reloading.
Dec 26 10:46:45 docker_web ntpd[16821]: 0.0.0.0 c615 05 clock_sync
Dec 26 10:47:01 docker_web systemd: Started Session 370295 of user root.
Dec 26 10:48:01 docker_web systemd: Started Session 370296 of user root.
Dec 26 10:49:01 docker_web systemd: Started Session 370297 of user root.
Dec 26 10:50:01 docker_web systemd: Started Session 370298 of user root.
Dec 26 10:50:01 docker_web systemd: Started Session 370299 of user root.
Dec 26 10:51:01 docker_web systemd: Started Session 370300 of user root.
Dec 26 10:51:04 docker_web systemd: Started Session 370301 of user root.
Dec 26 10:51:04 docker_web systemd-logind: New session 370301 of user root.
Dec 26 10:51:04 docker_web dbus[2169]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Dec 26 10:51:04 docker_web dbus[2169]: [system] Successfully activated service 'org.freedesktop.problems'
Dec 26 10:52:01 docker_web systemd: Started Session 370302 of user root.
Dec 26 10:53:01 docker_web systemd: Started Session 370303 of user root.
Dec 26 10:54:01 docker_web systemd: Started Session 370304 of user root.
Dec 26 10:54:40 docker_web systemd: Started Session 370305 of user root.
Dec 26 10:54:40 docker_web systemd-logind: New session 370305 of user root.
Dec 26 10:54:40 docker_web dbus[2169]: [system] Activating service name='org.freedesktop.problems' (using servicehelper)
Dec 26 10:54:40 docker_web dbus[2169]: [system] Successfully activated service 'org.freedesktop.problems'
Dec 26 10:55:01 docker_web systemd: Started Session 370306 of user root.
Dec 26 10:56:01 docker_web systemd: Started Session 370307 of user root.
Dec 26 10:56:23 docker_web systemd-logind: Removed session 370305.

本原创文章未经允许不得转载 | 当前页面:kilig-linux mint » 记一次docker mysql被oom排查

评论