具体报错关键字参数

Too many open files

排查思路查看具体的服务open files实际值

cat /proc/18430/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             30629                30629                processes
Max open files            1024                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       30629                30629                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

具体可以看到实际设置的是65535但是limits下级还有特殊的配置
Max open files 1024 4096

cat limits.conf
# End of file
root soft nofile 65535
root hard nofile 65535
* soft nofile 65535
* hard nofile 65535
* soft   nofile  65535
* hard   nofile  65535
root soft   nofile  655350
root hard   nofile  655350

系统(其实是pam_limits这个模块)会先读入/etc/security/limits.conf，然后读入/etc/security/limits.d/下面的文件，所以是 /etc/security/limits.d/20-nproc.conf这个文件设定的值生效。你可以理解成直接把它们按照读入顺序拼接即可，对于相同的用户指定，越靠后优先级越高。
所以还需要修改20-nproc.conf

cat /etc/security/limits.d/20-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     655350
root       soft    nproc     unlimited

同时修改相关服务的参数如supervisord.conf

cat /etc/supervisord.conf
minfds=655350 ; (min. avail startup file descriptors;default 1024)
minprocs=65535 ; (min. avail process descriptors;default 200)

重启服务然后通过命令再次检查是否生效

cat /proc/19997/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             65535                65535                processes
Max open files            655350               655350               files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       30629                30629                signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

最终问题解决。