今天遇到一个问题,server上无法开启一个服务。查找了以往类似的问题然后也尝试了相关的措施,问题一直没有得到解决。又仔细查了下log,发现一条进程创建时的error
pthread_create() failed: ret=12
这是一个No Memory的错误,首先想到的是查看Memory的使用情况,但不抱太大希望,因为系统相关的performance的问题。
# free
total used free shared buffers cached
Mem: 12126832 2170100 9956732 0 481604 836292
-/+ buffers/cache: 852204 11274628
Swap: 115306496 0 115306496
# cat /proc/meminfo
MemTotal: 12126832 kB
MemFree: 9955332 kB
Buffers: 481604 kB
Cached: 836292 kB
SwapCached: 0 kB
Active: 1026648 kB
Inactive: 360464 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 12126832 kB
LowFree: 9955332 kB
SwapTotal: 115306496 kB
SwapFree: 115306496 kB
Dirty: 144 kB
Writeback: 0 kB
AnonPages: 67356 kB
Mapped: 19364 kB
Slab: 280296 kB
CommitLimit: 121369912 kB
Committed_AS: 224800 kB
PageTables: 2148 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 335604 kB
VmallocChunk: 34359402487 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB
检查后看上去没什么memory问题。折腾了好半天,才发现问题出在stack size的limit上。
# ulimit -Sa
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 24576
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 63579881472 <<<<< HUGE !!!
cpu time (seconds, -t) unlimited
max user processes (-u) 24576
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
常规情况下,stack size设置一般为8192,不知道为何被设置成了如此大一个数字。将其设置为常规参数8192后,一切恢复正常
ulimit -s 8192
|