Jump to content


Uncontrolled spawninig od IDLE proceses

IDLE 2013.3 server P4V P4VS

  • Please log in to reply
1 reply to this topic

#1 rogathy

rogathy

    Newbie

  • Members
  • Pip
  • 9 posts

Posted 02 April 2014 - 03:43 PM

We are using P4D/FREEBSD70X86_64/2013.3/770073 (2014/01/14)

Last week I encountered a strange behavior on our server.
IDLE proceses are spawninig zombie (IDLE) proceses at an alarming rate.
It is like 500 processes in 30-60min sometimes even faster.
In peak: >2k processes in less than 30min.
In that group only 3-5 processes are acitive - running, all others are in IDLE.
Some of them disappears just after creation but most of them hangs and remains on the server doing nothing for hours, with handles to db.* files:

lsof: WARNING: compiled for FreeBSD release 8.4-RELEASE; this is 8.4-RELEASE-p4.
COMMAND  PID USER   FD   TYPE DEVICE  SIZE/OFF NODE NAME
p4d 1854 p4admin  cwd   VDIR    0,95   3072 2 /p4directory
p4d 1854 p4admin  rtd   VDIR    0,89    512 2 /
p4d 1854 p4admin  txt   VREG    0,93   4533160  6385959 /usr/local/sbin/p4d
p4d 1854 p4admin  txt   VREG    0,89 250672    11 /libexec/ld-elf.so.1
p4d 1854 p4admin  txt   VREG    0,93   1040520 25082950 /usr/lib/libstdc++.so.6
p4d 1854 p4admin  txt   VREG    0,89 59360 23970 /lib/libgcc_s.so.1
p4d 1854 p4admin  txt   VREG    0,89   1301864 23993 /lib/libc.so.7
p4d 1854 p4admin  txt   VREG    0,89 155232 23991 /lib/libm.so.5
p4d 1854 p4admin 0r  VCHR    0,28    0t0    28 /dev/null
p4d 1854 p4admin 1w  VCHR    0,28    0t0    28 /dev/null
p4d 1854 p4admin 2w  VCHR    0,28    0t0    28 /dev/null
p4d 1854 p4admin 3w  VREG    0,95 662241878    27 /p4directory/journal
p4d 1854 p4admin 4u  IPv4 0xffffff011b8bf760    0t0   TCP ourdnsname.com:netview-aix-6->172.16.14.4:51946 (ESTABLISHED)
p4d 1854 p4admin 5u  VREG    0,95 16384 18818050 /p4directory/db.depot
p4d 1854 p4admin 6u  VREG    0,95   1572864 18818063 /p4directory/db.view
p4d 1854 p4admin 7u  VREG    0,95 114688 18818106 /p4directory/db.monitor
p4d 1854 p4admin 8u  VREG    0,95 73728 18818058 /p4directory/db.group
p4d 1854 p4admin 9u  VREG    0,95 16384 18818103 /p4directory/db.protect
p4d 1854 p4admin   10u  VREG    0,95 81920 18818057 /p4directory/db.user
p4d 1854 p4admin   11u  VREG    0,95 286720 18818060 /p4directory/db.domain
p4d 1854 p4admin   13u  VREG    0,95 373743616 18818087 /p4directory/db.working

I cant' even kill them:
p4 monitor terminate or
p4 monitor pause
doesn't work at all (status of this proces is still "I" -> IDLE)

In most cases proceses are spawned in packs (5-25) like:
82899 P4V/NTX86/2011.1/373947/v70 172.16.14.26   I user user_client 00:48:57 IDLE none
82900 P4V/NTX86/2011.1/373947/v70 172.16.14.26   I user user_client 00:48:57 IDLE none
82901 P4V/NTX86/2011.1/373947/v70 172.16.14.26   I user user_client 00:48:57 IDLE none
82902 P4V/NTX86/2011.1/373947/v70 172.16.14.26   I user user_client 00:48:57 IDLE none
82961 P4V/NTX86/2011.1/373947/v70 172.16.14.26   I user user_client 00:48:53 IDLE none
82962 P4V/NTX86/2011.1/373947/v70 172.16.14.26   I user user_client 00:48:53 IDLE none

Sometimes one user can generate >1000 processes.

Connections come from P4V, P4VS and couple of ours tools using both P4API and commandline (p4) connections.

After couple of hours (when there is about 1500-2500 processes) it is not possible to connect to p4d anymore:
Database open error on db.logger!
open: db.logger: Too many open files in system
(Currently limit on the number of opened files is set to 25000)

At this moment the only way to get rid of zombie processes is to restart the server p4d deamon.

There were no such problems prior to last week. Number of processes never went above 20-30 at the same time.
There were no updates, major network changes. All tools used are the same.

Is there any way to coonfigure serwer to get rid of zombie processes safely and efectively?

#2 P4Matt

P4Matt

    Advanced Member

  • Members
  • PipPipPip
  • 1383 posts

Posted 02 April 2014 - 06:46 PM

I'd reach out to support about this if you have access to them; something is wrong if that many idle process are being created and not cleaned up. The type of debugging involved will require data you probably don't want to publish to the forums.

If the commands were just coming from internal tools I'd have some thoughts about what might be happening, but the fact V is doing it too is very suspicious. Investigation is definitely needed.





Also tagged with one or more of these keywords: IDLE, 2013.3 server, P4V, P4VS

0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users