django - Issues with celery daemon -
we're having issues our celery daemon being flaky. use fabric deployment script restart daemon whenever push changes, reason causing massive issues.
whenever deployment script run celery processes left in pseudo dead state. (unfortunately) still consume tasks rabbitmq, won't anything. confusingly brief inspection indicate seems "fine" in state, celeryctl status shows 1 node online , ps aux | grep celery shows 2 running processes.
however, attempting run /etc/init.d/celeryd stop manually results in following error:
start-stop-daemon: warning: failed kill 30360: no such process
while in state attempting run celeryd start appears work correctly, in fact nothing. way fix issue manually kill running celery processes , start them again.
any ideas what's going on here? don't have complete confirmation, think problem develops after few days (with no activity test server currently) on it's own no deployment.
i can't know what's ailing setup, i've used supervisord run celery -- maybe issue has upstart? regardless, i've never experienced celery running on top of supervisord.
for measure, here's sample supervisor config celery:
[program:celeryd] directory=/path/to/project/ command=/path/to/project/venv/bin/python manage.py celeryd -l info user=nobody autostart=true autorestart=true startsecs=10 numprocs=1 stdout_logfile=/var/log/sites/foo/celeryd_stdout.log stderr_logfile=/var/log/sites/foo/celeryd_stderr.log ; need wait executing tasks finish @ shutdown. ; increase if have long running tasks. stopwaitsecs = 600
restarting celeryd in fab script simple issuing sudo supervisorctl restart celeryd
.
Comments
Post a Comment