1

I am running a celery - rabbitmq configuration between a client and a server. The client process opens a connection on the server at port 5672. Using netstat -np on the server I confirm that since the socket reports ESTABLISHED.

The process on the client crashed and it doesn't exist anymore. But netstat -np still shows the connection ESTABLISHED. I also run netstat -np on the client and I confirm that no process has connection to the server on that port.

Why does the socket remains in ESTABLISHED status?

Demetris
  • 187
  • how long does it stay that way? my guess is less than 30-60 seconds, based on your server OS configuration. – Grega Bremec Jan 26 '16 at 16:32
  • Unfortunately it stays there more than that, at least 1 hour before I had to restart the service. By the way, I don't know if it is relevant, this is a rabbitmq server, and the client is connecting with celery. If needed, I should update the question. – Demetris Jan 26 '16 at 16:40
  • 1
    Ah, then this is a completely different story. AMQP requires the server to stay open if the client does not close the connection properly (for long-running tasks). Read more here: http://stackoverflow.com/questions/16835138/rabbitmq-doesnt-close-connections-server-side – Grega Bremec Jan 26 '16 at 16:41
  • Thank you, that makes sense. So should those connections be closed or they will time-out after a period of time? – Demetris Jan 26 '16 at 18:01
  • no problem ;) now, about the stale connections - you can close them manually using rabbitmqctl, write a script that does that for you from a cron job (both ugly) or if you can, switch on heartbeat and set the timeout to, say, 15-20% more than your longest known processing task. that should recycle those connections after heartbeat timeout, but it will also disconnect any client that has been busy for too long, so be careful. – Grega Bremec Jan 26 '16 at 18:40
  • If you can provide this information as an answer, in order for me to accept it. Thanks. – Demetris Jan 26 '16 at 19:08

1 Answers1

1

AMQP requires the server-side of the connection to stay open if the client does not close the connection properly (for long-running tasks). Read more in this post.

You can close those connections manually using rabbitmqctl or write a script that does that for you from a cron job, but both are rather ugly outside approaches to an application-level problem.

If you can, switch on heartbeat and set the timeout to, say, 15-20% more than your longest known processing task. That should recycle those "dead" connections after a heartbeat timeout, but be careful as it will also disconnect any client that has been busy for too long.

More on heartbeat and rabbitmqctl (see section Miscellaneous).

Community
  • 1
Grega Bremec
  • 226