Last time i experienced problems with Zenoss. Zenoss suddenly stopped getting information from the clients, stopped graphing client data and i did not know what the reason was for this situation. I tried removing and readding the client but the job of adding a new client was even not processed. When i tried to re-model the client i saw an error saying “ZenHub is down”.
I started to dig in and started to view all of those logs Zenoss produces (there are just too many!) and found that zenstatus log is also reporting “ZenHub is down” error. Here is what i found and how i resolved the “ZenHub is down” Zenoss problems.
Let’s fix the “ZenHub is down” problem!
1. Check ZenHub Status
You should see that ZenHub is actually up and running which is surprising due to the fact the only error you can find says “ZenHub is down”.
Be sure to run the following commands as “zenoss” user!
[zenoss@geekpeek ~]$ zenhub status program running; pid=19774
2. Enable ZenHub Debug Mode
You can enable ZenHub debug mode easily and on the fly with the following command.
[zenoss@geekpeek ~]$ zenhub debug Sending SIGUSR1 to 19774
3. Check Zenoss Logs
With ZenHub in debug mode you should see additional information from zenhub.log file. If you find the error “zen.ZenHub: all workers are busy” you are experiencing exactly the same problem i did.
/opt/zenoss/log/zenhub.log
2015-02-19 11:27:31,902 DEBUG zen.ZenHub: all workers are busy 2015-02-19 11:27:36,903 DEBUG zen.ZenHub: worklist has 27 items
Zenstatus.log file is still saying that “ZenHub may be disconnected” and “ZenHub is down”.
/opt/zenoss/log/zenstatus.log
2015-02-19 11:20:01,113 WARNING zen.zenstatus: No service named 'EventService': ZenHub may be disconnected 2015-02-19 11:20:01,113 ERROR zen.maintenance: Maintenance failed. Message from hub: ZenHub is down 2015-02-19 11:20:02,653 ERROR zen.zenstatus: No event service: None 2015-02-19 11:20:07,657 ERROR zen.zenstatus: No event service: None 2015-02-19 11:20:12,354 INFO zen.pbclientfactory: Initial connect timed out after 60 seconds 2015-02-19 11:20:12,658 ERROR zen.zenstatus: No event service: None 2015-02-19 11:20:17,663 ERROR zen.zenstatus: No event service: None 2015-02-19 11:20:22,664 ERROR zen.zenstatus: No event service: None
4. Check RabbitMQ Connections
An crucial advice here is to check RabbitMQ connections on Zenoss server. RabbitMQ is processing events for Zenoss server.
[root@geekpeek log]# rabbitmqctl list_connections Listing connections ... zenoss 127.0.0.1 41928 blocked zenoss 127.0.0.1 39100 blocked zenoss 127.0.0.1 46164 blocked zenoss 127.0.0.1 39083 blocking zenoss 127.0.0.1 39069 blocked zenoss 127.0.0.1 39101 blocked zenoss 127.0.0.1 39063 blocking zenoss 127.0.0.1 39077 blocked zenoss 127.0.0.1 39052 blocked zenoss 127.0.0.1 39062 blocking ...done.
If you see all RabbitMQ connections in “blocked” or “blocking” status this is the root of all your problems!
5. Reconfigure RabbitMQ
The problem is RabbitMQ has a free disk space limit set for the partition on which it is storing data and when available disk space falls below this limit, flow control is triggered. If you are aware of the disk space and RAM memory available you can reconfigure this value by editing “/etc/rabbitmq/rabbitmq.config” file.
[root@geekpeek ~]# cat /etc/rabbitmq/rabbitmq.config
[
{rabbit, [{disk_free_limit, {mem_relative, 0.1}}]}
].You can set the disk free limit relative to the memory available as shown above.
6. Restart RabbitMQ and Zenoss
After RabbitMQ reconfiguration you should restart RabbitMQ and Zennos for the configuration changes to apply.
[root@geekpeek ~]# /etc/init.d/rabbitmq-server restart [root@geekpeek ~]# /etc/init.d/zenoss restart
You should now have a working Zenoss without “ZenHub is down” error messages.