Hypervisor appearing Offline to the Management Plane - Error "ETIMEDOUT"

Problem

A Platform9 managed host shows as offline in Clarity UI.
On the host, the following traceback and subsequent error are observed in /var/log/pf9/hostagent.log.

    session.py WARNING - Not sending status message because channel is closed
    session.py INFO - Using the default virtual host '/' on the AMQP broker localhost
    slave.py ERROR - Connection error. Retrying in 10 seconds.
    Traceback (most recent call last):
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/bbslave/slave.py", line 83, in reconnect_loop
     channel_retry_period=retry_period)
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/bbslave/session.py", line 712, in start
     socket_timeout=socket_timeout) # in secs
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/bbcommon/amqp.py", line 238, in dual_channel_io_loop
     conn.ioloop.start()
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 354, in start
     self.poll()
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 602, in poll
     self._process_fd_events(fd_event_map, write_only)
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 443, in _process_fd_events
     handler(fileno, events, write_only=write_only)
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adaptersbase_connection.py", line 364, in _handle_events
     self._handle_read()
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 412, in _handle_read
     return self._handle_disconnect()
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 288, in _handle_disconnect
     self._adapter_disconnect()
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/select_connection.py", line 95, in _adapter_disconnect
     super(SelectConnection, self)._adapter_disconnect()
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 154, in _adapter_disconnect
     self._check_state_on_disconnect()
     File "/opt/pf9/hostagent/lib/python2.7/site-packages/pika/adapters/base_connection.py", line 169, in _check_state_on_disconnect
     raise exceptions.IncompatibleProtocolError
    IncompatibleProtocolError

Additionally, in /var/log/pf9/comms.log, an 'ETIMEDOUT' error is observed for the IPv4 address of the Management Plane.

    [ERROR] sni-broker-openstack-v1-5673-3 - Server socket for client 32003 error: { [Error: connect ETIMEDOUT<IPv4_addrress>:443]
    code: 'ETIMEDOUT',
    errno: 'ETIMEDOUT',
    syscall: 'connect',
    address: '<IPv4_addrress>',
    port: 443 }
    Error: connect ETIMEDOUT<IPv4_addrress>:443
    at Object.exports._errnoException (util.js:907:11)
    at exports._exceptionWithHostPort (util.js:930:20)
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1081:14)
    [DEBUG] sni-broker-openstack-v1-5673-3 - Server socket for client 32003 closed with error ... remaining: 17
    [WARN] sni-broker-v1-5672-2 - Forcefully destroying server socket for client 32025
    [DEBUG] sni-broker-v1-5672-2 - Server socket for client 32025 closed normally ... remaining: 3
    [WARN] sni-broker-openstack-v1-5673-3 - Forcefully destroying client 32003
    [DEBUG] sni-broker-openstack-v1-5673-3 - Client 32003 socket closed normally ... remaining: 17
    [ERROR] sni-broker-openstack-v1-5673-3 - Server socket for client 32005 error: { [Error: connect ETIMEDOUT<IPv4_addrress>:443]
    code: 'ETIMEDOUT',
    errno: 'ETIMEDOUT',
    syscall: 'connect',
    address: '<IPv4_addrress>',
    port: 443 }

Environment

Platform9 Managed OpenStack - All Versions
Platform9 Managed Kubernetes - All Versions

Cause

Bi-directional communication is required on port 443 between the host and Platform9 Management Plane in order to orchestrate operations necessary to provision cloud resources.
Resolution

  1. Disable any firewalls temporarily.
  2. Otherwise, ensure bi-directional traffic between the host and Management Plane is allowed on port 443.