Troubleshooting Genetec Mission Control™ - Mission Control 2.11 - 3.2.0.0

Product
Mission Control
Content type
Troubleshooting
Version
3.2
3.1
3.0
2.13
2.12
2.11
Language
English
Last updated
2023-05-10

Resetting passwords for RabbitMQ user accounts

If you lose the password for the RabbitMQ account that was created during the Genetec Mission Control™ installation, you can change the password from the RabbitMQ command prompt.

Symptoms

The Incident Manager role in Config Tool is red.

Cause

Without valid RabbitMQ logon credentials, you cannot connect to the RabbitMQ service from Config Tool. Without establishing a connection, the Incident Manager role goes offline and turns red.

Solution

To reset the password for your RabbitMQ account:
  1. From the server machine on which RabbitMQ is installed, open an elevated command prompt in Windows:
    1. In the Windows search box, type cmd.
    2. Press Ctrl+Shift+Enter.

      The User Account Control window opens and prompts you to run as administrator.

    3. Click Yes.
  2. Navigate to the RabbitMQ sbin folder.
    NOTE: It is usually found in C:\Program Files\RabbitMQ Server\rabbitmq_server-x.y.z\sbin.
  3. To see the current list of users in RabbitMQ, execute the following command:

    rabbitmqctl list_users

  4. Reset the password of your specific RabbitMQ account by executing the following command:

    rabbitmqctl change_password <user> <new password>

    NOTE: If an Erlang cookie authentication failure prevents the rabbitmqctl commands from being executed, you must ensure the following locations have the same .erlang.cookie files:
    • C:\Windows\ServiceProfiles\LocalService\
    • C:\
    • C:\Users\<username>\
    • C:\Windows\System32\config\systemprofile\
  5. Update the RabbitMQ logon credentials in Config Tool under Incident Manager > RabbitMQ.
    NOTE: For versions before Mission Control 3.0.2.0, the RabbitMQ logon details would be under Incident Manager > Properties.

Incident Manager role not connecting to RabbitMQ

If the Incident Manager role in Config Tool goes offline because it cannot connect to the RabbitMQ service, you can resolve the issue by checking a few pertinent settings such as firewall ports, user credentials, and so on.

Symptoms

The Incident Manager role in Config Tool is red.

Depending on the error message your receive when you click Maintenance > Diagnose, you can infer the cause of the error and implement the appropriate solution.

Error message

RabbitMQ connection failed - <RabbitMQ server IP address> Will retry in x seconds

Cause

The server running the Incident Manager role cannot connect to the RabbitMQ service running on port 5671 (default port) using an SSL connection.

Solution
Try one or a combination of the following solutions:
  • In Windows, go to Control Panel > Services and ensure that the RabbitMQ service has successfully started on the server where it is installed.
  • From the Config Tool under Incident Manager > RabbitMQ tab, ensure that you are using the correct Username and Password for the RabbitMQ connection. The Username and Password would have been created during the Genetec Mission Control™ installation.
    NOTE: For versions before Mission Control 3.0.2.0, the RabbitMQ logon details would be under Incident Manager > Properties.
  • Replace the RabbitMQ server name with the IP address of the server, or ensure that the server name is clear. The following image shows an example of the RabbitMQ logon screen:
    NOTE: For versions before Mission Control 3.0.2.0, the RabbitMQ logon details would be under Incident Manager > Properties.
  • Ensure that the server running the Incident Manager role can access the server running RabbitMQ on port5671 & port 15671.
    Tip: Open the command prompt on the Incident Manager Role server and enter the following:

    telnet <RabbitMQ server IP address> 5671

    telnet <RabbitMQ server IP address> 15671

    If the servers are communicating properly, you will get a response.
  • Ensure that all firewall ports allow TCP connections both ways between the Mission Control server and RabbitMQ server, on port 5671 & port 15671.

Error message

RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. Will try to reconnect in x seconds.

Cause
The credentials supplied in the Incident Manager's RabbitMQ tab do not match the RabbitMQ server credentials.
NOTE: For versions before Mission Control 3.0.2.0, the RabbitMQ logon details would be under Incident Manager > Properties.
Solution
Try one or a combination of the following solutions:
  • Verify your RabbitMQ credentials:
    1. Go to https://<RabbitMQ server IP address>:<RabbitMQ server web port>. The default web port is 15671.
      NOTE: Internet Explorer is no longer a supported browser for the RabbitMQ web portal.
    2. Enter your RabbitMQ credentials and ensure they are correct.
  • Update the RabbitMQ host name or IP address in the RabbitMQ page of the Incident Manager role to match the RabbitMQ server credentials.
  • If you have forgotten your credentials:
    1. Log on to the sever machine with RabbitMQ installed.
    2. Open the RabbitMQ command prompt and enter rabbitmqctl.bat change_password <username> <new password> .

For more details, see Resetting passwords for RabbitMQ user accounts.

Error message

One of the following error messages:
  • RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> Connection failed: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond <RabbitMQ server IP address>:<RabbitMQ server port>. Will try to reconnect in x seconds.
  • RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> Connection failed: No connection could be made because the target machine actively refused it. Will try to reconnect in x seconds.
Cause
Any one or combination of the following:
  • The host names or IP addresses supplied in Incident Manager's RabbitMQ page do not point to a valid RabbitMQ server.
  • The ports supplied in the Incident Manager's RabbitMQ page do not point to the correct port where RabbitMQ is hosted.
  • If the connection was working before, your RabbitMQ server might be down or unreachable due to network related problems.
NOTE: For versions before Mission Control 3.0.2.0, the RabbitMQ logon details would be under Incident Manager > Properties.
Solution
Try one or a combination of the following solutions:
  • Update the RabbitMQ host name or IP address in the Incident Manager's RabbitMQ page to match the RabbitMQ server credentials.
  • Update the RabbitMQ port number in the Incident Manager's RabbitMQ page to match the port number in your RabbitMQ server.
    NOTE: For versions before Mission Control 3.0.2.0, the RabbitMQ logon details would be under Incident Manager > Properties.
  • Access the machine where your RabbitMQ server is running to ensure that it is working.
  • See if any additional errors are reported by RabbitMQ in https://<RabbitMQ server IP address>:<RabbitMQ server web port>. The default web port is 15671.
    NOTE: Internet Explorer is no longer a supported browser for the RabbitMQ web portal.
  • If RabbitMQ is not the problem, check the following:
    • Firewall settings
    • Network settings
    • Network equipments

Error message

RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> The remote certificate is invalid according to the validation procedure. Will try to reconnect in x seconds.

Cause

The certificates on the RabbitMQ server and the Incident Manager do not match.

Solution

Make sure the certificates within the folder C:\Program Files (x86)\Genetec\RabbitMQ\ on both the Incident Manager and the RabbitMQ server machine are the same.

Error message

RabbitMQ connection failed. Will try to reconnect in x seconds.

Cause

Changing the host name after the installation of RabbitMQ causes a RabbitMQ certificate authentication failure.

Solution
Implement one of the following solutions:
  • Perform a clean uninstall of RabbitMQ. See, Uninstalling RabbitMQ.
  • Revert the host name back to the name set during installation of RabbitMQ.

Error message

RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> RabbitMQ's server does not have the Virtual Host "Genetec.MissionControl" usually created by the Mission Control RabbitMQ installer. You can create it manually from the RabbitMQ web management interface, but please note that from Mission Control 2.13.4 onwards, minimum accepted RabbitMQ version is 3.8.3. Using the Mission Control installer to update your RabbitMQ server is strongly recommended. Will try to reconnect in x seconds.

Applicable versions: Mission Control 3.0.1.0 and below.
NOTE: For Mission Control versions 3.0.2.0 and later, the virtual host is created automatically when the Incident Manager role is initialized.
Cause

The RabbitMQ's server does not have the virtual host Genetec.MissionControl.

Solution
Add a new virtual host.
  • Add new virtual host through web browser:
    • Log on to https://<RabbitMQ server IP address>:<RabbitMQ server web port>. The default web port is 15671.
      NOTE: Internet Explorer is no longer a supported browser for the RabbitMQ web portal.
    • Navigate to Admin > Virtual Hosts > Add new virtual host.
    • Enter Genetec.MissionControl as a name and click Add virtual host.
    • Set permissions:
      • Navigate to Admin > Users.
      • Select your user.
      • Select the Genetec.MissionControl virtual host.
      • Click Set permissions.
  • Add new virtual host through command prompt:
    • Log on to the RabbitMQ server machine.
    • Open the RabbitMQ command prompt and enter the following commands:
      1. rabbitmqctl.bat add_vhost Genetec.MissionControl
      2. rabbitmqctl.bat set_permissions -p Genetec.MissionControl admin ".*" ".*" ".*"

Error message

RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> RabbitMQ's server version (<older version than 3.8.3>) does not meet the minimum version requirement. From Mission Control 2.13.4 onwards, minimum accepted RabbitMQ version is 3.8.3. Use Mission Control installer to update your RabbitMQ server. Will try to reconnect in 4 seconds.

Cause

The RabbitMQ's server is running a outdated version lower than 3.8.3 which is the minimum required version by the Incident Manager starting as of Mission Control 2.13.4.0.

Solution

Update the RabbitMQ server using the Mission Control installer.

Error message

RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> INTERNAL_ERROR - Cannot declare a queue 'queue 'IncidentService.IncidentCommands' in vhost 'Genetec.MissionControl'' of type 'rabbit_quorum_queue' on node 'rabbit@VM1800': the 'quorum_queue' feature flag is disabled

Applicable versions: Mission Control 2.12.0.0 to 2.13.4.0

Cause

In a specific upgrade scenario, your RabbitMQ could have the quorum queue feature flag disabled which is needed by the Incident Manager.

Solution

Log on to the RabbitMQ server machine, open a RabbitMQ command prompt and enter the following command: rabbitmqctl.bat enable_feature_flag quorum_queue

You can also enable the feature flag through the web interface:
  1. Log on to https://<RabbitMQ server IP address>:<RabbitMQ server web port>. The default web port is 15671.
    NOTE: Internet Explorer is no longer a supported browser for the RabbitMQ web portal.
  2. In the Admin tab, Feature Flags section, click the Enable button next to the Quorum_queue feature flag.

Error message

[MissionControlServer][BrokerConnection] RabbitMQ management API network failure (timed out): verify address, port, certificate and status of the RMQ service. (When executing GET on /overview received response with status TimedOut and HTTP code 0. Request's body: )

Cause
Any one or combination of the following:
  • The host names or IP addresses supplied in Incident Manager's RabbitMQ page do not point to a valid RabbitMQ server.
  • The ports supplied in the Incident Manager's RabbitMQ page do not point to the correct HTTPS API port where RabbitMQ is hosted.
Solution
Try one or a combination of the following solutions:
  • Update the RabbitMQ host name or IP address in the Incident Manager's RabbitMQ page to match the RabbitMQ server credentials.
  • Update the RabbitMQ HTTPS API port number in the Incident Manager's RabbitMQ page to match the port number in your RabbitMQ server.
    NOTE: If you want to change the default port numbers used by Mission Control, you must either change it during the installation process or change the port in the RabbitMQ.conf file. After modifying the RabbitMQ.conf, you must update the details here to match. For details, see Modifying RabbitMQ ports.

Incident Manager tasks disappeared

All Genetec Mission Control™ tasks that were available in Security Desk and Config Tool, such as Incident monitoring and Incident configuration, have disappeared. This can occur when the Incidents (Mission Control) feature is disabled.

Cause

If Mission Control is properly licensed and correctly installed on the Directory server, any expansion server running the Incident Manager role, and all client machines, all related tasks will be visible in Security Desk and Config Tool by default. If the Incidents (Mission Control) feature is disabled, these tasks will be hidden until the feature is re-enabled.

Solution

  1. In Config Tool, open System > General settings > Features
  2. In the Features list, enable Incidents (Mission Control).
  3. Click Apply > Restart.
    NOTE: If any other Security Desk or Config Tool is connected to the system, it must also be restarted to see the changes.

RabbitMQ connections blocked

If the Incident monitoring task in Security Desk or Incident configuration task in Config Tool display an error message related to server connections it might be because of blocked RabbitMQ connections.

Symptom

Incident monitoring task in Security Desk or Incident configuration task in Config Tool display the following error message with an empty diagnostics dialog box:

Waiting for connection to server.

This can be verified from the RabbitMQ web portal:
  1. Log on to https://<RabbitMQ server IP address>:<RabbitMQ server web port>. The default web port is 15671.
    NOTE: Internet Explorer is no longer a supported browser for the RabbitMQ web portal.
  2. Enter your RabbitMQ credentials.
  3. Select the Connections tab.

The Virtual host State will show blocked or blocking.

Cause

Low server resources.

This can be verified from the RabbitMQ logs found in C:\ProgramData\Genetec\RabbitMQ\log.
**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************
2020-12-01 11:48:06.702 [info] <0.358.0> vm_memory_high_watermark clear. Memory used:3383427072 allowed:3435547852
2020-12-01 11:48:06.702 [warning] <0.356.0> memory resource limit alarm cleared on node 'rabbit@SCIGCLB-S*****'
2020-12-01 11:48:06.702 [warning] <0.356.0> memory resource limit alarm cleared across the cluster

For more information on blocked RabbitMQ connections, see Blocked Connection Notifications.

Solution

Resolve this issue using one of the following methods:
  • Restart the RabbitMQ service:
    1. On the RabbitMQ server, open Microsoft Management Console Services (services.msc).
    2. Select the RabbitMQ from the list of services and click Restart.

    This releases memory resources.

  • Increase virtual machine memory limit through the command line:
    • To dedicate a percentage of RAM to be used by RabbitMQ: rabbitmqctl set_vm_memory_high_watermark <fraction>

      For example, to dedicate 60% of RAM for RabbitMQ operations use the following command in the command prompt: rabbitmqctl set_vm_memory_high_watermark 0.6.

    • To dedicate exact amount of RAM to be used by RabbitMQ: rabbitmqctl set_vm_memory_high_watermark absolute "<memory_limit>"

      For example, to dedicate 6GB of RAM for RabbitMQ: rabbitmqctl set_vm_memory_high_watermark absolute "6G".

  • Increase server resources.

System takes too long to trigger incidents

When the system takes too long to trigger incidents, you can resolve the issue by filtering out unnecessary events from the RabbitMQ event queue.

Symptoms

Incidents are not triggered in a timely manner in Genetec Mission Control™.

Cause

Depending on the configuration of the system and its connected entities, the directory role might be sending too many events to the Mission Control RabbitMQ server. This makes the event queue too large to process new events, thereby creating a delay in triggering incidents. You can find details on the number of events in queue and response rate of the RabbitMQ server at https://<RabbitMQ server IP address>:<RabbitMQ server web port>.
NOTE: The default web port is 15671.

Solution

  1. Stop the Incident Manager role in Config Tool:
    • Go to System > Roles > Incident Manager.
    • Right-click Incident Manager.
    • Select Maintenance > Deactivate role.
  2. Log on to the RabbitMQ service from https://<RabbitMQ server IP address>:<RabbitMQ server web port> The default web port is 15671..
    NOTE: Internet Explorer is no longer a supported browser for the RabbitMQ web portal.
  3. Click the Queues tab and then click ExternalEventsQueue.
  4. Click Purge and then click Purge messages to clear the queue.
  5. Activate the Incident Manager role again:
    • Go to System > Roles > Incident Manager.
    • Right-click Incident Manager.
    • Select Maintenance > Activate role.
  6. Filter the events that you do not need in your Mission Control configuration:
    1. In Config Tool, go to System > Roles > Incident Manager.
    2. Click the Events tab.
    3. Deselect the events that you do not need.