Troubleshooting Genetec Mission Control™ - Genetec Mission Control™

series
Genetec Mission Control™
revised_modified
2021-03-25
category_custom
Guides
Troubleshooting topics

Resetting passwords for RabbitMQ user accounts

If you lose the password for the RabbitMQ account that was created during the Genetec Mission Control™ installation, you can change the password from the RabbitMQ command prompt.

Symptoms

The Incident Manager role in Config Tool is red.

Cause

Without valid RabbitMQ logon credentials, you cannot connect to the RabbitMQ service from Config Tool. Without establishing a connection, the Incident Manager role goes offline and turns red.

Solution

To reset the password for your RabbitMQ account:
  1. From the server machine on which RabbitMQ is installed, open an elevated command prompt in Windows:
    1. In the Windows search box, type cmd.
    2. Press Ctrl+Shift+Enter.

      The User Account Control window opens and prompts you to run as administrator.

    3. Click Yes.
  2. Navigate to the RabbitMQ sbin folder.
    NOTE: It is usually found in C:\Program Files\RabbitMQ Server\rabbitmq_server-x.y.z\sbin.
  3. To see the current list of users in RabbitMQ, execute the following command:

    rabbitmqctl list_users

  4. Reset the password of your specific RabbitMQ account by executing the following command:

    rabbitmqctl change_password <user> <new password>

    NOTE: If an Erlang cookie authentication failure prevents the rabbitmqctl commands from being executed, you must synchronize the .erlang.cookie files on your machine. For more information, refer to the RabbitMQ installation instructions.
  5. Update the RabbitMQ logon credentials in Config Tool under Incident Manager > Properties.

Incident Manager role not connecting to RabbitMQ

If the Incident Manager role in Config Tool goes offline because it cannot connect to the RabbitMQ service, you can resolve the issue by checking a few pertinent settings such as firewall ports, user credentials, and so on.

Symptoms

The Incident Manager role in Config Tool is red.

Depending on the error message your receive when you click Maintenance > Diagnose, you can infer the cause of the error and implement the appropriate solution.

Error message

RabbitMQ connection failed - <RabbitMQ server IP address> Will retry in x seconds

Cause

The server running the Incident Manager role cannot connect to the RabbitMQ service running on port 5671 (default port) using an SSL connection.

Solution
Try one or a combination of the following solutions:
  • In Windows, go to Control Panel > Services and ensure that the RabbitMQ service has successfully started on the server where it is installed.
  • From the Incident Manager role configuration tabs in Config Tool, ensure that you are using the correct Username and Password for the RabbitMQ connection. The Username and Password would have been created during the Genetec Mission Control™ installation.
  • Replace the RabbitMQ server name with the IP address of the server, or ensure that the server name is clear. The following image shows an example of the RabbitMQ logon screen:
  • Ensure that the server running the Incident Manager role can access the server running RabbitMQ on port 5671.
    Tip: Open the command prompt and enter the following:

    telnet <server IP address> 5671

    If the servers are communicating properly, you will get a response.
  • Ensure that all firewall ports allow TCP connections both ways between the Genetec Mission Control™ server and RabbitMQ server, on port 5671.

Error message

RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. Will try to reconnect in x seconds.

Cause

The credentials supplied in RabbitMQ section of the Incident Manager's Properties page do not match the RabbitMQ server credentials.

Solution
Try one or a combination of the following solutions:
  • Verify your RabbitMQ credentials:
    1. Go to https://<RabbitMQ server IP address>:<RabbitMQ server web port>.
      NOTE: The default web port is 15671.
    2. Enter your RabbitMQ credentials and ensure they are correct.
  • Update the RabbitMQ host name or IP address in the Properties page of the Incident Manager role to match the RabbitMQ server credentials.
  • If you have forgotten your credentials:
    1. Log on to the sever machine with RabbitMQ installed.
    2. Open the RabbitMQ command prompt and enter rabbitmqctl.bat change_password <username> <new password>.

For more details, see Resetting passwords for RabbitMQ user accounts.

Error message

One of the following error messages:
  • RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> Connection failed: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond <RabbitMQ server IP address>:<RabbitMQ server port>. Will try to reconnect in x seconds.
  • RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> Connection failed: No connection could be made because the target machine actively refused it. Will try to reconnect in x seconds.
Cause
Any one or combination of the following:
  • The host names or IP addresses supplied in RabbitMQ section of the Incident Manager's Properties page do not point to a valid RabbitMQ server.
  • The ports supplied in RabbitMQ section of the Incident Manager's Properties page do not point to the correct port where RabbitMQ is hosted.
  • If the connection was working before, your RabbitMQ server might be down or unreachable due to network related problems.
Solution
Try one or a combination of the following solutions:
  • Update the RabbitMQ host name or IP address in the Properties page of the Incident Manager role to match the RabbitMQ server credentials.
  • Update the RabbitMQ port number in the Properties page of the Incident Manager role to match the port number in your RabbitMQ server.
  • Access the machine where your RabbitMQ server is running to ensure that it is working.
  • See if any additional errors are reported by RabbitMQ in https://<RabbitMQ server IP address>:<RabbitMQ server web port>.
    NOTE: The default web port is 15671.
  • If RabbitMQ is not the problem, check the following:
    • Firewall settings
    • Network settings
    • Network equipments

Error message

RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> The remote certificate is invalid according to the validation procedure. Will try to reconnect in x seconds.

Cause

The certificates on the RabbitMQ server and the Incident Manager do not match.

Solution

Make sure the certificates within the folder C:\Program Files (x86)\Genetec\RabbitMQ\ on both the Incident Manager and the RabbitMQ server machine are the same.

Error message

RabbitMQ connection failed. Will try to reconnect in x seconds.

Cause

Changing the host name after the installation of RabbitMQ causes a RabbitMQ certificate authentication failure.

Solution
Implement one of the following solutions:
  • Perform a clean uninstall of RabbitMQ. See, Uninstalling RabbitMQ.
  • Revert the host name back to the name set during installation of RabbitMQ.

Error message

RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> RabbitMQ's server does not have the Virtual Host "Genetec.MissionControl" usually created by the Mission Control RabbitMQ installer. You can create it manually from the RabbitMQ web management interface, but please note that from Mission Control 2.13.4 onwards, minimum accepted RabbitMQ version is 3.8.3. Using the Mission Control installer to update your RabbitMQ server is strongly recommended. Will try to reconnect in x seconds.

Cause

The RabbitMQ's server does not have the virtual host Genetec.MissionControl.

Solution
Add a new virtual host.
  • Add new virtual host through web browser:
    • Log on to https://<RabbitMQ server IP address>:<RabbitMQ server web port>.
      NOTE: The default web port is 15671.
    • Navigate to Admin > Virtual Hosts > Add new virtual host.
    • Enter Genetec.MissionControl as a name and click Add virtual host.
    • Set permissions:
      • Navigate to Admin > Users.
      • Select your user.
      • Select the Genetec.MissionControl virtual host.
      • Click Set permissions.
  • Add new virtual host through command prompt:
    • Log on to the RabbitMQ server machine.
    • Open the RabbitMQ command prompt and enter the following commands:
      1. rabbitmqctl.bat add_vhost Genetec.MissionControl
      2. rabbitmqctl.bat set_permissions -p Genetec.MissionControl admin ".*" ".*" ".*"

Error message

RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> RabbitMQ's server version (<older version than 3.8.3>) does not meet the minimum version requirement. From Mission Control 2.13.4 onwards, minimum accepted RabbitMQ version is 3.8.3. Use Mission Control installer to update your RabbitMQ server. Will try to reconnect in 4 seconds.

Cause

The RabbitMQ's server is running a outdated version lower than 3.8.3 which is the minimum required version by the Incident Manager starting as of Genetec Mission Control™ 2.13.4.0.

Solution

Update the RabbitMQ server using the Genetec Mission Control™ installer.

Error message

RabbitMQ connection failed. - <RabbitMQ server IP address>:<RabbitMQ server port> INTERNAL_ERROR - Cannot declare a queue 'queue 'IncidentService.IncidentCommands' in vhost 'Genetec.MissionControl'' of type 'rabbit_quorum_queue' on node 'rabbit@VM1800': the 'quorum_queue' feature flag is disabled

Cause

In a specific upgrade scenario, your RabbitMQ could have the quorum queue feature flag disabled which is needed by the Incident Manager.

Applicable versions: Genetec Mission Control™ 2.12.0.0 to 2.13.4.0

Solution

Log on to the RabbitMQ server machine, open a RabbitMQ command prompt and enter the following command: rabbitmqctl.bat enable_feature_flag quorum_queue

You can also enable the feature flag through the web interface:
  1. Open the RabbitMQ web portal (https://localhost:15671).
  2. In the Admin tab, Feature Flags section, click the Enable button next to the Quorum_queue feature flag.

Incident Manager tasks disappeared

All Genetec Mission Control™ tasks that were available in Security Desk and Config Tool, such as Incident monitoring and Incident configuration, have disappeared. This can occur when the Incidents (Mission Control) feature is disabled.

Cause

If Genetec Mission Control™ is properly licensed and correctly installed on the Directory server, any expansion server running the Incident Manager role, and all client machines, all related tasks will be visible in Security Desk and Config Tool by default. If the Incidents (Mission Control) feature is disabled, these tasks will be hidden until the feature is re-enabled.

Solution

  1. In Config Tool, open System > General settings > Features
  2. In the Features list, enable Incidents (Mission Control).
  3. Click Apply > Restart.
    NOTE: If any other Security Desk or Config Tool is connected to the system, it must also be restarted to see the changes.

RabbitMQ connections blocked

If the Incident monitoring task in Security Desk or Incident configuration task in Config Tool display an error message related to server connections it might be because of blocked RabbitMQ connections.

Symptom

Incident monitoring task in Security Desk or Incident configuration task in Config Tool display the following error message with an empty diagnostics dialog box:

Waiting for connection to server.

This can be verified from the RabbitMQ web portal:
  1. Go to https://<RabbitMQ server IP address>:<RabbitMQ server web port>.
    NOTE: The default web port is 15671.
  2. Enter your RabbitMQ credentials.
  3. Select the Connections tab.

The Virtual host State will show blocked or blocking.

Cause

Low server resources.

This can be verified from the RabbitMQ logs found in C:\ProgramData\Genetec\RabbitMQ\log.
**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************
2020-12-01 11:48:06.702 [info] <0.358.0> vm_memory_high_watermark clear. Memory used:3383427072 allowed:3435547852
2020-12-01 11:48:06.702 [warning] <0.356.0> memory resource limit alarm cleared on node 'rabbit@SCIGCLB-S*****'
2020-12-01 11:48:06.702 [warning] <0.356.0> memory resource limit alarm cleared across the cluster

For more information on blocked RabbitMQ connections, see Blocked Connection Notifications.

Solution

Resolve this issue using one of the following methods:
  • Restart the RabbitMQ service:
    1. On the RabbitMQ server, open Microsoft Management Console Services (services.msc).
    2. Select the RabbitMQ from the list of services and click Restart.

    This releases memory resources.

  • Increase virtual machine memory limit through the command line:
    • To dedicate a percentage of RAM to be used by RabbitMQ: rabbitmqctl set_vm_memory_high_watermark <fraction>

      For example, to dedicate 60% of RAM for RabbitMQ operations use the following command in the command prompt: rabbitmqctl set_vm_memory_high_watermark 0.6.

    • To dedicate exact amount of RAM to be used by RabbitMQ: rabbitmqctl set_vm_memory_high_watermark absolute "<memory_limit>"

      For example, to dedicate 6GB of RAM for RabbitMQ: rabbitmqctl set_vm_memory_high_watermark absolute "6G".

  • Increase server resources.

System takes too long to trigger incidents

When the system takes too long to trigger incidents, you can resolve the issue by filtering out unnecessary events from the RabbitMQ event queue.

Symptoms

Incidents are not triggered in a timely manner in Genetec Mission Control™.

Cause

Depending on the configuration of the system and its connected entities, the directory role might be sending too many events to the Genetec Mission Control™ RabbitMQ server. This makes the event queue too large to process new events, thereby creating a delay in triggering incidents. You can find details on the number of events in queue and response rate of the RabbitMQ server at https://<RabbitMQ server IP address>:<RabbitMQ server web port>.
NOTE: The default web port is 15671.

Solution

  1. Stop the Incident Manager role in Config Tool:
    • Go to System > Roles > Incident Manager.
    • Right-click Incident Manager.
    • Select Maintenance > Deactivate role.
  2. Log on to the RabbitMQ service from https://<RabbitMQ server IP address>:<RabbitMQ server web port>.
    NOTE: The default web port is 15671.
  3. Click the Queues tab and then click ExternalEventsQueue.
  4. Click Purge and then click Purge messages to clear the queue.
  5. Activate the Incident Manager role again:
    • Go to System > Roles > Incident Manager.
    • Right-click Incident Manager.
    • Select Maintenance > Activate role.
  6. Filter the events that you do not need in your Genetec Mission Control™ configuration:
    1. In Config Tool, go to System > Roles > Incident Manager.
    2. Click the Events tab.
    3. Deselect the events that you do not need.

    To learn more about event filtering, see Specifying allowed events for Genetec Mission Control™.

Genetec Mission Control™ Web API not updated by Genetec™ Update Service

Genetec™ Update Service does not automatically update Genetec Mission Control™ Web API when updating Genetec Mission Control™ version.

Cause

Genetec Mission Control™ and Genetec Mission Control™ Web API have separate installers.

Solution

You must manually update Genetec Mission Control™ Web API to the compatible version.

For instructions, see Downloading and installing the Genetec Mission Control™ Web API in the Genetec Mission Control™ Deployment Guide.

Conditional activities error after Genetec Mission Control™ upgrade

The Conditional activities configured for an incident type in Genetec Mission Control™ 2.13.3.0 does not carry over the configured latest aggregated event setting when Genetec Mission Control™ is upgraded.

Symptom

The last aggregated event name selected in the Conditional activities configured for an incident type automation workflow in Genetec Mission Control™ 2.13.3.0 is not automatically selected when Genetec Mission Control™ is upgraded. This means the automation workflow is not executed as desired.

Cause

The event selection is reset to none by code.

Solution

You must reconfigure the Conditional activities for the incident type by selecting the awaited event name.

Uninstalling RabbitMQ

If your RabbitMQ server service is unable to start or handle messages due to corrupted RabbitMQ installation files, the best option is to perform a clean uninstall of RabbitMQ and then reinstall it.

Procedure

Applicable to Genetec Mission Control™ versions 2.13.4.0 and above.

  1. Go to Start > Control panel > Add or remove programs.
  2. Uninstall the Genetec RabbitMQ Server program.
  3. Verify the all installation directories for RabbitMQ have been cleared in the following locations:
    • C:\Program Files\erl<erlang version>\
    • C:\Program Files\RabbitMQ Server\
    • C:\Program Files (x86)\Genetec\RabbitMQ\
    • %ProgramData%\Genetec\RabbitMQ\
      NOTE: By default, this path leads to C:\ProgramData\Genetec\RabbitMQ
    • %AppData%\RabbitMQ\
      NOTE: By default, this path leads to C:\Users\<Username>\AppData\Roaming\RabbitMQ

Procedure

Applicable to Genetec Mission Control™ versions below 2.13.4.0.

  1. Go to Start > Control panel > Add or remove programs and uninstall the following programs:
    • RabbitMQ Server <version>
    • Erlang OTP <version>
    • OpenSSL <version>
  2. Press Ctrl+Shift+Esc to open the Task manager.
    • Right click on erl.exe and select End task.
    • Right click on epmd.exe and select End task.
  3. Delete all the installation directories for RabbitMQ and Erlang. This includes the following files and folders:
    • C:\Program Files\erl<erlang version>\
    • C:\Program Files\RabbitMQ Server\
    • C:\Program Files (x86)\Genetec\RabbitMQ\
    • %ProgramData%\Genetec\RabbitMQ\
      NOTE: By default, this path leads to C:\ProgramData\Genetec\RabbitMQ
    • %AppData%\RabbitMQ\
      NOTE: By default, this path leads to C:\Users\<Username>\AppData\Roaming\RabbitMQ
  4. Use Regedit to remove the following registries recursively. You will need to delete the folders and everything inside.
    NOTE: Open the Registry editor by typing Regedit in the Windows search box. Delete files from the following paths:
    • HKEY_LOCAL_MACHINE\SOFTWARE\Ericsson\
    • HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Ericsson\
  5. Delete the .erlang.cookie files from the following locations:
    • C:\.erlang.cookie
    • C:\Windows\.erlang.cookie
    • C:\Windows\System32\config\systemprofile\.erlang.cookie
    • C:\Users\<Username>\.erlang.cookie
    • C:\Windows\ServiceProfiles\LocalService\.erlang.cookie

This completes the manual uninstall of RabbitMQ.

Uninstalling Genetec Mission Control™

You can uninstall Genetec Mission Control™ from Windows Control Panel.

Procedure

  1. Go to Start > Control Panel > Add or Remove Programs.
  2. Select Genetec Mission Control™ Installer <version>.
    CAUTION:
    This will uninstall RabbitMQ also.

    If you want to uninstall only Genetec Mission Control™, select Genetec Mission Control™<version>

  3. Click Uninstall.
  4. Reboot the machine after uninstalling Genetec Mission Control™.