Troubleshooting the Archiver - Security Center 5.7 - 5.11

Applies to
Security Center 5.7 - 5.11
Last updated
2023-03-06
Content type
Troubleshooting
Language
English
Product
Security Center
Version
5.11
5.10
5.9
5.8

Troubleshooting the Archiver

If you are experiencing problems with your Archiver, learn about the symptoms, potential causes, and solutions to help you troubleshoot the issue.

Archiver role is offline

If the Archiver role is offline (icon is red), check whether the server that is hosting the Archiver role is online and operational. To help you troubleshoot this issue, learn about its possible causes and their respective solutions.

Host server is offline

Description of cause: The server hosting the Archiver role is offline (server icon is red in Config Tool). To check the status of the server, open Config Tool, and then click System > Roles > [Your role] > Resources.

Solution: In Windows Services, ensure that the Genetec Server service is running on that server. If the service is running, make sure that the expansion server has connected successfully to the Directory. If the server fails to start, troubleshoot the server issue.

Incorrect NIC priority

Description of cause: The priority for the network interface cards (NIC) is incorrect. The order in which the NICs appear might be incorrect in either the network settings in your Windows Control Panel or in the Network View task in Config Tool.

To check the NIC priority, open Config Tool, and then click Network view > [Select your server from the Networks list] > Properties.

Solution: In both your Windows Control Panel and Config Tool, make sure that NIC used for the Genetec Service appears at the top of the list of network interface cards.

Archiver ports are closed

Description of cause: The ports used by the Archiver role might be either closed or in use by other applications.

You can use Telnet Client or a Powershell command to confirm whether communication between servers is open:
  • Using Telnet Client, open the command prompt as administrator and enter the following command: telnet <IP address> <port number>. For example, if your IP address is 172.10.10.10 and you want to check port 4502, enter: telnet 172.10.10.10 4502. To enable Telnet Client, in Windows, click Control Panel > Programs and Features > Turn Windows features on or off, and then select Telnet Client from the list of features.
  • Using Powershell, open the command prompt as administrator and enter the following command: tnc -computer name (IP or DNS name of the server) -port (port number)

Solution: Make sure that the Archiver role's ports are open and not in use by other applications. For a list of required ports used in Security Center, click here.

Antivirus or firewalls are blocking communication

Description of cause: Your antivirus or Windows firewall is obstructing communication between servers.

Solution: Check that your exceptions or exclusions are properly configured for your antivirus and firewall:

System process or application is affecting Archiver role

Description of cause: A Windows process or application is negatively impacting the Archiver role. If this is the case, there might be Archiver-related errors in the Windows event logs or dump files that have been created.

To check the Windows logs, open the command prompt as administrator and enter the following command: eventvwr.msc. Make sure to check application, system, and Genetec event logs. To check whether dump files have been created, open C:\Program Data\Genetec Security Center [5.x], and then open the Dumps or Logs folders.

Solution: Contact technical support and provide the event logs and dump files.

Host server has low disk space

Description of cause: The server hosting the Archiver role is running out of disk space on its C drive. Low disk space can have negative effects on your server's performance and cause processes to falter or fail.

Solution: Free up disk space on the server's C drive and restart the Genetec Server Service. Make sure that the archive database, backups, and logs are not on C drive.

Tip: Use an external tool to analyze the used disk space and to discover new locations where the space is being consumed.

High CPU or memory usage on host server

Description of cause: The CPU or memory of the server hosting the Archiver role is overloaded. Using the Task Manager, check whether the machine is running within the Security Center specifications. To check CPU and memory usage, open the command prompt as administrator and enter the following command: taskmgr.

Solution: Do the following:
  1. Make sure the server is set up according to the Security Center specifications.
  2. Investigate processes and applications that use the most CPU or consume excessive amounts of physical memory (RAM). Consult the vendors of the respective applications or softwares, if required.
  3. Close all unwanted applications that might affect CPU and RAM usage.

Archiver.gconfig file corrupted

Description of cause: If the Archiver role is offline but the Genetec™ Server service is running on the host server, the Archiver's .gconfg file might be corrupted or missing.

Locate the Archiver.gconfig file in the Configuration files folder of your installation and check that they are readable. If the Archiver.gconfig file does not exist, generate it as follows:
  1. Log on to Server Admin and select the server hosting the Archiver role.
  2. Click Actions > Console > Commands > Archiver Agent commands > GenerateConfigFile.
Solution: If the file is corrupted, do the following:
  1. Move the corrupted file to another folder and keep it as a backup.
  2. Create an Archiver role on the same server and use the same database. Make sure that the new role has the same retention period as the old role.
  3. If the new Archiver role connects, move the video units from the old Archiver role to the new one, using the Move Unit tool in Config Tool.
  4. After the video units have been moved, deactivate the old Archiver role and wait until its retention period is done before deleting it.

Archiver database is not connected

If the Archiver database is not connected, check whether you have sufficient access rights. To help you troubleshoot this issue, learn about its possible causes and their respective solutions.

Missing permissions

Description of cause: The Genetec Service logon user does not have the necessary rights on the Archiver database in SQL.

Solution: To check this, open SQL Management Studio and do the following:
  1. Click Security > Logins > [Right-click the user] > Properties.
  2. Click Server roles and check that all server-level roles for the Archiver are selected.
    SQL Management Studio showing the server-level user permissions for the Archiver database.
  3. Click User mapping, select Archiver, and check that all database-level roles for the Archiver are selected.
    SQL Management Studio showing the database-level user permissions for the Archiver database.
For more information on the required permissions, see the basic administrative privileges.

Archiver database is full

Description of cause: The Archiver database has reached its size limit. In SQL Server 2008 Express and later, the size limit is 10 GB.

Solution: Upgrade SQL Server Express to the Standard or Enterprise editions. As a workaround, you can do the following:
  • Create a new database
  • Reduce the retention period of the recordings
  • Contact tech support to help you determine why the database has reached its storage limit
NOTE: Databases can suddenly fill up due to cameras sending too many events. You can have the system monitor incoming camera events to help detect when an unusually high number of events are being received. For more information, see Archiver: Resources tab.

Database transaction log is too big

Description of cause: The transaction log file (LDF) of the SQL database is too big. The LDF should not be bigger than the main data file (MDF).

Solution: In SQL Management Studio, do the following:
  1. Switch the recovery model of the database to Simple.
  2. Shrink the transaction log file.
  3. Set a maximum size limit for the transaction log file.

Remote database ports are not open

Description of cause: The remote database ports, TCP 1433 and UDP 1434, are not open or they are being used by other applications and causing conflicts.

Solution: Open the ports and make sure that they are reserved for the SQL Server.

Cannot add network drives for Archiver storage

If you cannot add network drives for Archiver storage, ensure that the user running the Genetec™ Server service has the required permissions and read/write access to the storage disk.

The user running the Genetec Server service lacks the required permissions

Description of cause: The user running the Genetec Server service does not have permission to access the network share or connect to the server.

Error message window that describes a network error due to lack of permission. The user cannot access video archives.

Solution: Make sure the user running the Genetec Server service has the required permissions.

To grant access to a specific folder:
  1. Right-click the folder that you want to share.
  2. Select Properties > Sharing > Advanced Sharing > Share this folder > Permissions.
  3. In the Group or user names section, select the user or group you want to share to. For example, MACHINE_NAME_ARCHIVER\admin.
  4. In the Permissions for section, select Full Control > Allow.
  5. Click OK.
For more information about user permissions for Security Center services, see Best practices for setting Windows user permissions for Security Center services.

The user running the Genetec Server service does not have read/write access

Description of cause: The user running the Genetec Server service does not have read/write access to the storage disk.

To check the read/write access on the storage disk in Windows, right-click the recording drive, click Properties > Security.

Solution: For both the storage disk and its subfolders, grant read/write access to the user running the Genetec Server service.

To assign full access privileges to the user running the Genetec Server service, click Properties > Security > Advanced, or click Properties > Security > Edit.

Watch this video to learn more. Click the Captions icon (CC) to turn on video captions in one of the available languages.

Archiver role cannot write to disk

If the Archiver role has stopped recording video and the logs indicate that the role cannot record to any drive, there might be an issue with the disk or the role configuration. To help you troubleshoot this issue, learn about its possible causes and their respective solutions.

Storage drives not accessible

Description of cause: The storage drives that are selected in the Resources tab of the Archiver role are inaccessible (icons are red) and Security Center cannot communicate with the drives.

Solution:
  • If the drives are local to the Archiver server, do the following:
    1. Log on to the server using the same credentials that the Genetec™ Server service uses (verified by running services.msc from the Windows Start menu).
    2. In Windows Explorer, find the folder where the archived footage is stored.

      Enable Show hidden files, folders, and drives in case the folder is hidden.

    3. Try creating a simple text file.

      If you cannot create the text file, you do not have the required write privileges. To get the required privileges, contact your administrator.

  • If the drives are hosted elsewhere on the network, do the following:
    1. Log on to the affected Archiver server using the same credentials that the Genetec Server service uses (verified by running services.msc from the Windows Start menu).
    2. Open Windows Explorer and try accessing the remote drives.
      • If you cannot access one ore more of the remote drives, either your access is denied because you are not in the list of permitted users for those drives, or there is a communication failure between the Archiver and the remote drives. This failure is most likely caused by a network issue. Try investigating and repairing the issue.
      • If you can access the remote drives, do the following:
        1. In Windows Explorer, find the folder where the archived footage is stored.

          Enable Show hidden files, folders, and drives in case the folder is hidden.

        2. Try creating a simple text file.

          If you cannot create the text file, you do not have the required write privileges. To get the required privileges, contact your administrator.

Recording disk full

Description of cause: All disks allotted for archiving are full and the Archiver is unable to free disk space by deleting existing video files. This event can occur when another application has used up all the disk space reserved for Security Center, or when the Delete oldest files when disks full option is not selected in the Server Admin. When this happens, archiving is stopped. The Archiver re-evaluates the disk space every 30 seconds.

You can verify the statistics of the drives by checking the free space compared to the minimum free space in the Resources tab of the Archiver role. You can also check for disk full or disk 80% events in the Archiver logs (C:\ArchiverLogs by default).

Solution: Do the following:
  • Ensure that the Archiver role is set to delete oldest files first when the disk is full.
  • In the Camera default settings tab of the Archiver role, reduce the Automatic Cleanup threshold so that older archives are deleted automatically.
  • Lower the retention period of the recordings.
  • In the Resources tab of the Archiver role, reduce the Min. free space space value. Make sure that the minimum free space value is at least 0.2% of the value of the total size.
  • Increase your storage capacity by adding more storage drives to the server.

Recording disk full due to orphan files

Description of cause: If the drives are full in Windows Explorer, there might be orphan files present. Orphan files are video files that exist on the drive but are not detected by Security Center. As a result, orphan files cannot be deleted when either of the following options are enabled:
  • Automatic cleanup (Camera default settings tab of Archiver role)
  • Delete oldest files when disks are full (Resources tab of Archiver role).

Solution: Check whether orphan files are present. If you find orphan files, delete them to free up disk space or re-integrate them into the database. For more information, see Finding orphan files on your system.

Missing read/write access

Description of cause: The user running the Genetec Server service does not have read/write access to the storage disk.

To check the read/write access on the storage disk in Windows, right-click the recording drive, click Properties > Security.

Solution: For both the storage disk and its subfolders, grant read/write access to the user running the Genetec Server service.

To assign full access privileges to the user, click Properties > Security > Advanced, or click Properties > Security > Edit.

Hardware or software issues

Description of cause: The disk on the server hosting the Archiver role is experiencing hardware or software issues.

Solution: Do the following:
  • In Windows, check if there are any disk errors in the Device Manager.
  • In Windows Event Viewer, check if there are any errors or unusual messages in the event logs.
  • Run a diagnostic check, using your storage's dedicated diagnostic tool.

Archiver performance is slow

If your Archiver performance is slow, check that your server specifications meet the Security Center requirements. To help you troubleshoot this issue, learn about its possible causes and their respective solutions.

Insufficient server specifications

Description of cause: The server specifications do not meet the Security Center requirements, thus causing performance issues on the server.

Solution: Make sure your Archiver server specifications meet the Security Center requirements.

Server is running low on memory

Description of cause: The SQL Server service is using most of the server's physical memory (RAM). By default, the SQL Server service is set to use all of the RAM available on the server. As a result, not much memory is left for the OS and other applications that are running on the server.

Solution: Limit the amount of RAM that the SQL Server service can use. For information on how to make changes to the server memory configuration, go to the Microsoft website.

Hard disk drives are slow

Description of cause: The hard disk drives of the server that the Archiver role is running on and recording to are slow.

Solution: In Windows, go to Resource Monitor > Disk to view the write speed of the server's hard disk drive. If required, contact your storage provider to upgrade your hard disk.
Tip: To see the response times for healthy drives, see PerfGuide: Analyzing Poor Disk Response Times.

Outdated drivers

Description of cause: The network card (NIC) drivers and hard disk drivers are outdated. To check the NIC drivers, open Device Manager in Windows, and then click Network Adapters > Properties. To check the hard disk drivers, open Device Manager in Windows, and then click Disk drives > Properties.

Solution: Update your network card drivers and hard disk drivers to their latest versions.

Archiver is handling motion detection

Description of cause: The Archiver is handling motion detection on manyof cameras.

Solution: Switch the motion detection settings your video units to Unit instead Archiver. If this is not doable, you can reduce the number of video units that detect motion and set them to record continuously instead.

BIOS configuration is not optimal

Description of cause: The BIOS settings are not set to Performance. Adjusting your server's BIOS settings can help increase the server's overall performance.

Solution: Set the server BIOS settings to Performance.

Archiver role is triggering MSMQ warnings

Microsoft Message Queuing (MSMQ) warnings display in Security Center to indicate problems with archiving camera events and video metadata into the Archiver database. Learn more about MSMQ, how it is used in Security Center, possible causes for the warnings, and their solutions.

Symptoms

  • You receive one or many of the following warnings:
    • Microsoft Message Queuing (MSMQ) is not running on the server___. Start the MSMQ service and restart the Archiver.
    • Microsoft Message Queuing (MSMQ) is delayed on the server___. Last message took ___ to arrive.
    • Archiver ___ on server ___ is unable to store messages in the Microsoft Message Queuing (MSMQ).
  • The Archiver role displays in a yellow state.

About MSMQ

Microsoft Message Queuing (MSMQ) is a feature in Windows that coordinates the transfer of messages between applications. The queue acts as an intermediary, which provides a common place for applications to send and receive messages from one another. It also temporarily stores messages until they are received.

Security Center uses this queue when archiving video. The recordings are stored immediately to the disk. However, the video metadata and the camera events linked to the recordings are sent in messages to the queue first before they are inserted into the Archiver database. This allows the Archiver to record continuously even when connection to the database is lost. The messages can stay in the queue until connection to the database is reestablished.
NOTE: MSMQ is not used for archiving from an Auxiliary Archiver.

MSMQ warnings

Microsoft Message Queuing (MSMQ) warnings might appear in Security Center for several reasons. Here are the possible causes for the warnings and how to proceed.

Microsoft Message Queuing (MSMQ) is not running on the server___. Start the MSMQ service and restart the Archiver.

Description of cause: MSMQ is not enabled on your system. The installer should have enabled it, but somehow did not.

Solution: Enable MSMQ manually:
  1. In Windows, click Start > Control Panel > Programs and Features.
  2. Click Turn Windows features on or off.
  3. Select Microsoft Message Queue (MSMQ) Server and click OK.

Microsoft Message Queuing (MSMQ) is delayed on the server___. Last message took ___ to arrive.

Description of cause: Messages have accumulated in the queue. The causes might include a slow database, slow disk response time, or other issues.

Solution: Contact the Genetec Technical Assistance Center (GTAC) for assistance. To help narrow down the problem, do the following and have the information readily available when you call in:

Archiver ___ on server ___ is unable to store messages in the Microsoft Message Queuing (MSMQ).

Description of cause: The ArchiverAgent could not send messages to the queue because the queue is full. When this happens, archiving stops. This could be due to the following:
  • The Archiver database has reached its size limit. In SQL Server 2008 Express and later, the size limit is 10 GB.
  • The queue fills up with messages faster than the database server can process them.
NOTE: If you receive this warning, the second warning likely displays too because messages are stuck in the queue.
NOTE: Databases can suddenly fill up due to cameras sending too many events. You can have the system monitor incoming camera events to help detect when an unusually high number of events are being received and be notified. For more information, see "Receiving notifications when databases are almost full" in the Security Center Administrator Guide.
Solution: Contact the Genetec Technical Assistance Center (GTAC) for assistance. To help narrow down the problem, do the following and have the information readily available when you call in:

Viewing database information

You can view the information about a role's database, such as the database server and database versions, how much disk space is available, and a summary of the data it holds.

What you should know

The database information provided varies depending on the role. You might be asked to provide information on a role’s database when you contact Genetec™ Technical Assistance Center.

All roles' database information are viewed from Config Tool, except for the Directory database, which must be viewed from the Server Admin - Main Server page. The procedures are similar, so only viewing from Config Tool is described here.

Procedure

  1. From the Config Tool homepage, open the System task and click the Roles view.
  2. Select a role, and click the Resources tab.
  3. Click Database info ().

    The following information can be displayed, depending on the role:

    Database server version
    Software version of the database server.
    Database version
    Schema version of the role’s database.
    Approximate number of events
    (Also called Approximate number of archived events and Event count) Number of events that are stored in the role’s database.
    Source count (Archiver and Auxiliary Archiver only)
    Number of video sources (cameras) that have archives.
    Video file count (Archiver and Auxiliary Archiver only)
    Number of video files.
    Size on disk
    Size of the Database files.
    Approximate number of entities (Directory only)
    Number of entities (areas, cameras, doors, schedules, and so on) in the system.
    Approximate number of active alarms (Directory only)
    Number of active alarms (not yet acknowledged) in the system.
    Approximate number of archived alarms (Directory only)
    Number of past alarms available for reporting, excluding the active ones.

Viewing your disk response time for MSMQ issues

To help determine if Microsoft Message Queuing (MSMQ) issues are caused by your disk performance, you can view the response time of the process that communicates with the MSMQ file.

Procedure

  1. In Windows, go to Resource Monitor > Disk.
  2. Expand the Disk Activity section and verify the response time for the mqsvc.exe process.