[ER] Stuck at "Please wait while the system is starting up"

Issue

When attempting to access the web console the start-up screen pictured below appears. This screen does not disappear despite waiting several minutes.

 

 

Reason

The most common reason for this behaviour is a corrupt database on the Master Server. Database corruption is rare, but can occur when:

  1. The Master Server was shut down while it was still busy and utilizing important data.
  2. The Master Server has run out of disk space resulting in file corruption.

 

Resolution

Before completing any of the below steps, it is important to determine which of the above reasons caused the original failure. Please check the available disk space; if it is low then please follow the instructions to increase your disk space. As a rule of thumb we would generally advise doubling the current amount of space, so if your Master Server has 20GB of space then it should be resized to 40GB (and so on).

It is important to note that recovery of the Master Server may not be possible if the corruption is severe - in which case the only other alternative is to rebuild your entire Master Server.

Follow the instructions below to attempt the recovery procedure:

  1. Login to your ER Master Server Terminal console as 'root'
  2. Stop the ER service using the command; /etc/init.d/er2-master stop
    This step may take awhile, we recommend waiting for 5 minutes before going to the next step
  3. Run the command; er2-recovery -i /var/lib/er2/db/root.kct -r -w /var/lib/er2/db/recovered.kct



  4. If it looks like all has gone well, run the following commands in sequence;
    rm /var/lib/er2/db/root.kct
    rm /var/lib/er2/db/root.kct.wal
    mv /var/lib/er2/db/recovered.kct /var/lib/er2/db/root.kct
  5. Start the ER service again using the command; /etc/init.d/er2-master start



  6. Open up the ER web console again and allow for 5-15 minutes for your Master Server to start up (a refresh may be needed)

Important notes

To prevent such problems from happening again, it is crucial to make sure no scans are running before you attempt to shutdown or make any major changes to your Master Server.

Also make sure to initiate a graceful shutdown using the commands;
/etc/init.d/er2-master stop
shutdown -h now

We also highly recommend backing up your Master Server by taking a VM snapshot or using the integrated ER backup tool which you may learn more about in this article.

 

All information in this article is accurate and true as of the last edited date.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.