ServerOwl™ How to Monitor Your Server's Background Processes

Last modified: March 19, 2019

ServerOwl not only allows you to monitor your website's usage, but also allows you to monitor your server's background processes such as a nightly database backup or file synchronization.

To properly maintain a professional website, there are often many background scripts deployed to routinely clean, backup or verify, that are run periodically by the cron manager. Some of these scripts can be run on disaster recovery (DR) severs that are firewalled to prevent public access during normal operation. In this situation, many of the scripts can fail for one reason or another and nobody will notice.

A DR process that was originally working at the time of creation, but is no longer effective because of an error, undermines the very purpose for that process to be. Also having many backup servers with limited disk space that can no longer receive the backups, requires a process to constantly scan resources and record a message on its success or failure. Alternatively having a MySQL cluster where it's possible for one of its slave databases to have a synchronisation error and cause intermittent errors in the system without identifying the cause or which slave is no longer receiving updates.

These periodic scripts can use the ServerOwl's REST interface to record the pulse of from each of these scripts and you can be alerted to any change in the message or the failure of ServerOwl to receive a pulse from your background process. This enables you to be informed of any change in status of a background process without the need to constantly manually check it.

Example of Monitoring Your MySQL Slave Database

Once you have created an Account or Campaign, you can then create an attached Server. Once the Server it created, there should exist an example tab underneath the Server details entitled "Bash Script". This should include your own personal Server's details and an example code to append to your bash scripts to send a REST message to ServerOwl's API.

One of the most typical applications is to monitor the status of your MySQL Slaves databases and send a message as soon as an error is identified.:-

#!/bin/bash

### VARIABLES ###
SERVER=$(hostname)
DB_USERS="<Your Database Username>"
DB_PASS="<Your Database Password>"
MYSQL_CHECK=$(mysql -u $DB_USERS -p$DB_PASS -e "SHOW VARIABLES LIKE '%version%';" || echo 1)
MYSQL_STATUS=$(mysql -u $DB_USERS -p$DB_PASS -e "SHOW SLAVE STATUS\G")
LAST_ERRNO=$(echo "${MYSQL_STATUS}" | grep "Last_Errno" | awk '{ print $2 }')
LAST_ERROR=$(echo "${MYSQL_STATUS}" | grep "Last_Error" | awk '{ $1=""; print $0 }')
LAST_IO_ERRNO=$(echo "${MYSQL_STATUS}" | grep "Last_IO_Errno" | awk '{ print $2 }')
LAST_IO_ERROR=$(echo "${MYSQL_STATUS}" | grep "Last_IO_Error" | awk '{ $1=""; print $0 }')
LAST_SQL_ERRNO=$(echo "${MYSQL_STATUS}" | grep "Last_SQL_Errno" | awk '{ print $2 }')
LAST_SQL_ERROR=$(echo "${MYSQL_STATUS}" | grep "Last_SQL_Error" | awk '{ $1=""; print $0 }')
SECONDS_BEHIND_MASTER=$(echo "${MYSQL_STATUS}" | grep "Seconds_Behind_Master" | awk '{ print $2 }')
IO_IS_RUNNING=$(echo "${MYSQL_STATUS}" | grep "Slave_IO_Running" | awk '{ print $2 }')
SQL_IS_RUNNING=$(echo "${MYSQL_STATUS}" | grep "Slave_SQL_Running" | awk '{ print $2 }')
ERRORS=()
MESSAGE=""
TYPE="info"

### Run Some Checks ###
## Check if I can connect to Mysql ##
if [ "$MYSQL_CHECK" == 1 ]
then
    ERRORS=("${ERRORS[@]}" "Can't connect to MySQL (Check username and password)")
fi
## Check For Last Error ##
if [ "$LAST_ERRNO" != 0 ]
then
    ERRORS=("${ERRORS[@]}" "Error when processing relay log (${LAST_ERRNO})")
    ERRORS=("${ERRORS[@]}" "${LAST_ERROR}")
fi
## Check For Last IO Error ##
if [ "$LAST_IO_ERRNO" != 0 ]
then
    ERRORS=("${ERRORS[@]}" "IO Error when processing relay log (${LAST_IO_ERRNO})")
    ERRORS=("${ERRORS[@]}" "${LAST_IO_ERROR}")
fi
## Check For Last SQL Error ##
if [ "$LAST_SQL_ERRNO" != 0 ]
then
    ERRORS=("${ERRORS[@]}" "SQL Error when processing relay log (${LAST_SQL_ERRNO})")
    ERRORS=("${ERRORS[@]}" "${LAST_SQL_ERROR}")
fi
## Check if IO thread is running ##
if [ "$IO_IS_RUNNING" != "Yes" ]
then
    ERRORS=("${ERRORS[@]}" "I/O thread for reading the master's binary log is not running (Slave_IO_Running)")
fi
## Check for SQL thread ##
if [ "$SQL_IS_RUNNING" != "Yes" ]
then
    ERRORS=("${ERRORS[@]}" "SQL thread for executing events in the relay log is not running (Slave_SQL_Running)")
fi
## Check how slow the slave is ##
if [ "$SECONDS_BEHIND_MASTER" == "NULL" ]
then
    ERRORS=("${ERRORS[@]}" "The Slave is reporting 'NULL' (Seconds_Behind_Master) - likely has stopped due to error")
elif (($SECONDS_BEHIND_MASTER > 60))
then
    ERRORS=("${ERRORS[@]}" "The Slave is at least 60 seconds behind the master (Seconds_Behind_Master - ${SECONDS_BEHIND_MASTER})")
fi

if ((${#ERRORS[@]} > 0))
then
    MESSAGE+="An error has been detected on <Slave Name> ${SERVER} involving the mysql replciation. Below is a list of the reported errors:\n\n"
    for i in "${ERRORS[@]}"
    do
        MESSAGE+="\t$i\n"
    done
    MESSAGE+="\nPlease correct this ASAP\n"
    TYPE="error"
fi

MESSAGE_CODED=""
MESSAGE_LENGTH="${#MESSAGE}"
for (( i = 0; i < MESSAGE_LENGTH; i++ )); do
    C="${MESSAGE:i:1}";
    case $C in
        [a-zA-Z0-9.~_-]) MESSAGE_CODED+="$C";;
        *) MESSAGE_CODED+=$(printf "%%%02X" "'$C'");;
    esac
done

wget -O- --header="Connection: close" --header="Authorization: Basic <Your Authorization Code>" --post-data="type=${TYPE}&message=${MESSAGE_CODED}" https://www.serverowl.net/rest.php/pulse/<Your Account Slug>/<Your Campaign Slug>/<Your Property Slug> > /dev/null 2>&1

For the example above, you'll need to replace the data inside the tags with your own personal information to work correctly.

There are some more examples to be found here.

← Back to FAQs