AMS

Introduction

The alert manager plugs into the AMS the server and allows alerting via email, SMS, and paging. Currently alerts are configured via a config file on the server, though a GUI is planned. An unlimited number of operators and alerts can be configured. And operators can be configured to receive one or more of each alert type. Alerts are configured using a combination of one or more of;

• Regular expression match
• Severity code - greater than or equal to (as per the GUI)
• One Facility code

The alert manager calls on external program to actually send alerts. Using sendmail for email, whilst for SMS and paging alerts it calls sendpage. Sendpage is freely available, and can be seen at http://sendpage.org/

Overview

The Alert manager (alertmgr) is responsible for detecting alert conditions and send the appriopiate message via either pager systems, SMS (GSM text messaging) or email. For pager and SMS it calls on sendpage, and for email sendmail.

This document uses the following terms extensively;

Operator

is notionally an individual to which alerts may be sent.

Alert

is defined set of criteria that cause a message to be sent.

Transport

is a method by which a message will be sent. Currently email, pager or sms

Transport Address

The address on a particular transport that should be used for an operator. For email this would be an email address, for a pager the pager number. If the underlying transport supports them aliases may be used.

When an alert condition is detected alertmgr sends a message which includes an identifying string (configurable) as well as the syslog record that raised the alert. The whole message is sent to one or more operators, by one or more transport. Thus each operator is able to receive a given alert by email, sms or pager and this is configurable on a per alert basis. Should the need arise a given alert may configured such that it is sent to the same operator by multiple transports.

The alert manager operates as a client of the main AMS server process and is normally started by the server when it, itself starts.

Configuration

The alertmgr is configured via a text file. By default this is located in /usr/local/ams/etc/alertmgr.cfg. There are three distinct sections to the configuration file and they must appear in this order;

1. Server
   currently this only defines the host and port the main ams_server process is running on.
2. Operators
   This section defines the transport details of all operators which may appear in the alert definition section.
3. Alerts
   Here each alert is defined using one of more of the facility code, severity or a regular expression match. The operators and transports to which the alerts should be sent also is defined here.

Operators

An operator is defined by a single line within the configuration file using a unique identifier ( say a name ), and one or more transports along with the transport address of that alert. All operators must be defined BEFORE the alert definitions.

Each operator can recieve one more alert types. Which alert types are sent is defined in the alerts section.

It is important to note that alertmgr simply passes the transport address to the appriopiate program and performs NO VALIDATION OF IT. This allows the use of aliases within sendmail and sendpage.

Normally for an email alert the address would be an email address (passed to sendmail), for a pager alert it would be the pager number (passed to sendpage).

Consider the following example operator definition for 'jon';
operator jon pager:312456 email:jon@hotmail.com

Alerts

An alert definition defines the criteria that raises a particular alert as well as the operators that will receive them. An alert is introduced by the keyword 'alert' followed by an identifying string which must be unique, e.g;
     alert Memory Error

The identifying string is sent as the first part of the alert message sent to the operator. In an email it forms the Subject line, in a pager message it is prepended to the main text. We recommend that the identifying string is short (less than 20 characters) owing to the limitations of many paging systems (SMS is limited to 160 characters).

Upto three lines define the criteria;

severity [value]

the minimum severity code that may trigger this alert (numeric 0-7)

facility [code]

the numeric facility code that may trigger this alert

regexp [expression]

A standard regular expression that will be matched against syslog record

If multiple criteria are defined then they must ALL hold true for an alert to be raised.

Troubleshooting

Upon startup the alertmgr redirects both STDOUT and STDERR to the log file /tmp/ams_alertmgr.log. This file is your first port of call when investigating problems with the alertmgr.

The alertmgr logs every alert event along with the transport and addresses the alert was sent to.

The current configuration of a running alertmgr can be dumped to the log file by sending it a SIGUSR1

Example configuration

# define where main AMS server is running ( hostname & port )
server localhost 4000
#
#
# define operators, message types, and addresses/pager numbers.

# *** OPERATOR DEFINITIONS ***

# andrew has a GSM phone(sms) and a local email address
operator andrew sms:6662781234 email:andrew

# Jon has a conventional pager and a remote email address
operator jon  pager:9875372  email:jon@rivan.demon.co.uk

# Fred never reads his email but is always sending sms...
operator fred sms:fred_sms

# Tim carries a pager and a phone around with him, pager gets
# more important alerts..
operator tim email:tim.r@bgep.co.uk sms:441234987314 pager:5284148
#

# *** ALERT DEFINITIONS ***

# we want memory errors regardless of severity/facility
alert Memory Error
regexp memory(\s+)error
sendto andrew:sms tim:pager andrew:email

# jon wants to be emailed any authentification warnings/errors
# Severity 4 catches WARNINGS and above
alert login
facility 4
severity 4
sendto jon:email

# Fred is interested in any kernel messages referring to
# solaris scsi disks...
alert solaris_scsi
facility 0
regexp sd
sendto fred:sms

# tim gets any CRITICAL (or above) messages emailed to him
alert critical
severity 2
sendto tim:email