Script Monitors


Script Monitor - An Overview

Custom script monitoring can be a tedious task if the output of the scripts that are run, are to be monitored manually. Applications Manager provides with script monitoring functionality to ease the process by automatically monitoring the output of in-house custom scripts (Windows/Linux) and by creating alarms as per the configuration. Script monitor allows you to monitor the script that is present in the local system or in the remote system, transfers the output to an Output File, parses the output and executes the actions configured.

Applications Manager provides Script Monitoring functionality to automatically monitor the output of ad-hoc Windows/Linux/Solaris scripts that are used in-house.

  • During creation of a new script monitor, you must give the location of the custom script (local / remote), attributes (numeric/string) to be monitored, the Output File in which the output is going to be redirected and the polling interval.
  • Based on the polling interval, Applications Manager executes the script.
  • The script will transfer its output to the Output File configured. The output of the script should be in a Key=Value format where '=' can be any delimiter.
  • Applications Manager parses the Output File and executes the actions configured. It enables you to alert an Administrator or automatically take corrective actions by way executing other OS scripts.
  • Reports for the attributes configured would be shown as graphs. Option to enable or disable reports is given.

Creating a new Script Monitor

To add a Script Monitor, follow the given steps:

  1. Click New Monitor. Choose Script Monitorstrong>
  2. Enter the Display Name of the Monitor.
  3. Specify if the script to be monitored is present in the Local Server or in a Remote Server.
  4. If it is Local Server, Give the absolute path of the Script to be Monitored and also the absolute path of the directory from which the script should be executed. The execution directory should in the same hierarchy of the 'script to be monitored' directory structure.
  5. Select the script location as command or script / batch file.
  6. Specify the absolute path of the script / command to be monitored.
  7. Specify the absolute path of the execution directory from which the script should be executed.. The execution directory should in the same hierarchy of the script to be monitored directory structure.
  8. Check the Output Settings checkbox, to configure output settings
    • Check the Get output from file checkbox and specify the path of the file. If the checkbox is not checked, Applications Manager takes the output from the output stream.
    • Enter the String and Numeric attributes (one attribute per line).
    • Enter the value of Delimiter used in the output file. By default, it is "=". If you don't specify a delimiter, then 'space' would be considered as a delimiter.
  9. If you want to monitor a tabular Output file, enter the details of the tables - Name, String and Numeric column attributes, delimiter and also specify which attribute is the Unique Column.
    • Inorder to identify a tabular output file, execute the following commands before and after the actual script.
      echo <--table <table-name> starts--> 
      [Script Commands]
      echo <--table <table-name> ends--> 
      This would enable Applications Manager to identify the Output File's table.
    • Even if you're using a command instead of a script as input, then the output of the command should be in the above format. It is also mandatory to have the headers and footer in the command output as the first line in the file. 
    • If a table row is unavailable in the next poll, you can configure appropriate table row actions (Retain, Delete or Unmanage) by enabling the Manage Table Row option.
    • If no table rows are available in the output file, you can configure appropriate health severity (Critical, Warning, or Clear) by enabling the Manage Table Data option.
  10. Specify the Arguments. For e.g., hostname 80 http
  11. Set the Polling Interval. By default, it is 5 minutes
  12. Specify the Timeout value in seconds. The value can be the maximum time taken by the script to execute.
  13. In Linux, Specify the mode in which script should be executed. By default, it is "sh".
  14. If the script is in a remote server, select the Host Name from the list
  15. If the remote server is a new host, then enter the server's Host Name / IP Address. Choose the Mode of MonitoringTelnet, SSH or Powershell.
  16. Enter the User Name and Password of the server.
  17. Enter the Port number - Default Telnet port no: 23, SSH: 22
  18. Specify the command prompt value, which is the last character in your command prompt. Default value is $ and possible values are >, #, etc.
  19. Upon adding the script monitor, you can view the details of the newly added Script Monitor
Note:
  • Remote Windows script monitoring is supported only in Powershell mode of monitoring.
  • Script/Command to monitor should not contain whitespace(s) for Powershell mode of monitoring.
  • Output file is mandatory when 'Output Settings' option is enabled for Powershell mode of monitoring.

Enable Configure Alarms Template for Script Monitor

You can enable a link to Configure Alarms (link will appear on the right top corner of each table in your Script Monitor page). Using the link, the user can set Predefined Thresholds for any attributes in the table.

Here is how you can enable Configure Alarms Template for Script Monitor:

  • Go to the Admin tab.
  • Click Performance Polling under Discovery and Data Collection.
  • Under Performance Data Collection, click on the Optimize Data Collection tab.
  • Choose Script / Database Query Monitor from the drop-down menu.
  • Check the Enable Configure Alarm Template for Script Monitor and Database Query Monitor option.
  • The Configure Alarm link will appear on the right top corner of each table in your Script Monitor page.

Tips

  1. If you choose the Script Location as command:
    • Command Length must not exceed 255 characters.
    • Linux commands must not contain backslash (\)
    • Here is an example.
  2. If you choose the Command type of script:
  3. If you choose the Script Location as script:
    • Any Script in a Local Windows server will be executed as,
         cmd /C [ScriptName]  [Arguments]
      Example: cmd /C D:\testScript.bat hostname username password.
  4. To execute a VBS, Powershell or any script that is not a batch file:
    • The script should write the output to command prompt or file.
    • You can also execute files by embedding them inside a Batch script and using the Batch script for monitoring. Here are some examples.
  5. You can use them directly by selecting "Command", without using another batch file:
              C:\windows\system32\cscript.exe D:\testFile.vbs hostname username > outputfile.txt
  6. Arguments can be used in the script.
Local & Remote Non-Windows Script Monitoring:
  • Provide proper "Shell Type" for executing the scripts. By default "sh" is used.
  • You can execute script directly as "Command": sh ./test.sh

Note: The user must have permission to execute the script and read/write permission for Output file.

Example Command

Command: netstat -an | grep "CLOSE_WAIT" | wc -l | awk '{print "CLOSE_WAIT=" $1}'

Example Scripts

To Execute VBS files:

  • To execute testFile.vbs, embed it inside a batch script testScript.bat
  • Contents of testScript.bat file:

c:\windows\system32\cscript.exe D:\testFile.vbs  > outputFile.txt

  • Add testScript.bat for monitoring and use outputFile.txt to get Output.

To Execute a Powershell file:

  • To execute test.ps1, embed it inside a batch script testScript.bat
  • Content of test.bat:

PowerShell.exe -NoProfile -ExecutionPolicy Bypass -Command

  • Add test.bat file for monitoring and use outputFile.txt to get Output.
Note:
  •  PowerShell script must be executed with -NoProfile -ExecutionPolicy Bypass
  •  Need to add "-encoding ascii" for output files created via powershell.

Sample Input

To monitor a script interfacestatus.bat that creates a user defined table called InterfaceStats and user defined parameters like DiskStatusDiskErrors and No. ofProcess, in the output file interfacestatusoutput.txt

  • In the Script Monitor creation form, give the absolute path of the script : c:\interfacestatus.bat
  • Under Output settings, give the absolute path of Output file: c:\interfacestatusoutput.txt
  • Give DiskStatus as the string attribute, No.ofProcess and DiskErrors are the numeric attributes with Delimiter "="
  • If you want to monitor statistics in a table format , select Tables in output file.
    • Here, we have the table InterfaceStats with the stats Name, IP, Status, In, Out where Name, IP, Status are string attributes; In & Out are numeric attributes. TheDelimiter is the separater between the two column names -> space. If tab is the delimiter, then give \t . Usually tab will be the delimiter for sql queries results.
  • Note: The starting tag of the table, InterfaceStats is "<--table InterfaceStats starts-->" and the end tag is "<--table InterfaceStats ends-->". Also, the first line of the table should contain the attribute names. The attribute names or the column names should also be separated by the same delimiter used to separate the data rows and that is specified as the column delimiter. In this case, they are Name, IP, Status, In and out. The remaining lines between the start and end tag should comprise of the actual data. Make sure that the delimiter for the table is unique and you should specify that as the column delimiter.
  • Unique Column is the attribute that doesn't repeat itself in the rows and identify the row by that value. Here it is IP.

Sample Output

DiskStatus=Up
No.ofProcess=1
DiskErrors=0

<--table InterfaceStats starts--> 
Name IP In Out
eth0 192.168.1.1 123451 234567
eth1 192.168.1.2 345633 453267
<--table InterfaceStats ends-->

Monitored Parameters

Go to the Monitors Category View by clicking the Monitors tab. Click on Script Monitor under the Custom Monitors Table. Displayed is the Script Monitor bulk configuration view distributed into three tabs:

  • Availability tab, gives the Availability history for the past 24 hours or 30 days.
  • Performance tab gives the Health Status and events for the past 24 hours or 30 days.
  • List view enables you to perform bulk admin configurations.

Overriding Availability and Response Time of the Script

This option would be useful, if you want to override the response time measured for custom scripts using a defined value in the output file. Applications Manager looks for few reserved keywords in the output file, if it matches then it will replace the corresponding attribute.

For eg., if the script output looks like this,

script_availability=1 (Allowed values are 0 or 1. "0" refers to success. "1" refers to failure.)
script_message=Server is not running.
script_responsetime=10

Then, while parsing the output file, script_availability value will be taken and based on that the availability is calculated.The same is the case for response time.

Examples:

Sample Scalar Usecase

Let's assume you want to monitor a script < filesystem.sh> present under /home/test-solaris/ in test-solaris machine. The output from this script is dumped to output.txtfile present under the same directory. The format of output.txt is as shown below :

SystemTemperature=37 
TimeSinceLastReboot=30 
TopPaginApp=sendmail
IOReads=1050

Setting up Script Monitor:

  • Login to the Applications Manager Web Client.
  • Click New Monitor. From the combo box, choose Script Monitor.
  • For the Display Name give some name. For e.g FileSystem
  • Choose whether the script to be monitored is present in the Local Server or in a Remote Server. If the script is in a Remote Server, then make sure you put the script in the Remote Server.
  • If it is Local Server/ Remote Server, give the absolute path of the Script to Monitor (/home/test-solaris/filesystem.sh) and also the absolute path of the directory from which the script should be executed(/home/test-solaris/).
  • Under Output Settings, give the Output file name (/home/test-solaris/output.txt) with absolute path. This is the file where the output from the script is dumped.
  • Enter the Name of the Non Numeric and Numeric attributes.

In the Numeric area, add

SystemTemperature
TimeSinceLastReboot
IOReads

In the String Attributes Text area, add :

TopPaginApp

  • Enter the value of Delimiter (=) used in the output file. By default, it is "=". If you don't specify a delimiter, then 'space' would be considered as a delimiter.
  • Specify the additional Arguments (if required to pass to the script). For e.g., hostname 80 http
  • Set the Polling Interval. By default, it is 5 minutes
  • Specify the Timeout value in seconds. The value can be the maximum time taken by the script to execute.
  • In Linux, Specify the mode in which script should be executed. By default, it is "sh".
  • If the script is in a Remote Server, select the Host Name from the list
  • If the remote server is a new host choose New Host, then enter the server's Host Name / IP Address (test-solaris). Choose the mode of monitoring - Telnet, SSH, or Powershell. (Telnet)
  • Enter the User Name(test) and Password(test) of the server.
  • Enter the Port number - Default Telnet port no: 23, SSH: 22
  • Specify the command prompt value, which is the last character in your command prompt. Default value is $ and possible values are >, #, etc.
  • Once all the values are entered, click Add Monitor(s).

The success message should be displayed. Click Monitor Details > Script Monitor and go to the create script, to view the details.

Note: Applications Manager uses either ANSI or UTF-8 encoding. Hence it is advised that you use the same format while writing output to the file.

Sample Table Usecase

Let's assume you want to monitor a script <prustat.sh> present under /home/test-solaris/ in test-solaris machine. The output from this script is dumped to output.txt file present under the same directory. The format of output.txt is as shown below

<--table prustat starts-->
PID CPU Mem Disk Net COMM
7176 0.88 0.70 0.00 0.00 dtrace
7141 0.00 0.43 0.00 0.00 sshd
7144 0.11 0.24 0.00 0.00 sshd
3 0.34 0.00 0.00 0.00 fsflush
7153 0.03 0.19 0.00 0.00 bash
99 0.00 0.22 0.00 0.00 nscd
7146 0.00 0.19 0.00 0.00 bash
52 0.00 0.17 0.00 0.00 vxconfigd
7175 0.07 0.09 0.00 0.00 sh
98 0.00 0.16 0.00 0.00 kcfd
<--table prustat ends-->

Note: the table headers <--table prustat starts-->. This is mandatory and should follow the same format as mentioned. Here "prustat" should be replaced by the Table Name explained below

Setting up Script Monitor

  • Follow the same instructions as mentioned for Scalar till point 6
  • Select Tables in output file check box
  • For the Table Name provide some name(prustat). Note that this same should be present in the table header(<--table prustat starts-->) in the output file.
  • For the Numeric Attributes area provide the column names in the script output that are numeric: CPU, Mem, Disk, Net
  • For the String Attributes provide the column names in the script output that are non numeric: PID, COMM
  • For the Unique Column provide the column names that can identify a row data. This can be a single value or multiple value: PID, COMM
  • For the Column Delimiter provide the column separator. The default value is a space.
  • In case you have scripts that output multiple tables then you can select More and configure the values.
  • Once all the values are entered, click Add Monitor(s).
You can use script monitor to monitor the SNMP OIDs

Please look at the steps below for creating a script monitor,

  • Create a script file ( say script.sh ) under the /opt/ManageEngine/AppManager_Home/ directory
  • Edit that file and type in the following content into that file, 
    snmpwalk -v 1 -c public app-w2k1 CPQHLTH-MIB::cpqHeFltTolPowerSupplyRedundant.0.1| awk '{ y = $1; x = $4 ; gsub(/[a-zA-Z()]/, "", x)} {print y " = " x}'>> output.txt 
    snmpwalk -v 1 -c public app-w2k1 CPQHLTH-MIB::cpqHeFltTolPowerSupplyRedundant.0.2 | awk '{ y = $1; x = $4 ; gsub(/[a-zA-Z()]/, "", x)} {print y " = " x}' >> output.txt
  • Click on "New Monitor" in applications manager and choose script monitor. Then create a new monitor using the following parameters,
    Script to Monitor   </opt/ManageEngine/AppManager_Home/script.sh>
    Directory from which the script should be executed   </opt/ManageEngine/AppManager_Home/>
    Under Output Settings, Output File   </opt/ManageEngine/AppManager_Home/output.txt>
    Numeric Attributes   <CPQHLTH-MIB::cpqHeFltTolPowerSupplyRedundant.0.1>
    <CPQHLTH-MIB::cpqHeFltTolPowerSupplyRedundant.0.2>
  • Use the default parameters for configuring the remaining attributes.

Note: If an attribute does not have any data, the script returns 'N.D', where 'N.D' means No Data.

REST API to get the HostName

This API gives host name for the given Script Monitor.

Syntax

http://[Host]:[Port]/AppManager/json/GetRemoteHost?apikey=[APIKEY]&resourceid=[RESOURCEID]

Request Parameters

The parameters involved in executing this API request are:

Field Description
apikey The key generated using the Generate API Key option in the 'Admin' tab.
resourceid The resource id of the monitor that needs to be polled.

Sample Request

http://app-windows:9090/AppManager/json/GetRemoteHost?apikey=095cb3835ff015b01a3b3a6c4ab2c38a&resourceid=10000293

Sample Response

{"response":{"result":[{"hostname":"app-w7-64-8"}],"uri":"/AppManager/json/GetRemoteHost"},"response-code":"4000"}

Custom Monitor Types

Custom Monitor Types in Applications Manager allow you to associate a monitor type to the inhouse scripts, for monitoring your own applications. You can define your own monitor types apart from the monitor types that are available by default. Know more.