HTTP URL Monitoring


In an environment, where the downtime of any website applications and services can cause a negative impact on the business performance, problems must be accurately identified and solved.

Applications Manager acts as a continuous URL monitoring service that keeps constant watch over a specified URL or website page. They test the website applications and web services to ensure that they are functioning properly. URL monitors verify the availability of specified, addressable, standard HTTP and HTTPS URLs. They scan the HTTP and HTTPS pages looking for a predefined keyword to check whether the website is available.

The difference between the two types of monitoring is that URL Monitoring monitors single HTTP and HTTPS URL, whereas URL sequence monitors a set of HTTP and HTTPS URLs in sequence. Also, any HTTP and HTTPS URL can be monitored using URL Monitoring.Please go through the following sections to know about the configuration details.

HTTP-URLs

To configure for URL monitoring, follow the given steps:

  1. Select New Monitor. Choose HTTP-URLs.
  2. Provide any display name for the HTTP-URL monitoring.
  3. Provide the HTTP/HTTPS URL address you want to monitor.
  4. Enter any keyword as match content. The URL monitoring searches the keyword in the content of the URL page to check the availability of the URL. This is optional.
    Note:
    • The content search is case-sensitive. If you provide 2 words, the content match is performed for the words separately. For example, if you specify the content as applications manager, the match is found for applications and manager separately. If you need the content match to be performed for the complete text, specify the 2 words in quotes, example "applications manager".
    • Space character will be acting as the delimiter.
    • Regex content check is supported only for "Should contain" field. The "Should not contain" field supports only plain text match.
  5. Provide the polling interval for which Applications Manager updates the status of the monitor.
  6. If you are adding a new monitor from an Admin Server, select a Managed Server.
  7. Choose between the POST and GET, which are the two types of Form Submission Method for any HTTP/HTTPS URL. If you choose POST, select URL Payload Type for 'POST' form submission.
  8. Click Add URL Monitor to initiate monitoring of the specified URL.

Apart from the basic URL Monitoring, Applications Manager also provides you with advanced options that furnish effective and more flexible URL Monitoring. This is optional and you need to choose these options only if the HTTP/ HTTPS URL requires Form-based authentication.

  1. Provide the request parameters, if any. The request parameters must be provided as name=value pairs for POST and GET methods. For example, if you want to monitor a URL like, http://appmanager:9090/showresource.do?haid=1&type=UrlMonitor&method=getMonitorForm, then provide "?haid=1&type=UrlMonitor&method=getMonitorForm" as request parameter.
  2. Enter the credential details like user name and password for authentication, or select the required credentials from the Credential Manager list after enabling the Select from Credential list option. (To add new credentials under Credential Manager, refer here.)
  3. Enter a keyword which when matches with the content in the URL must be notified of error in Error If Match field. Note: The search is case sensitive. Multiple keywords should be put within quotes. Otherwise, it will be treated as 'Any' one of the multiple keywords.
  4. Enter the Response Code details by choosing them from the combo box. By default, it is greater than 200. Hence the error will be notified once the criteria mentioned are met.
  5. Enter the time, in minutes, for which the URL monitor should wait for a page to complete downloading before timing out in Timeout field.
  6. Check "If monitor detects error, re-try immediately to verify error", where the monitoring is automatically performed when it detects an error, i.e. when an error is detected, the monitor will immediately be scheduled to run again once.
  7. Sometimes you may want to customize the default HTTP request header. You have the option to customize the "User-Agent" field. Check the Custom HTTP Headers and User Agent checkbox and enter the HTTP Header and User Agent in the text boxes. You may add multiple HTTP Headers and User Agents.

HTTP-URL Sequence Recorder (Transaction Recorder Tool)

The purpose of URL Sequence is to monitor multiple web pages of an online application. It checks pages with dynamically generated information enhancing interactive transactions such as logging into a login page, creating an account using a web form, instructing the application to perform some action, etc.

URL Sequencing starts with a specific URL such as the Login page and then followed by additional links/URL in it. The URL Sequence monitoring thus performs end-to-end verification of particular transactions helping you to troubleshoot any problems while monitoring.

Note:
If you are connecting to an URL using a proxy server, then you must configure proxy to initiate the URL sequence monitoring. Refer to the Configuring Proxy section of Performing Admin Activities, for further details.

Transaction Recorder Tool records all user online transactions in their exact sequence. These transactions will then be replayed at regular intervals of time and notifications will be sent when any error is detected.

Browse through the following topics to use the Transaction Recorder tool:

Note:
We have set a limit of 25 URLs per URL Sequence monitor.

Getting Started

You must download and install the recorder tool to record the sequence of user transactions. Every step of the transaction will be captured and recorded. You also can modify or edit your transactions.

Hardware/Software Requirements

The Transaction Recorder is developed using the Microsoft .NET Platform version 2.0. This is the minimum software requirement. A minimum of 2 MB of disk space is also required for installing the software.

Download and Installation

Download the new Recorder and start the installation process. The installation wizard takes you through a four step process after which the Recorder will be started:

  • A welcome screen is displayed, after which you will be asked for confirmation of the license requirements. Click Yes.
  • After confirmation, a screen to Chose Destination Location is displayed. The user is allowed to select the drive and directory where he wishes to install the Transaction Recorder. Click Browse and select the installation destination. Click Next.
  • Review the settings in the confirmation dialog and click Back if you wish to make changes. If not, click Next. The Recorder will be installed in your machine.
  • After the installation is completed, click Finish. Make sure the "Start Transaction Recorder" option is checked, if you would like to start the tool automatically after the installation is complete.

Recording a New Transaction

After installing, the Transaction Recorder can be accessed under All Programs in Windows Start Menu. You can record a new transaction following these steps:

  • Record the Sequence.
    • Click on the New button to start a new recording.
    • Enter the URL of the application you wish to monitor then press enter to load the corresponding web page.
    • Follow the sequence of steps that you wish to monitor.
  • Edit the captured URL's
    • After you have completed the sequence, click Preview & Save. A Login Credentials dialog box is displayed.
    • Provide the login credentials - Host Name, Port Number, User Name and Password. Check the Show Password checkbox to see the password
    • Click on Next to open the Preview Transactions screen. Here the user can view the recorded URLs and configure monitoring settings for each URL individually.
    • In the Preview Transactions screen, click the Filter URLlink on the top right corner to open a URL Filter Screen. URLs can be filtered based on the following three categories:
      • File Type / URL Extensions -- The captured URLs can be filtered based on their extensions. For example, .gif, png, jpg etc. for images, .js for scripts and more.
      • Content Type / MIME Type -- URLs can be filtered based on the MIME type like image, video, audio, text and application.
      • Domain Components -- You can also filter URLs based on the domains from which they originate. You have the option to include / exclude websites from a particular domain in the Domain Filter list.

        Note:
        These domains are not stored for future references. You have to re-enter such preferences if you re-start the tool.

    • You can add a new Extension, Content Type or Domain Component to the URL filter using the ADD button at the bottom of each list.
    • Delete URL: Click on a URL from the list of transactions and click the Delete URL link on the top right corner to delete it.
    • Basic Configuration: You can edit basic URL details like URL link, Display name, header and post data by selecting settings Basic option at the bottom of the list of recorded URLs.
    • Advanced Configuration:
      • Alert Configuration: You can configure the recorder such that alerts are generated when specific keywords are present/not present. Enter the keywords in the text boxes provided.
      • Basic Authentication Details: Enter Authentication details like username and password.
      • Response Code Configuration: You can configure the recorder to generate alerts if response time is greater than, equal to, not equal to, less than, less than or equal to a specified value.
    • After making all the required changes, click next to go to the save transactions screen.
  • Saving the transaction: In the save transactions screen, you can configure or edit the following fields:
    • URL sequence name: The wizard generates a sequence name. You can edit the default sequence name.
    • Poll interval: Poll interval is the time in minutes between two consecutive attempts to check for each recorded URL.
    • Host name
    • Port number
    • Username and Password. (The username and password used for Applications Manager with administrative privilege should be used).
  • Click Save to save the changes.

Modify an existing transaction

You can modify an existing transaction

  • Click Re-record.
  • Record the Sequence.
  • Edit the captured URLs
  • Select the transaction which needs to be modified from the drop-down menu and the click Fetch URLs.
  • Click next to perform a side-by-side comparison of the list of old URL's and the list of newly recorded URL's.
  • Click Update to update the existing transaction with the currently recorded sequence in the server.
Note:
  • If you would like to set User Agent for monitoring URL sequence, add the following key in AMServer.Properties file located in your Applications Manager installation folder.
    am.httpclient.useragent=<Browser User Agent which you would like to set>
Example: am.httpclient.useragent=Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; Q312461; SV1)
Save the file and restart Applications Manager for the changes to take effect.

Monitoring Dynamic URLs:

Suppose your login page has hidden parameters that needs to be submitted while logging into your application and the value for the hidden parameter is different for each login request. In this case, the hidden parameter value in the login page needs to be fetched dynamically and passed to the subsequent URL.

For monitoring these kind of dynamic URL Sequences, you can use $<Parameter name> [eg., $sessionID = 12] in the Request Parameter text box of the URL. So, while monitoring the URL Sequence, $< Parameter name> will fetch the dynamic values and pass it to the next URL in the sequence.

While using the Recorder tool, you cannot specify the parameters in the Request Parameter text box, so instead, after adding the URL Sequence, go to corresponding URL's 'Edit Monitor' page and enter the parameters. Consider the following URL Sequence :

  • URL 1 : http://api.appmanager.com/oauth/token?client_id=32faff282a68a03c2eb0f41aeca63a88&grant_type=client_credentials&client_secret=9de7337627ab85f3
  • URL 2 : http://api.appmanager.com/bestoffer/97232/354992/offers.xml?access_token=fcdee0e94da9b934a4cef6c2dee70384/li>
  • URL 3 : http://api.appmanager.com/selectoffer/97232/3549/offers.xml?access_token=fcdee0e94da9b934a4cef6c2dee70384
  • ...
  • ...
  • URL N : ...

In the above example access_token is a dynamic parameter whose value changes every time we login. In order to monitor the above sequence, follow the below mentioned steps :

  • Download the Transaction Recorder tool and record a new transaction sequence.
  • The recorder tool will have recorded all the steps along with dynamic request parameter for that session.
  • After adding the URL Sequence, click 'Edit Monitor' for the corresponding URL to enter the parameters.
  • In the Edit Monitor page, for the field 'Request Parameters (if any)' provide the dynamic request parameter as $parameter=TEXT:-:<parameter_name>:-:

    For Eg - In URL 1: The 'parameter' is 'access_token' and 'parameter_name' is 'access_token='. So, for the text box correponding to the 'Request Parameters' provide $access_token=TEXT:-:access_token=:-: and save the page

  • Click 'Edit Monitor' for the second URL, remove the dynamic request parameter from the 'URL address' field and use the same parameter to fill the 'Request Parameters (if any)' field

    For Eg - In URL 2: Edit http://api.appmanager.com/bestoffer/97232/354992/offers.xml?access_token=fcdee0e94da9b934a4cef6c2dee70384, remove the dynamic request parameter (access_token) and change the 'URL address' to http://api.appmanager.com/bestoffer/97232/354992/offers.xml?. Provide the 'Request Parameters' as $access_token=fcdee0e94da9b934a4cef6c2dee70384

  • Repeat the above steps for the rest of the URLs in the sequence where the dynamic parameter is present

So for every polling interval Applications Manager will search for this dynamically generated parameter and it will replace it.

Applications Manager acts as a continuous URL monitoring service that keeps a constant watch over the specified URL or web site pages. URL monitors verify the availability of specified, addressable, standard HTTP and HTTPS URLs. They scan the HTTP and HTTPS pages looking for a predefined keyword to check whether the web site is available.

There are two ways of URL monitoring provided by Applications Manager.

  • URL Monitoring
  • URL Sequence (Record & Playback)

In URL Sequence, click on the individual URL listed, to view its monitored parameters.

Monitored Parameters

Availability tab, gives the Availability history for the past 24 hours or 30 days.Performance tab gives the Health Status and events for the past 24 hours or 30 days.List view enables you to perform bulk admin configurations. Click on the individual monitors listed, to view the following information.

Parameter Description
Monitor Information
Health Specifies the health of the monitor based on its dependencies.
Type Type of Monitoring
URL address Specifies the URL being monitored
Match Content The string that is searched in the resulting html page.
Request Method The request method sent to the HTTP/ HTTPS URL (Get or Post)
Monitored Parameters
Availability The current status of the URL / URL Sequence- whether it is up or down.
Response Time The response time for accessing an URL or total response time of URL Sequence (takes into account the response time of the individual URLs in the Sequence)
Current Status Current status of the response time. Click on the icon to know its RCA details.
Current Page Size The current page size of the URL in bytes (only in URL monitoring)
% Change in Page Size The percentage change between the current page size and the previous page size. (only in URL monitoring)
Response Time Split Up
DNS Time The time that it takes to resolve the website’s hostname to a certain IP address. If the DNS lookup time is high, it means that you or your hosting provider has a problem with their DNS servers.
Connection Time Once the URL has been resolved to an IP address, the connection time shows how long it takes to establish a connection to your server. The connection time helps in discovering network latency, routing issues, and server bandwidth problems.
First Byte Time The first byte shows how long it takes from the moment the connection was created until the first byte is about to be transferred. The time to perform any negotiations with the server and the time needed for the server to calculate the result are also included.
Last Byte Time Once the first byte of HTML has been received, the Web server continues to send the HTML that represents the layout of the Web page, including CSS and Java Script. Last byte time includes time till the last byte for all documents with the HTML to load. Also commonly referred to as the "Download Time" or "HTML Content Time".