Monitoring and Configuration

Monitoring

You can monitor the current status of each model in the model manager. The card for a model shows the current mode for this model (such as production mode) and whether it is in the Active (deployed) or Inactive state.

If a model failed to deploy or failed while running, is shown on the card for the model. To find out whether a model has failed while processing data, reload all models in the model manager to show their latest states. See also Reloading all models.

• Monitoring periodic status

• Monitoring dropped events

• Monitoring the model life-cycle

Monitoring periodic status

In addition to the status that is shown on the card for a model, it is possible to enable generation of periodic status published as Cloud of Things operations or events. See Configuration on setting the status_device_name and status_period_secs tenant options.

Each operation has the following parameters:

Model status

The following information is published for each deployed model that is currently running or has failed:

This information about each model provides insight into the performance or working of models. For example, a model with a much larger number of numBlockEvaluations than another model might indicate that it is consuming most resources even though it might have low numModelEvaluations. Similarly, it can be used to find out whether a model is producing output at the expected rate relative to the number of times it is evaluated.

The following is an example of the status operation data that is published by Cloud of Things:

{
    "creationTime": "2018-07-23T21:48:54.620+02:00",
    "deviceId": "6518",
    "deviceName": "apama_status",
    "id": "8579",
    "self": "https://yourtenant.ram.m2m.telekom.com/devicecontrol/operations/8579",
    "status": "PENDING",
    "models_running": {
        "Package Tracking": {
            "mode": "SIMULATION",
            "modeProperties":{"startTime":1533160604, "endTime":1533160614},
            "numModelEvaluations": 68,
            "numBlockEvaluations": 967,
            "avgBlockEvaluations": 14.2,
            "numOutputGenerated": 50
        }
    },
    "models_failed": {
        "Build Pipeline ": {
            "mode": "PRODUCTION",
            "numModelEvaluations": 214,
            "numBlockEvaluations": 671,
            "avgBlockEvaluations": 3.13,
            "numOutputGenerated": 4
        }
    },
    "apama_status": {
        "user-analytics-oldEventsDropped": "1",
        "numJavaApplications": "1",
        "numMonitors": "27",
        "user-httpServer.eventsTowardsHost": "1646",
        "numFastTracked": "183",
        "user-httpServer.authenticationFailures": "4",
        "numContexts": "5",
        "slowestReceiverQueueSize": "0",
        "numQueuedFastTrack": "0",
        "mostBackedUpInputContext": "<none>",
        "user-httpServer.failedRequests": "4",
        "slowestReceiver": "<none>",
        "numInputQueuedInput": "0",
        "user-httpServer.staticFileRequests": "0",
        "numReceived": "1690",
        "user-httpServer.failedRequests.marginal": "1",
        "numEmits": "1687",
        "numOutEventsUnAcked": "1",
        "user-httpServer.authenticationFailures.marginal": "1",
        "user-httpServer.status": "ONLINE",
        "numProcesses": "48",
        "numEventTypes": "228",
        "virtualMemorySize": "3177968",
        "numQueuedInput": "0",
        "numConsumers": "3",
        "numOutEventsQueued": "1",
        "uptime": "1383561",
        "numListeners": "207",
        "numOutEventsSent": "1686",
        "mostBackedUpICQueueSize": "0",
        "numSnapshots": "0",
        "mostBackedUpICLatency": "0",
        "numProcessed": "1940",
        "numSubListeners": "207"
    }
}

You can monitor the status using the Apama REST API or the Management interface which is an EPL plug-in. See the following topics in the Apama product documentation for further information:

• “Managing and Monitoring over REST” in Deploying and Managing Apama Applications, and
• “Using the Management interface” in Developing Apama Applications.

Monitoring dropped events

When a model receives an event, it may be dropped if the correlator delivers or processes it too late. See Input blocks and event timing.

The total number of dropped events across all models is periodically published as part of the status operation. The count of the number of dropped events is available as a user-defined status value with the name user-analytics-oldEventsDropped in the apama_status parameter of the status operation. See also Monitoring periodic status for details about the operation.

All dropped input events are also sent to channel AnalyticsDroppedEvents, allowing you to implement your own monitoring of the dropped events. A dropped input event sent to the channel AnalyticsDroppedEvents is packaged inside an event of type apama.analyticsbuilder.DroppedEvent. This allows you to extract the original dropped event and perform any analysis on it, for example, categorizing the number of dropped events per device. This can be achieved by writing EPL that listens for the DroppedEvent events, aggregates by device identifier and/or time, and sends measurements to Cloud of Things that can be monitored.

Monitoring the model life-cycle

Life-cycle messages are written to the correlator log whenever a model is created or deleted, or when it fails. The log messages may look as follows:

Model "Build Pipeline" with PRODUCTION mode has started.
    
Model "Build Pipeline" with PRODUCTION mode has ended.
    
Model "Build Pipeline" with PRODUCTION mode has failed with an error: 
IllegalArgumentException - Error while validating parameters for the 
block "toggle" of type "apama.analyticskit.blocks.core.Toggle": 
The "Set Delay" must be finite and positive: -1.

See Accessing the correlator log for information on where to find the correlator log.

Configuration

You can customize the settings of Analytics Builder, the so-called “tenant options”, by sending REST requests to Cloud of Things. The key names that you can use with the REST requests are listed in the topics below. A category name is needed along with the key name; this is always analytics.builder.

You can find some concrete examples in Using curl commands for setting various tenant options. However, you can use any tool you like.

CAUTION:

After you have changed a tenant option using a REST request, the correlator will automatically restart. An alarm with a MAJOR severity will be created in this case; you can view it on the Alarms page of the Cockpit application(see the User guide for information on how to work with alarms).

• Keys for status reporting

• Keys for model timeouts

• Other keys

• Logged tenant options

• Using curl commands for setting various tenant options

Keys for status reporting

Keys for model timeouts

Other keys

Logged tenant options

The values for some of the tenant options are logged. These are the following:

• status_device_name
• status_period_secs
• timedelay_secs
• numWorkerThreads

If you want to find out which values are currently used for these tenant options, you can look them up in the correlator log. See also Accessing the correlator log.

Using curl commands for setting various tenant options

You can set or change various tenant options by sending POST requests to Cloud of Things. This topic explains how you can do this using the curl command-line tool. See https://curl.haxx.se/ for detailed information on curl. See also the information on tenants in the Reference guide.

The syntax of the curl command depends on the environment in which you are working. The syntax for a Bash UNIX shell, for example, is as follows:

curl --user username -X POST -H 'Content-Type: application/json' -d '{"category": "analytics.builder", "key": "keyname", "value": "value"}' -k https://hostname/tenant/options

where:

• username is the name of a user who has the permission to change the tenant options in Cloud of Things. curl will prompt for a password. Or you can provide a password in the username argument by appending it with a colon (:) and the password. For example:

  --user User123:secretpw

• keyname is one of the keys listed in the previous topics.

• value is the value that is to be set for the key, which can be a number or a string, depending on the key.

• hostname is the host name of your tenant where your user application is deployed.

• The category is always analytics.builder.

For example (Bash shell):

curl --user User123 -X POST -H 'Content-Type: application/json' -d '{"category": "analytics.builder", "key": "numWorkerThreads", "value": "4"}' -k https://mytenant/tenant/options

Accessing the correlator log

The location of the correlator log depends on the environment in which you are working:

• Cloud of Things:

  The correlator log is accessible via Cloud of Things Administration application. You can find it on the Logs tab of the Apama-ctrl microservice. You have to subscribe to the microservice so that you can see the logs. For more information on microservices and log files, see the User guide.

• Cloud of Things Private:

  See the Cloud of Things Private guide for detailed information on the log file location.