Edge operations

Restarting the Edge appliance

When you apply a patch or update your Edge appliance to a newer version, you must restart your Edge appliance after the update. Before restarting your Edge appliance, ensure that your appliance is in a safe state.

Restarting the Edge appliance using the UI

To restart your Edge appliance:

1. Log in to the Management tenant using the Edge administrator credentials created during the installation.

   • Username: management/<Edge admin username>
   • Password: password provided during the installation

2. Switch to the Administration application using the application switcher at the right of the top bar .

3. Click Edge > Reboot in the navigator.

4. Click Reboot.

Restarting the Edge appliance using the REST APIs

To restart the Edge appliance using the REST APIs, see the edge/reboot API in the Cloud of Things Edge OpenAPI Specification.

Starting and stopping services

Karaf processes

In the Edge appliance, run the following commands as admin user.

opcua-mgmt-service

In the Edge appliance, run the following commands as admin user.

opcua-device-gateway

In the Edge appliance, run the following commands as admin user.

smartrule

In the Edge appliance, run the following commands as admin user.

Apama

In the Edge appliance, run the following commands as admin user.

Expanding the disk size

You can expand the disk size of the installation disk and the data disk using the UI and REST APIs. You can either expand the disk size for both the disks or any one of the disk at a time. There is no limit on the number of the disk expansion process. Before expanding the disk size, you must set or edit the disk size in the hypervisor. See the hypervisor specific documentation for editing the disk size.

Expanding the disk size using the UI

1. Shut down your Edge appliance.

2. Increase the size of the installation and data disks in you hypervisor.

3. Restart your Edge appliance.

4. Log in to the Management tenant using the Edge administrator credentials created during the installation.

   • Username: management/<Edge admin username>
   • Password: password provided during the installation

5. Switch to the Administration application using the application switcher at the right of the top bar .

6. Click Edge > Expand disk size in the navigator.

7. Click Expand.

Expanding the disk size using the REST APIs

To expand the disk size using the REST APIs, see the edge/expand-disk API in the Cloud of Things Edge OpenAPI Specification.

Health check

Network

The following instructions show how to check the network connectivity of the platform.

Check network interface of the node

The following commands will show the interface and network settings of the machine:

ip a

This will list all interfaces and its current configuration.

Example:

[admin@iot-edge-server ~]$ ip a

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
   valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
   valid_lft forever preferred_lft forever

2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:88:e7:de brd ff:ff:ff:ff:ff:ff
inet 10.0.2.15/24 brd 10.0.2.255 scope global noprefixroute dynamic enp0s3
   valid_lft 85338sec preferred_lft 85338sec
inet6 fe80::a00:27ff:fe88:e7de/64 scope link noprefixroute
   valid_lft forever preferred_lft forever

3: enp0s8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 08:00:27:81:fe:9d brd ff:ff:ff:ff:ff:ff
inet 192.168.56.120/24 brd 192.168.56.255 scope global noprefixroute enp0s8
   valid_lft forever preferred_lft forever
inet6 fe80::5b3a:bc65:40b5:f9ea/64 scope link noprefixroute
   valid_lft forever preferred_lft forever

You must make sure that the node has an external interface (ethX) and the loopback interface configured (lo). The loopback interface needs to have the fixed IP 127.0.0.1 with subnet mask 255.0.0.0, and the IP address of the external interface must reside in the correct subnet with the correct subnet mask (in this example 255.255.252.0).

The following command lists the local routing information.

netstat -rn

Example:

[admin@iot-edge-server ~]$ netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
0.0.0.0         10.0.2.2        0.0.0.0         UG        0 0          0 enp0s3
10.0.2.0        0.0.0.0         255.255.255.0   U         0 0          0 enp0s3
192.168.56.0    0.0.0.0         255.255.255.0   U         0 0          0 enp0s8

Make sure you have the destination 0.0.0.0 in the list which then also has the gateway flag (G) set.

Check access to the internet

Try to reach a well-known address in the internet with the following command:

ping -s 1500 8.8.8.8

Example:

[admin@iot-edge-server ~]$ ping -s 1500 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 1500(1528) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=2.61 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=2.80 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=56 time=2.82 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=56 time=2.75 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=56 time=2.79 ms

As when checking the internal reach ability, you must make sure that you can see the replies from the address you tried to reach. Use Ctrl-C to end the ping command.

Processing

This section lists the required services and processes on the Edge appliance.

Check platform status

You can check the status of the platform by running the following command:

curl -v http://localhost:8181/tenant/health

* About to connect() to localhost port 8181 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 8181 (#0)
> GET /tenant/health HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost:8181
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Sat, 05 May 2018 18:13:28 GMT
< Transfer-Encoding: chunked
< Server: Jetty(8.1.19.v20160209)
<
* Connection #0 to host localhost left intact
{
    "status": "UP",
    "services": {
        "details": {},
        "status": "UP"
    },
    "mongodb": {
        "details": {},
        "status": "UP"
    },
    "mongodb-cluster": {
        "details": {
            "mongodb-cluster-enabled": true
        },
        "noSuppressibleDown": false,
        "status": "UP"
    },
    "tenant": {
        "details": {},
        "status": "UP"
    }
}

The ‘status:“UP”’ shows that all services are running.

If something fails, the endpoint should respond a different error code. The response should contain something like:

{
    "status":"DOWN",
    "services":{
        "details": {},
        "status":"UP"
    },
    "mongodb":{
        "details": {},
        "status":"UP"
    },
    "mongodb-cluster": {
        "details": {
            "mongodb-cluster-enabled": true
        },
        "noSuppressibleDown": false,
        "status": "UP"
    },
    "tenant":{
      "details":{
         "notFullyInitializedTenants":{
            "savenindia":"PAYPAL_REFUND",
            "nowa":"PAYPAL_REFUND",
            "ttc":"PAYPAL_REFUND",
            "controllogichh":"PAYPAL_REFUND",
            "jonathan":"PAYPAL_REFUND",
            "barcotrail":"PAYPAL_REFUND",
            "ngservices":"PAYPAL_REFUND",
            "sixsenses":"PAYPAL_REFUND",
            "trav2":"PAYPAL_REFUND",
            "lucaslocatrix":"PAYPAL_REFUND"
         }
      },
      "status":"DOWN"
   }
}

The response contains checks for the most important components:

• services - status of OSGI services running within karaf container
• mongo - status of connection to mongo database (for clustered mongo shows status of connection to mongos)
• mongodb-cluster - status of mongo cluster, status DOWN doesn’t have bad impact for overall health status (because it is marked as “noSuppressibleDown”: false) Mongo-cluster shows status as follow:
  • “details”: {“mongodb-cluster-enabled”: true}, “status”: “UP” - mongo cluster is enabled and works correctly
  • “details”: {“mongodb-cluster-enabled”: false}, “status”: “DOWN” - mongo cluster is disabled and mongo works in single mode (for health status check section “mongo”)
  • “details”: {“mongodb-cluster-enabled”: true}, “status”: “DOWN” - mongo cluster is enabled and works incorrectly, additionally mongo exception is presented
• tenant - tenant initialization status

The list describes the tenants which on the core node have not been fully initialized and the initialization status they are in.

Checking the system services

Checking REST API availability

Run the following command to check the REST API availability:

curl -u 'edge/<username>:<password>' -v -X GET http://<base_url>/platform

Example: [admin@iot-edge-server ~]$ curl -u ‘edge/:’ -v -X GET http://<base_url>/platform

* About to connect() to <base_url> port 80 (#0)
*   Trying 52.29.189.245... connected
* Connected to <base_url> (52.29.189.245) port 80 (#0)
* Server auth using Basic with user 'management/<username>'
> GET /platform HTTP/1.1
> Authorization: Basic bWFuYWdlbWVudC90c3NjaHVlbDohITQ3TmV1bjI3MQ==
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: <base_url>
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 23 Aug 2016 15:39:29 GMT
< Content-Type: application/vnd.com.nsn.cumulocity.platformApi+json; charset=UTF-8; ver=0.9
< Transfer-Encoding: chunked
< Connection: keep-alive
<
{"alarm":{"alarms":{"alarms":null,"self":"http://management.<base_url>/alarm/alarms"},"alarmsForSource":"http://management.<base_url>/alarm/alarms?source={source}","alarmsForSourceAndStatus":"http://management.<base_url>/alarm/alarms?source={source}&status={status}","alarmsForSourceAndStatusAndTime":"http://management.<base_url>/alarm/alarms?source={source}&status={status}&dateFrom={dateFrom}&dateTo={dateTo}","alarmsForSourceAndTime":"http://management.<base_url>/alarm/alarms?source={source}&dateFrom={dateFrom}&dateTo={dateTo}","alarmsForStatus":"http://management.<base_url>/alarm/alarms?status={status}","alarmsForStatusAndTime":"http://management.<base_url>/alarm/alarms?status={status}&dateFrom={dateFrom}&dateTo={dateTo}","alarmsForTime":"http://management.<base_url>/alarm/alarms?dateFrom={dateFrom}&dateTo={dateTo}","self":"http://management.<base_url>/alarm"},…}

This example shows the correct response of the platform. The username and password must have full read access to the Management tenant. The <base_url> needs to be given to connect to the correct platform and the Management tenant must not be blocked from outside.

Monitoring

Monit is a small open-source utility for managing and monitoring the Unix systems. The Monit utility conducts automatic maintenance and repair, and can execute meaningful causal actions in error situations.

In Edge, Monit is used to monitor the processes and perform a restart if any of the processes is down or inactive. The usual Monit interval (cycle) is 30 seconds.

Using Monit

Monit can provide a quick status report of all the configured services and processes by running the following command:

sudo monit summary

There might be cases where Monit has stopped monitoring some resources because of timeout on constant failures or dependency issues.

A specific component, for example, apama-ctrl_proc, can be restarted using the following command:

sudo monit restart apama-ctrl_proc

The status of a specific component, for example, apama-ctrl_proc, can be viewed using the following command:

sudo monit status apama-ctrl_proc

The Monit status can be checked by running:

```shell
sudo systemctl status monit

Monit can be restarted by running:

sudo systemctl restart monit

The log file for monit is located in /var/log/monit.log.

Monitor

The monitor can be started by running the run_monitor.py script located under /opt/c8y/utilities/diagnostic-utility/src/.

The monitor script supports only one optional startup parameter:

-s or --skipDataCollector allows you to skip the data collection even if one or more monitored components are not working.

Diagnostic utility

The diagnostic utility is a script to collect, for example, the journal logs and performance metrics of various components running on the Edge appliance, which are essential for you and the DT IoT support team to troubleshoot the problems. The diagnostic utility is enabled by default and is scheduled to run periodically. However, you can also trigger it manually using the command line (below) or the user interface.

cd /opt/c8y/utilities/diagnostic-utility
sudo ./run_data_collector.py

Hardware information

The basic hardware information of the target system is captured. These reports are placed under the ‘hardware’ directory.

The following hardware information is available:

Software information

The basic software information of the target system is captured. These reports are placed under the ‘software’ directory.

The following software information is available:

Cloud of Things information

This section contains information on the running Cloud of Things processes, health endpoint check result, Cloud of Things logs, and so on.

The following Cloud of Things information is collected:

Jstat dumps can provide information on the following options:

• class - Statistics on the behavior of the class loader
• compiler - Statistics of the behavior of the HotSpot Just-in-Time compiler
• gc - Statistics of the behavior of the garbage collected heap
• gccapacity - Statistics of the capacities of the generations and their corresponding spaces
• gccause - Summary of garbage collection statistics (same as -gcutil), with the cause of the last and current (if applicable) garbage collection events
• gcnew – statistics of the behavior of the new generations
• gcnewcapacity - Statistics of the sizes of the new generations and its corresponding spaces
• gcold - Statistics of the behavior of the old and permanent generations
• gcoldcapacity - Statistics of the sizes of the old generations
• gcpermcapacity - Statistics of the sizes of the permanent generations
• gcutil – Summary of garbage collection statistics
• printcompilations – HotSpot compilation method statistics

As per the current implementation, the 5 statistics counts are collected at an interval of 50ms. To improve performance, this task is executed via threads.

Optional startup parameters

Following are the supported startup parameters for the monitor and the data collector.

Data collector

The data collector can be started by running the “run_data_collector.py” script located under “/opt/c8y/utilities/diagnostic-utility/".

Following are the supported command line arguments which can be used while invoking the script. More than one of the supported arguments can be used simultaneously.

• -hw or –hardware: Allows the script to collect only the hardware information
• -sw or –software: Allows the script to collect only the software information
• -c8y or –cumulocity: Allows the script to collect only the cumulocity information
• -h: Displays the help message

Microservices log file locations

The logs of the Kubernetes components are captured at: /tmp/diagnostic-utility/diagnostic_report_XXXXX/cumulocity/log_archive/kubernetes_logs.zip.

The kubernetes_logs.zip file contains the logs of all Kubernetes platform components at “kube-system” path in the archive. The components captured are:

• heapster-XX
• kube-apiserver-server
• kube-controller-manager-server
• kube-dns-XX
• kube-flannel-ds-XX
• kube-proxy-XX
• kube-scheduler-server

The hosted microservices are captured at cumulocity-single-node path in the archive. The pre-installed component kube-registry-persistent-secure-xx-xx is already available in the archive. The logs of any additional microservices that are uploaded will also be available at this path.

Utility configuration file

The diagnostic utility can be customized using a properties file located under “/etc/diagnostic-utility/diagnostic_utility.properties”.

Following are the available keys used in the configuration file:

Downloading diagnostics using the UI

1. Log in to the Management tenant using the Edge administrator credentials created during the installation.

   • Username: management/<Edge admin username>
   • Password: password provided during the installation

2. Switch to the Administration application using the application switcher at the right of the top bar .

3. Click Edge > Diagnostics in the navigator.

4. Click Request diagnostics report to download the diagnostics.

To download the diagnostics report remotely, see Remote connectivity > Downloading diagnostics remotely.

Downloading diagnostics using the REST APIs

To download the diagnostics report using the REST APIs, see the edge/diagnostics API in the Cloud of Things Edge OpenAPI Specification.

Logging

The solution stores log files at the following locations for the different nodes.

Core node log file locations

Apama log file locations

To access the apama-ctrl log files, run the command:

sudo docker logs apama-ctrl-edge

MongoDB log file locations

Agent log file locations

DataHub log file locations

See Log files for details on DataHub log files.

Log rotation

The components for which log rotation is configured are as follows:

Under /etc/ they are configured via logrotate.conf and the configuration files under /etc/logrotate.d.

The components for which log rotation is configured are as follows:

For microservices, there currently is no specific log rotation configured.

Troubleshooting the system

In case of any issues, DT IoT recommends you to follow these steps:

• Perform a health check, see Health check.
• Monitor the system processes, see Monitoring.
• Check the log files, see Log files.

These steps help you analyze the issue and provide a fix. If you need to contact DT IoT support, include the output of the diagnostics dump. For more information about using the diagnostics, see Diagnostics.

Troubleshooting Microservices

In case of any microservices related issues, we recommend you to check if _microservice_hosting_ is disabled using the command sudo monit status microservice_hosting

If it shows up as disabled, enable the microservice hosting feature through the GUI or the REST API. If enabling the microservice hosting feature fails, check the status of the Kubernetes system pods using the command below:

sudo kubectl get pods --namespace kube-system

If any pod in the list show its status as evicted, the most probable cause is that the system ran out of disk space. Expand the disk size through the GUI or the REST API. Enable the microservice hosting feature after disk expansion.

When running any kubectl command (for example sudo kubectl get pods --namespace kube-system), if the message -bash: kubectl: command not found is seen then the microservice hosting feature has not been enabled or the process to enable the microservice hosting feature has failed. Attempt to enable the microservice hosting feature through the GUI or the REST API.