General aspects

Availability

Microservices Runtime for Cloud of Things is available as an add-on option for

• Cloud of Things SMART
• Cloud of Things ENTERPRISE
• Cloud of Things PRIVATE

In order to be used with your Cloud of Things tenant, Microservices Runtime for Cloud of Things needs to be booked and enabled. Please contact us for further information.

Important: To use this feature, you need a valid subscription. Please contact our Sales team at iot4you@telekom.de to activate this feature.

Introduction

Microservices use standard REST APIs with full authentication and authorization to communicate with Cloud of Things. They are, in most cases, multi-tenant, meaning they must be able to strictly separate tenants and connect to multiple tenants at the same time. Microservices may offer their own endpoints that can be used by Cloud of Things and Cloud of Things-based applications, for example, for system integration purposes. Examples of such microservices are the Jasper Control Center integration and the SMS integration for sending SMS notifications to end users.

You can extend the Cloud of Things platform with customer-specific functionality by deploying microservices. For instance, you can develop integrations to third-party software or provide server-side business logic. A microservice-based architecture introduces change that is often well received by those developing modern applications, and solutions can be delivered much more quickly to those requesting flexible and scalable applications. Microservices bring significant benefits such as deployability, reliability, availability, scalability, modifiability and management.

Cloud of Things microservices have the following properties:

• By default, they provide REST or Websocket APIs.
• Inbound REST and Websocket endpoints are secured by Cloud of Things core built-in API gateway functionality.
• Requests from one microservice to the Cloud of Things REST API can be executed by either using the original user account (of the inbound request) or by using a service user.
• Multi-tenant support.

The following management features are supported:

• Microservices can be registered to individual tenants and super-tenants (that is, tenants with subtenants).
• Multi-tenant microservices can be subscribed to other tenants.

Technically, microservices are Docker containers hosted by Cloud of Things and they follow specific conventions. They are typically accessed using Cloud of Things REST API available under /service/<microservice-name>.

Developers are not restricted to any programming language when developing a microservice for Cloud of Things. However, a microservice must serve as an HTTP server working on port 80 and must be encapsulated in a Docker image. Refer to the relevant chapters in this guide for further information for the development of microservices.

Containerization and orchestration

Images and containers

Docker is a platform to develop, deploy and run applications with containers. An image is an executable package that includes everything needed to run an application (that is, the code, a runtime, libraries, environment variables and configuration files). A container is a runtime instance of an image (that is, what the image becomes in memory when executed). Refer to the Docker documentation for more information about Docker.

Cloud of Things microservices are based on Docker. Hence, a microservice must be packaged as a Docker image in order to run on the Cloud of Things platform. A microservice is executed in a Docker container during runtime. The Docker container ensures that a microservice does not harm other microservices running in Cloud of Things.

Containers have an upper thread limit of 10240 for microservices.

Pods

Kubernetes is the container orchestration engine for automating deployment, scaling and management of containerized applications. A Pod is the basic building block of Kubernetes and it represents a running process on your cluster. A Pod encapsulates an application container, storage resources, a unique network IP and options that govern how the container should run.

Docker is the most common container runtime used in a Kubernetes Pod. Moreover, Kubernetes is used to orchestrate Docker containers and it provides many enterprise-grade features for hosting Docker containers such as auto-scaling and load balancing. Refer to the Kubernetes documentation for more information about Kubernetes.

When Docker faces some issues, for example, a Pod synchronization error, an alarm is created and can be seen in Alarms in the Cockpit application. Refer to Troubleshooting in this section to learn about common issues.

Requirements and interactions

The following requirements towards Cloud of Things microservices must be met:

• A microservice must be a (linux/amd64) Docker image run.
• The Docker image must be packaged as image.tar and must include a manifest file (cumulocity.json).
• A microservice must be stateless, that means, it must contain only ephemeral state. The reason is that a microservice must be able to survive (random) restarts due to hardware reasons (server failure) and operations reasons (upgrade, migration).
• All persistent states must be stored at the Cloud of Things platform via inventory, binary, tenant options and other APIs. Persistent volumes are not supported.
• A microservice cannot access the database directly and must use the Cloud of Things API.
• A microservice must provide one inbound REST API. Additional inbound ports are not supported.
• A microservice can use multiple outbound ports.
• The request lifetime must have the maximum. The infrastructure might terminate too long running requests.
• All log information must be sent to the standard output in order to be captured and persisted by the infrastructure.

Microservices interact with Cloud of Things and the outside world as shown in the following diagram:

The Microservices’ lifecycle is managed using the microservice subscription API. This allows registration and subscription of Microservices. External actors (for example web user interfaces, integrations or other microservices) can invoke a microservice by sending REST or Websocket requests to its endpoints /service/<microservice-name>/<path>. A microservice can issue requests to external endpoints or to the Cloud of Things REST APIs. Microservices can store logs and metrics in the associated developer tenant.

Privacy and Security Assessment (PSA)

The Privacy and Security Assessment (PSA) process is a core element in safeguarding security and data privacy at Deutsche Telekom.

Important: Microservices need to pass Deutsche Telekom the Privacy and Security Assessment (PSA) process before they can be deployed on Cloud of Things.

Data privacy and security are continuing to grow in significance. This fact is illustrated by the rapid rise in politically and criminally motivated cyber attacks. Around the world, individuals, organizations, and corporation, such as Deutsche Telekom, are being targeted. Data privacy and security are particularly pivotal for Deutsche Telekom, since our services are based on our customers’ trust in security.

Deutsche Telekom recognized the resulting challenges at an early stage and responded by developing and introducing a tailored solution: the Privacy and Security Assessment (PSA).

The PSA process guarantees that all development projects meet the strictest technical security and data privacy requirements.

The PSA process has the following goals:

• A consistent and adequately high security and data privacy level in all products, systems and platforms.
• An integrated and standardized process for technical security and data privacy as a deeply rooted element of product and system development processes.
• A support level adapted to project complexity and criticality through the introduction of categorization at the start of each development project.

A first important step is the identification of relevant privacy and security requirements. The requirements are compiled by the security and privacy experts of Deutsche Telekom on basis of vendor specifications and industry standards as well as valid laws and regulations. They are group-wide aligned and enforced. Next step of the PSA process is implementing, testing and documentation of the requirements. For not implementable requirements a automatic risk assessment on basis of the protection need of the individual system will be performed. If necessary an action plan will be prepared together with the project team.

The PSA process guarantees for all relevant projects and systems the implementation of groups security and privacy standards and an approval by security and privacy experts before bringing in service.

Security requirements

Note: Security requirements can be downloaded here (zip, 2.0 MB).

Microservice manifest

The application manifest provides the required settings to manage microservice instances and the application deployment in the Cloud of Things platform. The definition is provided within the cumulocity.json file in the binary uploaded to the Cloud of Things platform.

Here is an example manifest:

{
    "apiVersion": "v2",
    "name": "my-microservice",
    "version": "1.0.0",
    "provider": {
        "name": "New Company Ltd.",
        "domain": "https://new-company.com",
        "support": "support@new-company.com"
    },
    "isolation": "MULTI_TENANT",
    "scale": "AUTO",
    "replicas": 2,
    "resources": {
        "cpu": "1",
        "memory": "1G"
    },
    "requestedResources":{
            "cpu": "100m",
            "memory": "128Mi"
    },
    "requiredRoles": [
        "ROLE_ALARM_READ"
    ],
    "livenessProbe": {
        "httpGet": {
            "path": "/health"
        },
        "initialDelaySeconds": 60,
        "periodSeconds": 10
    },
    "readinessProbe": {
        "httpGet": {
            "path": "/health",
            "port": 80

        },
        "initialDelaySeconds": 20,
        "periodSeconds": 10
    },
    "settingsCategory": "my-ms",
    "settings": [
        {
            "key": "tracker-id",
            "defaultValue": "1234"
        }
    ]
}

See below for detailed information about available settings.

Settings

Version

The version has an impact on the microservice upload behavior:

• If a new ZIP file for a microservice is uploaded but the version is the same as the previous, for example, “1.1.0”, then there is no guarantee that the Docker image for the microservice will be updated.
• If the version is a snapshot, for example, “1.1.0-SNAPSHOT”, then Docker will update the image on each ZIP upload.

The snapshot postfix means that the image build is a snapshot of your application at a given time and it is still under development. When your microservice is ready for production release, you can remove the postfix and just use the final version of your application.

Provider

Resources

RequestedResources

Option

Probe

ExecAction

TCPSocketAction

HTTPGetAction

HttpHeader

Extension

Isolation and scaling

The following isolation levels are available for microservices:

• Multi-tenant: Single microservice Docker container instantiated for all subscribed tenants, unless the microservice is scaled.
• Single-tenant: Dedicated microservice Docker container instantiated for each subscribed tenant.

See Microservice manifest > Settings for the isolation setting.

In case the scale setting is set to NONE, the platform guarantees that there is one instance of the service per isolation level by default. The number of guaranteed instances can be increased by defining the replicas setting in the manifest. If scaling is enabled (set to AUTO), the microservice will be horizontally auto-scaled (creating more instances of the microservice) in case of high CPU usage. Auto-scaling monitors the microservices to make sure that they are operating at the desired performance levels, and it will automatically scale up your cluster as soon as you need it and scale it back down when you don’t.

See Microservice manifest > Settings for the scale and replicas setting.

Microservice migration to API Version 2

With release 10.15, DT IoT announces the availability of microservice API Version 2 and the deprecation of API Version 1 to comply with new security requirements. Microservice API Version 2 provides an improved microservice container security context restricting the invocation of privileged Linux Kernel APIs. In details this means that with microservice API Version 2 the microservice container is granted only specific capabilities.

Refer to the Linux manual page for more information on Kernel capabilities. With the API Version change, the microservice is granted the capability NET_BIND_SERVICE.

Microservice migration

Perform the following steps to migrate your microservice to the new API Version.

1. Change the API Version to 2 in the microservice manifest. See Microservice manifest.

2. Deploy your microservice to the test environment.

3. Test the functionality of your microservice and analyze possible errors.

In the simplest case it is sufficient to set the API Version to 2 in your microservice manifest.

However, for microservices which currently make use of specific privileges of the Linux Kernel API that are not granted anymore, you additionally need to refactor the source code so that the service doesn’t require the invocation of these privileges.

Affected microservices

Set the API Version field in the microservice manifest to 2 and deploy this service to your Cumulocity test environment. This environment must be in version 10.15 or higher. Verify that the functionality provided by the microservice still works as expected.

Microservices still using old user privileges after the environment upgrade

If your microservice is using the deprecated API Version 1, it may be affected by the change depending on the configuration of this environment. In this case, you will not be able to upload or subscribe the microservice.

Security

Microservices typically provide a REST API and Cloud of Things provides a light API gateway (Proxy) for inbound REST requests. Inbound WebSocket requests are supported. The API gateway, located between the client and the microservice container, provides:

• Authorization: All calls are authenticated using Cloud of Things credentials with basic or OAuth authorization.
• TLS Termination: TLS inbound calls are terminated and only HTTP is used inside the cluster.
• Metering: The API calls are metered in the API calls tenant statistics.
• Routing: The API gateway routes requests for /service/<name> to the microservice <name>. The request routed to the microservice container and tenant options are added to the request headers. Note that therefore a space (” “) is not allowed as a tenant option value. If contextPath is defined in the application manifest, the API gateway routes requests for /service/<contextPath>.

Authentication and authorization

A request to a microservice can be authenticated using various authentication mechanisms supported by Cloud of Things such as:

1. Basic authentication
2. OAI-Secure
3. SSO
4. JWT token authentication (deprecated)

If you use Java for your development, we recommend you to use the Microservice SDK version 10.4.6 or later to gain support for all available authentication ways in Cloud of Things.

In case of other programming languages, for which Cloud of Things does not provide an SDK, we recommend developers to ensure support for all authentication mechanisms available in the Cloud of Things platform.

The Cloud of Things exposes the REST endpoint /user/currentUser. The microservice retrieves the Cloud of Things address from the C8Y_BASEURL operating system environment variable. The microservice can verify if the credentials embedded in the HTTP request, which is handled by the microservice, are valid or not by using the REST endpoint /user/currentUser. If the credentials are correct, the response status from the /user/currentUser endpoint is 200 OK. In case of incorrect credentials, the response contains a 401 Unauthorized status code. In order to verify if the credentials are correct, the microservice copies the credentials from ongoing requests and sends them to the /user/currentUser endpoint.

Depending on the authentication method, the credentials can be passed to the microservice in various ways, such as:

1. Authorization HTTP header
2. Cookie called authorization
3. Custom HTTP header called X-XSRF-TOKEN
4. Custom HTTP header called tfatoken

If the incoming request contains the cookie authorization, the microservice has to copy the cookie and the header X-XSRF-TOKEN to the request to the /user/currentUser endpoint. In other cases, the header Authorization has to be copied. This is necessary, if a request contains the header tfatoken, which always needs to be included in the request to /user/currentUser.

You can see the credential validation flow on the sequence diagram below:

Example

The microservice can receive the HTTP requests with the following header (see example below). The authentication process is based on basic authentication and SMS based TFA.

GET https://cumulocity.default.svc.cluster.local/test HTTP/1.1
accept: */*
authorization: Basic dGVuYW50X2lkL3VzZXJuYW1lOnBhc3N3b3JkCg==
connection: close
content-length: 0
cookie: REQUEST_ORIGIN=
host: auth-scope-management.default.svc.cluster.local:80
tfatoken: 23b75292468e0ba7fe03245d502d9c29e21e8a997fc7dd7e1a1df7fe31cbfb17
x-forwarded-host: cumulocity.default.svc.cluster.local:80
x-forwarded-prefix: /service/auth
x-forwarded-proto: http
x-real-ip: 192.168.1.20

The microservice sends the following requests to validate the credentials (provided that c8y_baseurl is equal to https://cumulocity:8111):

GET https://cumulocity:8111/user/currentUser HTTP/1.1
Accept: text/plain, application/json, application/*+json, */*
Authorization: Basic dGVuYW50X2lkL3VzZXJuYW1lOnBhc3N3b3JkCg==
Content-Length: 0
tfatoken: 23b75292468e0ba7fe03245d502d9c29e21e8a997fc7dd7e1a1df7fe31cbfb17

If the user is authenticated with OAI-Secure, the following REST request can reach the microservice:

GET https://cumulocity.default.svc.cluster.local/test HTTP/1.1
accept: */*
authorization: Basic dGVuYW50X2lkL3VzZXJuYW1lOjxmYWtlIHBhc3N3b3JkPgo=
connection: close
content-length: 0
cookie: authorization=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI0YzAwMGVlOC1hYjY5LTRiMzYtYWNhNC0xNzM4ZGYwZWNhZGIiLCJpc3MiOiJjdW11bG9jaXR5LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJhdWQiOiJjdW11bG9jaXR5LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJzdWIiOiJhZG1pbiIsInRjaSI6IjMxMWM3YjZiLWM1MmQtNGEzMC1iZDFlLWNiODdlMjNlODQyOSIsImlhdCI6MTY0NjkyODgxMywibmJmIjoxNjQ2OTI4ODEzLCJleHAiOjE2NDgxMzg0MTMsInRmYSI6ZmFsc2UsInRlbiI6Im1hbmFnZW1lbnQiLCJ4c3JmVG9rZW4iOiJHU0l6VFB4b2JQS1FJQmV2Uk1SWiJ9.NL5b9-iHSc3SLaPu87oazBd2_kjNiwO9tM_bXS3qCUd0_wTZ-BKc3q6sRHTKO_LNbtCQg3G6-7ZhD6TyvqjTLsyiZTsgMRtAE7ZiRPtXaFOA0_SDQ9kG_MztAZ3U9O008akgXcjEEAdphVv5eey_lADrg1BmIlqiBFoKts2JGSv1xFtXKIxpcVtRDGUkb-2qb8OhaHamT7b3HL628NSiH4VqfO38vkLLkimHEz-roqmbFGQ355TvA3-s_erO96j3rHbBPDsluFqFN0eOTCidBffKt6OvyKw-_64MaHHs6R9Ulsv-LuY-YAvlTZVxYwFAi3yn3mWlpXEAzvGYHMrx8A
host: auth-scope-management.default.svc.cluster.local:80
user-agent: insomnia/2022.1.1
x-forwarded-host: cumulocity.default.svc.cluster.local:80
x-forwarded-prefix: /service/auth
x-forwarded-proto: http
x-real-ip: 192.168.1.20
x-xsrf-token: GSIzTPxobPKQIBevRMRZ

To verify the credentials from the request above, the microservice sends the following request:

GET https://cumulocity:8111/user/currentUser HTTP/1.1
Accept: text/plain, application/json, application/*+json, */*
Content-Length: 0
Cookie: authorization=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI0YzAwMGVlOC1hYjY5LTRiMzYtYWNhNC0xNzM4ZGYwZWNhZGIiLCJpc3MiOiJjdW11bG9jaXR5LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJhdWQiOiJjdW11bG9jaXR5LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJzdWIiOiJhZG1pbiIsInRjaSI6IjMxMWM3YjZiLWM1MmQtNGEzMC1iZDFlLWNiODdlMjNlODQyOSIsImlhdCI6MTY0NjkyODgxMywibmJmIjoxNjQ2OTI4ODEzLCJleHAiOjE2NDgxMzg0MTMsInRmYSI6ZmFsc2UsInRlbiI6Im1hbmFnZW1lbnQiLCJ4c3JmVG9rZW4iOiJHU0l6VFB4b2JQS1FJQmV2Uk1SWiJ9.NL5b9-iHSc3SLaPu87oazBd2_kjNiwO9tM_bXS3qCUd0_wTZ-BKc3q6sRHTKO_LNbtCQg3G6-7ZhD6TyvqjTLsyiZTsgMRtAE7ZiRPtXaFOA0_SDQ9kG_MztAZ3U9O008akgXcjEEAdphVv5eey_lADrg1BmIlqiBFoKts2JGSv1xFtXKIxpcVtRDGUkb-2qb8OhaHamT7b3HL628NSiH4VqfO38vkLLkimHEz-roqmbFGQ355TvA3-s_erO96j3rHbBPDsluFqFN0eOTCidBffKt6OvyKw-_64MaHHs6R9Ulsv-LuY-YAvlTZVxYwFAi3yn3mWlpXEAzvGYHMrx8A
X-XSRF-TOKEN: GSIzTPxobPKQIBevRMRZ

If a request to the /user/currentUser REST endpoint contains the correct credentials, the response body contains various information, such as user roles assigned to the user account. This information can be very useful in order for the microservice to carry out the authorization process.

When a microservice receives an HTTP request, it can become necessary to use the REST API exposed by Cloud of Things to fulfil the request. To access the REST API, the microservice must provide a valid credential. It can use credentials that are present in the incoming REST request or use a dedicated user account, created to access resources belonging to tenants which are subscribed to the microservice. The choice between the credentials provided together with the incoming request to the microservice REST or a dedicated account depends on the use case. In order to utilize the user credentials received by the microservice in an incoming HTTP request, the microservice must copy the credentials according to the rules described above.

Furthermore, most of the HTTP requests which are sent by microservices must contain the HTTP header X-Cumulocity-Application-Key. The header indicates that an action is performed by an application. When the header is missing Cloud of Things assumes that the HTTP request is originated by an IoT device. Therefore, a request without X-Cumulocity-Application-Key affects for example, the device availability state, or the billing data.

Thus, the microservice must only include X-Cumulocity-Application-Key if the microservice proxy requests from an IoT device to the Cloud of Things platform.

If the microservice includes the header X-Cumulocity-Application-Key, the header must contain the correct application key. Then, the microservice retrieves the application key by sending a REST request to the endpoint /application/currentApplication exposed by the Cloud of Things platform. The REST request must contain the credentials for basic authentication in the following format: tenantId/username:password. The tenant ID, username and password are read by the microservice from the following operating system environment variables: C8Y_BOOTSTRAP_TENANT, C8Y_BOOTSTRAP_USER, and C8Y_BOOTSTRAP_PASSWORD. In order to increase the performance, the microservice has to implement a caching mechanism related to the user credentials.

Microservice authentication and multi-tenancy

In general, microservices use the standard Cloud of Things authentication mechanisms. This is executed in two steps:

1. The microservice can be created in any tenant that has feature-microservice-hosting enabled.
2. The microservices access the tenant API.

At the installation time of the microservice, an application is created in the owner tenant (usually a Management tenant or Enterprise tenant) reflecting the new microservice. In addition, a bootstrap user is created in the owner tenant that allows the microservice to retrieve subscriptions. Whenever required, a platform administrator will subscribe a customer to the new microservice. As part of the subscription, a service user in the customer tenant is created using random credentials.

Microservice authorization

Authorization is relevant on two levels:

1. On the owner tenant level, the only authorization of a microservice is to access its own subscriptions.
2. For accessing customer tenants, the microservice installs a set of required permissions for being able to operate.

A microservice is associated with a bootstrap user in the owner tenant, which will make sure that only its subscriptions are returned. A microservice is also associated with a set of permissions that it requires for carrying out its function on a customer tenant. These permissions are visualized in the Administration application. The permissions are associated with the service user that is created when a platform administration associates a microservice with a tenant (subscribes to it). The owner tenant must also be subscribed to if you intend to use microservice features on it.

Users and roles

There are three types of users:

• Tenant user: The user that invokes a microservice through its REST API endpoints /service/<microservice-name>/<path> passed through by the proxy.
• Service user: A generated user that allows a microservice to access a subscribed tenant independent of a REST API invocation, for example, for initialization or regular jobs.
• Microservice bootstrap user: A user passed to the microservice for requesting subscribed tenants and service users.

The following role types are defined for users:

• Required roles: The roles that are predefined to allow access to Cloud of Things Rest APIs. For instance, if a microservice creates measurements using the service user, measurement admin role must be added as a required role of the application. Required roles are added to the service users.
• Roles: The custom roles provided to tenant platform users by the microservice developer. These roles can be assigned or revoked to the tenant platform users or groups using the Administration application.

Custom roles must adhere to this name format in order to be shown in the UI:

ROLE__(READ|ADMIN|CREATE)

You can add them to the application manifest in the roles properties as follows:

"roles": [
    "ROLE_MY_MICROSERVICE_READ",
    "ROLE_MY_MICROSERVICE_ADMIN"
]

The roles are set in the Microservice manifest. For more details about users and roles, review the User API in the Cloud of Things OpenAPI Specification.

Microservice bootstrap

Each microservice receives a dedicated bootstrap user which ensures that a microservice can be identified by the platform and can have access only to the allowed resources. A microservice runtime provides bootstrap user and service user credentials as environment variables which can also be acquired via platform API. Note that depending on the isolation level, the environment variables differ.

In multi-tenant scope, there is a single microservice deployment reused by multiple tenants and that is why service user credentials are not provided as hardcoded environment properties. However, a microservice running in multi-tenant isolation can retrieve all subscriptions via a GET request and using bootstrap credentials as follows:

GET /application/currentApplication/subscriptions
Host: ...
Authorization: Basic ...

Bootstrap user credentials can be retrieved with a GET request authorized with application owner credentials:

GET /application/applications/<APPLICATION_ID>/bootstrapUser
Host: ...
Authorization: Basic ...

An example of a typical user switching in multi-tenant isolation is presented below, where – in a hypothetical scenario – there is a need to send an alarm to each tenant subscribed to a microservice.

The user wants to employ microservice capabilities to raise alarms to all subscribed tenants calls.

Steps:

1. The user makes a request to the platform’s endpoint /service/<microservice>/createAlarms.
2. The platform verifies the user credentials and redirects the request to the microservice.
3. The microservice reads the bootstrap credentials (from environment variables) and uses them to retrieve the service user credentials for all subscribed tenants.
4. The microservice iterates over the service user credentials and uses them to create alarms to each tenant.
5. The microservice returns the result to the platform, and the platform to the invoking user.

Encryption

There is a mechanism to encrypt the tenant options that afterwards are automatically decrypted when injecting them into microservices requests.

If a tenant option is created with a key name that starts with “credentials.", it is automatically encrypted and can be fetched as unencrypted only by system users. For instance, when you create a tenant option in a category that matches to the application context path, the value is passed to the microservice by the microservice proxy on the platform as a header (key => value). Note that therefore a space (” “) is not allowed as a tenant option value. All encrypted options are decrypted and passed. Moreover, the options can be fetched via REST using the options endpoint at microservice runtime.

Refer to tenant options in the Tenant API in the Cloud of Things OpenAPI Specification for more details.

Microservice runtime

Microservices deployed on the platform have a specific runtime environment and they must understand certain details about the specific Cloud of Things cluster they run in. For example, a microservice needs to know the endpoint address of the Cloud of Things REST APIs. This information is provided by environment variables and they are injected by Cloud of Things when the container is started.

Environment variables

The following environment variables are available for microservices:

Example

Prerequisite: The microservice has been packed and deployed in the Docker repository. Get the microservice image name and tag with the following command:

$ docker images

Run the Docker container for the microservice providing the environment variables:

$ docker run -–cap-drop=ALL -–cap-add=NET_BIND_SERVICE \
   -e C8Y_BOOTSTRAP_TENANT=<BOOTSTRAP_TENANT> \
   -e C8Y_BOOTSTRAP_USER=<BOOTSTRAP_USERNAME> \
   -e C8Y_BOOTSTRAP_PASSWORD=<BOOTSTRAP_USER_PASSWORD> \
   -e C8Y_MICROSERVICE_ISOLATION=MULTI_TENANT \
   -e C8Y_BASEURL=<URL> -i -t <DOCKER_REPOSITORY_IMAGE>:<TAG>

Use a backslash (\) before special characters such as &, !, ;, \.

Timezone variable

The timezone variable allows configuring a default timezone used by the microservice. The microservice installer injects the TZ environment variable into the microservice according to the following settings:

• Tenant option in the microservice owner tenant
• Platform application environment variables (MICROSERVICE_RUNTIME_TIMEZONE)

The tenant option has higher priority, that means, if the parameter is set in both places, the value from the tenant option is taken.

Example

Assuming that the microservice owner has the tenant option:

{
    "category" : "microservice.runtime",
    "key" : "timezone",
    "value" : "Europe/Warsaw"
}

Deploying and running the microservice inside Docker will result in passing the following variables into the microservice environment:

TZ=Europe/Warsaw

When using Java-based microservices this variable is automatically read and applied to the Java process, no additional work is required. Microservices developed with other programming languages may require some manual work, that is, loading the TZ value from the environment and using it to configure the time zone on the language level programmatically.

Proxy variables

Proxy variables are used to set a proxy URL for different protocols. For the microservices written in Java, setting each variable will result in passing the corresponding parameter into the JVM runtime (for detailed information see the Java Networking and Proxies webpage).

Proxy variables are passed into the microservice environment during installation. The microservice installer passes the variables into the environment according to the following settings:

• Tenant options in the microservice owner tenant
• Platform application environment variables

Tenant options have higher priority, that means, if the parameter is set in both places, the value from the tenant option is taken.

The table below contains the options and proxy variables:

All tenant options have the same category: microservice.runtime

For each protocol (HTTP, HTTPS, Socks), microservice environment variables are passed into runtime only if the HOST parameter is set. If the HOST parameter is missing, other parameters for the same protocol are not processed.

Examples

The microservice owner tenant has the tenant options:

{ "category" : "microservice.runtime", "key" : "proxy.http.host", "value" : "10.11.12.13" }
{ "category" : "microservice.runtime", "key" : "proxy.http.port", "value" : "8080" }

and there is an environment variable in the platform application:

MICROSERVICE_RUNTIME_PROXY_HTTP_PORT=8181

Deploying and running the microservice inside Docker will result in passing the following variables into the microservice environment (notice PORT value):

PROXY_HTTP_HOST=10.11.12.13
PROXY_HTTP_PORT=8080

{ "category" : "microservice.runtime", "key" : "proxy.https.host", "value" : "10.11.12.13" }

and there is an environment variable in the platform application:

MICROSERVICE_RUNTIME_PROXY_HTTPS_PORT=8181

Deploying and running the microservice inside Docker will result in passing the following variables into the microservice environment:

PROXY_HTTP_HOST=10.11.12.13
PROXY_HTTP_PORT=8181        

{ "category" : "microservice.runtime", "key" : "proxy.http.port", "value" : "8080" }
{ "category" : "microservice.runtime", "key" : "proxy.http.non.proxy.hosts", "value" : "localhost" }

and the proxy HOST is not set (neither in tenant option, nor env variable).

Deploying and running the microservice inside Docker will not pass any proxy environment variable.

{ "category" : "microservice.runtime", "key" : "socks.http.host", "value" : "10.11.12.13" }  

Deploying and running the microservice inside Docker will result in passing the following variable into the microservice environment (only host parameter):

SOCKS_HTTP_HOST=10.11.12.13

Platform access and other microservices

To execute requests against the Cloud of Things platform running a microservice, you must send requests to the host specified by the C8Y_BASEURL variable.

A microservice does not have direct access to other microservices running on the platform. Instead, a microservice must use the platform as a proxy. The endpoint used to access other applications is <C8Y_BASEURL>/service/<OTHER_APPLICATION_NAME>/.

Request routing

The request is redirected to a microservice depending on the isolation level (auto-scaling is ignored at this moment for clarity), subscription and authorization. A typical request to the platform looks like

<METHOD>  /service/<MICROSERVICE>/<MICROSERVICE_ENDPOINT>
Host: ...
Authorization: Basic ...

Credentials are used to verify if a requesting user is authorized to access the microservice, and tenant subscription is verified afterwards. If both checks pass, the request is routed to a dedicated microservice deployment in case that the isolation level is per tenant, or to a shared deployment in case of multi-tenant isolation.

The routed request is stripped of /service/<MICROSERVICE> part. However, the Authorization header is not modified, thus a request is still executed as a tenant platform user.

<METHOD>  /<MICROSERVICE_ENDPOINT>
Host: ...
Authorization: Basic ...

Microservice utility tool

Cloud of Things provides you a utility tool for easy microservice packaging, deployment and subscription. The script requires a local installation of Docker and jq, which is a lightweight and flexible JSON processor.

Prerequisites

Docker

Verify that you have a Docker installation. The Docker version must be >= 1.2.6

$ docker version
Client:
 Version:         1.12.6
 API version:     1.24
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 OS/Arch:         linux/amd64

JSON processor

Execute the following command to install the JSON processor on Linux systems:

$ sudo yum install jq

For macOS, use the following command:

$ brew install jq

Bash

The microservice utility tool (script) needs Bash version 4+ to run. Verify your Bash version with the following command:

$ bash --version

GNU bash, version 5.0.3(1)-release (x86_64-apple-darwin18.2.0)
Copyright (C) 2019 Free Software Foundation, Inc.

macOS systems come with a preinstalled Bash version 3.x. Hence, you must update it in order to execute the microservice script. To do so, execute the following commands:

$ brew install bash
$ chsh -s /usr/local/bin/bash

If your Bash version has not changed while executing bash --version, you may need to restart your system. Note that the updated interpreter gets installed at /usr/local/bin/bash and you must modify the first line of the microservice utility tool (script) as follows:

#!/usr/local/bin/bash

or

#!

Configure the microservice utility tool

The script can be found in the GitHub repository: cumulocity-examples.

Change the mode to allow the script to be executed:

$ chmod +x microservice

Use the help option to see all the available functions (goals) and options:

$ ./microservice help

Packing

A microservice must be packed as a Docker image in order to be deployed. It requires a Docker image.tar and cumulocity.json files packed into a ZIP file.

The following directory structure is required to pack a microservice:

/docker/Dockerfile      # Instructions to build the Docker image
/docker/*               # All files within the directory will be included in the Docker build
/cumulocity.json        # The application manifest file

The script can be run in a parent folder holding such structure, or by passing the path to the directory using the -dir option. For instance, to pack a “Hello World” microservice application, execute:

$ ./microservice pack --name hello-world

It will create a ZIP file named hello-world.zip and an intermediate image.tar which is an exported Docker image.

Deploying

A microservice becomes available once it has been successfully deployed on the Cloud of Things platform. This is done by uploading a ZIP file with the microservice packed as specified above. A user cannot directly push an image to the Docker registry.

Deploying your microservice application is rather easy, just execute the following command:

$ ./microservice deploy -n hello-world -d <URL> -u <username> -p <password> -te <tenant>

Note that you must have a tenant and user credentials in order to deploy your microservice.
The successful execution will create an application on the Cloud of Things platform with the specified name, if it does not exist yet. Then it will upload the hello-world.zip file into the platform. Once it has been uploaded, your application will be listed in Ecosystem > Microservices in the Administration application.

If the name of the microservice application is not provided in the manifest file, it will be automatically inferred from the ZIP file name without the version number.

For further information on deploying microservices to Cloud of Things, refer to Administration > Managing and monitoring microservices in the User guide.

Subscribing

You must subscribe to the application in order to use it. Execute the following command to subscribe your tenant to the deployed microservice:

$ ./microservice subscribe -n hello-world -d <URL> -u <username> -p <password> -te <tenant> -id <APPLICATION_ID>

It will result in tenant subscription to an application specified by the ID parameter. If the user has already been subscribed, a warning message will be displayed.

Multiple goals

Goals can be executed together to pack, deploy and subscribe the application in a single line. In this case, the application ID will be automatically pulled by the script.

$ ./microservice pack deploy subscribe -n hello-world -d <URL> -u <username> -p <password> -te <tenant>

Operating microservices

Cloud of Things manages microservices by monitoring the microservice instance and storing the metrics. In case a microservice exceeds the memory limit, it is restarted automatically. Also, microservices can be auto-scaled in case of high CPU usage. For more information, review the scaling details above.

Troubleshooting

Some common issues have been identified and documented below.

I get an error saying “Microservice application name is incorrect” on uploading a microservice application

When naming your microservice application only use lower-case letters, digits and dashes. The maximum length for the name is 23 characters.

I deployed my microservice but requests to any endpoint returns an error message “Microservice not available Connection refused”

After uploading the microservice, the internal deployment and container run may take a couple of minutes. Once completed, the error message will disappear. Meanwhile, enjoy your coffee ☕️

An alarm was created with the message “Failed to pull image … rpc error: code = Canceled desc = context canceled”

This may happen occasionally when pulling large Docker images for containers within a pod. It will go to normal once it has been successfully pulled.

An alarm was created with the message “Pod synchronization error.”

This may happen when Kubernetes is performing the auto-scaling and is trying to restart the pods.

An alarm was created with the message “No nodes are available that match all of the predicates: [Insufficient cpu …]”

There are additional containers running besides yours and they are provisioned by default with Kubernetes. These additional containers might be taking up of the CPU quota of the single node. Kubernetes will manage this and change the pod status from Pending to Running.