Availability

Microservices Runtime for Cloud of Things is available as an add-on option for

In order to be used with your Cloud of Things tenant, Microservices Runtime for Cloud of Things needs to be booked and enabled. Please contact us for further information.

Important: To use this feature, you need a valid subscription. Please contact our Sales team at iot4you@telekom.de to activate this feature.

Introduction

Microservices use standard REST APIs with full authentication and authorization to communicate with Cloud of Things. They are, in most cases, multi-tenant, meaning they must be able to strictly separate tenants and connect to multiple tenants at the same time. Microservices may offer their own endpoints that can be used by Cloud of Things and Cloud of Things-based applications, for example, for system integration purposes. Examples of such microservices are the Jasper Control Center integration and the SMS integration for sending SMS notifications to end users.

You can extend the Cloud of Things platform with customer-specific functionality by deploying microservices. For instance, you can develop integrations to third-party software or provide server-side business logic. A microservice-based architecture introduces change that is often well received by those developing modern applications, and solutions can be delivered much more quickly to those requesting flexible and scalable applications. Microservices bring significant benefits such as deployability, reliability, availability, scalability, modifiability and management.

Cloud of Things microservices have the following properties:

The following management features are supported:

Technically, microservices are Docker containers hosted by Cloud of Things and they follow specific conventions. They are typically accessed using Cloud of Things REST API available under /service/<microservice-name>.

Developers are not restricted to any programming language when developing a microservice for Cloud of Things. However, a microservice must serve as an HTTP server working on port 80 and must be encapsulated in a Docker image. Refer to the relevant chapters in this guide for further information for the development of microservices.

Containerization and orchestration

Images and containers

Docker is a platform to develop, deploy and run applications with containers. An image is an executable package that includes everything needed to run an application (that is, the code, a runtime, libraries, environment variables and configuration files). A container is a runtime instance of an image (that is, what the image becomes in memory when executed). Refer to the Docker documentation for more information about Docker.

Cloud of Things microservices are based on Docker. Hence, a microservice must be packaged as a Docker image in order to run on the Cloud of Things platform. A microservice is executed in a Docker container during runtime. The Docker container ensures that a microservice does not harm other microservices running in Cloud of Things.

Containers have an upper thread limit of 10240 for microservices.

Pods

Kubernetes is the container orchestration engine for automating deployment, scaling and management of containerized applications. A Pod is the basic building block of Kubernetes and it represents a running process on your cluster. A Pod encapsulates an application container, storage resources, a unique network IP and options that govern how the container should run.

Docker is the most common container runtime used in a Kubernetes Pod. Moreover, Kubernetes is used to orchestrate Docker containers and it provides many enterprise-grade features for hosting Docker containers such as auto-scaling and load balancing. Refer to the Kubernetes documentation for more information about Kubernetes.

When Docker faces some issues, for example, a Pod synchronization error, an alarm is created and can be seen in Alarms in the Cockpit application. Refer to Troubleshooting in this section to learn about common issues.

Requirements and interactions

The following requirements towards Cloud of Things microservices must be met:

Microservices interact with Cloud of Things and the outside world as shown in the following diagram:

Microservices interactions

The Microservices’ lifecycle is managed using the microservice subscription API. This allows registration and subscription of Microservices. External actors (for example web user interfaces, integrations or other microservices) can invoke a microservice by sending REST or Websocket requests to its endpoints /service/<microservice-name>/<path>. A microservice can issue requests to external endpoints or to the Cloud of Things REST APIs. Microservices can store logs and metrics in the associated developer tenant.

Privacy and Security Assessment (PSA)

The Privacy and Security Assessment (PSA) process is a core element in safeguarding security and data privacy at Deutsche Telekom.

Important: Microservices need to pass Deutsche Telekom the Privacy and Security Assessment (PSA) process before they can be deployed on Cloud of Things.

Data privacy and security are continuing to grow in significance. This fact is illustrated by the rapid rise in politically and criminally motivated cyber attacks. Around the world, individuals, organizations, and corporation, such as Deutsche Telekom, are being targeted. Data privacy and security are particularly pivotal for Deutsche Telekom, since our services are based on our customers’ trust in security.

Deutsche Telekom recognized the resulting challenges at an early stage and responded by developing and introducing a tailored solution: the Privacy and Security Assessment (PSA).

The PSA process guarantees that all development projects meet the strictest technical security and data privacy requirements.

The PSA process has the following goals:

A first important step is the identification of relevant privacy and security requirements. The requirements are compiled by the security and privacy experts of Deutsche Telekom on basis of vendor specifications and industry standards as well as valid laws and regulations. They are group-wide aligned and enforced. Next step of the PSA process is implementing, testing and documentation of the requirements. For not implementable requirements a automatic risk assessment on basis of the protection need of the individual system will be performed. If necessary an action plan will be prepared together with the project team.

The PSA process guarantees for all relevant projects and systems the implementation of groups security and privacy standards and an approval by security and privacy experts before bringing in service.

Security requirements

Note: Security requirements can be downloaded here (zip, 2.0 MB).

Microservice manifest

The application manifest provides the required settings to manage microservice instances and the application deployment in the Cloud of Things platform. The definition is provided within the cumulocity.json file in the binary uploaded to the Cloud of Things platform.

Here is an example manifest:

{
    "apiVersion": "v2",
    "name": "my-microservice",
    "version": "1.0.0",
    "provider": {
        "name": "New Company Ltd.",
        "domain": "https://new-company.com",
        "support": "support@new-company.com"
    },
    "isolation": "MULTI_TENANT",
    "scale": "AUTO",
    "replicas": 2,
    "resources": {
        "cpu": "1",
        "memory": "1G"
    },
    "requestedResources":{
            "cpu": "100m",
            "memory": "128Mi"
    },
    "requiredRoles": [
        "ROLE_ALARM_READ"
    ],
    "livenessProbe": {
        "httpGet": {
            "path": "/health"
        },
        "initialDelaySeconds": 60,
        "periodSeconds": 10
    },
    "readinessProbe": {
        "httpGet": {
            "path": "/health",
            "port": 80

        },
        "initialDelaySeconds": 20,
        "periodSeconds": 10
    },
    "settingsCategory": "my-ms",
    "settings": [
        {
            "key": "tracker-id",
            "defaultValue": "1234"
        }
    ]
}

See below for detailed information about available settings.

Settings

Name Type Description Required
apiVersion String Document type format discriminator. The accepted values are positive integer numbers proceeded by an optional "v", such as "v2" and "2". Values which do not conform to this convention are considered "v1". Yes
name String Application name No
contextPath String Microservice contextPath is used to define extension points.
Default: Microservice name
No
version String Application version. Must be a correct SemVer value but the "+" sign is disallowed. Yes
provider Provider Application provider information. Simple name allowed for predefined providers, for example, c8y. Detailed object for external provider. Yes
billingMode Enum Values: RESOURCES, SUBSCRIPTION
Default: RESOURCES

In case of RESOURCES, the number of resources used is exposed for billing calculation per usage. In case of SUBSCRIPTION, all resources usage is counted for the microservice owner and the subtenant is charged for subscription.
No
isolation Enum Values: MULTI_TENANT, PER_TENANT
Default: MULTI_TENANT

Deployment isolation. In case of PER_TENANT, there is a separate instance for each tenant; otherwise, there is one single instance for all subscribed tenants. Should be overridable on subscription and should affect billing.
No
scale Enum Values: AUTO, NONE
Default: NONE

Enables scaling policy. See Isolation and scaling for more details.
No
replicas Integer Value range: 1 - 5

Defines the number of microservice instances. For auto-scaled microservices, the value represents the minimum number of microservices instances.
No
resources Resources Configuration for resources limits.
Default limits are CPU=0.5, Memory=512MB. Different default values may be configured by the system administrator.
No
requestedResources RequestedResources Intended configuration for minimal required resources.
The values may be over-written based on system settings.
Default values are CPU=0.25, Memory=256MB. Different default values may be configured by the system administrator.
No
settings Option[ ] Set of tenant options available to define the configuration of a microservice.
Default: [ ] (empty list)
No
settingsCategory String Allows to specify custom category for microservice settings. By default contextPath is used. No
requiredRoles String[ ] List of permissions required by a microservice to work. No
roles String[ ] Roles provided by the microservice.
Default: [ ] (empty list)
No
livenessProbe Probe Defines the strategy used to verify if a microservice is alive or requires a restart. No
readinessProbe Probe Defines the strategy used to verify if a microservice is ready to accept traffic. No
extensions Extension[ ] Defines a set of extensions that should be enabled for a microservice.
Default: [ ] (empty list)
No

Version

The version has an impact on the microservice upload behavior:

The snapshot postfix means that the image build is a snapshot of your application at a given time and it is still under development. When your microservice is ready for production release, you can remove the postfix and just use the final version of your application.

Provider

Name Type Description Required
name String Company name of the provider Yes
domain String Website of the provider No
support Email Email of the support person No

Resources

Name Type Description Required
cpu String Limit for number of CPUs or CPU time
Default CPU: 0.5, min: 0.1
Default CPU time: 500m, min: 100m
A different default value may be configured by the system administrator.
No
memory String Limit for microservice memory usage
Default: 512M, Min: 10M
Possible units are: E, P, T, G, M, K, Ei, Pi, Ti, Gi, Mi, Ki
A different default value may be configured by the system administrator.
No

RequestedResources

Name Type Description Required
cpu String Intended minimal requirements for number of CPUs or CPU time
The value may be over-written based on system settings.
Default: 250m
A different default value may be configured by the system administrator.
No
memory String Intended minimal requirements for microservice memory usage
The value may be over-written based on system settings.
Default: 256M
Possible postfix values are: E, P, T, G, M, K, Ei, Pi, Ti, Gi, Mi, Ki
A different default value may be configured by the system administrator.
No

Option

Name Type Description Required
key String Key of the option Yes
defaultValue String Default value Yes
editable Boolean Defines if the option can be changed by a subscribed tenant on runtime
Default: false
No
overwriteOnUpdate Boolean Defines if an editable option is reset upon microservice update
Default: true
No
inheritFromOwner Boolean Specifies if an option should be inherited from the owner
Default: true
No

Probe

Name Type Description Required
exec ExecAction Commands to be executed on a container to probe the service No
tcpSocket TCPSocketAction TCP socket connection attempt as a probe No
httpGet HTTPGetAction HTTP request to be executed as a probe No
initialDelaySeconds Number Tells the platform for how long it should wait before performing
the first probe
Default: 200
No
periodSeconds Number Defines in which interval the probe should be executed
Default: 10
No
successThreshold Number Minimum consecutive successes for the probe to be considered
successful after having failed
Default: 1
No
timeoutSeconds Number Number of seconds after which the probe times out
Default: 1
No
failureThreshold Number Number of failed probes after which an action should be taken
Default: 3
No

ExecAction

Name Type Description Required
command String[ ] Commands to be executed on a container to probe the service Yes

TCPSocketAction

Name Type Description Required
host String Host to verify Yes
port Number Port to verify
Default:80
Yes

HTTPGetAction

Name Type Description Required
host String Host name to connect to Yes
path String Path to access on the HTTP server Yes
port Number Port to verify
Default: 80
No
scheme String Scheme to use for connecting to the host (HTTP or HTTPS)
Default: HTTP
No
headers HttpHeader HTTP headers to be added to a request No

HttpHeader

Name Type Description Required
name String Header name Yes
value String Header value Yes

Extension

Name Type Description Required
type String Type ID of the extension Yes
* any Configuration parameters No

Isolation and scaling

The following isolation levels are available for microservices:

See Microservice manifest > Settings for the isolation setting.

In case the scale setting is set to NONE, the platform guarantees that there is one instance of the service per isolation level by default. The number of guaranteed instances can be increased by defining the replicas setting in the manifest. If scaling is enabled (set to AUTO), the microservice will be horizontally auto-scaled (creating more instances of the microservice) in case of high CPU usage. Auto-scaling monitors the microservices to make sure that they are operating at the desired performance levels, and it will automatically scale up your cluster as soon as you need it and scale it back down when you don’t.

See Microservice manifest > Settings for the scale and replicas setting.

Microservice migration to API Version 2

With release 10.15, DT IoT announces the availability of microservice API Version 2 and the deprecation of API Version 1 to comply with new security requirements. Microservice API Version 2 provides an improved microservice container security context restricting the invocation of privileged Linux Kernel APIs. In details this means that with microservice API Version 2 the microservice container is granted only specific capabilities.

Refer to the Linux manual page for more information on Kernel capabilities. With the API Version change, the microservice is granted the capability NET_BIND_SERVICE.

Microservice migration

Perform the following steps to migrate your microservice to the new API Version.

  1. Change the API Version to 2 in the microservice manifest. See Microservice manifest.

  2. Deploy your microservice to the test environment.

  3. Test the functionality of your microservice and analyze possible errors.

In the simplest case it is sufficient to set the API Version to 2 in your microservice manifest.

However, for microservices which currently make use of specific privileges of the Linux Kernel API that are not granted anymore, you additionally need to refactor the source code so that the service doesn’t require the invocation of these privileges.

Affected microservices

Set the API Version field in the microservice manifest to 2 and deploy this service to your Cumulocity test environment. This environment must be in version 10.15 or higher. Verify that the functionality provided by the microservice still works as expected.

Microservices still using old user privileges after the environment upgrade

If your microservice is using the deprecated API Version 1, it may be affected by the change depending on the configuration of this environment. In this case, you will not be able to upload or subscribe the microservice.

Security

Microservices typically provide a REST API and Cloud of Things provides a light API gateway (Proxy) for inbound REST requests. Inbound WebSocket requests are supported. The API gateway, located between the client and the microservice container, provides:

Authentication and authorization

A request to a microservice can be authenticated using various authentication mechanisms supported by Cloud of Things such as:

  1. Basic authentication
  2. OAI-Secure
  3. SSO
  4. JWT token authentication (deprecated)

If you use Java for your development, we recommend you to use the Microservice SDK version 10.4.6 or later to gain support for all available authentication ways in Cloud of Things.

In case of other programming languages, for which Cloud of Things does not provide an SDK, we recommend developers to ensure support for all authentication mechanisms available in the Cloud of Things platform.

The Cloud of Things exposes the REST endpoint /user/currentUser. The microservice retrieves the Cloud of Things address from the C8Y_BASEURL operating system environment variable. The microservice can verify if the credentials embedded in the HTTP request, which is handled by the microservice, are valid or not by using the REST endpoint /user/currentUser. If the credentials are correct, the response status from the /user/currentUser endpoint is 200 OK. In case of incorrect credentials, the response contains a 401 Unauthorized status code. In order to verify if the credentials are correct, the microservice copies the credentials from ongoing requests and sends them to the /user/currentUser endpoint.

Depending on the authentication method, the credentials can be passed to the microservice in various ways, such as:

  1. Authorization HTTP header
  2. Cookie called authorization
  3. Custom HTTP header called X-XSRF-TOKEN
  4. Custom HTTP header called tfatoken

If the incoming request contains the cookie authorization, the microservice has to copy the cookie and the header X-XSRF-TOKEN to the request to the /user/currentUser endpoint. In other cases, the header Authorization has to be copied. This is necessary, if a request contains the header tfatoken, which always needs to be included in the request to /user/currentUser.

You can see the credential validation flow on the sequence diagram below:

OAuth

Example

The microservice can receive the HTTP requests with the following header (see example below). The authentication process is based on basic authentication and SMS based TFA.

GET https://cumulocity.default.svc.cluster.local/test HTTP/1.1
accept: */*
authorization: Basic dGVuYW50X2lkL3VzZXJuYW1lOnBhc3N3b3JkCg==
connection: close
content-length: 0
cookie: REQUEST_ORIGIN=
host: auth-scope-management.default.svc.cluster.local:80
tfatoken: 23b75292468e0ba7fe03245d502d9c29e21e8a997fc7dd7e1a1df7fe31cbfb17
x-forwarded-host: cumulocity.default.svc.cluster.local:80
x-forwarded-prefix: /service/auth
x-forwarded-proto: http
x-real-ip: 192.168.1.20

The microservice sends the following requests to validate the credentials (provided that c8y_baseurl is equal to https://cumulocity:8111):

GET https://cumulocity:8111/user/currentUser HTTP/1.1
Accept: text/plain, application/json, application/*+json, */*
Authorization: Basic dGVuYW50X2lkL3VzZXJuYW1lOnBhc3N3b3JkCg==
Content-Length: 0
tfatoken: 23b75292468e0ba7fe03245d502d9c29e21e8a997fc7dd7e1a1df7fe31cbfb17

If the user is authenticated with OAI-Secure, the following REST request can reach the microservice:

GET https://cumulocity.default.svc.cluster.local/test HTTP/1.1
accept: */*
authorization: Basic dGVuYW50X2lkL3VzZXJuYW1lOjxmYWtlIHBhc3N3b3JkPgo=
connection: close
content-length: 0
cookie: authorization=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI0YzAwMGVlOC1hYjY5LTRiMzYtYWNhNC0xNzM4ZGYwZWNhZGIiLCJpc3MiOiJjdW11bG9jaXR5LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJhdWQiOiJjdW11bG9jaXR5LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJzdWIiOiJhZG1pbiIsInRjaSI6IjMxMWM3YjZiLWM1MmQtNGEzMC1iZDFlLWNiODdlMjNlODQyOSIsImlhdCI6MTY0NjkyODgxMywibmJmIjoxNjQ2OTI4ODEzLCJleHAiOjE2NDgxMzg0MTMsInRmYSI6ZmFsc2UsInRlbiI6Im1hbmFnZW1lbnQiLCJ4c3JmVG9rZW4iOiJHU0l6VFB4b2JQS1FJQmV2Uk1SWiJ9.NL5b9-iHSc3SLaPu87oazBd2_kjNiwO9tM_bXS3qCUd0_wTZ-BKc3q6sRHTKO_LNbtCQg3G6-7ZhD6TyvqjTLsyiZTsgMRtAE7ZiRPtXaFOA0_SDQ9kG_MztAZ3U9O008akgXcjEEAdphVv5eey_lADrg1BmIlqiBFoKts2JGSv1xFtXKIxpcVtRDGUkb-2qb8OhaHamT7b3HL628NSiH4VqfO38vkLLkimHEz-roqmbFGQ355TvA3-s_erO96j3rHbBPDsluFqFN0eOTCidBffKt6OvyKw-_64MaHHs6R9Ulsv-LuY-YAvlTZVxYwFAi3yn3mWlpXEAzvGYHMrx8A
host: auth-scope-management.default.svc.cluster.local:80
user-agent: insomnia/2022.1.1
x-forwarded-host: cumulocity.default.svc.cluster.local:80
x-forwarded-prefix: /service/auth
x-forwarded-proto: http
x-real-ip: 192.168.1.20
x-xsrf-token: GSIzTPxobPKQIBevRMRZ

To verify the credentials from the request above, the microservice sends the following request:

GET https://cumulocity:8111/user/currentUser HTTP/1.1
Accept: text/plain, application/json, application/*+json, */*
Content-Length: 0
Cookie: authorization=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJqdGkiOiI0YzAwMGVlOC1hYjY5LTRiMzYtYWNhNC0xNzM4ZGYwZWNhZGIiLCJpc3MiOiJjdW11bG9jaXR5LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJhdWQiOiJjdW11bG9jaXR5LmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJzdWIiOiJhZG1pbiIsInRjaSI6IjMxMWM3YjZiLWM1MmQtNGEzMC1iZDFlLWNiODdlMjNlODQyOSIsImlhdCI6MTY0NjkyODgxMywibmJmIjoxNjQ2OTI4ODEzLCJleHAiOjE2NDgxMzg0MTMsInRmYSI6ZmFsc2UsInRlbiI6Im1hbmFnZW1lbnQiLCJ4c3JmVG9rZW4iOiJHU0l6VFB4b2JQS1FJQmV2Uk1SWiJ9.NL5b9-iHSc3SLaPu87oazBd2_kjNiwO9tM_bXS3qCUd0_wTZ-BKc3q6sRHTKO_LNbtCQg3G6-7ZhD6TyvqjTLsyiZTsgMRtAE7ZiRPtXaFOA0_SDQ9kG_MztAZ3U9O008akgXcjEEAdphVv5eey_lADrg1BmIlqiBFoKts2JGSv1xFtXKIxpcVtRDGUkb-2qb8OhaHamT7b3HL628NSiH4VqfO38vkLLkimHEz-roqmbFGQ355TvA3-s_erO96j3rHbBPDsluFqFN0eOTCidBffKt6OvyKw-_64MaHHs6R9Ulsv-LuY-YAvlTZVxYwFAi3yn3mWlpXEAzvGYHMrx8A
X-XSRF-TOKEN: GSIzTPxobPKQIBevRMRZ

If a request to the /user/currentUser REST endpoint contains the correct credentials, the response body contains various information, such as user roles assigned to the user account. This information can be very useful in order for the microservice to carry out the authorization process.

When a microservice receives an HTTP request, it can become necessary to use the REST API exposed by Cloud of Things to fulfil the request. To access the REST API, the microservice must provide a valid credential. It can use credentials that are present in the incoming REST request or use a dedicated user account, created to access resources belonging to tenants which are subscribed to the microservice. The choice between the credentials provided together with the incoming request to the microservice REST or a dedicated account depends on the use case. In order to utilize the user credentials received by the microservice in an incoming HTTP request, the microservice must copy the credentials according to the rules described above.

Furthermore, most of the HTTP requests which are sent by microservices must contain the HTTP header X-Cumulocity-Application-Key. The header indicates that an action is performed by an application. When the header is missing Cloud of Things assumes that the HTTP request is originated by an IoT device. Therefore, a request without X-Cumulocity-Application-Key affects for example, the device availability state, or the billing data.

Thus, the microservice must only include X-Cumulocity-Application-Key if the microservice proxy requests from an IoT device to the Cloud of Things platform.

If the microservice includes the header X-Cumulocity-Application-Key, the header must contain the correct application key. Then, the microservice retrieves the application key by sending a REST request to the endpoint /application/currentApplication exposed by the Cloud of Things platform. The REST request must contain the credentials for basic authentication in the following format: tenantId/username:password. The tenant ID, username and password are read by the microservice from the following operating system environment variables: C8Y_BOOTSTRAP_TENANT, C8Y_BOOTSTRAP_USER, and C8Y_BOOTSTRAP_PASSWORD. In order to increase the performance, the microservice has to implement a caching mechanism related to the user credentials.

Microservice authentication and multi-tenancy

In general, microservices use the standard Cloud of Things authentication mechanisms. This is executed in two steps:

  1. The microservice can be created in any tenant that has feature-microservice-hosting enabled.
  2. The microservices access the tenant API.

At the installation time of the microservice, an application is created in the owner tenant (usually a Management tenant or Enterprise tenant) reflecting the new microservice. In addition, a bootstrap user is created in the owner tenant that allows the microservice to retrieve subscriptions. Whenever required, a platform administrator will subscribe a customer to the new microservice. As part of the subscription, a service user in the customer tenant is created using random credentials.

Microservice authorization

Authorization is relevant on two levels:

  1. On the owner tenant level, the only authorization of a microservice is to access its own subscriptions.
  2. For accessing customer tenants, the microservice installs a set of required permissions for being able to operate.

A microservice is associated with a bootstrap user in the owner tenant, which will make sure that only its subscriptions are returned. A microservice is also associated with a set of permissions that it requires for carrying out its function on a customer tenant. These permissions are visualized in the Administration application. The permissions are associated with the service user that is created when a platform administration associates a microservice with a tenant (subscribes to it). The owner tenant must also be subscribed to if you intend to use microservice features on it.

Users and roles

There are three types of users:

The following role types are defined for users:

Custom roles must adhere to this name format in order to be shown in the UI:

ROLE__(READ|ADMIN|CREATE)

You can add them to the application manifest in the roles properties as follows:

"roles": [
    "ROLE_MY_MICROSERVICE_READ",
    "ROLE_MY_MICROSERVICE_ADMIN"
]

The roles are set in the Microservice manifest. For more details about users and roles, review the User API in the Cloud of Things OpenAPI Specification.

Microservice bootstrap

Each microservice receives a dedicated bootstrap user which ensures that a microservice can be identified by the platform and can have access only to the allowed resources. A microservice runtime provides bootstrap user and service user credentials as environment variables which can also be acquired via platform API. Note that depending on the isolation level, the environment variables differ.

Variable Description Per tenant scope Multi-tenant scope
C8Y_BOOTSTRAP_TENANT Application owner tenant ID x x
C8Y_BOOTSTRAP_USER Username of the bootstrap user x x
C8Y_BOOTSTRAP_PASSWORD Password of the bootstrap user x x
C8Y_TENANT Subscribed tenant ID x  
C8Y_USER Username of the service user of a subscribed tenant x  
C8Y_PASSWORD Password of the service user of a subscribed tenant x  

In multi-tenant scope, there is a single microservice deployment reused by multiple tenants and that is why service user credentials are not provided as hardcoded environment properties. However, a microservice running in multi-tenant isolation can retrieve all subscriptions via a GET request and using bootstrap credentials as follows:

GET /application/currentApplication/subscriptions
Host: ...
Authorization: Basic ...

Bootstrap user credentials can be retrieved with a GET request authorized with application owner credentials:

GET /application/applications/<APPLICATION_ID>/bootstrapUser
Host: ...
Authorization: Basic ...

An example of a typical user switching in multi-tenant isolation is presented below, where – in a hypothetical scenario – there is a need to send an alarm to each tenant subscribed to a microservice.

microservice_user_switch_example

The user wants to employ microservice capabilities to raise alarms to all subscribed tenants calls.

Steps:

  1. The user makes a request to the platform’s endpoint /service/<microservice>/createAlarms.
  2. The platform verifies the user credentials and redirects the request to the microservice.
  3. The microservice reads the bootstrap credentials (from environment variables) and uses them to retrieve the service user credentials for all subscribed tenants.
  4. The microservice iterates over the service user credentials and uses them to create alarms to each tenant.
  5. The microservice returns the result to the platform, and the platform to the invoking user.

Encryption

There is a mechanism to encrypt the tenant options that afterwards are automatically decrypted when injecting them into microservices requests.

If a tenant option is created with a key name that starts with “credentials.", it is automatically encrypted and can be fetched as unencrypted only by system users. For instance, when you create a tenant option in a category that matches to the application context path, the value is passed to the microservice by the microservice proxy on the platform as a header (key => value). Note that therefore a space (” “) is not allowed as a tenant option value. All encrypted options are decrypted and passed. Moreover, the options can be fetched via REST using the options endpoint at microservice runtime.

Refer to tenant options in the Tenant API in the Cloud of Things OpenAPI Specification for more details.

Microservice runtime

Microservices deployed on the platform have a specific runtime environment and they must understand certain details about the specific Cloud of Things cluster they run in. For example, a microservice needs to know the endpoint address of the Cloud of Things REST APIs. This information is provided by environment variables and they are injected by Cloud of Things when the container is started.

Environment variables

The following environment variables are available for microservices:

Name Details
APPLICATION_NAME The name of the microservice application
APPLICATION_KEY The key of the microservice application
SERVER_PORT Default open port (80)
MICROSERVICE_SUBSCRIPTION_ENABLED Default value: true
C8Y_BASEURL Platform address (contains port number)
C8Y_BASEURL_MQTT Platform address of the MQTT server (contains port number)
C8Y_MICROSERVICE_ISOLATION Isolation level (MULTI_TENANT or PER_TENANT)
C8Y_BOOTSTRAP_REGISTER Indicator whether the microservice should perform self registration or not.
Default value: false
C8Y_BOOTSTRAP_TENANT Bootstrap user tenant, for MULTI_TENANT - microservice owner
C8Y_BOOTSTRAP_USER Bootstrap username
C8Y_BOOTSTRAP_PASSWORD Bootstrap user password
C8Y_TENANT Application user tenant (available only for PER_TENANT isolation)
C8Y_USER Application username (available only for PER_TENANT isolation)
C8Y_PASSWORD Application user password (available only for PER_TENANT isolation)
MEMORY_LIMIT Max memory that can be used. Default value: 256M
TZ Timezone from the host machine or configurable tenant options
LOG4J_FORMAT_MSG_NO_LOOKUPS Disables the vulnerable Log4j lookup feature (see CVE-2021-44228)
Default value: true
Example

Prerequisite: The microservice has been packed and deployed in the Docker repository. Get the microservice image name and tag with the following command:

$ docker images

Run the Docker container for the microservice providing the environment variables:

$ docker run -–cap-drop=ALL -–cap-add=NET_BIND_SERVICE \
   -e C8Y_BOOTSTRAP_TENANT=<BOOTSTRAP_TENANT> \
   -e C8Y_BOOTSTRAP_USER=<BOOTSTRAP_USERNAME> \
   -e C8Y_BOOTSTRAP_PASSWORD=<BOOTSTRAP_USER_PASSWORD> \
   -e C8Y_MICROSERVICE_ISOLATION=MULTI_TENANT \
   -e C8Y_BASEURL=<URL> -i -t <DOCKER_REPOSITORY_IMAGE>:<TAG>

Use a backslash (\) before special characters such as &, !, ;, \.

Timezone variable

The timezone variable allows configuring a default timezone used by the microservice. The microservice installer injects the TZ environment variable into the microservice according to the following settings:

The tenant option has higher priority, that means, if the parameter is set in both places, the value from the tenant option is taken.

Example

Assuming that the microservice owner has the tenant option:

{
    "category" : "microservice.runtime",
    "key" : "timezone",
    "value" : "Europe/Warsaw"
}

Deploying and running the microservice inside Docker will result in passing the following variables into the microservice environment:

TZ=Europe/Warsaw

When using Java-based microservices this variable is automatically read and applied to the Java process, no additional work is required. Microservices developed with other programming languages may require some manual work, that is, loading the TZ value from the environment and using it to configure the time zone on the language level programmatically.

Proxy variables

Proxy variables are used to set a proxy URL for different protocols. For the microservices written in Java, setting each variable will result in passing the corresponding parameter into the JVM runtime (for detailed information see the Java Networking and Proxies webpage).

Proxy variables are passed into the microservice environment during installation. The microservice installer passes the variables into the environment according to the following settings:

Tenant options have higher priority, that means, if the parameter is set in both places, the value from the tenant option is taken.

The table below contains the options and proxy variables:

Tenant option Platform env variable Microservice env variable
proxy.http.host MICROSERVICE_RUNTIME_PROXY_HTTP_HOST PROXY_HTTP_HOST
proxy.http.port MICROSERVICE_RUNTIME_PROXY_HTTP_PORT PROXY_HTTP_PORT
proxy.http.non.proxy.hosts MICROSERVICE_RUNTIME_PROXY_HTTP_NON_PROXY_HOSTS PROXY_HTTP_NON_PROXY_HOSTS
proxy.https.host MICROSERVICE_RUNTIME_PROXY_HTTPS_HOST PROXY_HTTPS_HOST
proxy.https.port MICROSERVICE_RUNTIME_PROXY_HTTPS_PORT PROXY_HTTPS_PORT
proxy.socks.host MICROSERVICE_RUNTIME_PROXY_SOCKS_HOST PROXY_SOCKS_HOST
proxy.socks.port MICROSERVICE_RUNTIME_PROXY_SOCKS_PORT PROXY_SOCKS_PORT

All tenant options have the same category: microservice.runtime

For each protocol (HTTP, HTTPS, Socks), microservice environment variables are passed into runtime only if the HOST parameter is set. If the HOST parameter is missing, other parameters for the same protocol are not processed.

Examples

The microservice owner tenant has the tenant options:

{ "category" : "microservice.runtime", "key" : "proxy.http.host", "value" : "10.11.12.13" }
{ "category" : "microservice.runtime", "key" : "proxy.http.port", "value" : "8080" }

and there is an environment variable in the platform application:

MICROSERVICE_RUNTIME_PROXY_HTTP_PORT=8181

Deploying and running the microservice inside Docker will result in passing the following variables into the microservice environment (notice PORT value):

PROXY_HTTP_HOST=10.11.12.13
PROXY_HTTP_PORT=8080

The microservice owner tenant has the tenant option:
{ "category" : "microservice.runtime", "key" : "proxy.https.host", "value" : "10.11.12.13" }

and there is an environment variable in the platform application:

MICROSERVICE_RUNTIME_PROXY_HTTPS_PORT=8181

Deploying and running the microservice inside Docker will result in passing the following variables into the microservice environment:

PROXY_HTTP_HOST=10.11.12.13
PROXY_HTTP_PORT=8181        

The microservice owner tenant has the tenant options:
{ "category" : "microservice.runtime", "key" : "proxy.http.port", "value" : "8080" }
{ "category" : "microservice.runtime", "key" : "proxy.http.non.proxy.hosts", "value" : "localhost" }

and the proxy HOST is not set (neither in tenant option, nor env variable).

Deploying and running the microservice inside Docker will not pass any proxy environment variable.


The microservice owner tenant has the tenant option:
{ "category" : "microservice.runtime", "key" : "socks.http.host", "value" : "10.11.12.13" }  

Deploying and running the microservice inside Docker will result in passing the following variable into the microservice environment (only host parameter):

SOCKS_HTTP_HOST=10.11.12.13

Platform access and other microservices

To execute requests against the Cloud of Things platform running a microservice, you must send requests to the host specified by the C8Y_BASEURL variable.

A microservice does not have direct access to other microservices running on the platform. Instead, a microservice must use the platform as a proxy. The endpoint used to access other applications is <C8Y_BASEURL>/service/<OTHER_APPLICATION_NAME>/.

Important
C8Y_BASEURL allows access only to microservices’ REST endpoints. Hence, a microservice cannot retrieve information from UI applications.
Important

When accessing the REST endpoints the connections may time out. This requires the HTTP client of a microservice to be prepared for retries to establish a connection.

The client used in the Microservice SDK performs the required retries. Using it avoids issues with timed-out connections.

Request routing

The request is redirected to a microservice depending on the isolation level (auto-scaling is ignored at this moment for clarity), subscription and authorization. A typical request to the platform looks like

<METHOD>  /service/<MICROSERVICE>/<MICROSERVICE_ENDPOINT>
Host: ...
Authorization: Basic ...

Credentials are used to verify if a requesting user is authorized to access the microservice, and tenant subscription is verified afterwards. If both checks pass, the request is routed to a dedicated microservice deployment in case that the isolation level is per tenant, or to a shared deployment in case of multi-tenant isolation.

The routed request is stripped of /service/<MICROSERVICE> part. However, the Authorization header is not modified, thus a request is still executed as a tenant platform user.

<METHOD>  /<MICROSERVICE_ENDPOINT>
Host: ...
Authorization: Basic ...

Microservice utility tool

Cloud of Things provides you a utility tool for easy microservice packaging, deployment and subscription. The script requires a local installation of Docker and jq, which is a lightweight and flexible JSON processor.

Prerequisites

Docker

Verify that you have a Docker installation. The Docker version must be >= 1.2.6

$ docker version
Client:
 Version:         1.12.6
 API version:     1.24
 OS/Arch:         linux/amd64

Server:
 Version:         1.12.6
 API version:     1.24
 OS/Arch:         linux/amd64

JSON processor

Execute the following command to install the JSON processor on Linux systems:

$ sudo yum install jq

For macOS, use the following command:

$ brew install jq

Bash

The microservice utility tool (script) needs Bash version 4+ to run. Verify your Bash version with the following command:

$ bash --version

GNU bash, version 5.0.3(1)-release (x86_64-apple-darwin18.2.0)
Copyright (C) 2019 Free Software Foundation, Inc.

macOS systems come with a preinstalled Bash version 3.x. Hence, you must update it in order to execute the microservice script. To do so, execute the following commands:

$ brew install bash
$ chsh -s /usr/local/bin/bash

If your Bash version has not changed while executing bash --version, you may need to restart your system. Note that the updated interpreter gets installed at /usr/local/bin/bash and you must modify the first line of the microservice utility tool (script) as follows:

#!/usr/local/bin/bash

or

#!

Configure the microservice utility tool

The script can be found in the GitHub repository: cumulocity-examples.

Change the mode to allow the script to be executed:

$ chmod +x microservice

Use the help option to see all the available functions (goals) and options:

$ ./microservice help

Packing

A microservice must be packed as a Docker image in order to be deployed. It requires a Docker image.tar and cumulocity.json files packed into a ZIP file.

The following directory structure is required to pack a microservice:

/docker/Dockerfile      # Instructions to build the Docker image
/docker/*               # All files within the directory will be included in the Docker build
/cumulocity.json        # The application manifest file

The script can be run in a parent folder holding such structure, or by passing the path to the directory using the -dir option. For instance, to pack a “Hello World” microservice application, execute:

$ ./microservice pack --name hello-world

It will create a ZIP file named hello-world.zip and an intermediate image.tar which is an exported Docker image.

Important
When naming your microservice application use only lower-case letters, digits and dashes. The maximum length for the name is 23 characters.

Deploying

A microservice becomes available once it has been successfully deployed on the Cloud of Things platform. This is done by uploading a ZIP file with the microservice packed as specified above. A user cannot directly push an image to the Docker registry.

Deploying your microservice application is rather easy, just execute the following command:

$ ./microservice deploy -n hello-world -d <URL> -u <username> -p <password> -te <tenant>

Note that you must have a tenant and user credentials in order to deploy your microservice.
The successful execution will create an application on the Cloud of Things platform with the specified name, if it does not exist yet. Then it will upload the hello-world.zip file into the platform. Once it has been uploaded, your application will be listed in Ecosystem > Microservices in the Administration application.

If the name of the microservice application is not provided in the manifest file, it will be automatically inferred from the ZIP file name without the version number.

For further information on deploying microservices to Cloud of Things, refer to Administration > Managing and monitoring microservices in the User guide.

Subscribing

You must subscribe to the application in order to use it. Execute the following command to subscribe your tenant to the deployed microservice:

$ ./microservice subscribe -n hello-world -d <URL> -u <username> -p <password> -te <tenant> -id <APPLICATION_ID>

It will result in tenant subscription to an application specified by the ID parameter. If the user has already been subscribed, a warning message will be displayed.

Multiple goals

Goals can be executed together to pack, deploy and subscribe the application in a single line. In this case, the application ID will be automatically pulled by the script.

$ ./microservice pack deploy subscribe -n hello-world -d <URL> -u <username> -p <password> -te <tenant>

Operating microservices

Cloud of Things manages microservices by monitoring the microservice instance and storing the metrics. In case a microservice exceeds the memory limit, it is restarted automatically. Also, microservices can be auto-scaled in case of high CPU usage. For more information, review the scaling details above.

Troubleshooting

Some common issues have been identified and documented below.

I get an error saying “Microservice application name is incorrect” on uploading a microservice application

When naming your microservice application only use lower-case letters, digits and dashes. The maximum length for the name is 23 characters.

I deployed my microservice but requests to any endpoint returns an error message “Microservice not available Connection refused”

After uploading the microservice, the internal deployment and container run may take a couple of minutes. Once completed, the error message will disappear. Meanwhile, enjoy your coffee ☕️

An alarm was created with the message “Failed to pull image … rpc error: code = Canceled desc = context canceled”

This may happen occasionally when pulling large Docker images for containers within a pod. It will go to normal once it has been successfully pulled.

An alarm was created with the message “Pod synchronization error.”

This may happen when Kubernetes is performing the auto-scaling and is trying to restart the pods.

An alarm was created with the message “No nodes are available that match all of the predicates: [Insufficient cpu …]”

There are additional containers running besides yours and they are provisioned by default with Kubernetes. These additional containers might be taking up of the CPU quota of the single node. Kubernetes will manage this and change the pod status from Pending to Running.