Thursday, August 11, 2022
HomeCloud ComputingChatOps: Managing Kubernetes Deployments in Webex

ChatOps: Managing Kubernetes Deployments in Webex


That is the third publish in a sequence about writing ChatOps companies on prime of the Webex API.  Within the first publish, we constructed a Webex Bot that obtained message occasions from a gaggle room and printed the occasion JSON out to the console.  In the second, we added safety to that Bot, including an encrypted authentication header to Webex occasions, and subsequently including a easy record of approved customers to the occasion handler.  We additionally added consumer suggestions by posting messages again to the room the place the occasion was raised.

On this publish, we’ll construct on what was accomplished within the first two posts, and begin to apply real-world use circumstances to our Bot.  The aim right here will probably be to handle Deployments in a Kubernetes cluster utilizing instructions entered right into a Webex room.  Not solely is that this a enjoyable problem to unravel, however it additionally offers wider visibility into the goings-on of an ops group, as they will scale a Deployment or push out a brand new container model within the public view of a Webex room.  Yow will discover the finished code for this publish on GitHub.

This publish assumes that you just’ve accomplished the steps listed within the first two weblog posts.  Yow will discover the code from the second publish right here.  Additionally, essential, you’ll want to learn the primary publish to learn to make your native growth atmosphere publicly accessible in order that Webex Webhook occasions can attain your API.  Make certain your tunnel is up and operating and Webhook occasions can circulation by way of to your API efficiently earlier than continuing on to the subsequent part.  On this case, I’ve arrange a brand new Bot known as Kubernetes Deployment Supervisor, however you should use your present Bot should you like.  From right here on out, this publish assumes that you just’ve taken these steps and have a profitable end-to-end information circulation.


Let’s check out what we’re going to construct:

Architecture Diagram

Constructing on prime of our present Bot, we’re going to create two new companies: MessageIngestion, and Kubernetes.  The latter will take a configuration object that provides it entry to our Kubernetes cluster and will probably be accountable for sending requests to the K8s management aircraft.  Our Index Router will proceed to behave as a controller, orchestrating information flows between companies.  And our WebexNotification service that we constructed within the second publish will proceed to be accountable for sending messages again to the consumer in Webex.

Our Kubernetes Sources

On this part, we’ll arrange a easy Deployment in Kubernetes, in addition to a Service Account that we are able to leverage to speak with the Kubernetes API utilizing the NodeJS SDK.  Be at liberty to skip this half if you have already got these assets created.

This part additionally assumes that you’ve got a Kubernetes cluster up and operating, and each you and your Bot have community entry to work together with its API.  There are many assets on-line for getting a Kubernetes cluster arrange, and getting kubetcl put in, each of that are past the scope of this weblog publish.

Our Check Deployment

To maintain factor easy, I’m going to make use of Nginx as my deployment container – an easily-accessible picture that doesn’t have any dependencies to rise up and operating.  When you have a Deployment of your personal that you just’d like to make use of as an alternative, be happy to exchange what I’ve listed right here with that.

# in assets/nginx-deployment.yaml
apiVersion: apps/v1
variety: Deployment
    identify: nginx-deployment
      app: nginx
  replicas: 2
      app: nginx
      app: nginx
    - identify: nginx
      picture: nginx:1.20
      - containerPort: 80

Our Service Account and Function

The following step is to verify our Bot code has a means of interacting with the Kubernetes API.  We are able to try this by making a Service Account (SA) that our Bot will assume as its identification when calling the Kubernetes API, and making certain it has correct entry with a Kubernetes Function.

First, let’s arrange an SA that may work together with the Kubernetes API:

# in assets/sa.yaml
apiVersion: v1
variety: ServiceAccount
  identify: chatops-bot

Now we’ll create a Function in our Kubernetes cluster that may have entry to just about every little thing within the default Namespace.  In a real-world utility, you’ll possible wish to take a extra restrictive strategy, solely offering the permissions that permit your Bot to do what you propose; however wide-open entry will work for a easy demo:

# in assets/position.yaml
variety: Function
  namespace: default
  identify: chatops-admin
- apiGroups: ["*"]
  assets: ["*"]
  verbs: ["*"]

Lastly, we’ll bind the Function to our SA utilizing a RoleBinding useful resource:

# in assets/rb.yaml
variety: RoleBinding
  identify: chatops-admin-binding
  namespace: default
- variety: ServiceAccount
  identify: chatops-bot
  apiGroup: ""
  variety: Function
  identify: chatops-admin
  apiGroup: ""

Apply these utilizing kubectl:

$ kubectl apply -f assets/sa.yaml
$ kubectl apply -f assets/position.yaml
$ kubectl apply -f assets/rb.yaml

As soon as your SA is created, fetching its information will present you the identify of the Secret by which its Token is saved.

Screenshot of the Service Account's describe output

Fetching information about that Secret will print out the Token string within the console.  Watch out with this Token, because it’s your SA’s secret, used to entry the Kubernetes API!

The secret token value

Configuring the Kubernetes SDK

Since we’re writing a NodeJS Bot on this weblog publish, we’ll use the JavaScript Kubernetes SDK for calling our Kubernetes API.  You’ll discover, should you have a look at the examples within the Readme, that the SDK expects to have the ability to pull from a neighborhood kubectl configuration file (which, for instance, is saved on a Mac at ~/.kube/config).  Whereas that may work for native growth, that’s not best for Twelve Issue growth, the place we usually go in our configurations as atmosphere variables.  To get round this, we are able to go in a pair of configuration objects that mimic the contents of our native Kubernetes config file and might use these configuration objects to imagine the identification of our newly created service account.

Let’s add some atmosphere variables to the AppConfig class that we created within the earlier publish:

// in config/AppConfig.js
// contained in the constructor block
// after earlier atmosphere variables

// no matter you’d like to call this cluster, any string will do
this.clusterName = course of.env['CLUSTER_NAME'];
// the bottom URL of your cluster, the place the API may be reached
this.clusterUrl = course of.env['CLUSTER_URL'];
// the CA cert arrange on your cluster, if relevant
this.clusterCert = course of.env['CLUSTER_CERT'];
// the SA identify from above - chatops-bot
this.kubernetesUserame = course of.env['KUBERNETES_USERNAME'];
// the token worth referenced within the screenshot above
this.kubernetesToken = course of.env['KUBERNETES_TOKEN'];

// the remainder of the file is unchanged…

These 5 strains will permit us to go configuration values into our Kubernetes SDK, and configure a neighborhood consumer.  To try this, we’ll create a brand new service known as KubernetesService, which we’ll use to speak with our K8s cluster:

// in companies/kubernetes.js

import {KubeConfig, AppsV1Api, PatchUtils} from '@kubernetes/client-node';

export class KubernetesService {
    constructor(appConfig) {
        this.appClient = this._initAppClient(appConfig);
        this.requestOptions = { "headers": { "Content material-type": 

    _initAppClient(appConfig) { /* we’ll fill this in quickly */  }

    async takeAction(k8sCommand) { /* we’ll fill this in later */ }

This set of imports on the prime provides us the objects and strategies that we’ll want from the Kubernetes SDK to rise up and operating.  The requestOptions property set on this constructor will probably be used after we ship updates to the K8s API.

Now, let’s populate the contents of the _initAppClient methodology in order that we are able to have an occasion of the SDK prepared to make use of in our class:

// contained in the KubernetesService class
_initAppClient(appConfig) {
    // constructing objects from the env vars we pulled in
    const cluster = {
        identify: appConfig.clusterName,
        server: appConfig.clusterUrl,
        caData: appConfig.clusterCert
    const consumer = {
        identify: appConfig.kubernetesUserame,
        token: appConfig.kubernetesToken,
    // create a brand new config manufacturing facility object
    const kc = new KubeConfig();
    // go in our cluster and consumer objects
    kc.loadFromClusterAndUser(cluster, consumer);
    // return the consumer created by the manufacturing facility object
    return kc.makeApiClient(AppsV1Api);

Easy sufficient.  At this level, we have now a Kubernetes API consumer prepared to make use of, and saved in a category property in order that public strategies can leverage it of their inside logic.  Let’s transfer on to wiring this into our route handler.

Message Ingestion and Validation

In a earlier publish, we took a have a look at the total payload of JSON that Webex sends to our Bot when a brand new message occasion is raised.  It’s price looking once more, since this can point out what we have to do in our subsequent step:

Message event body

For those who look by way of this JSON, you’ll discover that nowhere does it record the precise content material of the message that was despatched; it merely provides occasion information.  Nevertheless, we are able to use the area to name the Webex API and fetch that content material, in order that we are able to take motion on it.  To take action, we’ll create a brand new service known as MessageIngestion, which will probably be accountable for pulling in messages and validating their content material.

Fetching Message Content material

We’ll begin with a quite simple constructor that pulls within the AppConfig to construct out its properties, one easy methodology that calls a few stubbed-out personal strategies:

// in companies/MessageIngestion.js
export class MessageIngestion {
    constructor(appConfig) {
        this.botToken = appConfig.botToken;

    async determineCommand(occasion) {
        const message = await this._fetchMessage(occasion);
        return this._interpret(message);

    async _fetchMessage(occasion) { /* we’ll fill this in subsequent */ }

    _interpret(rawMessageText) { /* we’ll discuss this */ }

We’ve obtained a superb begin, so now it’s time to write down our code for fetching the uncooked message textual content.  We’ll name the identical /messages endpoint that we used to create messages within the earlier weblog publish, however on this case, we’ll fetch a particular message by its ID:

// in companies/MessageIngestion.js
// contained in the MessageIngestion class

// discover we’re utilizing fetch, which requires NodeJS 17.5 or increased, and a runtime flag
// see earlier publish for more information
async _fetchMessage(occasion) {
    const res = await fetch("" +, {
        headers: {
            "Content material-Sort": "utility/json",
            "Authorization": `Bearer ${this.botToken}`
        methodology: "GET"
    const messageData = await res.json();
    if(!messageData.textual content) {
        throw new Error("Couldn't fetch message content material.");
    return messageData.textual content;

For those who console.log the messageData output from this fetch request, it can look one thing like this:

The messageData object

As you may see, the message content material takes two varieties – first in plain textual content (identified with a purple arrow), and second in an HTML block.  For our functions, as you may see from the code block above, we’ll use the plain textual content content material that doesn’t embrace any formatting.

Message Evaluation and Validation

This can be a advanced subject to say the least, and the complexities are past the scope of this weblog publish.  There are lots of methods to research the content material of the message to find out consumer intent.  You can discover pure language processing (NLP), for which Cisco presents an open-source Python library known as MindMeld.  Or you possibly can leverage OTS software program like Amazon Lex.

In my code, I took the straightforward strategy of static string evaluation, with some inflexible guidelines across the anticipated format of the message, e.g.:

<tagged-bot-name> scale <name-of-deployment> to <number-of-instances>

It’s not essentially the most user-friendly strategy, however it will get the job accomplished for a weblog publish.

I’ve two intents accessible in my codebase – scaling a Deployment and updating a Deployment with a brand new picture tag.  A swap assertion runs evaluation on the message textual content to find out which of the actions is meant, and a default case throws an error that will probably be dealt with within the index route handler.  Each have their very own validation logic, which provides as much as over sixty strains of string manipulation, so I gained’t record all of it right here.  For those who’re excited about studying by way of or leveraging my string manipulation code, it may be discovered on GitHub.

Evaluation Output

The joyful path output of the _interpret methodology is a brand new information switch object (DTO) created in a brand new file:

// in dto/KubernetesCommand.js
export class KubernetesCommand {
    constructor(props = {}) {
        this.kind = props.kind;
        this.deploymentName = props.deploymentName;
        this.imageTag = props.imageTag;
        this.scaleTarget = props.scaleTarget;

This standardizes the anticipated format of the evaluation output, which may be anticipated by the varied command handlers that we’ll add to our Kubernetes service.

Sending Instructions to Kubernetes

For simplicity’s sake, we’ll give attention to the scaling workflow as an alternative of the 2 I’ve obtained coded.  Suffice it to say, that is not at all scratching the floor of what’s attainable along with your Bot’s interactions with the Kubernetes API.

Making a Webex Notification DTO

The very first thing we’ll do is craft the shared DTO that may comprise the output of our Kubernetes command strategies.  This will probably be handed into the WebexNotification service that we in-built our final weblog publish and can standardize the anticipated fields for the strategies in that service.  It’s a quite simple class:

// in dto/Notification.js
export class Notification {
    constructor(props = {}) {
        this.success = props.success;
        this.message = props.message;

That is the thing we’ll construct after we return the outcomes of our interactions with the Kubernetes SDK.

Dealing with Instructions

Beforehand on this publish, we stubbed out the general public takeAction methodology within the Kubernetes Service.  That is the place we’ll decide what motion is being requested, after which go it to inside personal strategies.  Since we’re solely wanting on the scale strategy on this publish, we’ll have two paths on this implementation.  The code on GitHub has extra.

// in companies/Kuberetes.js
// contained in the KubernetesService class
async takeAction(k8sCommand) {
    let outcome;
    swap (k8sCommand.kind) {
        case "scale":
            outcome = await this._updateDeploymentScale(k8sCommand);
            throw new Error(`The motion kind ${k8sCommand.kind} that was 
decided by the system shouldn't be supported.`);
    return outcome;

Very easy – if a acknowledged command kind is recognized (on this case, simply “scale”) an inside methodology known as and the outcomes are returned.  If not, an error is thrown.

Implementing our inside _updateDeploymentScale methodology requires little or no code.  Nevertheless it leverages the K8s SDK, which, to say the least, isn’t very intuitive.  The info payload that we create consists of an operation (op) that we’ll carry out on a Deployment configuration property (path), with a brand new worth (worth).  The SDK’s patchNamespacedDeployment methodology is documented within the Typedocs linked from the SDK repo.  Right here’s my implementation:

// in companies/Kubernetes.js
// contained in the KubernetesService class
async _updateDeploymentScale(k8sCommand) {
    // craft a PATCH physique with an up to date duplicate depend
    const patch = [
            "op": "replace",
            "value": k8sCommand.scaleTarget
    // name the K8s API with a PATCH request
    const res = await 
"default", patch, undefined, undefined, undefined, undefined, 
    // validate response and return an success object to the
    return this._validateScaleResponse(k8sCommand, res.physique)

The tactic on the final line of that code block is accountable for crafting our response output.

// in companies/Kubernetes.js
// contained in the KubernetesService class
_validateScaleResponse(k8sCommand, template) {
    if (template.spec.replicas === k8sCommand.scaleTarget) {
        return new Notification({
            success: true,
            message: `Efficiently scaled to ${k8sCommand.scaleTarget} 
situations on the ${k8sCommand.deploymentName} deployment`
    } else {
        return new Notification({
            success: false,
            message: `The Kubernetes API returned a reproduction depend of 
${template.spec.replicas}, which doesn't match the specified 

Updating the Webex Notification Service

We’re virtually on the finish!  We nonetheless have one service that must be up to date.  In our final weblog publish, we created a quite simple methodology that despatched a message to the Webex room the place the Bot was known as, primarily based on a easy success or failure flag.  Now that we’ve constructed a extra advanced Bot, we want extra advanced consumer suggestions.

There are solely two strategies that we have to cowl right here.  They may simply be compacted into one, however I choose to maintain them separate for granularity.

The general public methodology that our route handler will name is sendNotification, which we’ll refactor as follows right here:

// in companies/WebexNotifications
// contained in the WebexNotifications class
// discover that we’re including the unique occasion
// and the Notification object
async sendNotification(occasion, notification) {
    let message = `<@personEmail:${occasion.information.personEmail}>`;
    if (!notification.success) {
        message += ` Oh no! One thing went mistaken! 
    } else {
        message += ` Properly accomplished! ${notification.message}`;
    const req = this._buildRequest(occasion, message); // a brand new personal 
message, outlined under
    const res = await fetch(req);
    return res.json();

Lastly, we’ll construct the personal _buildRequest methodology, which returns a Request object that may be despatched to the fetch name within the methodology above:

// in companies/WebexNotifications
// contained in the WebexNotifications class
_buildRequest(occasion, message) {
    return new Request("", {
        headers: this._setHeaders(),
        methodology: "POST",
        physique: JSON.stringify({
            roomId: occasion.information.roomId,
            markdown: message

Tying All the pieces Collectively within the Route Handler

In earlier posts, we used easy route handler logic in routes/index.js that first logged out the occasion information, after which went on to answer a Webex consumer relying on their entry.  We’ll now take a distinct strategy, which is to wire in our companies.  We’ll begin with pulling within the companies we’ve created to date, retaining in thoughts that this can all happen after the auth/authz middleware checks are run.  Right here is the total code of the refactored route handler, with adjustments happening within the import statements, initializations, and handler logic.

// revised routes/index.js
import categorical from 'categorical'
import {AppConfig} from '../config/AppConfig.js';
import {WebexNotifications} from '../companies/WebexNotifications.js';
import {MessageIngestion} from "../companies/MessageIngestion.js";
import {KubernetesService} from '../companies/Kubernetes.js';
import {Notification} from "../dto/Notification.js";

const router = categorical.Router();
const config = new AppConfig();
const webex = new WebexNotifications(config);
const ingestion = new MessageIngestion(config);
const k8s = new KubernetesService(config);

// Our refactored route handler
router.publish('/', async operate(req, res) {
  const occasion = req.physique;
  strive {
    // message ingestion and evaluation
    const command = await ingestion.determineCommand(occasion);
    // taking motion primarily based on the command, presently stubbed-out
    const notification = await k8s.takeAction(command);
    // reply to the consumer 
    const wbxOutput = await webex.sendNotification(occasion, notification);
    res.statusCode = 200;
  } catch (e) {
    // reply to the consumer
    await webex.sendNotification(occasion, new Notification({success: false, 
message: e}));
    res.statusCode = 500;
    res.finish('One thing went terribly mistaken!');
export default router;

Testing It Out!

In case your service is publicly accessible, or if it’s operating regionally and your tunnel is exposing it to the web, go forward and ship a message to your Bot to try it out.  Do not forget that our take a look at Deployment was known as nginx-deployment, and we began with two situations.  Let’s scale to 3:

Successful scale to 3 instances

That takes care of the joyful path.  Now let’s see what occurs if our command fails validation:

Failing validation

Success!  From right here, the probabilities are infinite.  Be at liberty to share your whole experiences leveraging ChatOps for managing your Kubernetes deployments within the feedback part under.

Observe Cisco Studying & Certifications

Twitter, Fb, LinkedIn and Instagram.




Most Popular

Recent Comments