Writing Amazon Kinesis Data to Domo

​

Intro

​

Prerequisites

• a Kinesis Stream receiving data in JSON format, with one JSON object per Kinesis record
• an active Domo instance

​

Step 1 – Create a JSON Webhook DataSet

1. In the Domo Appstore, find and click on the JSON Webhook connector.
2. Click Get the Data.
3. Copy and save the link in the Webhook URL field. (Note that you cannot change this field.)
   
4. (Optional) If you want a secret credential to be required for the Webhook when data is sent to it, enter it into the Webhook Secret field. (For the remainder of these steps, we will assume a Webhook Secret is not being used.)
5. Select whether the DataSet will Append or Replace. It is set to Append by default, which is typically what Kinesis users need.
6. Click Next.
7. Name your DataSet and click Save.

​

Step 2 – Create an Amazon Lambda Function

1. In the AWS Management Console, choose the Lambda service. (Note that your AWS UI may be slightly different.
   
2. On the menu on left menu, select Functions.
   
3. Click Create function.
   
4. Fill out the fields to author the function from scratch:
   • Function name – Give your function a descriptive, memorable name.
   • Runtime – Select the language your function will use. This tutorial uses Node.js 20.x. Note that it is also possible to create functions using Java, C#, Go, Python, or Ruby.
   • Role – Select Choose an existing role. Work with your AWS administrator to make sure that there is a role appropriate for this function. The role needs to be able to read from Kinesis, write to CloudWatch (for logging), and make HTTPS calls.
     
5. Click Create function.
6. After the Lambda function has been created, add Kinesis as a trigger to it by selecting Kinesis from the menu on the left of the screen.
   
7. Configure the trigger by filling out the required fields. Generally, the default values are acceptable:
   • Kinesis stream – Select the Stream that you want to use to trigger this function. This is the stream that is getting the data you want to copy to Domo.
   • Consumer – Leave this as No consumer.
   • Batch Size – You can adjust this based on how much data your Kinesis Stream has to handle. To start, leave it at 100.
   • Starting Position – Set this to Latest. That way, the function will be given the most recent data in the Kinesis Stream.
   • Activate Trigger – Make sure this box is checked.
     
8. Click Add.
9. Click the name of the Function so that the Function Code panel appears.
   
10. Set the code fields to the following:
    • Code entry type – Edit code inline
    • Runtime – Node.js 20.x
    • Handler – index.handler
11. The code window should show a file called index.js. Replace all of the code in that file with the following:
    Copy

    Ask AI

    /**
    * Stream data from AWS Kinesis to Domo via Webhook
    */
    
    'use strict';
    
    const https = require('https');
    
    // This is called when the Lambda function is triggered.
    // It should be configured to be triggered by data in a Kinesis Stream
    exports.handler = (event, context, callback) => {
       let count = 0;
    
       // Build an array of all of the JSON objects that came in from Kinesis
       let collectedData = [];
       event.Records.forEach((record) => {
          // Kinesis data is base64 encoded so decode here
          const data = new Buffer(record.kinesis.data, 'base64').toString('ascii');
       // console.log('Decoded payload:', JSON.stringify(data, null, 2));
       console.log('Decoded payload:', data);
    
       // Assumes that each Kinesis record contains only 1 JSON object
       let item = JSON.parse(data);
       collectedData[count] = item;
    
       count += 1;
    });
    
       // The array of objects will be passed to the webhook in a single POST call
       let postData = JSON.stringify(collectedData);
       let options = {
          host: process.env.WEBHOOK_HOST, // Should generally be "webhooks.domo.com"
          path: process.env.WEBHOOK_ENDPOINT, // Should be the rest of the webhook URL given by Domo
          method: 'POST',
          headers: {
                'Content-Type': 'application/json',
                'Content-Length': postData.length
          }
       };
    
       const req = https.request(options, (res) => {
          console.log('statusCode: ' + res.statusCode);
    
       res.on('data', (d) => {
          console.log('data: ' + d);
    })
    
    });
    
       req.on('error', (e) => {
          console.error('*** ERROR ***');
       console.error(e);
    });
    
       req.write(postData);
    
       req.end();
    
       console.log('Pushed ' + count + ' record(s) to the webhook');
    };
    
12. Below the code window, add two environment variables:
    • WEBHOOK_HOST – This should be the domain in the Webhook URL given in the DataSet you created in step 1 of these instructions. It will typically be webhooks.domo.com.
    • WEBHOOK_ENDPOINT – This should be everything in the Webhook URL starting with “/api”. This will be a very long alphanumeric string of characters.
      
13. From the menu in the editor, select File —> Save.
14. At the top of the screen, click Save to save all changes to your Lambda function.