Stream SmartThings data to Cosmos DB and PowerBI - Part 1

Azure, IOT, Smart Home, SmartThings

I’ve recently added a whole load of home automation functionality using Samsung’s SmartThings platform. Part of this is a number of sensors to trigger automation based on motion, temperature, open/close etc. The SmarTthings platform is able to respond to events from these sensors to undertake tasks, and this works fine. What the platform doesn’t really do however is collect this data to give you historical information. For example, I wanted to collect temperature values from various sensors around the house over time to give me a sense of temperature fluctuations throughout the day and the effectiveness of my heating. Whilst there’s no built-in way to collect this data in SmartThings, one of the good things about this platform is that it is easily extensible through its app and smart app interface. Knowing this I set about looking to collect this data and store and report on it in Azure.

My plan was to use the following tools:

  • Event hubs to capture the data as a live stream from SmartThings
  • Stream Analytics to collect the data from the Event Hub and Transfer it to long-term storage
  • Cosmos DB to store the data
  • Power BI to report on the data

It should be noted that my approach here is to send the Data to Cosmos DB for long-term storage and then have Power BI read and report on that data from Cosmos. It is possible to stream data to PowerBI and have real-time reporting, but as all I am looking for here is historical reporting I didn’t feel this is necessary. If you want to do this you could certainly do so, and you would follow the same steps to start with to get the data to Event Hub.

This might seem a bit overkill for what we are doing (and I’m sure it is), but it’s also a good learning experience for these tools in Azure and a fun project to boot. There’s quite a lot of steps here, so I’m going to break this post into a few parts. In this first part, we will tackle getting the data from SmartThings to Event Hub.

Data Collection

The first thing we need to do is create an event hub to which we are going to stream the data. Use the Azure Portal to create an Event Hub in a region closest to you. For the volume of data, we are going to get from SmarThings the basic SKU should be enough.

This will go ahead and create an Event Hub namespace. Once that completes, select the created item in the Azure portal then go to the “Event Hubs” tab, we now need to create an actual event hub. Click the + icon and then name your event hub (I generally give it the same name as the namespace we created), you can accept the defaults.

Once this has created, click on the “Shared Access Policies” tab and create an access policy for this Event Hub. It will need to allow Send and Listed. Then copy both the primary key and the connection string for use later. You can call this what you want, but remember the name for later.

Create SmartThings App

Now we have created the event hub we need to get SmartThings to send data to it. This is done through creating a SmartApp. The code we are using for this is based on an article by Ben Crouse from 2015 but I have updated this code slightly to fix some issues and set it up in Github to make it easy to deploy. To add the smart app you need to use the SmartThings web-based IDE at https://account.smartthings.com and log in with your SmartThings credentials. Behind the scenes there are actually multiple regional SmartThings instances, this URL should take you to the right one, but to check you can go to the hubs tab and check that you can see your Smart things hub here.

Now we are logged into the IDE we can add the app. The easiest way to do this is through Github. In the IDE web app go to the “SmartApps” tab, in here you should see a settings button.

 

Click on the settings button then in the window that opens add the details of the Github repository. The owner should be sam-cogan, the name st-event-hub and the branch master. You can view the content of the repo here, and obviously for a copy yourself if you wish. Click Save.

Now click on the “Update from Repo” button in the IDE. You should see the st-event-hub repo in the list, click on this. This will open a window with 3 columns, the only content should be in the far right one labelled “New” and it should look like this:

Check the box and the box that says “publish”, then click the “Execute Update” button. Once this completes you should now see:

This indicates that the app is installed and ready for use.

App Settings

Before we can start using this solution, we do need to add some additional data so that it knows how to connect to the event hub. Click on the edit button on the left of the app name, in the window that opens expand the settings section, there should two settings with no values:

The EventHubURL setting is fairly easy to get, it will follow this format:

https://.servicebus.windows.net//messages

So in my setup, I have a namespace and an event hub call STEventHub so my URL is:

https://steventhub.servicebus.windows.net/steventhub/messages

The EventHub Secret is a bit more painful, as you need to generate an Event Hub SAS token. The easiest way to do this is with this PowerShell Script create by Marcel Meurer

[Reflection.Assembly]::LoadWithPartialName("System.Web")| out-null 
#Parameters 
$URI="<eventhub namespacename>.servicebus.windows.net/<eventhub name>"
$Access_Policy_Name="<access policy name>" 
$Access_Policy_Key="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" 
$Expires=([DateTimeOffset]::Now.ToUnixTimeSeconds())+31622400 
#Building Token 
$SignatureString=[System.Web.HttpUtility]::UrlEncode($URI)+ "`n" + [string]$Expires 
$HMAC = New-Object System.Security.Cryptography.HMACSHA256 
$HMAC.key = [Text.Encoding]::ASCII.GetBytes($Access_Policy_Key) 
$Signature = $HMAC.ComputeHash([Text.Encoding]::ASCII.GetBytes($SignatureString)) 
$Signature = [Convert]::ToBase64String($Signature) 
$SASToken = "SharedAccessSignature sr=" + [System.Web.HttpUtility]::UrlEncode($URI) + "&sig=" + [System.Web.HttpUtility]::UrlEncode($Signature) + "&se=" + $Expires + "&skn=" + $Access_Policy_Name 
$SASToken

You’ll need to update the parameters as follows:

  • URI – This is the URI of your Event Hub, so it’s the same as the one we created for the EventHub URL app setting, but without the “/messages” bit at the end
  • Access_Policy_Name – This is the name of the access policy you created for your event hub earlier
  • Access_Policy_Key – This is the Primary Key for the access policy you created
  • Expres – This is the number of seconds until this key expires. As we want to send this data over a long period we need to make this quite long, in this example, I am using 1 year

Run the PowerShell script and it should generate a key which you can add to this App Setting and save.

Adding the App

Now we have created and configure the app we need to add it to our SmartThings instance and authorise it to get data. You’ll need to do this through the SmartThing mobile application.  Open the application and go to the “Automation” tab, and then click on the “Smart Apps” tab. Scroll down and click on “Add a SmartApp”. On the window that opens scroll all the way down to “My Apps”.

In the window that opens, click on the “ST-Event-Hub” app

This will open the app and ask you to select which devices you want to send data to SmartThings. Go through each category and select the devices you wish to send data. Then give it a name and select save.

Confirm Data Transmission

At this point, we should now be sending our selected SmartThings data to our event hub. You can check the stats for your event hub in the portal to make sure that you are sending the data ok. You can also use Service Bus Explorer to connect directly to your Event Hub and view the data.

If you aren’t seeing data sent OK then you can look at the logs in SmartThings. To do this go back to the web IDE and click on the “Live Logging” tab. You can then filter it by the ST-Event-Hub app. One thing to be aware of this by default the logging of failures to send data to the Event Hub is disabled, this is because for some reason SmartThings/Groovy treats a 200 response from the Event Hub as an error, rather than success, so we have to disable error logging to prevent large amounts of 200 errors. To enable it again, go into the smart app and look at the code, then un-comment the highlighted line below:

try { 
    httpPost(params) { resp -> log.debug "response message ${resp}"
    }
} 
catch (e) { 
    // For some reason SmartThings treats 200 as an error response, so we need to comment this out to avoid errors. Uncomment the line below to debug errors 
    //log.error "something went wrong: $e" 
}

Getting this all working was probably the hardest part of this project. I had to make a number of tweaks to the code in the SmartThings app, and getting the Event Hub Secret in the right format took a good deal of time, so hopefully, this article will save you some of these pitfalls. In part 2 of this article, we’ll look at how we can take the data from Event Hub and get it into Cosmos DB ready to be reported on.