Last Watch AI - Extend Automations with Node-RED

Last Watch is a standalone application for creating if-then automations based on AI object detection.

Last Watch comes with a few automations out of the box which allow you to do simple things like send a Telegram photo message if a person is detected on camera. The list of integrations continues to grow (MQTT is now available!), but the automation options are still very basic.

What if you want more advanced options, like turning on a sensor when a car is stationary, and a different sensor when a car is in motion? While you could create different Detection Profiles for each scenario, this can become very tedious with dozens of profiles to manage in LW. Instead, why not offload the automation logic to a real automation platform like Home Assistant or Node-RED?

As of version 0.8.0, Last Watch allows you to do just that. Using the MQTT automation in LW, you can publish entire detection events to be used in any platform you want. Node-RED is perfect in this case for managing automations and allows you to build new and interesting profiles that are not possible natively in Last Watch. In this post I will demonstrate how to set this up and begin building automations.

Setting up MQTT

I am running Home Assistant with the Node-RED and MQTT Broker addons.

In this demo I am going to send events from Last Watch to Node-RED using MQTT. First, I need to define an Automation config in Last Watch for my MQTT topic.

Here I entered the IP of my Home Assistant server and the default MQTT port. The Topic and Client ID can be set to anything you want. Username and Password must be set unless you enabled Anonymous authentication in the MQTT add-on. Finally, make sure to not tick Custom Payload. We want Last Watch to automatically generate the event payload.

Now we can save the config and create a new Detection Profile to trigger the messages.

Since I am handling all of my automation in Node-RED, I'll create a profile to capture every type of object with a very low level of confidence.

Now I can save my profile and subscribe it to my MQTT automation. Now messages should start flowing to the broker.

Node-RED

Next, we will set up an MQTT listener in Node-RED.

I'll start by connecting an Mqtt-in node to a debug node and setting the topic to match the string configured in Last Watch. We also need a json node in the middle to de-serialize the payload into a usable object. Detection Events should begin recording in the debug window.

The payload has 3 properties. Two contain metadata about the event and the detection profile, and the third is an array of predictions. This array tells us what kinds of objects were detected, how confident the AI was, and whether or not the object was masked or filtered.

At this point we can begin building automations based on this message structure. I will use a split node to turn the prediction array into a stream of messages, and handle each prediction separately.

A switch node is used to split out different object classes so I have different paths for people, cars, trucks, etc. I am also using switch nodes to check if objects are masked or filtered (the smart filter flags objects which have not moved).

Now I can hook each path into whatever automation I want. I can trigger scripts or turn on sensors in Home Assistant, send push notifications, control smart home devices, etc. I could also add additional logic and filtering based on things like AI confidence, object size/position, proximity of objects or total number of objects. Some of this is possible natively in Last Watch, but it is much easier to manage and debug using the visual editor in Node-RED.

[{"id":"87975ec6.d3688","type":"mqtt in","z":"644db6a5.3303b8","name":"","topic":"mqtt/lastwatch/driveway","qos":"0","datatype":"auto","broker":"e7767a8a.ec2a28","x":940,"y":260,"wires":[["2464c0b4.008c8"]]},{"id":"2464c0b4.008c8","type":"json","z":"644db6a5.3303b8","name":"Parse Json","property":"payload","action":"","pretty":false,"x":1140,"y":260,"wires":[["dc32e8ef.1c3c88"]]},{"id":"dc32e8ef.1c3c88","type":"change","z":"644db6a5.3303b8","name":"Get Predictions","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.predictions","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":1310,"y":260,"wires":[["5f3453aa.b97d4c"]]},{"id":"5f3453aa.b97d4c","type":"split","z":"644db6a5.3303b8","name":"","splt":"\\n","spltType":"str","arraySplt":1,"arraySpltType":"len","stream":false,"addname":"","x":1470,"y":260,"wires":[["e1cf19e1.e60798"]]},{"id":"e1cf19e1.e60798","type":"switch","z":"644db6a5.3303b8","name":"Object Class","property":"payload.object_class","propertyType":"msg","rules":[{"t":"eq","v":"car","vt":"str"},{"t":"eq","v":"truck","vt":"str"},{"t":"eq","v":"person","vt":"str"}],"checkall":"true","repair":false,"outputs":3,"x":910,"y":460,"wires":[["e7159af6.13ab38"],["3a6e3c56.c56164"],["beb71c4d.ed465"]]},{"id":"beb71c4d.ed465","type":"debug","z":"644db6a5.3303b8","name":"Person in driveway!","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1100,"y":520,"wires":[]},{"id":"e7159af6.13ab38","type":"switch","z":"644db6a5.3303b8","name":"Not Masked","property":"payload.is_masked","propertyType":"msg","rules":[{"t":"eq","v":"0","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":1080,"y":400,"wires":[["d4b0f034.d300e"]]},{"id":"3a6e3c56.c56164","type":"switch","z":"644db6a5.3303b8","name":"Not Masked","property":"payload.is_masked","propertyType":"msg","rules":[{"t":"eq","v":"0","vt":"num"}],"checkall":"true","repair":false,"outputs":1,"x":1080,"y":460,"wires":[["c3f2847d.23f778"]]},{"id":"4be162e5.fa3cbc","type":"debug","z":"644db6a5.3303b8","name":"Car moving in driveway!","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1500,"y":420,"wires":[]},{"id":"bef2ccdc.a5b72","type":"debug","z":"644db6a5.3303b8","name":"Car parked in driveway!","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1500,"y":360,"wires":[]},{"id":"c3f2847d.23f778","type":"debug","z":"644db6a5.3303b8","name":"Truck in driveway!","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":1280,"y":460,"wires":[]},{"id":"d4b0f034.d300e","type":"switch","z":"644db6a5.3303b8","name":"Smart Filtered","property":"payload.is_smart_filtered","propertyType":"msg","rules":[{"t":"eq","v":"1","vt":"num"},{"t":"eq","v":"0","vt":"num"}],"checkall":"true","repair":false,"outputs":2,"x":1280,"y":400,"wires":[["bef2ccdc.a5b72"],["4be162e5.fa3cbc"]]},{"id":"e7767a8a.ec2a28","type":"mqtt-broker","name":"HA-Broker","broker":"192.168.1.53","port":"1883","clientid":"","usetls":false,"compatmode":false,"keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","closeTopic":"","closeQos":"0","closePayload":"","willTopic":"","willQos":"0","willPayload":""}]

Thanks for reading!