Full Code of andrewjfreyer/presence for AI

Repository: andrewjfreyer/presence
Branch: master
Commit: f14d3d7c5e32
Files: 2
Total size: 20.4 KB

Directory structure:
gitextract_zicas_p1/

├── README.md
└── presence.sh

================================================
FILE CONTENTS
================================================

================================================
FILE: README.md
================================================
presence
=======

***Note: as of `presence` 0.5.1, triggered scans, guest scanning, and beacon scanning are removed from `presence.sh` for simplification. Please consider using [monitor](http://github.com/andrewjfreyer/monitor) instead for beacon scanning and detection and for generic device detection. It is unlikely that `presence` will receive substantive updates after version 0.5.1.***

____


***TL;DR***: *Bluetooth-based presence detection useful for [mqtt-based](http://mqtt.org) home automation. More granular, responsive, and reliable than device-reported GPS. Cheaper, more reliable, more configurable, and less mqtt-spammy than Happy Bubbles. Does not require any app to be running or installed. Does not require device pairing. Designed to run as service on a [Raspberry Pi Zero W](https://www.raspberrypi.org/products/raspberry-pi-zero-w/).*

Note that the more frequently you scan for devices, the more 2.4GHz bandwidth you will use. This script may cause interference with Wi-Fi or other bluetooth devices for particularly short delays between scans. 

<h1>Summary</h1>

A JSON-formatted MQTT message is reported to a broker whenever a specified bluetooth device responds to a **name** query. If the device responds, the JSON message includes the name of the device and a **confidence** of 100. 

After a delay, another **name** query is sent and, if the device does not respond, a verification-of-absence loop begins that queries for the device (on a shorter interval) a set number of times. Each time, the device does not respond, the **confidence** is reduced, eventually to 0. 

A configuration file defines 'owner devices' that contains the mac addresses of the devices you'd like to regularly ping to determine presence. 

Topics are formatted like this:

     location/pi_zero_location/00:00:00:00:00:00 

Messages are JSON formatted and contain **name** and **confidence** fields, including a javascript-formatted timestamp and a duration of a particular scan (in ms):

     { confidence : 100, name : Andrew’s iPhone, scan_duration_ms: 500, timestamp : Sat Apr 21 2018 11:52:04 GMT-0600 (MDT)}
     { confidence : 0, name : Andrew’s iPhone or Unknown, scan_duration_ms: 5000, timestamp : Sat Apr 21 2018 11:52:04 GMT-0600 (MDT)}

___

<h1>Example Use with Home Assistant</h1>

The presence script can be used as an input to a number of [mqtt sensors](https://www.home-assistant.io/components/sensor.mqtt/) in [Home Assistant.](https://www.home-assistant.io). Output from these sensors can be averaged to give a highly-accurate numerical occupancy confidence. 

In order to detect presence in a home that has three floors and a garage, we might include one Raspberry Pi per floor. For average houses, a single well-placed sensor can probably work, but for more reliability at the edges of the house, more sensors are better. 


```
- platform: mqtt
  state_topic: 'location/first floor/00:00:00:00:00:00'
  value_template: '{{ value_json.confidence }}'
  unit_of_measurement: '%'
  name: 'Andrew First Floor'

- platform: mqtt
  state_topic: 'location/second floor/00:00:00:00:00:00'
  value_template: '{{ value_json.confidence }}'
  unit_of_measurement: '%'
  name: 'Andrew Second Floor'

- platform: mqtt
  state_topic: 'location/third floor/00:00:00:00:00:00'
  value_template: '{{ value_json.confidence }}'
  unit_of_measurement: '%'
  name: 'Andrew Third Floor'

- platform: mqtt
  state_topic: 'location/garage/00:00:00:00:00:00'
  value_template: '{{ value_json.confidence }}'
  unit_of_measurement: '%'
  name: 'Andrew Garage'
```

These sensors can be combined/averaged using a [min_max](https://www.home-assistant.io/components/sensor.min_max/):

```
- platform: min_max
  name: "Andrew Home Occupancy Confidence"
  type: mean
  round_digits: 0
  entity_ids:
    - sensor.andrew_garage
    - sensor.andrew_third_floor
    - sensor.andrew_second_floor
    - sensor.andrew_first_floor
```

So, as a result of this combination, we use the entity **sensor.andrew_home_occupancy_confidence** in automations to control the state of an **input_boolean** that represents a very high confidence of a user being home or not. 

As an example:

```
- alias: Andrew Occupancy 
  hide_entity: true
  trigger:
    - platform: numeric_state
      entity_id: sensor.andrew_home_occupancy_confidence
      above: 10
  action:
    - service: homeassistant.turn_on
      data:
        entity_id: input_boolean.andrew_occupancy
```

___


<h1>Installation Instructions (Raspbian Jessie Lite Stretch):</h1>

<h2>Setup of SD Card</h2>

1. Download latest version of **jessie lite stretch** [here](https://downloads.raspberrypi.org/raspbian_lite_latest)

2. Download etcher from [etcher.io](https://etcher.io)

3. Image **jessie lite stretch** to SD card. [Instructions here.](https://www.raspberrypi.org/magpi/pi-sd-etcher/)

4. Mount **boot** partition of imaged SD card (unplug it and plug it back in)

5. **[ENABLE SSH]** Create blank file, without any extension, in the root directory called **ssh**

6. **[SETUP WIFI]** Create **wpa_supplicant.conf** file in root directory and add Wi-Fi details for home Wi-Fi:

```
country=US
    ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
    update_config=1

network={
    ssid="Your Network Name"
    psk="Your Network Password"
    key_mgmt=WPA-PSK
}
```

 7. **[FIRST STARTUP]** Insert SD card and power on Raspberry Pi Zero W. On first boot, the newly-created **wpa_supplicant.conf** file and **ssh** will be moved to appropriate directories. Find the IP address of the Pi via your router. One method is scanning for open ssh ports (port 22) on your local network:
```
nmap 192.168.1.0/24 -p 22
```

<h2>Configuration and Setup of Raspberry Pi Zero W</h2>

1. SSH into the Raspberry Pi (password: raspberry):
```
ssh pi@theipaddress
```

2. Change the default password:
```
sudo passwd pi
```

3. **[PREPARATION]** Update and upgrade:

```
sudo apt-get update
sudo apt-get upgrade -y
sudo apt-get dist-upgrade -y
sudo rpi-update
sudo reboot
```

5. **[BLUETOOTH]** Install Bluetooth Firmware:
```
#install bluetooth drivers for Pi Zero W
sudo apt-get install pi-bluetooth

#verify that bluetooth is working
sudo service bluetooth start
sudo service bluetooth status
```

6. **[OPTIONAL: BLUEZ UPGRADE]** Compile/install latest **bluez** 

This is not strictly required anymore, as **pi-bluetooth** appears to have been updated to support bluez 0.5.43.

```

#purge old bluez
sudo apt-get --purge remove bluez

#get latest version number from: https://www.kernel.org/pub/linux/bluetooth/
#current version as of this writing is 5.49
cd ~; wget https://www.kernel.org/pub/linux/bluetooth/bluez-5.49.tar.xz
tar xvf bluez-5.49.tar.xz

#update errythang again
sudo apt-get update

#install necessary packages
sudo apt-get install libusb-dev libdbus-1-dev libglib2.0-dev libudev-dev libical-dev libreadline-dev

#move into new unpacked directory
cd bluez-5.49

#set exports
export LDFLAGS=-lrt

#configure 
./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var --enable-library -disable-systemd

#make & install
make
sudo make install

#cleanup
cd ~
rm -r bluez-5.49/
rm bluez-5.49.tar.xz

#update again
sudo apt-get update
sudo apt-get upgrade

#verify bluez version 
bluetoothd -v
```

7. **[REBOOT]**
```
sudo reboot
```

8. **[INSTALL MOSQUITTO]**
```

# get repo key
wget http://repo.mosquitto.org/debian/mosquitto-repo.gpg.key

#add repo
sudo apt-key add mosquitto-repo.gpg.key

#download appropriate lists file 
cd /etc/apt/sources.list.d/
sudo wget http://repo.mosquitto.org/debian/mosquitto-stretch.list

#update caches and install 
apt-cache search mosquitto
sudo apt-get update
sudo aptitude install libmosquitto-dev mosquitto mosquitto-clients
```


9. **[INSTALL PRESENCE]**
```
#install git
cd ~
sudo apt-get install git

#clone this repo
git clone git://github.com/andrewjfreyer/presence

#enter presence directory
cd presence/
```

10. **[CONFIGURE PRESENCE]** create file named **mqtt_preferences** and include content:
```
nano mqtt_preferences
```

Then...

```
mqtt_address="ip.address.of.broker"
mqtt_port="optional broker network port number. Defaults to 1883"
mqtt_user="your broker username"
mqtt_password="your broker password"
mqtt_topicpath="location"
mqtt_room="your pi's location"
```

11. **[CONFIGURE PRESENCE]** create file named **owner_devices** and include mac addresses of devices on separate lines. 

```
nano owner_devices
```

Then...

```
00:00:00:00:00 #comments 
00:00:00:00:00
```

12. **[CONFIGURE SERVICE]** Create file at **/etc/systemd/system/presence.service** and include content:

```
sudo nano /etc/systemd/system/presence.service
```

Then...

```
[Unit]
Description=Presence service

[Service]
User=root
ExecStart=/bin/bash /home/pi/presence/presence.sh &
WorkingDirectory=/home/pi/presence
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

```

13. **[CONFIGURE SERVICE]** Enable service by:
```
sudo systemctl enable presence.service
sudo systemctl start presence.service
```

That's it. Your broker should be receiving messages and the presence service will restart each time the Raspberry Pi boots.  



================================================
FILE: presence.sh
================================================

#!/bin/bash

# ----------------------------------------------------------------------------------------
# GENERAL INFORMATION
# ----------------------------------------------------------------------------------------
#
# Written by Andrew J Freyer
# GNU General Public License
# http://github.com/andrewjfreyer/presence
#
# ----------------------------------------------------------------------------------------

# ----------------------------------------------------------------------------------------
# INCLUDES & VARIABLES
# ----------------------------------------------------------------------------------------

#VERSION NUMBER
VERSION=0.5.1

#COLOR OUTPUT FOR RICH DEBUG 
ORANGE='\033[0;33m'
RED='\033[0;31m'
NC='\033[0m'
GREEN='\033[0;32m'
PURPLE='\033[1;35m'

#BASE DIRECTORY REGARDLESS OF INSTALLATION; ELSE MANUALLY SET HERE
base_directory=$(dirname "$(readlink -f "$0")")

#FIND MQTT PATH, ELSE MANUALLY SET HERE
mosquitto_pub_path=$(which mosquitto_pub)
mosquitto_sub_path=$(which mosquitto_sub)

#ERROR CHECKING FOR MOSQUITTO PUBLICATION 
[ -z "$mosquitto_pub_path" ] && echo "Required package 'mosquitto_pub' not found. Please install." && exit 1
[ -z "$mosquitto_sub_path" ] && echo "Required package 'mosquitto_sub' not found. Please install." && exit 1

# ----------------------------------------------------------------------------------------
# LOAD PREFERENCES
# ----------------------------------------------------------------------------------------

#OR LOAD FROM A SOURCE FILE
if [ ! -f "$base_directory/behavior_preferences" ]; then 
	echo -e "${GREEN}presence $VERSION ${RED}WARNING:  ${NC}Behavior preferences are not defined:${NC}"
	echo -e "/behavior_preferences. Creating file and setting default values.${NC}"
  	echo -e "" 

  	#DEFAULT VALUES
  	echo "
#DELAY BETWEEN SCANS OF OWNER DEVICES WHEN AWAY FROM HOME
delay_between_owner_scans_away=6

#DELAY BETWEEN SCANS OF OWNER DEVICES WHEN HOME 
delay_between_owner_scans_present=30

#HOW MANY VERIFICATIONS ARE REQUIRED TO DETERMINE A DEVICE IS AWAY 
verification_of_away_loop_size=6

#HOW LONG TO DELAY BETWEEN VERIFICATIONS THAT A DEVICE IS AWAY 
verification_of_away_loop_delay=3

#PREFERRED HCI DEVICE
hci_device='hci0'" > "$base_directory/behavior_preferences"
fi 

# ----------------------------------------------------------------------------------------
# VARIABLE DEFINITIONS 
# ----------------------------------------------------------------------------------------

#SET PREFERENCES FROM FILE
DELAY_CONFIG="$base_directory/behavior_preferences" ; [ -f $DELAY_CONFIG ] && source $DELAY_CONFIG

#LOAD DEFAULT VALUES IF NOT PRESENT
[ -z "$hci_device" ] && hci_device='hci0' 
[ -z "$name_scan_timeout" ] && name_scan_timeout=5
[ -z "$delay_between_owner_scans_away" ] && delay_between_owner_scans_away=6 
[ -z "$delay_between_owner_scans_present" ] && delay_between_owner_scans_present=30
[ -z "$verification_of_away_loop_size" ] && verification_of_away_loop_size=6
[ -z "$verification_of_away_loop_delay" ] && verification_of_away_loop_delay=3

#LOAD PREFERENCES IF PRESENT
MQTT_CONFIG=$base_directory/mqtt_preferences ; [ -f $MQTT_CONFIG ] && source $MQTT_CONFIG
[ ! -f "$MQTT_CONFIG" ] && echo "warning: please configure mqtt preferences file. exiting." && echo "" > "$MQTT_CONFIG" && exit 1

#FILL ADDRESS ARRAY WITH SUPPORT FOR COMMENTS
[ ! -f "$base_directory/owner_devices" ] && "" > "$base_directory/owner_devices"
macaddress_owners=($(cat "$base_directory/owner_devices" | grep -oiE "([0-9a-f]{2}:){5}[0-9a-f]{2}" ))
[ -z "$macaddress_owners" ] && echo "warning: no owner devices are specified. exiting." && exit 1


#NUMBER OF CLIENTS THAT ARE MONITORED
number_of_owners=$((${#macaddress_owners[@]}))

# ----------------------------------------------------------------------------------------
# HELP TEXT
# ----------------------------------------------------------------------------------------

show_help_text() {
	echo "Usage:"
	echo "  presence -h 	show usage information"
	echo "  presence -d 	print debug messages and mqtt messages"
	echo "  presence -b 	binary output only; either 100 or 0 confidence"
	echo "  presence -c 	only post confidence status changes for owners/guests"
	echo "  presence -V		print version"
}

# ----------------------------------------------------------------------------------------
# PROCESS OPTIONS (technique: https://stackoverflow.com/questions/192249/how-do-i-parse-command-line-arguments-in-bash)
# ----------------------------------------------------------------------------------------

OPTIND=1

# INITIALIZE OUR OWN VARIABLES:
debug=0
binary_only=0
changes_only=0

while getopts "h?Vdbct:" opt; do
    case "$opt" in
    h|\?)
        show_help_text
        exit 0
        ;;
    V)  
		echo "$VERSION"
		exit 0
		;;
    d)  debug=1
		;;
    b)  binary_only=1
		;;
	c)  changes_only=1
		;;
	*)	echo "warning: unknown or depreciated option: $opt"
    esac
done

#RESET OPTION INDEX
shift $((OPTIND-1))

#SHIFT IF NECESSARY
[ "$1" = "--" ] && shift

# ----------------------------------------------------------------------------------------
# DEBUG FUNCTION
# ----------------------------------------------------------------------------------------

debug_echo () {
	if [ "$debug" == "1" ]; then 
		(>&2 echo -e "${ORANGE}DEBUG MSG:	$1${NC}")
	fi 
}

# ----------------------------------------------------------------------------------------
# SCAN 
# ----------------------------------------------------------------------------------------

scan () {
	if [ ! -z "$1" ]; then 
		local result=$(hcitool -i $hci_device name "$1" 2>&1 | grep -v 'not available' | grep -vE "hcitool|timeout|invalid|error" )
		debug_echo "Scan result: [$result]"
		echo "$result" 
	fi
}

# ----------------------------------------------------------------------------------------
# PUBLISH MESSAGE
# ----------------------------------------------------------------------------------------

publish () {
	if [ ! -z "$1" ]; then 

		#SET NAME FOR 'UNKONWN'
		local name="$3"

		#IF NO NAME, RETURN "UNKNOWN"
		if [ -z "$3" ]; then 
			name="Unknown"
		fi 

		#TIMESTAMP
		stamp=$(date "+%a %b %d %Y %H:%M:%S GMT%z (%Z)")

		#DEBUGGING 
		[ "$debug" == "1" ] && (>&2 echo -e "${PURPLE}$mqtt_topicpath$1 { confidence : $2, name : $name, scan_duration_ms: $4, timestamp : $stamp} ${NC}")

		#POST TO MQTT
		$mosquitto_pub_path -h "$mqtt_address" -p "${mqtt_port:=1883}" -u "$mqtt_user" -P "$mqtt_password" -t "$mqtt_topicpath$1" -m "{\"confidence\":\"$2\",\"name\":\"$name\",\"scan_duration_ms\":\"$4\",\"timestamp\":\"$stamp\"}"
	fi
}

# ----------------------------------------------------------------------------------------
# MAIN LOOP
# ----------------------------------------------------------------------------------------

device_statuses=()			#STORES STATUS FOR EACH BLUETOOTH DEVICES
device_names=()				#STORES DEVICE NAMES FOR BOTH BEACONS AND BLUETOOTH DEVICES
one_owner_home=0 			#FLAG FOR AT LEAST ONE OWNER BEING HOME

# ----------------------------------------------------------------------------------------
# START THE OPERATIONAL LOOP
# ----------------------------------------------------------------------------------------

#MAIN LOOP
while true; do 

	#RESET AT LEAST ONE DEVICE HOME
	one_owner_home=0

	#--------------------------------------
	#	UPDATE STATUS OF ALL USERS WITH NAME QUERY
	#--------------------------------------
	for ((index=0; index<${#macaddress_owners[*]}; index++));
	do
		#CLEAR PER-LOOP VARIABLES
		name_scan_result=""
		name_scan_result_verify=""
		ok_to_publish=1

		#OBTAIN INDIVIDUAL ADDRESS
		current_device_address="${macaddress_owners[$index]}"

		#CHECK FOR ADDITIONAL BLANK LINES IN ADDRESS FILE
		if [ -z "$current_device_address" ]; then 
			continue
		fi

		#MARK BEGINNING OF SCAN OPERATION
		start_timer=$(date +%s%N)

		#OBTAIN RESULTS AND APPEND EACH TO THE SAME
		name_scan_result=$(scan $current_device_address)
		
		#MARK END OF SCAN OPERATION
		end_time=$(date +%s%N)
		
		#CALCULATE DIFFERENCE
		duration_timer=$(( (end_time - start_timer) / 1000000 )) 

		#THIS DEVICE NAME IS PRESENT
		if [ "$name_scan_result" != "" ]; then

			#STATE IS SAME && ONLY REPORT CHANGES THEN DISABLE PUBLICATION
			[ "${device_statuses[$index]}" == '100' ] && [ "$changes_only" == 1 ] && ok_to_publish=0

			#NO DUPLICATE MESSAGES
			[ "$ok_to_publish" == "1" ] && publish "/$mqtt_room/$current_device_address" '100' "$name_scan_result" "$duration_timer"

			#USER STATUS			
			device_statuses[$index]="100"

			#SET AT LEAST ONE DEVICE HOME
			one_owner_home=1

			#SET NAME ARRAY
			device_names[$index]="$name_scan_result"

		else

			#USER STATUS			
			status="${device_statuses[$index]}"

			if [ -z "$status" ]; then 
				status="0"
			fi 

			#BY DEFAULT, SET REPETITION TO PREFERENCE
			repetitions="$verification_of_away_loop_size"

			#IF WE ARE JUST STARTING OR, ALTERNATIVELY, WE HAVE RECORDED THE STATUS 
			#OF NOT HOME ALREADY, ONLY SCAN ONE MORE TIME. 
			if [ "$status" == 0 ];then 
				repetitions=1
			fi 

			#SHOULD VERIFY ABSENSE
			for repetition in $(seq 1 $repetitions); 
			do 
				#RESET OK TO PUBLISH
				ok_to_publish=1

				#VERIFICATION LOOP DELAY
				sleep "$verification_of_away_loop_delay"

				#GET PERCENTAGE
				percentage=$(($status * ( $repetitions - $repetition) / $repetitions))

				#ONLY SCAN IF OUR STATUS IS NOT ALREADY 0
				if [ "$status" != 0 ];then 

					#MARK BEGINNING OF SCAN OPERATION
					start_timer=$(date +%s%N)

					#PERFORM SCAN
					name_scan_result_verify=$(scan $current_device_address)

					#MARK END OF SCAN OPERATION
					end_time=$(date +%s%N)
					
					#CALCULATE DIFFERENCE
					duration_timer=$(( (end_time - start_timer) / 1000000 )) 

					#CHECK SCAN 
					if [ "$name_scan_result_verify" != "" ]; then						

						#STATE IS SAME && ONLY REPORT CHANGES THEN DISABLE PUBLICATION
						[ "${device_statuses[$index]}" == '100' ] && [ "$changes_only" == 1 ] && ok_to_publish=0

						#PUBLISH
						[ "$ok_to_publish" == "1" ] && publish "/$mqtt_room/$current_device_address" '100' "$name_scan_result_verify" "$duration_timer"

						#SET AT LEAST ONE DEVICE HOME
						one_owner_home=1

						#WE KNOW THAT WE MUST HAVE BEEN AT A PREVIOUSLY-SEEN USER STATUS
						device_statuses[$index]="100"

						#UPDATE NAME ARRAY
						device_names[$index]="$name_scan_result_verify"

						#MUST BREAK CONFIDENCE SCANNING LOOP; 100' ISCOVERED
						break
					fi
				fi 

				#RETREIVE LAST-KNOWN NAME FOR PUBLICATION; SINCE WE OBVIOUSLY DIDN'T RECEIVE A NAME SCAN RESULT 
				expectedName="${device_names[$index]}"

				if [ "$percentage" == "0" ]; then 
					#STATE IS SAME && ONLY REPORT CHANGES THEN DISABLE PUBLICATION
					[ "${device_statuses[$index]}" == '0' ] && [ "$changes_only" == 1 ] && ok_to_publish=0

					#PRINT ZERO CONFIDENCE OF A DEVICE AT HOME
					[ "$ok_to_publish" == "1" ] && publish "/$mqtt_room/$current_device_address" "0" "$expectedName" "$duration_timer"
				else 
					#STATE IS SAME && ONLY REPORT CHANGES THEN DISABLE PUBLICATION
					[ "${device_statuses[$index]}" == '$percentage' ] && [ "$changes_only" == 1 ] && ok_to_publish=0

					#IF BINARY ONLY, THEN DISABLE PUBLICATION
					[ "$binary_only" == "1" ] && ok_to_publish=0

					#REPORT CONFIDENCE DROP
					[ "$ok_to_publish" == "1" ] && publish "/$mqtt_room/$current_device_address" "$percentage" "$expectedName" "$duration_timer"
				fi 

				#UPDATE STATUS ARRAY
				device_statuses[$index]="$percentage" 
			done
		fi
	done

	#CHECK STATUS ARRAY FOR ANY DEVICE MARKED AS 'HOME'
	wait_duration=0

	#DETERMINE APPROPRIATE DELAY
	if [ "$one_owner_home" == 1 ]; then 
		 wait_duration=$delay_between_owner_scans_present
	else
		wait_duration=$delay_between_owner_scans_away
	fi

	sleep "$wait_duration"
done