Sunday, June 26, 2016

Wifi device detection with Domoticz and OpenWRT

I've recently started using Domoticz to control my various home automation systems. One of the bits of information about the house that I wanted to gather was the presence of mobile phones on the wireless network. Both my partner and I always have our phones with us, so if they are present on the network then it's a good bet that we're at home. This allows constructing domoticz rules based on who's in the house.

There are many examples of scripts on the internet that show how to do this in Domoticz using either ping, arp-scan or a combination of the two. However the success of them seems to be varied at best. The main issue appears to be that mobile devices attempt to conserve power by not being 'active' (whatever that means) on the wireless network. I tried to make my scripts more accurate by adding more and more layers of complexity and complex rules. Eventually I decided the active scanning approach was doomed.

Luckily I run OpenWRT on both of my wireless access points. Logging into the OpenWRT boxes and running

/usr/sbin/iw dev wlan0 station dump
will give a dump of information about the various devices. The next question was how to get this information from the OpenWRT system to the system running Domoticz. I decided to use xinitd on OpenWRT, which allows binding the output of an arbitrary command to a TCP port. It's not at all secure, but I'm happy to take the risk as this isn't going to be facing the public internet. To configure OpenWRT do the following:
  1. Install xinitd:
    opkg update && opkg install xinitd
  2. Edit
    to contain the following line:
    wlanap 8216/tcp
  3. Create

    service wlanap
    socket_type     = stream
    protocol        = tcp
    wait            = no
    user            = root
    group           = root
    server          = /usr/sbin/dump_ap
    disable         = no
    only_from       = 192.168.XXX.XXX

    with the IP of the machine running Domoticz.
  4. Create

    /usr/sbin/iw dev wlan0 station dump
    Note: If your router has multiple wireless interfaces (e.g. 2.4GHz and 5GHz) then you should repeat the last line in the script for the other interfaces, such as wlan1.
  5. Mark it as executable:
    chmod a+x /usr/sbin/dump_ap
  6. Restart xinetd on OpenWRT
  7. Repeat for each OpenWRT box

It should be noted that the help for the iw tool does give the following warning:
Do NOT screenscrape this tool, we don't consider its output stable
. So every time I update OpenWRT I am going to be at risk of things breaking, but that will just add to the excitement of applying updates!

So the last piece in the puzzle was to write a script to periodically query my wireless access points and toggle virtual switches in Domoticz as appropriate. I created a couple of virtual switches in Domoticz for each of the devices of interest and then created a cronjob to run the following Python script periodically:
#!/usr/bin/env python
# Detects ARP addresses

import argparse
import json
import logging
import os
import re
import socket
import subprocess
import sys
import urllib2

log = logging.getLogger()

DEFAULT_ARP_SCAN = "/usr/bin/arp-scan"
DEFAULT_CONFIG_LOCATION = os.path.join(os.environ['HOME'], ".detect-arp", "config.json")

DOMOTICZ_URL = "http://localhost"

    "12:34:56:78:9A:BC": 31, # Dave's Phone
    "DE:AD:BE:EF:00:12": 37, # Bob's Phone

WLANAP_HOSTS = ["AccessPoint1", "AccessPoint2"]
ARP_LINE_RE = re.compile(r"^Station ([0-9a-f:]{17}) \(on .+\)$")

def getArps():
    foundArp = set()
    for host in WLANAP_HOSTS:
        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        s.connect((host, WLANAP_PORT))
        data = ""
        while True:
            dataFrag = s.recv(4096)
            if len(dataFrag) == 0:
            data += dataFrag
        for line in data.split("\n"):
            m =
            if m is None:
            arpAddr =
            log.debug("Found ARP %s", arpAddr)
    return foundArp

def scan(args):
    log.debug("Starting scan")
    foundArp = getArps()
    # Update the switch state as needed
    log.debug("Toggling switches")
    for arpAddr, idx in ARP_TO_IDX.iteritems():
        # Get the current state
        stateUrl = "%s/json.htm?type=devices&rid=%d" % (
            DOMOTICZ_URL, idx)
        resp = urllib2.urlopen(stateUrl)
        jsonStr =
        stateData = json.loads(jsonStr)
        if stateData.get("status", None) != "OK":
            log.err("Reply from '%s': %s", stateUrl, jsonStr)

        currentState = stateData["result"][0]["Status"]
        if arpAddr in foundArp:
            switchCmd = "On"
            switchCmd = "Off"
        if switchCmd == currentState:
            log.debug("Not switching %d due to unchanged state (%s)", idx, switchCmd)
        log.debug("Switching %s (%d) to %s", arpAddr, idx, switchCmd)
        switchUrl = "%s/json.htm?type=command¶m=switchlight&idx=%d&switchcmd=%s" % (
            DOMOTICZ_URL, idx, switchCmd)
        resp = urllib2.urlopen(switchUrl)
        data =
        switchData = json.loads(data)
        if switchData.get("status", None) != "OK":
            log.err("Reply from '%s': %s", switchUrl, switchData)

def install(args):
    log.debug("Starting install")

def uninstall(args):
    log.debug("Starting uninstall")

def getParser():
    parser = argparse.ArgumentParser(description="Detects devices on the network based on ARP")
    parser.add_argument("--arpscan", dest="arpScanPath", default=DEFAULT_ARP_SCAN,
                        help="Location of arpscan, default %(default)s")
    parser.add_argument("--config", dest="configPath", default=DEFAULT_CONFIG_LOCATION,
                        help="Location of the config file, default %(default)s")
    parser.add_argument("-v", "--verbose", dest="verbose", action="store_true",
                        help="Enable verbose output")
    subParser = parser.add_subparsers(title="action", dest="action")

    scanParser = subParser.add_parser("scan", help="scan for devices")

    installParser = subParser.add_parser("install", help="install script into crontab")

    uninstallParser = subParser.add_parser("uninstall", help="uninstall script from crontab")

    return parser

def main():
    parser = getParser()
    args = parser.parse_args()
    if args.verbose:
    commandFn = globals()[args.action]

if __name__ == "__main__":

Note: You'll want to customise the MAC address to switch indexes defined in
and make sure that
contains each of the host names for your WiFi access points.

Thursday, August 21, 2014

Truncated Logs and Trees

When asking someone else for help it can be tricky to adjust between the information rich and bandwidth constrained worlds of the real and the virtual. If I have a tree at the bottom of my garden which loses all its leaves one summer then the conversation with an expert on tree disease would go something like this:

"Hello Dave the tree expert," I would say.

"Hi Toby, what's up?" he would reply.

"I think the tree at the bottom of my garden is dying."

Dave would then start to investigate, "Why do you think that?"

"Because it's lost all its leaves," I'd say.

"Hmmm," he would say as he stroked his chin with his hand, "did they all drop over a couple of days? Or was it over a period of weeks?"

The conversation would then continue to go back and forth between us with Dave asking various questions to narrow down the issue. Unless I'd had too much coffee that day I would be very unlikely to approach Dave and say:

"Hi Dave," I'd blurt out.

He would just have time to say, "Oh Hello Toby-"

I would then interrupt him and say, "I have a tree. It's dying. It's at the end of my garden. I have just got a new cat. All the leaves fell off the tree. They were green, but went brown before dropping off. They have a strange white haze on them. The neighbour has just had some building work done right next to the corner the tree is in. I had waffles for breakfast. I just planted a row of lavender at the other end of my garden. The tree trunk looks the same as always. It's been a dry summer, but the winter was very damp."

Trying to throw all this information at the person in one go, without knowing what is relevant and what isn't, just doesn't work as an efficient way to communicate. In fact it is generally the case that the best problem solvers in any given area aren't the experts with the most in-depth knowledge but the ones who know the right questions to ask, as well as what order to ask them in. However with computer systems this is not generally the case.

Most pieces of computer software produce logs. These contain a history of events and observations that the software has made about the world. They can be very useful for tracking down why a program is behaving in a particular way, especially if the information which is chosen to be logged describes the state of the system sufficiently.

When asking someone for help you should always give them full access to the logs. First you should described the problem; you should say what's happened, not what you think has gone wrong. Think "I crashed my car as the brake peddle didn't slow the vehicle down" rather than "I think my brake pads are worn".

Once you've done that you should provide the full logs. If you have some thoughts on which lines in the logs are most important then do explicitly mention them, but don't delete everything from the log apart from what you think is important. You are asking for help. You don't fully understand the system. You will cut important information out of the logs because you don't think it's relevant because you don't appreciate the full issue. If you did know enough about what was going on to know which log lines are relevant then you wouldn't need to ask for help!

So please, when asking for help, send full logs and describe what the issue is, don't just make guesses and provide incomplete information to someone who is trying to aid you.

Friday, September 06, 2013


Fed up with MythTvFs breaking on my Linux MythTV backend when I upgraded MythTV I created PyMythTvFs.

PyMythTvFs is a Fuse based file system written in Python for accessing MythTv recordings as a file system. It's still very much work in progress but it is functional and allows playback of recorded programs.

In the future I intend to add several other features, such as folders, delete support and custom format strings. That's still to come, but for now it allows easy watching of MythTV programs.

Wednesday, February 01, 2012

Reverting or rolling back accidental commits in SVN

When using SVN it's sometimes the case that you'll type:
svn commit
Only to realise that you've just accidentally committed the wrong changes, or even some files you didn't want in the subversion repository.

Obviously the svn revert command won't help as that's for reverting local changes. If you want to undo and roll back your changes from r12345 then the magic SVN runes are:
svn merge -c -12345 .

Then you can review and then commit those changes (with a suitable commit log message about rolling back or undoing the revision).

The option -c -<#> is equivalent to -m <#>:<#-1>. Notice that there is a dash before the number, without it you would get -m <#-1>:<#>. The lone dot just tells SVN to work on the repository in the current directory.

Further details about the merge command options can be found here.

Tuesday, October 25, 2011

Fixing Samba failing to find canonical path for autofs mounted share locations

Following on from my previous post I was chatting with a friend and they pointed out that I could also make use of preexec to get my desired behaviour.

So I removed the entries for wide links and unix extensions from my smb.conf, allowing them to go back to being the default. I then changed my mythtv share to contain:

        comment = Myth TV Recordings
        path = /auto/mythtv
        guest ok = Yes
        preexec = ls /auto/mythtv

This then solves the issue of Samba not allowing the share to be opened due to the "canonicalize_connect_path failed for service " error.

Tuesday, September 27, 2011

Sharing an autofs mount point over Samba

Since upgrading to Samba 2:3.5.11~dfsg-1 (or possibly slightly earlier) on my Debian based home server I found that some of the shares were no longer working. I could see them when browsing the network device, it just wouldn't let me access them.

This seemed to only be happening to my Samba shares which were pointing directly at the root of an autofs mount point, which in my case was a mythtvfs mount at /auto/mythtv.

What made this more puzzling was that if I ssh'ed into the machine then I could happily do

ls /auto/mythtv
and see all my files. The final piece of the puzzle fitted into place when I realised that the Samba share would work if the mount was already mounted.

A look at the samba logs indicated the following:
  canonicalize_connect_path failed for service mythtv, path /auto/mythtv

On a hunch I tried the wide links option (which requires disabling unix extensions) by adding the following to the global section of my /etc/samba/smb.conf:

# Need to disable unix extensions and allow wide links to allow /auto to be mounted
unix extensions = no
wide links = yes
And then restarted samba with "/etc/init.d/samba restart". This then fixed the issue, allowing my autofs mount points to be mounted correctly. The big downside of this is that unix extensions which allow symlinks and hard links over samba are then disabled. If I get the time I might look into seeing if I can patch canonicalize_connect_path to work correctly with autofs mount points without exposing any security flaws.

Possibly the most irritating thing about this issue was that searching for the terms "Samba" and "autofs" gives lots of hits for the other side of the situation; using autofs to mount Samba shares automatically.

Friday, June 17, 2011

Tearing screen even with VSyncing on NVIDIA ION

I've got a home media machine which I to watch MythTV and various online video streaming services. It's been annoying me for a while (possibly since I updated to Ubuntu 11.04) that there was tearing on the screen. It looked exactly like it wasn't vsyncing correctly, but I had vsyncing enabled in all the players I was trying.

The machine I'm using is an Acer Revo with NVIDIA ION chipset, so it isn't the most powerful system. Searching the internet for 'tearing' turned up results which were mostly about people not being able to play 720p or 1080p videos. However that certainly wasn't my issue as I'd done all of the following:

  • Giving the graphics card 512MB of RAM (called UMA in the BIOS setting).
  • Enabling VDPAU acceleration.
  • Disabling the compiz window manager (selecting "Ubuntu classic (no effects)" as the session).
  • Setting export __GL_SYNC_TO_VBLANK=1 and ignoring issues with vsync occurring for the wrong display.
  • Checking the refresh rate was correctly setup with nvidia-settings utility.
  • Updating to the latest NVIDIA drivers

But none of this helped. I then realised that all rendering to the screen was suffering from this, not just video playback. I wondered if the ION couldn't handle the memory bandwidth, but turning down the resolution didn't help. I then thought that maybe it was my TV being a bit broken with HDMI, but testing that would require finding a VGA cable or moving a monitor downstairs, so I didn't try that.

I even got to the point of thinking about installing Windows XP to see if it had the same issue and, if it didn't, giving up on Ubuntu. However after one last search I discovered a post on the XBMC forum. It contained the wisdom of adding the following to /etc/X11/xorg.conf:
Section "Extensions"
    Option         "Composite" "Disable"

So I went ahead and did that, restarted X and all tearing was gone. I don't know where the bug is in the system, but I'm just glad of the ability in Linux to control things like this.

Creative Commons License
The words and photos on this webpage which are created by Toby Gray are licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 2.0 England & Wales License.