Making your own smart ‘machine learning’ thermostat using Arduino, AWS, HBase, Spark, Raspberry PI and XBee

This blog post describes making your own smart thermostat using machine learning (K-means clustering) and a bunch of hardware: Arduino, Raspberry PI, two XBee’s and an Amazon Cloud sever (see: idea in brief). I have to start with a disclaimer. I am not a programmer good programmer and certainly not an electrical engineer. However, this research project yielded a working thermostat that is able to learn over time how to improve energy efficiency.

For those who want to directly go to the smart ‘learning’ part, you can skip to Part 7: Learning and adapting temperature scenarios in the Amazon cloud (SPARK). Otherwise, the structure of this post, after a short introduction and overview, follows the path of data, control and necessary feedback loops as shown in figure 1. Every part contains specific code examples. The full source code can found on github

  1. Introduction & overview
  2. Reading data form a ‘dumb’ thermostat and various temperature sensors (Arduino)
  3. Sending data, at 1,000 values per second, to a Raspberry PI (Python)
  4. Storing data in the Amazon Cloud (HBase)
  5. Turning the boiler on and off at the right time (Arduino)
  6. Using outside temperature and scenarios to control an Arduino from a Raspberry PI
  7. Learning and adapting temperature scenarios in the Amazon cloud (SPARK)
Idea in brief
This blog post describes building and programming your own smart thermostat. The smart part is based on machine learning in the form of K-means clustering to optimize when, how often and how long the boiler/furnace turns on. The thermostat is built on the concept of feedback loops (figure 1).

  1. The first feedback loop is an Arduino directly controlling the boiler (furnace).
  2. The second feedback loop is a Raspberry PI that uses XBee to wirelessly receive temperature data and boiler status information from the Arduino and send instructions back to the Arduino.
  3. The third and last feedback loop runs on a server in the Amazon cloud. This server uses the Spark Machine Learning Library (MLlib) and HBase to optimize the boiler control model that is running on the Raspberry PI.

Figure 1: Overview

Continue reading


Exploring the future of information part 3 – Aristotle on the value of information

The value of information is the level to which it can extend our ability to sense our environment over time and place.

Idea in brief

In this third post I explore a complex question: ‘What is the value of information?’ I have come to the conclusion that to answer this question we have to use the philosophy of Aristotle. More specifically, I discovered that the value of information is the level to which it can extend our ability to sense our environment.

Drawing from my experience as the founder of a digital media company in the 90’s, being an information consultant in the 00’s and working with today’s big data, I first explore the reinvention of the value of information throughout the last three decades. I find that existing theories s do not provide a sufficient answer. The fundamental value of information is not in its application. Rather, the economic theory about ‘making decisions’ reduces the value to choosing between preset options and the business case theory only explains that we have to find value, not how to find it.

To find the value of information I dive deeper into what it means to be a human being. By discovering that Aristotle defines the human soul as a ‘natural sensory body’.,as a result, the value of information is the level to which it can extend our ability to sense our environment of time and place. Applying this theory to existing uses of information, ranging from paper documents to accounting systems, I come to the conclusion that this not only provides a much better explanation of what the value of information is, but that it also gives us a new way in getting more value out of new technology.

Continue reading

Exploring the future of information part 2

Why ancient Egyptian building concepts should not be applied to using information.

This is the second post in a series in which I explore the future of information. In these posts I combine a variety of concepts ranging from the impact of the philosophy of Aristotle on using information to the Internet of Things.

In this second post I explore my own career in the field of Business Intelligence, explain why the pyramid should not be used to represent an organization and how information can be applied in its context instead of a hierarchy. Previous post: Introduction, The rise and deception of post-modernism. Upcoming posts are:

3. Aristotle on the value of information.
4. Consumerization of Information
5. Consumer Data Governance
6. The right app-platform for the Internet of Things
7. The stars in the future field of information technology.

Five years ago I made a career change. I moved from the digital media industry to the field of Business Intelligence (BI). Part of this move was going from an organization of 8 people to a company with 80,000 employees. Starting in this new field I was struck by the abundant use of a particular geometric form: the pyramid. Almost to the extent that the shape itself holds the truth to using information in organizations, which overly simplified what the field of Business Intelligence is about. The shape was used for describing the hierarchy of information itself: data, information, knowledge and wisdom. The shape was used to describe the transformation from raw data to usable information dashboards. Most importantly, the shape was used to represent an organization, often divided into the three levels of operational, tactical and strategic. Concepts often used in the BI paradigm like ‘one version of the truth’, ‘the enterprise data warehouse’ (with an emphasis on enterprise), ‘management information’ and ‘operational BI’ comply with the idea of the organization as a pyramid. When I moved towards to field of Performance Management (PM), I found that here the pyramid had an even greater importance. According do the PM paradigm information flows and narrows to the top by aggregating Key Performance Indicators and targets and actions cascade from the top to the ‘operational’ level.

From the start I felt uneasy with using a pyramid to represent an organization, because it did not fit with the theory from my MBA education. Modern organizations do not function according to a strict hierarchy. Also, the idea of ‘centralized control’ is replaced by empowerment and internal markets. At first I attributed this feeling to moving from a tiny company to one of the largest IT firms in the world. After five years of working as a BI Consultant, however, I have never seen an organization that could be represented by a pyramid. Even when an attempt is made to make it look like one, it always disappoints. One of the best examples comes from when I was working on an assignment for the Academy of a big government institution. This academy, seen as an education related HR department, operated completely independent of the primary activities of the rest of the organization, which dealt mostly with handling large amounts of money. Yet the academy was managed in the same way as any other business unit. Not only did it have to use the same IT systems and other facilities that were designed for these primary activities, but it also had to use the same metrics. This led to such great inefficiencies that any business unit preferred using an external company for training or HR services if they could. This was not only because of cost but also because it was better equipped for doing the task it was set-out to do.

So how should we look at an organization? First a bit of theory. In 1979 the renowned professor Henry Mintzberg published his most famous book ‘The Structuring of Organizations’. In Mintzberg’s model (figure 1) the basic organization consists of five parts: operating core, middle line, strategic apex, techno structure and support staff. To be honest, even here the outline of a pyramid can still be seen. But at leas two parts are added, the techno structure and the support staff. Looking at Mintzberg’s five variations of organizational structure (figure 2) the pyramid almost completely disappears. This is most apparent in the diversified, or divisionalised organization. In the Harvard Business Review article following the book (1981) Minztberg states that most Fortune 500 firms have adopted this divisionalised form. And according to him, “The Divisionalised Form differs from the other four structural configurations in one important respect. It is not a complete structure from the strategic apex to the operating core, but rather a structure superimposed on others”. He further explains that each division has its own structure.

In my article ‘Beyond agility, evolutionary IT-systems and business processes’ I argue that by using divisions organizations can adapt to their environment. Using this form is essential for survival.

Taking a glance on the current organizational structure of the 2012 list of Fortune 500 firms reveals that the divisionalised structure is still most popular amongst these firms. For example ExxonMobil has an organizational structure based on 12 separate global businesses. Philips, the Dutch electronics giant, has three business units based on market sectors and one unit called ‘Innovation & Emerging Businesses’, which contains all support staff. Philips needs a total of six separate organizational charts to explain its organization. Starbucks changed in 2011 to a structure containing five separate organizations: three based on region (China and Asia Pacific, Americas), EMEA and two on the promising brands Seattle’s Best Coffee and Tazo tea. Also in the nonbusiness sector the Divisionalised form is well known. Take for example the Dutch Justice Department. Its organizational “chart” is so complex-containing at least three IT related business units-that it only exists in words and complete sentences. Designing an actual chart would be too complex.

Now what does this mean for the future of information? First of all when using information in organizations the actual complexity of the organization has to be taken into account. This means that the ideal of an ‘Enterprise Data Warehouse’ is no longer attainable. The buying up and selling off of smaller companies by larger corporations not only makes this impossible, but completely integrating business units is also not desirable. Furthermore, ‘one version of the truth’ is a goal that should not be attained. Every business unit is its own organization and operates in its own environment. Every part of an organization, therefore, has its own ‘version of the truth’.

From my own experience I have found it is much better to focus on the actual application of information, regardless of the structure. Information can flow from and to all parts of the organization, as displayed in figure 3. This can be up, down, lateral across business units, diagonal across business units or even from or to outside the organization. A good example of this is the use of information by a well-known large Dutch retailer. They embedded an algorithm in their supply chain to automatically send last minute updates to the warehouses. This algorithm uses data from all corners of the business. By using this diverse information their replenishment process is one of the most efficient in the world. The application of information in its context instead of a hierarchy is something I will further explore in my next posts: ‘Aristotle on the value of information’, ‘the consumerization of information’.

The series continues with:
Part 3 Aristotle on the value of information.

Modified on April 2, 2013 Added link to part 3 in reference to future posts.

Exploring the future of information part 1

Introduction: The rise and deception of post-modernism

This is the first post in a series in which I explore the future of information. In these posts I combine a variety of concepts ranging from the impact of the philosophy of Aristotle on using information to the Internet of Things.

The introduction explores the deception of post-modernism and how it relates to the future of information. Upcoming posts are:

2. Why ancient Egyptian building concepts should not be applied to using information.
3. Aristotle on the value of information.
4. Consumerization of Information
5. Consumer Data Governance
6. The right app-platform for the Internet of Things
7. The stars in the future field of information technology.

“The future of information” might be one of the most pretentious titles for a series of blog posts. I came up with this title after reading an article in F@stCompany called ’10 Tips From Boing Boing On Making Online Content Sing’. Since I wrote my previous post more then a year ago I felt I definitely needed to use some ‘singing content’. The title is based on tip nr. 4, “get an attitude”. I promise, however, that the content of this series will adhere to the concepts of good Storytelling. These posts will be based on my own stories and ideas about the future of information.

It seems like all current articles or blog post start with a famous quote. Most popular are quotes from Steve Jobs, as if just by quoting the Apple guru will make any piece of text more respectable. Trying to find a quote that is as pretentious as the title of this series I remembered a Mac OS application. This application presented you with a new famous quote every time you restarted your computer. I must admit I have seen this software only once, when my cousin bought a second-hand computer for his mother. He bought it at an Apple event in the mid-90’s from a Filemaker salesman. This was during the ‘sabbatical’ years of Jobs from Apple and most people believed there was little future for Macs. The floor size of the exhibitors hall had been reduced by half from the previous year and Filemaker had a clear strategy of moving their database product to the Windows platform. The salesperson might have thought that selling his Mac made more business sense than selling Apple compatible products. After bringing it to my cousins’ mother we booted the machine to see if we had to clean it up. In addition to the pictures that my cousin deemed unfit for his mother, we noticed that the startup screen had been replaced by software showing a quote from a famous philosopher. We immediately removed both pictures and software. Remembering using quotes as a cheap sales tactic, I decided to use a more personal story as an introduction.

Like all 20 something’s I (re)invented ‘post modernism’. This might sound pretentious as well, but it is not. From my experience, all adolescents at some point in their lives come to the conclusion that ‘all truths are relative’. I distinctly remember when I had my ‘revelation’. When I was 22 my parents took my brother and me on a four-week vacation to Japan. The last city we visited was Kyoto, and at the end of our trip we were completely templed-out. So instead of visiting yet another beautiful ancient temple we went to the Miho museum. This is, in my opinion, one of the most beautiful museums in the world. The art is not particularly exciting and I have almost no memory of the paintings and sculptures I saw. The museum itself however left a lasting impression. The entrance is through a pedestrian tunnel that takes you from an ordinary bus station to the museum building that is literally carved out of a mountain. It must have been the lighting in the museum, which feels like you can actually touch a ray of sunlight-I vaguely remember looking at a painting painted by an Russian artists during the communist era of that country-that I came to the conclusion that people’s belief systems are formed by earlier experiences and that the way people see and judge the world is therefore also based these experiences. As a result people see the world through their own lens, and behold the (re)invention (by my 22-year-old self) of post modernism. Two years later I came to the conclusion, like most people, that this insight is useless. The scene in which I had this insight could be described as the opposite from the one above, as it was the ‘not so clean’ kitchen in my student apartment. Nevertheless, no less true, I found out that both ends of the spectrum, complete dogma and completely individual, are so self evident that they are useless, in and of themselves, to explain anything.

Now what do the rise and deception of post-modernism have to do with the future of information? For that I have to go to a more recent museum visited. This will be my last museum story, I promise. After a long renovation the Stedelijk Museum in Amsterdam reopened. A couple of days ago I revisited the museum, and I was especially drawn to the paintings related to the magazine De Stijl. De Stijl was founded in 1917 in the Netherlands and has had many famous contributors, including the painter Piet Mondrian and architect Gerrit Rietveld. Being Dutch myself I have long had an interest in the painters from ‘De Stijl’, but being redrawn to them I wondered what got me so exited. According to its Wikipedia entry the members of De Stijl aimed for a radical reform of the art to keep pace with the technical, scientific and social changes in the world. After recent technical innovations in information technology, like tablets, smartphones, cheap sensors or technology related to big data I believe that it is worthwhile to aim for a conceptual view on the future of information. Now, the magazine De Stijl was founded many years before the creation of post-modernism and is attributed to have had a significant influence in Dutch modernism. Likewise the ‘big-data’ movement, which almost like a modernistic movement is highly technology driven is being countered by almost post-modernistic ideas that it all depends on the context in which technology is used. This however does not do justice to the potential lasting effect of this technology. Both sides are so self evident that they are useless in explaining anything. So while exploring the future of information I will do my best to present a nuanced view and find the middle ground between new information technological development an the context in which information is applied.

The series continues with:
Part 2 Why ancient Egyptian building concepts should not be applied to using information.
Part 3 Aristotle on the value of information.

Modified on April 2, 2013 Added links to parts 2 and 3 in reference to future posts.