Over the last few years, CloudAMQP has strived to make it easier for developers to move into architecture with microservices. This tutorial will guide you through the jungle.

Content in general

The tutorial shows three microservices that, when used together, build a weather reporting web page. The basic structure consists of 1.) A Raspberry Pi that sends temperature data to another application via a message queue; 2.) A receiving application that stores the temperature data in a database; 3.) A final web application displays the data on a simple web page.

This tutorial includes five steps:

1. An explanation of the microservice architecture and why you should use a message queue (RabbitMQ in our examples).
2. The platform Heroku (PaaS) and how to set up an application on that platform.
3. A Raspberry Pi and a DHT11 - a temperature and humidity module, which collects weather data and sends this information to the message queue. (Examples with randomly generated data will also be shown.)
4. The application that grabs messages from the queue and adds them into a database, which is PostgreSQL in this tutorial.
5. Finally, a web application that displays the collected data on a web site. In this tutorial, we use the Python framework Flask.

Prerequisites

For test purposes, this tutorial includes explanations, sample code, and TODO sections, telling you what you need to do before proceeding to the next section.

Ensure that you have basic knowledge of Git, how it works as well as knowledge of how to use the command line/terminal.

All code examples starting with a dollar sign ($) are code that should be written into a terminal. The dollar sign is not typed as part of the command.

TODO: Familiarize yourself with Git.

Note: All programming in this tutorial is in Python. To get the most out of this guide, make sure you have Python3 installed on your machine. Install it before you proceed from the Python official website.

$ python3
Python 3.7.4 (v3.7.4:e09359112e, Jul  8 2019, 14:54:52)

TODO: Make sure you have Python3 installed

Install the pip tool. pip is a PyPA recommended tool for installing Python packages. More information on how to install pip can be found here: pip installation.

TODO: Make sure you have the pip tool installed.

Table of contents

• Part 1: Microservices and message queues
• Part 1.1: What is a message queue and why would I use one?
• Part 1.2: Why not insert the data from the sensor straight into the database?
• Part 1.3: The architecture of a message queue

• Part 2: Heroku (PaaS)
• Part 2.2: Set up your first application (Microservice #2, “queue-to-database”)
• Part 2.3: Deploy to Heroku
• Part 2.4: Bind a message queue to the application

• Part 3: Raspberry Pi and the DHT11
• Part 3.1: Collect temperature data and publish collected data to the message queue

• Part 4: Database - Access temperature data and insert it into a database
• Part 4.1: What is a database?
• Part 4.2: Add a database to the microservice: “queue-to-database”
• Part 4.3: Access temperature data
• Part 4.4: Insert temperature data into the database

• Part 5: Flask Web Application - Show temperature data
• Part 5.1: Set up the web application in Heroku (Microservice 3)
• Part 5.2: Set up flask
• Part 5.3: Show data from the database in your web application

Part 1: Microservices and message queues

An individual microservice is a service that exists for a single purpose; one that is self-contained and independent of other instances and services. When building an application in a microservice architectural style, the approach is to develop a single application consisting of two or more small services (microservices). Each microservice should be developed separately, and the finished application is the sum of its microservices.

In a microservice architecture, applications are built and deployed as highly decoupled focused services. Decoupling is used to describe how much one piece of a system relies on another part of the system. Decoupling is the process of separating the services so that their functionality will be more self-contained. A decoupled system is achieved when two or more systems are able to communicate without being connected. The systems can remain wholly autonomous and unaware of each other. A change in one service shouldn't require a modification in the other services. In this case, the Raspberry Pi can send weather data to the message queue, even if the service that inserts the data into the database is under maintenance.

1.1 What is a message queue, and why would I use one?

Message queuing is a style of service-to-service communication. It allows applications to communicate by sending messages to each other. One service must put a message on a predefined queue to be able to communicate with another service. The other service retrieves the message from the queue and processes the requests and the information contained in the message. The message queue provides temporary storage when the destination program is busy or not connected.

1.2 Why not insert the data from the sensor straight into the database?

Less code has less potential to break
Imagine that you have a complex app, with a sensor that is placed somewhere far away.
If something breaks in the code, you need to get out there to fix it. Placing a message with the collected data to a message queue only requires a few lines of code and has less potential to break.

Persisting data until it has been fully processed
Sometimes a process fails while processing data. Unless that data is persisted, it’s lost forever. Queues mitigate this by keeping data until it has been processed and sometimes even acknowledged.
Many message queues require the process to explicitly indicate that the processing has finished before the message is removed from the queue. This ensures that your data is kept safe until it can be handled.

The database might be busy
If you have many applications connecting to the database at the same time, your database might be busy processing other requests. The database could also be blocked due to a request that takes a long time to process. A busy database might lead to time outs on other requests.

A message queue adds a middle layer; all information is first of all added to the queue. The application that adds the data to the database pulls messages from this queue when the application is ready to handle it.

Achieve decoupling - transfer data across different technologies
Message queues achieve decoupling, which keeps the architecture flexible. It also makes it very easy to connect two applications even if they are written in different languages. For example, you can write the code that collects data in C and the service that moves the data into a database in Python.

Scaling is as simple as adding more power
If the number of messages received by the queue is growing, you always have the option to add more subscribers to the queue. No code needs to be changed and no configurations need to be tweaked. Scaling is as simple as adding more consumers.

1.3 Architecture of a message queue

The underlying architecture of a message queue is simple; there are client applications called producers that create messages and deliver them to the message queue. Another client application called consumer connects to the queue and gets the messages to be processed. Messages placed onto the queue are stored until the consumer retrieves them.

Part 2: Heroku (PaaS)

Publishing the application on the internet and providing a public URL to it is easy with Heroku. Heroku is a Platform as a service (PaaS) that avoids the hassle of dealing with servers. Heroku enables developers to build, run, and operate applications entirely in the cloud. It is a platform for apps, with app management and instant scaling, for development and production.

All developers have to do is sign up, download a few tools, upload the code to the platform, and voila, the app is online.

Part 2.1: Create a Heroku account

Start by creating a Heroku account if you do not already have one here: https://www.heroku.com/home

TODO: Create your Heroku account

Part 2.2: Set up your first application (Microservice #2)

Microservice #2 will be the heart of the architecture in this guide. It grabs weather data from the message queue and adds it to a database as illustrated in the image:

You need to install the Heroku Command Line Interface (CLI). This CLI can be used to manage and scale your applications, provision add-ons, view your application logs, and run your application locally. Once installed, you can use Heroku commands from your command shell.
Use the Heroku login command to log on to the Heroku CLI:

$ heroku login

TODO: Download Heroku CLI and login to Heroku.

In Heroku, an application can be created via the Web UI or the terminal. Name the application (in this example called “dht11module-queue-to-db”) and selected Python as language.

Navigate to the “Create New App” page, as in the image below, or write the following line in the terminal using your name to create the app:

$ heroku create dht11module-queue-to-db --buildpack heroku/python

TODO: Create a new app on Heroku (using CLI or the Web UI), use Python as language (buildpack)

The next step is to set up the structure of the application. First, a requirements file is needed for Heroku to detect requirements for this application, and to install appropriate Python packages. The file can be created by using the freezing command in pip. Freezing reads the versions of all installed packages in a local virtual environment and then produces a text file with the package version for each Python package specified.

$ pip freeze > requirements.txt

Create another file named app.py and insert the following “Hello world” code into that file. This code validates that Python3 is installed and working and that you can run your application locally.

print("Hello world. This line will be printed.")

$ python3 app.py
> Hello world. This line will be printed.

TODO: Enter the folder for your new application and create a requirements file.
Test run the application by entering “python3 app.py” in the terminal:

TODO: Create an app.py file and add the "Hello world" code and make sure it is able to run via command line

Part 2.3: Deploy to Heroku

You are now able to run your code locally, and it’s time to try to run the same code from Heroku.

Set git remote to Heroku via your heroku git url, in this case: https://git.heroku.com/dht11module-queue-to-db. Write the following in the project catalogue for the service, where dht11module-queue-to-db is your app name:

$ heroku git:remote -a dht11module-queue-to-db

Push your code to Heroku:

$ git add *
$ git commit -m "init"
$ git push heroku master

You can now test run the Heroku app by connecting to Heroku and running it in the cloud.

$ heroku run bash --app dht11module-queue-to-db
> Running bash on ⬢ dht11module-queue-to-db... up, run.4825 (Free)
$ python3 app.py
> Hello world. This line will be printed.

TODO: Add Heroku git as remote, push the code to Heroku, and run the service on Heroku

Part 2.4: Bind a message queue to the application

A great feature of Heroku is that it is straightforward to bind a service to your application. There are 150+ Heroku Add-ons to choose from in the Elements Marketplace. You will find everything you need to speed up your development cycle and make your team more efficient.

This example will show you how to bind a message queue to the application and how to use RabbitMQ via the Heroku Add-on service called CloudAMQP. CloudAMQP is a Software as a service (SaaS) and, in this case, RabbitMQ as a service. SaaS is simply a model for the distribution of software.

The RabbitMQ message queue addon can also be added to your application via the terminal:

$heroku addons:create cloudamqp:lemur

TODO: Bind the CloudAMQP message queue addon to your application

Further reading about Python and Heroku can be found here: devcenter.heroku.com/articles/getting-started-with-python

Part 3: Raspberry Pi and the DHT11 (Microservice #1)

It’s time to look into the Raspberry Pi and the DHT11 weather module. If you don’t have access to a Raspberry Pi, it is still useful to read through this section. Part 4 includes full code that can be used to replace the Raspberry Pi.

The DHT11 sensor is a module that provides temperature and humidity. It is easy to set up and only requires one wire for the data signal. We recommend this guide for setup: www.circuitbasics.com/how-to-set-up-the-dht11-humidity-sensor-on-the-raspberry-pi/

TODO: Set up the Raspberry Pi and the DHT11 sensor

Part 3.1: Collect temperature data and publish collected data to the message queue

You need to set up a CloudAMQP environment variable in your Pi, and add the CloudAMQP connection URL into it. Details about your CloudAMQP instance, such as connection URL, server name, user/vhost and password can be seen on the details page.

The following code collects temperature and humidity and adds it into a message, and sends it to RabbitMQ.

#microservice number 1, Raspberry Pi sending data to Microservice number 2
import pika
import os
import time
import json

message_interval = 20  # seconds
reading_interval = 5  # seconds
sensor = 11 # might need to be changed depending on the pi setup
pin = 4 # might need to be changed depending on the pi setup

#Access the CLODUAMQP_URL environment variable
url = os.environ.get('CLOUDAMQP_URL')
params = pika.URLParameters(url)
connection = pika.BlockingConnection(params)
# start a channel
channel = connection.channel()
# Declare a queue
channel.queue_declare(queue='weather')

isSimulation = 1
if isSimulation:
  import random
  def genrand():
      return random.random(), random.random() * 10
else:
  import Adafruit_DHT

while True:
  body = []
  timeout = time.time() + message_interval
  while True:
    if time.time() > timeout:
      break
    if isSimulation:
      humidity, temperature = genrand()
    else:
        humidity, temperature = Adafruit_DHT.read_retry(sensor, pin)

    read_time = time.time()
    d = {'t': read_time, 'T': temperature, 'H': humidity}
    body.append(d)
    time.sleep(reading_interval)

    print('sending data')
    channel.basic_publish(exchange='',
                        routing_key=''weather'',
                        body=json.dumps(body))

connection.close()

TODO: Collect the weather data and send it to RabbitMQ

You can now view the message flow in the message queue by entering the management interface for RabbitMQ. A link to the management interface can be found by opening up the CloudAMQP addon and clicking on RabbitMQ Manager.

RabbitMQ Manager is a user-friendly interface that let you monitor and handle your RabbitMQ server from a web browser. Among other things, queues and connections can be created, deleted, and listed. You can monitor message rates and send and receive messages manually.

Learn more about the management interface from our blog series: Getting started guide - The RabbitMQ Management Interface

TODO: View the message queue in the RabbitMQ management interface

Part 4: Database - Access temperature data and insert it into a database

We have now successfully sent data from one application (from the Raspberry Pi application) to the message queue. The next step is to store it in a database, we are still working in microservice #2.

Part 4.1: What is a database?

A database allows you to store information related to a specific topic in an organized way. A database is excellent when you need to store a searchable collection of data in a computer system. Today's systems often use databases in their architecture. For example, almost every webshop has a database with its products and nearly every web site with user logins uses a database to store user information.

Most databases contain one or more tables which may each include several different fields. Its name uniquely identifies a table. Each table is made up of rows and columns, like a grid just like an Excel sheet. The example shows one single table that stores the timestamp and the temperature and later on the humidity:

Timestamp	Temperature	Humidity
2019-06-10 10:00:00	15	95
2019-06-09 10:00:00	17	98

Part 4.1: Add a database to the microservice

This example uses ElephantSQL, which is PostgreSQL as a Service, to add a database to the application using the free plan.

TODO: Go to ElephantSQL and create an account. Create a free plan named turtle.

Part 4.2: Connect to the database from the microservice

We recommend the client psycopg2 to connect to PostgreSQL. Psycopg is the most popular PostgreSQL database adapter for Python, and should be added to your requirements file.

TODO: Add psycopg2 to your requirements file

Set the environment DATABASE_URL variable to point to the remote ElephantSQL and to the local machine as a backup. The connection string for ElephantSQL can be found on the details page.

TODO: Add the connection string to the DATABASE_URL environment variable

Below is an example code that connects the application to the database and prints information about the database. Run this code to make sure that the connection is correctly set up and working.

#example of how to store data in a database with pyscopg2
import pika, os
import urllib.parse as up
import psycopg2

try:
  up.uses_netloc.append("postgres")
  url = up.urlparse(os.environ["DATABASE_URL"])
  connection = psycopg2.connect(database=url.path[1:],
    user=url.username,
    password=url.password,
    host=url.hostname,
    port=url.port
  )

#Print PostgreSQL Connection properties
print ( connection.get_dsn_parameters(),"\n")
#Print PostgreSQL version
cursor.execute("SELECT version();")
record = cursor.fetchone()
print("You are connected to - ", record,"\n")
except (Exception, psycopg2.Error) as error :
print ("Error while connecting to PostgreSQL", error)
finally:
  #closing database connection.
  if(connection):
    connection.close()
    print("PostgreSQL connection is closed")

TODO : Validate that you can set up a connection from Microservice #2 to the database

Part 4.2: Collect the temperature data and insert it into the database

It’s time to collect the temperature data from the message queue and permanently store this data in the database. The code below connects to RabbitMQ and accesses a message, and stores the data in a database.

#microservice number 2, receives data from a message queue and stores in a database
import pika, os
import urllib.parse as up
import psycopg2
from datetime import datetime
import json

#Access the CLODUAMQP_URL environment variable and parse it (fallback to localhost)
url = os.environ.get('CLOUDAMQP_URL', 'amqp://guest:guest@localhost:5672/%2f')
params = pika.URLParameters(url)
connection = pika.BlockingConnection(params)
channel = connection.channel() # start a channel
channel.queue_declare(queue='weather') # Declare a queue

def store(ch, method, properties, body):
body = json.loads(body)
print("[X] Received time:" + str(body["t"]) + " and temperature: " + str(body["T"]))
try:
  up.uses_netloc.append("postgres")
  url = up.urlparse(os.environ["ELEPHANTSQL_URL"])
  connection = psycopg2.connect(database=url.path[1:],
    user=url.username,
    password=url.password,
    host=url.hostname,
    port=url.port
  )
  cursor = connection.cursor()
  cursor.execute("CREATE TABLE IF NOT EXISTS weather (id SERIAL PRIMARY KEY, time TIMESTAMP, temperature integer);")

  postgres_insert_query = """ INSERT INTO weather (TIME, TEMPERATURE) VALUES (%s,%s)"""
  record_to_insert = ((body["t"]), int(body["T"]))
  cursor.execute(postgres_insert_query, record_to_insert)
  connection.commit()
  count = cursor.rowcount
  print (count, "Record inserted successfully into weather table")

  except (Exception, psycopg2.Error) as error :
    print ("Error while connecting to PostgreSQL", error)
  finally:
    #closing database connection.
    if(connection):
      connection.close()
      print("PostgreSQL connection is closed")

  channel.basic_consume('weather',
    store,
    auto_ack=True)

  print(' [*] Waiting for messages:')
  channel.start_consuming()
  connection.close()

Full code (access + subscribe data + store data in the database) can be found here: https://github.com/lovisajohansson/python-cloudamqp-elephantsql. Remove the part of the code that refers to publishing a message if you have a Raspberry Pi.

TODO: Download the code and make sure that it connects to the message queue, accesses a message from the queue, and inserts the data into the database

You can view the data inserted to the database via the ElephantSQL browser. Write “SELECT * from WEATHER” and all data in that table will be listed.

TODO: Navigate to the ElephantSQL browser and view the stored data

Part 5: Show temperature data

It’s time to create Microservice #3 - the final microservice with the task to show all data stored in the database on a web page. This microservice is built using Flask, which is a Python web development framework used to build web applications.

Part 5.1: Set up the web application in Heroku (Microservice #3)

Microservice #3 will also be set up on Heroku. As said in Part 1, a new application can be created via the web UI or the terminal. Name the application (example “mytemperature”) and selected Python as language.

Navigate to the “Create New App” page as in the image below or write the following line in the terminal (using your name):

$ heroku create mytemperature --buildpack heroku/python

TODO: Create the new microservice in Heroku use the Python buildpack

Install dependencies for Flask, in this case the gunicorn libraries. Gunicorn "Green Unicorn" is a Python Web Server Gateway Interface (WSGI) HTTP server; a server that runs Python web application code. You can install those dependencies by using pip:

$ pip install flask gunicorn

A requirements file is needed, which can be created by running the freezing command again. You will see that both Flask and gunicorn are added to the requirements file.

$ pip freeze > requirements.txt

TODO: Install flask and gunicorn and create the requirements file

This application needs a text file named Procfile which is to be placed in the root of the application to provide a list of the process types. Create the Procfile file and add the code shown below. This code specifies that the application uses a web dyno with gunicorn as the http server.

web: gunicorn app:app --log-file -

TODO: Create the Procfile file

Set git remote to Heroku via https://git.heroku.com/mytemperature and write the following in the project catalogue for the service:

$ heroku git:remote -a mytemperature

Then push the new code to Heroku:

$ git add *
$ git commit -m "init"
$ git push Heroku master

TODO: Add Heroku git remote to the project and push the new code

Part 5.2: Set up the Flask application

Create a base Flask application and verify that everything is correctly installed. Create a file named app.py and copy the code below:

from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello_world():
  return 'Hello World!'

if __name__ == '__main__':
  app.run()

Test run the application using the following command:

$ python3 app.py

Your app should now be running on localhost:5000.

TODO: Write a Hello World Flask application and view the result in the browser on the localhost

You can run this app on the web by pushing the code to Heroku and running the following command:

$ heroku open

You can view the app at your web URL, in this case at https://showtemperature.herokuapp.com/

TODO: Run the Hello world code at Heroku

Part 5.3: Show data from the database in your web application

Add the connection string for the database URL to the environment variable DATABASE_URL.

TODO: Add the database connection URL to the environment variable DATABASE_URL

We recommend that you download the full code for this part of the application, which can be found here: https://github.com/lovisajohansson/python-flask-elephantsql

Below is all the code in sections according to files needed:

Create a config.py file that reads the connection URL from the environment variable. Add the following lines to the file:

import os
class Config(object):
SQLALCHEMY_DATABASE_URI = os.environ['DATABASE_URL']

TODO: Create a config.py file read the DATABASE_URL from the file

Create the file app.py. Insert the code below, which connects to the database and routes to a (to be created) page that shows the weather data.

from flask import Flask, render_template, request
from flask_sqlalchemy import SQLAlchemy

database_uri = 'postgresql+psycopg2://{dbuser}:{dbpass}@{dbhost}/{dbname}'.format(
  dbuser=os.environ['DBUSER'],
  dbpass=os.environ['DBPASS'],
  dbhost=os.environ['DBHOST'],
  dbname=os.environ['DBNAME']
)

app = Flask(__name__)

app.config.update(
  SQLALCHEMY_DATABASE_URI=database_uri,
  SQLALCHEMY_TRACK_MODIFICATIONS=False,
)

#initialize the database connection
db = SQLAlchemy(app)

@app.route('/')
def view_registered_weathers():
  from models import Weather
  weathers = Weather.query.all()
  return render_template('weather.html', weathers=weathers)

if __name__ == '__main__':
  app.run()

TODO: Create the file app.py and insert the code from above.

Create a model file named models.py, which determines the logical structure of the database, using the code below.

from app import db

class Weather(db.Model):
  """Simple database model to track weather."""

  __tablename__ = 'weather'
  id = db.Column(db.Integer, primary_key=True)
  time = db.Column(db.DateTime)
  temperature = db.Column(db.Integer)

  def __init__(self, time=None, email=None):
  self.time = time
  self.temperature = temperature

TODO: Create models.py and insert the code for the weather model

Create the HTML page that shows the data by first creating a new folder named templates. Two files are added into this folder - base.html , and weather.html. Base.html includes the base structure for all HTML pages and weather.html includes the table containing weather data.

base.html

<!DOCTYPE html>
<html lang="en">
<head>
  {% block head %}
  <title>{% block title %}{% endblock %}</title>
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap.min.css" integrity="sha384-BVYiiSIFeK1dGmJRAkycuHAHRg32OmUcww7on3RYdg4Va+PmSTsz/K68vbdEjh4u" crossorigin="anonymous">
  <link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/css/bootstrap-theme.min.css" integrity="sha384-rHyoN1iRsVXV4nD0JutlnGaslCJuC7uwjduW9SVrLvRYooPp2bWYgmgJQIXwl/Sp" crossorigin="anonymous">
  <script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.7/js/bootstrap.min.js" integrity="sha384-Tc5IQib027qvyjSMfHjOMaLkfuWVxZxUPnCJA7l2mCWNIpG9mGCD8wGNIcPD7Txa" crossorigin="anonymous"></script>
  {% endblock %}
</head>
<body>
    <div id="content" style="max-width: 500px; padding:10px;">{% block content %}{% endblock %}</div>
</body>
</html>

weather.html

{% extends "base.html" %}
{% block title %}Registered weather data{% endblock %}
{% block head %}
{{ super() }}
{% endblock %}
{% block content %}
<h1>Registered weather data</h1>
<table class="table table-striped">
<thead>
<tr>
  <th>Time</th>
  <th>Temperature</th>
</tr>
</thead>
<tbody>
{% for weather in weathers %}
  <tr>
  <td>{{ weather.time }}</td>
  <td>{{ weather.temperature }}</td>
  </tr>
{% endfor %}
</table>
{% endblock %}

You can now run the code! It will show a table of data stored in the database.

$python3 app.py

Summary

In this blog we have shown that you how to use three microservices to create a weather reporting web page:

• Python and the pip tool (language and installing tool)
• App on Heroku using CLI or the Web UI (Platform as a service, PaaS)
• Raspberry Pi (small single-board computer) and DHT11 (temperature and humidity sensor).
• CloudAMQP free account. (Message queue)
• ElephantSQL free account. (Database as a service)
• Flask (micro web framework)

You have created an application with a microservice style framework that:

• Has less potential to break.
• Keeps your data safe until it has been processed to mitigate data loss.
• Avoids too much database traffic leading to time outs on other requests.
• Facilitates the transfer of data across different technologies.
• Scales quickly and easily.

We hope that you like this blog post. If you have any questions or comments, reach out to us at contact@cloudamqp.com

Full code for Microservice #2 (including publishing av weather data): https://github.com/lovisajohansson/python-cloudamqp-elephantsql

Full code for Microservice #3: https://github.com/lovisajohansson/python-flask-elephantsql