Monitoring The Viable Blog

February 28, 2020

Is the Viable Blog online? How well are search engines indexing the Viable Blog? Are people finding the content engaging? These are carefully formed questions that are designed to monitor the Viable Blog’s success in terms of the goal. And to support the continued search for improvements that will move this endeavor towards achieving the goal better.

In this second iteration (and post) of the Viable Blog, we will explore these questions as requirements. Here are the requirements we are addressing:

REQ4: The website must be online
All missed opportunities to make money must be minimal.
Fit Criterion
An alert must be sent to a system's administrator whenever the website is offline.
The Viable Blog is online (at a minimum) 99% of the time.
REQ5: Monitor search engine
Search engines provide helpful tips on opportunities to make improvements.
Fit Criterion
The Viable Blog is properly configured on the Google Search Console.
The Viable Blog is properly configured on the Bing Webmaster Tools.
Any improvements identified by the services must be acted upon.
REQ6: Where possible user engagement must be monitored
It's important to measure user engagement in order to keep moving towards the goal.
Fit Criterion
A Google Analytics account is created.
The account's tracking script is present on all pages.
REQ7: The website must be compliant with the cookies law
Activities such as measuring user engagement is likely to require the setting of cookies.
Fit Criterion
Before cookies are set, consent is obtained.
A cookies policy is provided which is in line with the law.

Is the Viable Blog online?

In this section, let’s address the requirement REQ4 - the website must be online.

The question “is the Viable Blog online?” can be fulfilled using a third-party website monitoring service. In the case of the Viable Blog I am using UptimeRobot. I like UptimeRobot because it’s a very affordable (if not free) simple to use website monitoring service.

Configuring UptimeRobot via Terraform

As we are embracing infrastructure-as-code in the development of the Viable Blog, let’s configure UptimeRobot in the infrastructure-as-code way. And let’s continue using Terraform as we did in The First Iteration of the Viable Blog. In the first iteration we configured CloudFlare using Terraform.

As there is no official UptimeRobot Terraform provider, the first task in configuring UptimeRobot via Terraform was to find a 3rd party provider. The UptimeRobot provider I found is named Terraform UptimeRobot provider. To add the 3rd party provider to Terraform, I copied the latest binary, from the project’s release page, into ~/.terraform.d/plugins as follows:

wget https://github.com/louy/terraform-provider-uptimerobot/releases/download/v0.4.3/terraform-provider-uptimerobot_0.4.3_linux_amd64.tar.gz -P ~/Downloads
tar -xf ~/Downloads/terraform-provider-uptimerobot_0.4.3_linux_amd64.tar.gz 
mkdir ~/.terraform.d/plugins
mv ~/Downloads/terraform-provider-uptimerobot ~/.terraform.d/plugins

For more information on installing 3rd party Terraform plugins see Terraform Third-party Plugins.

Now that we have an UptimeRobot Terraform provider, let’s begin configuring a website monitor. I began by adding the following Terraform configuration to a new file named uptimerobot.tf.

provider "uptimerobot" {
}

As I am sure you can guess, this block simply tells Terraform to look for a binary that declares itself to be the UptimeRobot Terraform provider.

Next, I added the website monitor configuration to the Terraform configuration as follows:

resource "uptimerobot_monitor" "main" {
  friendly_name = "The Viable Blog"
  type          = "keyword"
  keyword_type  = "not exists"
  keyword_value = "Welcome to the Viable Blog"
  url           = "https://viable.blog"
  interval      = 300
}

The ‘uptimerobot_monitor’ block instructs UptimeRobot to add a monitor that requests the Viable Blog’s homepage every 5 minutes. And the monitor is configured to go into a state of alert if the keyword phrase “Welcome to the Viable Blog” does not exist.

In case you are wondering, why not simply look for an okay status? Experience has taught me that a status ‘200 OK’ monitor is not reliable. Once I had a website monitor suggest everything was fine, meanwhile the web-server’s own default welcome page was showing “Welcome to Nginx!” - that was a monitoring fail. Hence I now look for something specific to each project.

The last bit of configuration to add was an alert channel. Or two alert channels in my case. One alert to be sent to my email and another alert to be sent via SMS to my mobile phone. It turned out that I could not simply add the following Terraform which attempts to configure a new SMS alert.

# This does not work!
resource "uptimerobot_alert_contact" "text_alert" {
  friendly_name = "Text Alert"
  type = "sms"
  value = "<REDACTED>"
}

I was attempting to configure a new SMS alert via Tarraform. However, after sometime I realised it’s not possible. According to the UptimeRobot API page, which states “[setting up] SMS alert contacts [via the API] are not supported yet”. Instead, I manually configured the alert channels via UptimeRobots’ GUI and then referenced the values as Terraform data.

data  "uptimerobot_alert_contact" "email_alerts" {
  friendly_name = "Viable Blog Email Alerts"
}

data  "uptimerobot_alert_contact" "sms_alerts" {
  friendly_name = "Viable Blog SMS Alerts"
}

As you can see, the Terraform data blocks reference the corresponding alert channel via the channel’s name.

Okay, so putting everything together, here is a copy of uptimerobot.tf used as configuration in deploying the second iteration of the Viable Blog.

provider "uptimerobot" {
}

data  "uptimerobot_alert_contact" "email_alerts" {
  friendly_name = "Viable Blog Email Alerts"
}

data  "uptimerobot_alert_contact" "sms_alerts" {
  friendly_name = "Viable Blog SMS Alerts"
}

resource "uptimerobot_monitor" "main" {
  friendly_name = "The Viable Blog"
  type = "keyword"
  keyword_type = "not exists"
  keyword_value = "Welcome to the Viable Blog"
  url = "https://viable.blog"
  interval = 300

  alert_contact {
    id = data.uptimerobot_alert_contact.email_alerts.id
  }

  alert_contact {
    id = data.uptimerobot_alert_contact.sms_alerts.id
  }
}

But before I could run terraform apply, I needed to get an UptimeRobot API key. Also export the UptimeRobot environment variable, named UPTIMEROBOT_API_KEY, with the value of the API key. Finally, I ran terraform init (this initialised Terraform with the new UptimeRobot provider) then at this point terraform apply was good to go.

Project Specific Environment Variables

Since the first post named The First Iteration, I have moved the environment variables into a file called environment.sh. The environment.sh is located in the root of this project, but because it’s a private file, the environment.sh file is untracked by Git (using .gitignore). Keeping environment variables closer to the project’s source code is one way of enabling the development and maintenance of more than one project on the same computer. Now, to load the environment variables into a terminal session, I simply run . environment.sh. I believe this is a clean way of working because each terminal session’s variables are isolated.

Terraform Apply via Make

Since the first post named The First Iteration, I have updated the project’s Makefile to be more generic in terms of changing infrastructure configuration. I have changed update-cloudflare to terraform-apply and added terraform-plan. Here is a copy of the changes in the Makefile.

# Apply all Terraform changes
terraform-apply:
	cd infrastructure/ && terraform apply -auto-approve -var-file="settings.tfvars"

# Print a plan of Terraform changes
terraform-plan:
	cd infrastructure/ && terraform plan -var-file="settings.tfvars"

How well are search engines indexing the Viable Blog?

In this section let’s address the requirement REQ5 - Monitor search engines.

Requesting on the Google Search Console

I began by going to the Google Search Console, where I added the Viable Blog as a ‘Domain Property’. The Google Search Console asked me to added a ‘google-site-verification’ DNS TXT record as proof of ownership. So I add the DNS TXT record via Terraform configuration to CloudFlare as follows:

variable "google_site_verification_code" {
  description = "The Google Search Console verification code"
  type = string
}

resource "cloudflare_record" "google_site_verification_code" {
  zone_id = var.cloudflare_zone_id
  name = "viable.blog"
  type = "TXT"
  value = "google-site-verification=${var.google_site_verification_code}"
}

The ‘google_site_verification_code’ variable block, that I added, was not strictly necessary (at this point). However, because Terraform variables can have descriptions, it makes sense to add a variable. This variable’s description clearly defines what a ‘google-site-verification’ DNS TXT record is for. That will be handy in the future when I have forgotten where it is used.

The second Terraform block, named ‘cloudflare_record’, simply adds the google-site-verification=... DNS root TXT record to CloudFlare’s DNS server.

With the Terraform ready to go, I ran make terraform-apply. I then headed back to the Google Search Console. And unsurprisingly, it confirmed that the ownership of the Viable Blog was verified. Being verified, I was then able to fully configure the Viable Blog on the Google Search Console.

It’s worth noting that when I submitted the Viable Blog’s XML site map, the Google Search console opened a dialogue that said “You will be notified if anything goes wrong with it in the future”. This message is a good indication that the requirement to monitor search engines is met - at least for Google.

Bing

In order to receive search engine indexing feedback from Bing, the Viable Blog needs to be registered on Bing’s Webmaster Tools. The verification process was much the same as the Google Search Console’s process. However instead of adding a DNS TXT record to CloudFlare, I opted to add a ‘BingSiteAuth.xml’ file to the root of the Viable Blog.

Just like on the Google Search Console, I added the Viable Blog’s XML site map. And with that the Viable Blog was configured on Bing Webmasters. In terms of the requirement to monitor Bing, I think it’s a case of periodically checking the Bing Webmaster’s Console. That is because, the Bing Web Master’s does not seem to send out notifications of any errors or improvements.

Whilst configuring the Viable Blog on Bing Web Masters, I noticed the Bing Web Masters’ API. The Bing Web Master’s API looks interesting, so I will likely look into it’s capabilities at some point soon.

Are people finding the content engaging?

Let’s address the requirement REQ6 Where possible user engagement must be monitored

The question “Are people finding the content engaging?” can be regularly analysed through a third-party analytics service. In the case of the Viable Blog I am using Google Analytics. I like Google Analytics because it’s a well known trusted service.

At this point, I simply created an account for the Viable Blog on Google Analytics. Subsequently I added the Google Analytics script to the Viable Blog’s template so that it’s included on every page. In the near future I will begin to make use of Google Analytics finer grain tooling, such as event tracking. But for now, the requirement is met. Let’s address legal compliance next.

Cookies law compliance

It’s time to address the requirement REQ7 The website must be compliant with the cookies law

Google Analytic sets cookies and setting cookies has a compliance overhead. In my case, the Viable Blog must comply with the EU Cookies Directive as enacted through the United Kingdom’s domestic legislation. In the United Kingdom, the ICO (Information Commissioner’s Office) enforces compliance.

Luckily the ICO have an easy to follow guide on cookie compliance.

Adding a Cookies Policy

The ICO’s guide suggests a website must provide people with information about the cookies set when visiting the Viable blog. So I bought a Cookies Policy template from Docular. Docular provide legal templates that are written for UK laws. The policy cost £10. I filled in the template via Docular’s easy to use online tool.

Docular’s website offered me several download formats. I found the Microsoft Word DOCX option to be the easiest format to convert to a web page. In my case I copied the contents of the Word document into RegexBuddy and converted it to HTML. Here are the five transforms I applied.

# 1 - Wrap the title in a H1 element
# Capture 
^([\w\s]+?)$
# Replace
<h1>$1</h1>

# 2 - Put the subsections into DL elements 
# Capture
^(\d+\.\d+)\s(.*?)$
# Replace
<dl class="row">
<dt class="col-sm-1 offset-sm-1">$1</dt>
<dd class="col-sm-10">$2</dd>
</dl>

# 3 - Put the top level sections into DL elements
# Capture
^(\d+\.)\s(.*?)$
# Replace
<dl class="row section-header">
<dt class="col-sm-1"><strong>$1</strong></dt>
<dd class="col-sm-11"><strong>$2</strong></dd>
</dl>

# 4 - Put lists into DL elements
# Capture
^(\(\w+\))\s(.*?)$
# Replace
<dl class="row">
<dt class="col-sm-1 offset-sm-2">$1</dt>
<dd class="col-sm-9">$2</dd>
</dl>

# 5 - Tidy up
# Remove
(?six)
(</dl>\s+?
<dl\sclass="row">)
(?!\s+?<dt\sclass="col-sm-1"><strong>\d+?\.)

The Regex above essentially is a five step “find and replace” that I used to transformed Docular’s Word format into the Viable Blog’s Cookies Policy.

The ICO’s guide suggests a website must obtain consent before setting unnecessary cookies. This obligation can be fulfilled via a third-party Cookie consent service. In the case of the Viable Blog I set up the Cookiebot service. The Cookiebot is a simple freemium service that provides a good cookie consent mechanism.

The set up has two parts. The first part is to configure via their website console how you want the cookies consent mechanism to appear. The second part was to add a JavaScript to the HTML head element of Viable Blog. The JavaScript automatically detects all non-required cookies on a web page and blocks them until a user has given their consent.

Once a month the Cookiebot service will scan the Viable Blog and update the list of Cookies used. This lets visitors see a list of the cookies used by the Viable Blog. This reduces a maintenance overhead, in terms of keeping the cookies list up to date.

Let’s ship the second iteration

So that’s the end of the second iteration of the Viable Blog. Basic monitoring has been added. The website’s administrator can relax in the safe knowledge the website is online. And the feedback from search engines and user engagement will begin to get collected so that the opportunities to grow the website into something awesome will hopefully be spotted.

In the next few iterations of the Viable Blog (and accompanying blog posts) we will look at: tests; templating; security; and more…

If you are finding the Viable Blog helpful please do buy me a coffee, as this will help motivate me - and because three coffees will cover the cost of the cookie policy.