Asset trackers are an often required application for any company that has trucks, buses, or other assets that are on the move. In this codelab you will build the backend components that are responsible for driving a Transport Tracker solution, based on the I/O Bus Tracker. We will show you how to use Google Maps APIs Web Services with the Node.js Client for Google Maps Services, along with Firebase Realtime Database, to build a system that can be used to track assets in near real time.

What you'll build

In this codelab, you'll build a Node.js application that's:

  • Scalable - you'll use Firebase Realtime Database to scale out communication load with the front ends.
  • Maintainable - you'll use Node.js programming language to make the back end code simple and easy to maintain.
  • Independent of the client-side tracking mechanism - you'll build a simulator that predicts the route of the vehicles you're tracking, and simulates their location on the map, so that you can test the app independently from any client-side vehicle locator.

What you'll learn

What you'll need

This codelab focuses on the Google Maps APIs Web Services. Non-relevant concepts and code blocks are glossed over and are provided for you to simply copy and paste.

Environment setup

If you don't already have a Google Account (Gmail or Google Apps), you must create one.

Sign in to Google Cloud Platform console (console.cloud.google.com) and create a new project. At the top of your screen, there is a Project drop down menu:

Click on the project drop down menu, then click the option to create a new project.

Enter a name for your project on the resulting form, and take note of your allocated project ID:

Remember your project ID, as you'll use it later. The project ID is a unique name across all Google Cloud projects. The name above has already been taken and will not work for you. Insert your own project ID wherever you see YOUR_PROJECT_ID in this codelab.

Google Cloud Shell

While Google Cloud can be operated remotely from your laptop, in this codelab you'll use Cloud Shell, a command line environment running in the Cloud. Cloud Shell provides an automatically provisioned Linux virtual machine that's loaded with all the development tools you need (gcloud, go, and more). It offers a persistent 5GB home directory, and runs on the Google Cloud, providing enhanced network performance and easier authentication.

To activate Cloud Shell, from the Cloud platform console click the button on the top right-hand side (it should only take a few moments to provision and connect to the environment):

Clicking on the above button will open a new shell in the lower part of your browser, after possibly showing an introductory interstitial:

Once connected to the Cloud Shell, type the following command to verify that you are already authenticated and that the project is already set to your YOUR_PROJECT_ID:

gcloud auth list
gcloud config list project

If for some reason the project is not set, run the following command:

gcloud config set project <YOUR_PROJECT_ID>

Starting developing with Hello World

In your Cloud Shell instance, you'll start by creating a Node.js app that will serve as the basis for the rest of the codelab.

In the toolbar of the Cloud Shell, click on the file icon to open a menu, and then select Launch code editor to open a code editor in a new tab. This web based code editor allows you to easily edit files in the Cloud Shell instance.

Create a new transport-tracker-server directory for your application in the code editor, by opening the File menu, and selecting New > Folder.

Name the new folder transport-tracker-server:

Next you'll create a small Node.js app to make sure everything's working. It is now time for Hello World!

In creating a minimal NPM Node.js application, we need to create two files. The first file, package.json, contains the information required by NPM to correctly download both development and runtime dependencies, as well as run the resulting application. The second file, main.js, contains the code for our application. Create the following 2 files in the transport-tracker-server directory:

package.json

{
  "name": "transport-tracker-server",
  "version": "1.0.0",
  "description": "The server component for Transport Tracker",
  "scripts": {
    "main": "node main.js"
  },
  "author": "Brett Morgan",
  "license": "Apache-2.0"
}

main.js

console.log('Hello, World!');

Back in the Cloud Shell, change directory to transport-tracker-server:

cd transport-tracker-server

Run your application as follows:

npm run main

Querying GTFS files

The core of this application is the timetable information for the Google I/O Bus Tracker. This timetable is stored in a set of CSV files that follow GTFS, the General Transit Feed Specification. To be able to easily use these files, we will load them into an in-memory SQLite database, so that we can then run SQL queries over the data to figure out where each bus should be at the given times.

Make a directory to store the data files:

mkdir gtfs

Change into the gtfs sub-directory:

cd gtfs

Now download the data files.

wget https://raw.githubusercontent.com/googlecodelabs/transport-tracker-codelab/master/backend/step-03/gtfs/agency.txt
wget https://raw.githubusercontent.com/googlecodelabs/transport-tracker-codelab/master/backend/step-03/gtfs/calendar_dates.txt
wget https://raw.githubusercontent.com/googlecodelabs/transport-tracker-codelab/master/backend/step-03/gtfs/routes.txt
wget https://raw.githubusercontent.com/googlecodelabs/transport-tracker-codelab/master/backend/step-03/gtfs/stops.txt
wget https://raw.githubusercontent.com/googlecodelabs/transport-tracker-codelab/master/backend/step-03/gtfs/stop_times.txt
wget https://raw.githubusercontent.com/googlecodelabs/transport-tracker-codelab/master/backend/step-03/gtfs/trips.txt

When the downloads have completed, confirm that you have all six data files in your gtfs directory. You are looking for agency.txt, calendar_dates.txt, routes.txt, stop.txt, stops_times.txt, and trips.txt.

ls

With all that out of the way, change directory back up one to where our source code lives:

cd ..

If you look at the downloaded files you will see that they contain the timetable is for Google I/O 2016. That is because the timetable for buses for Google I/O 2017 were not finalised at the time this tutorial was authored. Sorry about that.

Now we can add a dependency to our application to enable us to use SQLite easily, before getting on to the fun task of writing SQL! In your Google Shell command line window, use npm to install sqlite3, along with bluebird, asyncawait and csv-parse.

npm install sqlite3 bluebird asyncawait csv-parse --save

We are using sqlite3 as a database engine to be able to store and query our GTFS data. We are using bluebird as a javascript promise library, along with asyncawait to be able to write code that deals with the asynchronous nature of database interactions easy. Later on we will again use bluebird to make interacting with Google Maps APIs Web Services easier as well. Lastly, we are using csv-parse to parse the comma separated value data in the above six data files easy to handle.

If you open the package.json file again, you will notice there is a new dependencies section that lists these new runtime dependencies.

package.json

{
  "name": "transport-tracker-server",
  "version": "1.0.0",
  "description": "The backend component of the Transport Tracker",
  "scripts": {
    "main": "node main.js"
  },
  "author": "Brett Morgan",
  "license": "Apache-2.0",
  "dependencies": {
    "asyncawait": "^1.0.6",
    "bluebird": "^3.5.0",
    "csv-parse": "^1.2.0",
    "sqlite3": "^3.1.8"
  }
}

We can now create a Node.js module that is responsible for reading in the GTFS content we downloaded earlier, along with providing the means to query the GTFS timetable. Create a file called gtfs.js in the transport-tracker-server directory, and put the following content into it.

gtfs.js

const fs = require('fs');
const parse = require('csv-parse/lib/sync');
const Promise = require('bluebird');
const sqlite3 = Promise.promisifyAll(require('sqlite3'));

exports.GTFS = class {
  constructor() {
    function tableColumns(properties) {
      return properties
        .map(prop => {
          return `${prop.name} ${prop.type}`;
        })
        .join(',');
    }

    function tablePlaceholders(properties) {
      return properties
        .map(() => {
          return '?';
        })
        .join(',');
    }

    function insertRecord(stmt, record, properties) {
      stmt.run(
        properties.map(prop => {
          return record[prop.name];
        })
      );
    }

    function load(db, filename, properties) {
      const createStmt = `CREATE TABLE ${filename} (
          ${tableColumns(properties)}
      )`;
      db.run(createStmt);
      const csv = fs.readFileSync(`${__dirname}/gtfs/${filename}.txt`);
      const records = parse(csv, {columns: true});
      const insertStmt = `INSERT INTO ${filename}
                          VALUES (${tablePlaceholders(properties)})`;
      const stmt = db.prepare(insertStmt);
      records.forEach(record => {
        insertRecord(stmt, record, properties);
      });
      stmt.finalize();
    }

    function loadAgency(db) {
      load(db, 'agency', [
        {name: 'agency_id', type: 'TEXT'},
        {name: 'agency_name', type: 'TEXT'},
        {name: 'agency_url', type: 'TEXT'},
        {name: 'agency_timezone', type: 'TEXT'},
        {name: 'agency_lang', type: 'TEXT'}
      ]);
    }

    function loadCalendarDates(db) {
      load(db, 'calendar_dates', [
        {name: 'service_id', type: 'TEXT'},
        {name: 'date', type: 'INTEGER'},
        {name: 'exception_type', type: 'INTEGER'}
      ]);
    }

    function loadRoutes(db) {
      load(db, 'routes', [
        {name: 'route_type', type: 'INTEGER'},
        {name: 'route_id', type: 'INTEGER'},
        {name: 'route_short_name', type: 'TEXT'},
        {name: 'route_long_name', type: 'TEXT'},
        {name: 'agency_id', type: 'TEXT'},
        {name: 'route_color', type: 'TEXT'},
        {name: 'route_text_color', type: 'TEXT'}
      ]);
    }

    function loadStopTimes(db) {
      load(db, 'stop_times', [
        {name: 'trip_id', type: 'INTEGER'},
        {name: 'arrival_time', type: 'TEXT'},
        {name: 'departure_time', type: 'TEXT'},
        {name: 'stop_id', type: 'INTEGER'},
        {name: 'stop_sequence', type: 'INTEGER'},
        {name: 'stop_headsign', type: 'TEXT'},
        {name: 'pickup_type', type: 'INTEGER'},
        {name: 'drop_off_type', type: 'INTEGER'},
        {name: 'shape_dist_traveled', type: 'TEXT'}
      ]);
    }

    function loadStops(db) {
      load(db, 'stops', [
        {name: 'stop_lat', type: 'REAL'},
        {name: 'stop_lon', type: 'REAL'},
        {name: 'stop_name', type: 'TEXT'},
        {name: 'stop_id', type: 'INTEGER'},
        {name: 'location_type', type: 'INTEGER'}
      ]);
    }

    function loadTrips(db) {
      load(db, 'trips', [
        {name: 'route_id', type: 'INTEGER'},
        {name: 'trip_id', type: 'INTEGER'},
        {name: 'trip_headsign', type: 'TEXT'},
        {name: 'service_id', type: 'TEXT'}
      ]);
    }

    this.db = new sqlite3.Database(':memory:');
    this.db.serialize(() => {
      loadAgency(this.db);
      loadCalendarDates(this.db);
      loadRoutes(this.db);
      loadStops(this.db);
      loadStopTimes(this.db);
      loadTrips(this.db);
    });
  }

  getCalendarDates() {
    return this.db.allAsync('SELECT * FROM calendar_dates ORDER BY service_id');
  }

  getCalendarDateById(service_id) {
    return this.db.getAsync(
      'SELECT * FROM calendar_dates WHERE service_id = ?',
      service_id
    );
  }

  getCalendarDateByDate(date) {
    return this.db.getAsync('SELECT * FROM calendar_dates WHERE date = $date', {
      $date: date
    });
  }
  getRoutes() {
    return this.db.allAsync('SELECT * FROM routes ORDER BY route_id');
  }

  getRouteById(route_id) {
    return this.db.getAsync('SELECT * FROM routes WHERE route_id = $route_id', {
      $route_id: route_id
    });
  }

  getStops() {
    return this.db.allAsync('SELECT * FROM stops ORDER BY stop_id');
  }

  getStopById(stop_id) {
    return this.db.getAsync('SELECT * FROM stops WHERE stop_id = $stop_id', {
      $stop_id: stop_id
    });
  }

  getStopsForRoute(route_id) {
    return this.db.allAsync(
      ` SELECT * FROM stops
        WHERE stop_id IN (
          SELECT stop_id FROM stop_times
          WHERE trip_id IN (
            SELECT trip_id FROM trips
            WHERE route_id = $route_id))`,
      {$route_id: route_id}
    );
  }

  getStopTimes() {
    return this.db.allAsync(
      'SELECT * FROM stop_times ORDER BY trip_id, stop_sequence'
    );
  }

  getStopInfoForTrip(trip_id) {
    return this.db.allAsync(
      ` SELECT st.arrival_time as arrival_time, st.departure_time as departure_time,
               st.stop_id as stop_id, st.stop_sequence as stop_sequence,
               s.stop_lat as lat, s.stop_lon as lng, s.stop_name as stop_name, cd.date as date
        FROM stop_times as st
          INNER JOIN stops as s
          INNER JOIN calendar_dates AS cd
          INNER JOIN trips AS t
        WHERE st.trip_id = $trip_id
          AND st.stop_id = s.stop_id
          AND st.trip_id = t.trip_id
          AND t.service_id = cd.service_id
        ORDER BY departure_time`,
      {$trip_id: trip_id}
    );
  }

  getStopsForTrip(trip_id) {
    return this.db.allAsync(
      ` SELECT *
        FROM stops
        WHERE stop_id IN (SELECT stop_id
                          FROM stop_times
                          WHERE trip_id = $trip_id
                          ORDER BY departure_time)`,
      {$trip_id: trip_id}
    );
  }

  getStopTimesForTrip(trip_id) {
    return this.db.allAsync(
      ` SELECT * FROM stop_times
        WHERE trip_id = $trip_id
        ORDER BY departure_time`,
      {$trip_id: trip_id}
    );
  }

  getStopTimesForStop(stop_id) {
    return this.db.allAsync(
      ` SELECT * FROM stop_times
        WHERE stop_id = $stop_id
        ORDER BY departure_time`,
      {$stop_id: stop_id}
    );
  }

  getTrips() {
    return this.db.allAsync('SELECT * FROM trips ORDER BY trip_id');
  }

  getTripById(trip_id) {
    return this.db.getAsync('SELECT * FROM trips WHERE trip_id = $trip_id', {
      $trip_id: trip_id
    });
  }

  getTripsForStop(stop_id) {
    return this.db.allAsync(
      ` SELECT * FROM trips
        WHERE trip_id IN (
          SELECT trip_id FROM stop_times
          WHERE stop_id = $stop_id)`,
      {$stop_id: stop_id}
    );
  }

  getTripsForCalendarDate(calendar_date) {
    return this.db.allAsync(
      ` SELECT t.route_id AS route_id, t.service_id AS service_id,
               t.trip_headsign AS trip_headsign, t.trip_id AS trip_id,
               c.date AS departure_date,
               ( SELECT departure_time FROM stop_times
                 WHERE trip_id = t.trip_id
                 ORDER BY departure_time ASC LIMIT 1) as departure_time,
               ( SELECT stop_id FROM stop_times
                 WHERE trip_id = t.trip_id
                 ORDER BY departure_time ASC LIMIT 1) as departure_stop_id
        FROM trips AS t INNER JOIN calendar_dates AS c
        WHERE t.service_id = c.service_id AND c.date = $calendar_date
        ORDER BY departure_date, departure_time`,
      {$calendar_date: calendar_date}
    );
  }

  getTripsOrderedByTime() {
    return this.db.allAsync(
      ` SELECT t.route_id AS route_id, t.service_id AS service_id,
               t.trip_headsign AS trip_headsign, t.trip_id AS trip_id,
               c.date AS departure_date,
             ( SELECT departure_time FROM stop_times
               WHERE trip_id = t.trip_id
               ORDER BY departure_time ASC LIMIT 1) as departure_time,
             ( SELECT stop_id FROM stop_times
               WHERE trip_id = t.trip_id
               ORDER BY departure_time ASC LIMIT 1) as departure_stop_id
        FROM trips AS t INNER JOIN calendar_dates AS c
        WHERE t.service_id = c.service_id
        ORDER BY departure_date, departure_time`
    );
  }

  getTripsForRouteOrderedByTime(route_id) {
    return this.db.allAsync(
      ` SELECT t.route_id AS route_id, t.service_id AS service_id, t.trip_headsign AS trip_headsign,
               t.trip_id AS trip_id, c.date AS departure_date,
             ( SELECT departure_time FROM stop_times
               WHERE trip_id = t.trip_id
               ORDER BY departure_time ASC LIMIT 1) as departure_time,
             ( SELECT stop_id FROM stop_times
               WHERE trip_id = t.trip_id
               ORDER BY departure_time ASC LIMIT 1) as departure_stop_id
        FROM trips AS t INNER JOIN calendar_dates AS c
        WHERE t.route_id = $route_id AND t.service_id = c.service_id
        ORDER BY departure_date, departure_time`,
      {$route_id: route_id}
    );
  }

  getNextThreeTripsForRoute(route_id, calendar_date, time) {
    return this.db.allAsync(
      ` SELECT t.route_id AS route_id, t.service_id AS service_id, t.trip_headsign AS trip_headsign,
               t.trip_id AS trip_id, c.date AS departure_date,
             ( SELECT departure_time FROM stop_times
               WHERE trip_id = t.trip_id
               ORDER BY departure_time ASC LIMIT 1) as departure_time,
             ( SELECT stop_id FROM stop_times
               WHERE trip_id = t.trip_id
               ORDER BY departure_time ASC LIMIT 1) as departure_stop_id
        FROM trips AS t INNER JOIN calendar_dates AS c
        WHERE t.route_id = $route_id AND t.service_id = c.service_id
          AND (departure_date > $calendar_date OR departure_date = $calendar_date and departure_time > $time)
        ORDER BY departure_date, departure_time
        LIMIT 3`,
      {
        $route_id: route_id,
        $calendar_date: calendar_date,
        $time: time
      }
    );
  }

  getTripsInServiceForRoute(calendar_date, time, route_id) {
    return this.db.allAsync(
      ` SELECT  t.route_id AS route_id, t.service_id AS service_id, t.trip_headsign AS trip_headsign,
                t.trip_id AS trip_id,
              ( SELECT   departure_time
                FROM     stop_times
                WHERE    trip_id = t.trip_id
                ORDER BY departure_time ASC limit 1) AS initial_departure_time,
              ( SELECT   arrival_time
                FROM     stop_times
                WHERE    trip_id = t.trip_id
                ORDER BY arrival_time DESC limit 1) AS final_arrival_time
        FROM trips AS t INNER JOIN calendar_dates AS c
        WHERE t.service_id = c.service_id
          AND c.date = $calendar_date
          AND initial_departure_time <= $time
          AND $time <= final_arrival_time
          AND t.route_id = $route_id
        ORDER BY trip_id ASC`,
      {
        $calendar_date: calendar_date,
        $time: time,
        $route_id: route_id
      }
    );
  }
};

The next step is to use this newly created gtfs module to find out how many bus trips were scheduled for a specific day. Add the following code to your main.js file:

main.js

/*eslint-disable unknown-require */
const _async = require('asyncawait/async');
const _await = require('asyncawait/await');
const {GTFS} = require('./gtfs.js');
const gtfs = new GTFS();

_async(() => {
  const trips = _await(gtfs.getTripsForCalendarDate('20160518'));
  console.log(trips.length);
})().catch(err => {
  console.error(err);
});

Now we can run this updated main.js:

npm run main

You should see the answer is 243. Quite a few trips, yeah?

Using the Google Maps Directions API to predict a path from A to B

Now that we know when the I/O buses will run, we need to know by which path they will get there. Google Maps Directions API gives us the capability to ask for directions between two points. We can use this to construct a prediction for where the buses will be, and when. We will use this information later when we build a simulator for the buses.

First, a bit of administrative setup. To use the Google Maps APIs, you need a project on the Google API Console and a Google Maps API key. Go to the Google API Console, and create a project. Click Continue to enable the Directions API. On the Credentials page, click Create credentials and get an API key. Copy the value of your API key.

We are going to get quite a few configuration items as we build out the transport tracker, so let's create a file to organize these. In the editor, make a tracker_configuration.json file, and insert your Google Maps API key into it - replace YOUR_API_KEY with the value of your API key:

tracker_configuration.json

{
        "mapsApiKey": "YOUR_API_KEY"
}

We are going to build a command line Node.js application which will query the Google Maps Directions API web service. We'll use the Node.js Client for Google Maps Services to build up a JSON structure, mapping out where all the buses should be at each time in the day. This reduces the complexity of the simulator and minimizes the amount of API usage while you are in the development phase. We also have dependencies on moment, for ease of working with time, and @mapbox/polyline for decoding polylines.

Add the dependencies using npm:

npm install @google/maps moment @mapbox/polyline --save

Next, add a generate_paths.js file, with the following content.

generate_paths.js

/*eslint-disable no-shadow-global, unknown-require, no-undef-expression*/
const mapsApiKey = require('./tracker_configuration.json').mapsApiKey;
const {GTFS} = require('./gtfs');
const gtfs = new GTFS();
const _async = require('asyncawait/async');
const _await = require('asyncawait/await');
const Promise = require('bluebird');
const moment = require('moment');
const polyline = require('@mapbox/polyline');
const fs = require('fs');
const readline = require('readline');

const googleMapsClient = require('@google/maps').createClient({
  key: mapsApiKey,
  Promise
});

function generate_paths() {
  const trips = _await(gtfs.getTripsOrderedByTime());
  const tripsWithLocations = [];
  trips.forEach((trip, tripIndex) => {
    logProgress(`Processing trip ${tripIndex + 1} of ${trips.length}`);
    const timeCursor = moment(
      `${trip.departure_date} ${trip.departure_time}`,
      'YYYYMMDD HH:mm:ss'
    );
    const tripPoints = [];
    const stopInfo = _await(gtfs.getStopInfoForTrip(trip.trip_id));
    const stops = [];
    stopInfo.forEach(stop => {
      stops.push({lat: stop.lat, lng: stop.lng});
    });
    const request = {origin: stops[0], destination: stops[stops.length - 1]};
    if (stops.length > 2) {
      request['waypoints'] = stops.slice(1, -1);
    }
    var response = _await(googleMapsClient.directions(request).asPromise())
      .json;
    if (response.status === 'OK') {
      const route = response.routes[0];
      route.legs.forEach(leg => {
        leg.steps.forEach(step => {
          const durationInSeconds = step.duration.value;
          const points = polyline.decode(step.polyline.points);
          const secondsPerPoint = durationInSeconds / points.length;
          points.forEach(point => {
            tripPoints.push({
              time: timeCursor.format('YYYYMMDD HH:mm:ss'),
              location: {lat: point[0], lng: point[1]}
            });
            timeCursor.add(secondsPerPoint, 'seconds');
          });
        });
      });
      tripsWithLocations.push({trip: trip, points: tripPoints});
    } else {
      logProgress(' ERROR: ' + response.status);
      process.stdout.write('\n');
      process.exit(1);
    }
  });
  fs.writeFileSync('paths.json', JSON.stringify(tripsWithLocations));
  logProgress('Paths written successfully to paths.json.');
  process.stdout.write('\n');
}

function logProgress(str) {
  // A little bit of readline magic to not fill the screen with progress messages.
  readline.clearLine(process.stdout, 0);
  readline.cursorTo(process.stdout, 0, null);
  process.stdout.write(str);
}

_async(() => {
  generate_paths();
})().catch(err => {
  console.error(err);
});

It is well worth spending a couple of minutes reading through this file to understand what's going on. The first require line pulls in the configuration file we just created, and grabs the Maps API key from it. The next two lines load the GTFS capability we created in the previous step, and instantiate the gtfs object to load the GTFS data files into memory.

The next three require statements add the capability to do async/await style development to Node.js. Next, we add moment and @mapbox/polyline for ease of dealing with time and decoding polylines respectively. The last require is for the Node.js Client for Google Maps Services. We configure it with an API key and a Promise constructor to make the exposed API async/await friendly.

Getting into the meat of this script, we see a function called generate_paths. Firstly we query the GTFS timetable for all of the trips that will happen, ordered by time of first departure. For each trip, we retrieve the stops that the bus will make on this trip, using this to construct a request to the Google Maps Directions API. When we receive the response, we inspect the JSON body to make sure it's an OK result, then we jump into the first (and usually only) route response, walking through each leg for the individual steps in the route. We decode the polyline associated with each step, so that we can plot a realistic path for the bus. We use moment's facility with time to construct a timeline for the trip.

Once we have completed this for each trip in our timetable, we write out the content as a single JSON file, named paths.json. Please do not open this in the editor, as it is a large text file and tends to give most text editors that aren't called vi or emacs reason to pause. The generated JSON is 42mb of text, all on a single line. It is possible to have JSON.stringify generate this content with newlines and appropriate spacing (increasing the generated file size to around 70mb), however attempting to do so causes node running on Cloud Shell to run out of RAM.

Now we can add a script entry to our package.json file to run this script.

package.json

{
  "name": "transport-tracker-server",
  "version": "1.0.0",
  "description": "The backend component of the Transport Tracker",
  "scripts": {
    "main": "node main.js",
    "generate_paths": "node generate_paths.js"
  },
  "author": "Brett Morgan",
  "license": "Apache-2.0",
  "dependencies": {
    "@google/maps": "^0.3.1",
    "@mapbox/polyline": "^0.2.0",
    "asyncawait": "^1.0.6",
    "bluebird": "^3.5.0",
    "csv-parse": "^1.2.0",
    "moment": "^2.18.1",
    "sqlite3": "^3.1.8"
  }
}

Generate the paths as follows:

npm run generate_paths

Making your code prettier, automatically

As a quick aside, I like having my build tooling keep my source code neat and tidy. JavaScript has a really neat tool for this, called prettier. We can install it using npm as follows:

npm install prettier --save-dev

Now add an entry to our package.json file's scripts section to run prettier across our JavaScript files:

package.json

{
  "name": "transport-tracker-server",
  "version": "1.0.0",
  "description": "The backend component of the Transport Tracker",
  "scripts": {
    "main": "node main.js",
    "generate_paths": "node generate_paths.js",
    "prettier": "for file in *.js; do prettier --write --single-quote --bracket-spacing=false $file; done"
  },
  "author": "Brett Morgan",
  "license": "Apache-2.0",
  "dependencies": {
    "@google/maps": "^0.3.1",
    "@mapbox/polyline": "^0.2.0",
    "asyncawait": "^1.0.6",
    "bluebird": "^3.5.0",
    "csv-parse": "^1.2.0",
    "moment": "^2.18.1",
    "sqlite3": "^3.1.8"
  },
  "devDependencies": {
    "prettier": "^1.1.0"
  }
}

Run prettier like this:

npm run prettier

Hopefully everything now looks neat and tidy.

A simulation starts with a heart beat

Now that we have a timetable for our buses, and paths for the buses to follow, we're going to build a simulator for the buses. A simulation starts with a clock that acts like a heartbeat for the rest of the system.

One of the eternal issues when developing systems is being able to peer inside the running system to figure out what is going on. We are going to use Firebase Realtime Database as a communication medium between the various parts of our server, meaning that you can inspect it by looking at the Firebase Realtime Database console.

First you need to set up a Firebase Realtime Database. Go to the Firebase console and click Add project to create a project, then enter a project name:

Click Create Project, then click Add Firebase to your web app.

Go to the Service Accounts tab in your project's settings. Click Generate New Private Key. A JSON file containing your service account's credentials will be downloaded. You'll need this to initialize the SDK in the next step.

Note also the Admin SDK configuration snippet supplied on the Firebase Service Accounts tab. You'll need the Node.js values from this snippet too.

We are going to be using the Firebase Admin SDK for Node.js so that we can use service accounts to write to the Firebase Realtime Database. This allows you to configure your Firebase Realtime Database such that only authenticated users are able to write to it. Properly securing the database is outside the scope of this tutorial, but luckily there is documentation on setting up Firebase Realtime Database Security Rules, along with a recorded Google I/O talk from 2016.

The first step is to make your service account credentials available to the application. Create a file called serviceAccountKey.json, and copy the content of the downloaded service account credentials into that file.

Add the runtime dependency on Firebase Admin SDK for Node.js with npm:

npm install firebase-admin --save

If you now look in your package.json file, you should see something like this:

package.json

{
  "name": "transport-tracker-server",
  "version": "1.0.0",
  "description": "The backend component of the Transport Tracker",
  "scripts": {
    "main": "node main.js",
    "generate_paths": "node generate_paths.js",
    "prettier": "for file in *.js; do prettier --write --single-quote --bracket-spacing=false $file; done"
  },
  "author": "Brett Morgan",
  "license": "Apache-2.0",
  "dependencies": {
    "@google/maps": "^0.3.1",
    "@mapbox/polyline": "^0.2.0",
    "asyncawait": "^1.0.6",
    "bluebird": "^3.5.0",
    "csv-parse": "^1.2.0",
    "firebase-admin": "^4.2.1",
    "moment": "^2.18.1",
    "sqlite3": "^3.1.8"
  },
  "devDependencies": {
    "prettier": "^1.1.0"
  }
}

Now we can move on to the code. We'll start by publishing the current time of the simulation in Firebase. Once we have this, we will start adding additional services that use the time to add additional state to Firebase, for example the upcoming bus trips from the GTFS timetable, or the simulated locations of the buses from the generated paths file.

Create a small class to act as the heartbeat of our simulation. Create a file called heart_beat.js and add the following content to it:

heart_beat.js

/* eslint-disable no-undef-expression */
const moment = require('moment');

const DATE_FORMAT = 'YYYYMMDD HH:mm:ss';

// HeartBeat generates a stream of updates to `timeRef`, with either
// simulated time updates, or real time updates, depending on the
// truthyness of `simulatedTime`
exports.HeartBeat = class {
  constructor(timeRef, simulatedTime) {
    this.simulationTime = moment.utc('2016-05-18 06:00', DATE_FORMAT);
    this.endOfSimulation = moment.utc('2016-05-20 18:00', DATE_FORMAT);
    this.timeRef = timeRef;
    this.simulated = simulatedTime;

    // Update the time once a second
    this.timeTimerId = setInterval(() => {
      this.timeAdvance();
    }, 1000);
  }

  timeAdvance() {
    if (this.simulated) {
      this.timeRef.set({
        display: this.simulationTime.format('h:mm A, MMM Do'),
        moment: this.simulationTime.valueOf()
      });
      this.simulationTime = this.simulationTime.add(30, 'seconds');
      if (this.simulationTime.diff(this.endOfSimulation, 'minutes') > 0) {
        // Reset simulation to start once we run out of bus trips.
        this.simulationTime = moment.utc('2016-05-18 06:00', DATE_FORMAT);
      }
    } else {
      const now = moment();
      this.timeRef.set({
        display: now.format('h:mm A, MMM Do'),
        moment: now.valueOf()
      });
    }
  }
};

Now that we have a class responsible for publishing time in our Firebase instance, it is time to update our main.js to instantiate an instance of HeartBeat, and wire it up with our Firebase Realtime Database. We need to add another couple of entries to our configuration file:

tracker_configuration.json

{
        "mapsApiKey": "YOUR_MAPS_API_KEY",
        "databaseURL": "https://<project-name>.firebaseio.com",
        "simulation": true
}

The mapsApiKey is carried over from the previous step.

The databaseURL is the address of your Firebase Realtime Database, and is dependant on the project you created above. You can find the URL in the Admin SDK configuration snippet supplied on the Firebase Service Accounts tab.

The simulation is used to configure HeartBeat, deciding whether to use simulated time, or report the real system time.

Update your main.js file to contain the following:

main.js

/*eslint-disable unknown-require */
const trackerConfig = require('./tracker_configuration.json');

const admin = require('firebase-admin');
const serviceAccount = require('./serviceAccountKey.json');

admin.initializeApp({
  credential: admin.credential.cert(serviceAccount),
  databaseURL: trackerConfig.databaseURL
});

// Database references
const timeRef = admin.database().ref('current-time');

// Library classes
const {HeartBeat} = require('./heart_beat.js');

new HeartBeat(timeRef, trackerConfig.simulation);

It is now time to tidy up your code:

npm run prettier

And re-run main as follows:

npm run main

You will now be able to open up your Firebase Realtime Database - click the Database tab in the Firebase console. You should see something like this:

We have the pieces, it's time to put them together

So far we have built an in-memory SQL database, loaded it with data from the GTFS time table, constructed a set of paths for our buses from Directions API results, and started publishing time into a Firebase Realtime Database. We can now take these individual pieces, and assemble a complete simulation. Allons-y!

We will now do something that might appear a little strange at first, but allows us to break up the functionality into individual units with minimal interdependencies. We are going to to subscribe to the time reference published by the HeartBeat instance in our TimeTable publishing class.

In the Transport Tracker HTML and JavaScript front end, there are three panels displayed, each with a group of routes. The configuration for the panel layout is in a JSON file, and we are going to need the configuration in TimeTable. So, let's create it now.

panels_config.json

[
  {
    "panel": 0,
    "routesGroups": [[0, 7, 1], [2, 3]],
    "markers": [
      {
        "lat": 37.44291,
        "lng": -122.164276,
        "iconPath": "/images/dashboard/placemarker_yellow.png",
        "name": "Palo Alto Caltrain"
      },
      {
        "lat": 37.40935,
        "lng": -122.12266,
        "iconPath": "/images/dashboard/placemarker_yellow.png",
        "name": "Hilton Garden Inn Palo Alto"
      },
      {
        "lat": 37.394108,
        "lng": -122.07663,
        "iconPath": "/images/dashboard/placemarker_gray.png",
        "name": "Mountain View Caltrain Station"
      },
      {
        "lat": 37.37613,
        "lng": -122.06106,
        "iconPath": "/images/dashboard/placemarker_green.png",
        "name": "Hotel Avante"
      },
      {
        "lat": 37.37091,
        "lng": -122.04287,
        "iconPath": "/images/dashboard/placemarker_green.png",
        "name": "Grand Hotel"
      },
      {
        "lat": 37.352013,
        "lng": -122.01318,
        "iconPath": "/images/dashboard/placemarker_pink.png",
        "name": "Wild Palms Hotel"
      },
      {
        "lat": 37.352524,
        "lng": -122.002,
        "iconPath": "/images/dashboard/placemarker_pink.png",
        "name": "The Domain Hotel"
      },
      {
        "lat": 37.335464,
        "lng": -122.03266,
        "iconPath": "/images/dashboard/placemarker_blue.png",
        "name": "Cupertino Inn"
      },
      {
        "lat": 37.33336,
        "lng": -122.01515,
        "iconPath": "/images/dashboard/placemarker_blue.png",
        "name": "Hilton Garden Inn Cupertino"
      },
      {
        "lat": 37.332687,
        "lng": -122.015,
        "iconPath": "/images/dashboard/placemarker_blue.png",
        "name": "Courtyard San Jose Cupertino"
      }
    ],
    "southWestLat": 37.332687,
    "southWestLng": -122.16427599999997,
    "northEastLat": 37.48,
    "northEastLng": -122.00200000000001
  },
  {
    "panel": 1,
    "routesGroups": [[6, 5, 4], []],
    "markers": [
      {
        "lat": 37.417946,
        "lng": -121.97653,
        "iconPath": "/images/dashboard/placemarker_lime.png",
        "name": "Aloft Santa Clara"
      },
      {
        "lat": 37.40988,
        "lng": -122.00213,
        "iconPath": "/images/dashboard/placemarker_lime.png",
        "name": "Country Inn & Suites By Carlson"
      },
      {
        "lat": 37.383606,
        "lng": -121.962326,
        "iconPath": "/images/dashboard/placemarker_red.png",
        "name": "Biltmore Hotel & Suites"
      },
      {
        "lat": 37.39203,
        "lng": -121.977776,
        "iconPath": "/images/dashboard/placemarker_red.png",
        "name": "Avatar Hotel"
      },
      {
        "lat": 37.38378,
        "lng": -121.9788,
        "iconPath": "/images/dashboard/placemarker_indigo.png",
        "name": "Embassy Suites Santa Clara"
      },
      {
        "lat": 37.387,
        "lng": -121.98322,
        "iconPath": "/images/dashboard/placemarker_indigo.png",
        "name": "Plaza Suites"
      }
    ],
    "southWestLat": 37.383606,
    "southWestLng": -122.15,
    "northEastLat": 37.4263,
    "northEastLng": -121.96232600000002
  },
  {
    "panel": 2,
    "routesGroups": [[9, 8, 10], [11, 12]],
    "markers": [
      {
        "lat": 37.794075,
        "lng": -122.395454,
        "iconPath": "/images/dashboard/placemarker_sf1.png",
        "name": "Hyatt Regency SF"
      },
      {
        "lat": 37.600426,
        "lng": -122.385864,
        "iconPath": "/images/dashboard/placemarker_gray.png",
        "name": "Millbrae BART Station"
      },
      {
        "lat": 37.788204,
        "lng": -122.40887,
        "iconPath": "/images/dashboard/placemarker_sf2.png",
        "name": "The Westin St. Francis SF"
      },
      {
        "lat": 37.616764,
        "lng": -122.38395,
        "iconPath": "/images/dashboard/placemarker_gray.png",
        "name": "San Francisco Airport (SFO)"
      },
      {
        "lat": 37.367252,
        "lng": -121.92668,
        "iconPath": "/images/dashboard/placemarker_gray.png",
        "name": "San José Airport (SJC)"
      }
    ],
    "southWestLat": 37.367252,
    "southWestLng": -122.5,
    "northEastLat": 37.81,
    "northEastLng": -121.92668000000003
  }
]

The panel configuration includes the locations of the stops that are relevant when displaying a certain panel, along with the routes for which we want to show the time tables. We are going to publish the time tables to Firebase in a configuration that mirrors this panel layout configuration, to make putting it all on screen easy for the HTML and JavaScript front end.

Our next step is to create a TimeTable class to handle the responsibility of querying the GTFS store in response to time change notifications from our HeartBeat instance. Let's create a new file called time_table.js and put the following content in it:

time_table.js

/* eslint-disable unknown-require */
const moment = require('moment');
const _async = require('asyncawait/async');
const _await = require('asyncawait/await');

const DATE_FORMAT = 'YYYYMMDD HH:mm:ss';

// TimeTable listens for updates on `timeRef`, and then publishes updated
// time table information on `panelsRef`, using `gtfs` as a source of
// next trips, `panelConfig` for the grouping of routes to panels, and
// `googleMapsClient` to access Directions API for Predicted Travel Times.
exports.TimeTable = class {
  constructor(timeRef, panelsRef, gtfs, panelConfig, googleMapsClient) {
    this.timeRef = timeRef;
    this.panelsRef = panelsRef;
    this.gtfs = gtfs;
    this.panelConfig = panelConfig;
    this.googleMapsClient = googleMapsClient;

    // Cache of Predicted Travel Times
    this.pttForTrip = {};
    // When we last issued a Predicted Travel Time request for a route.
    this.pttLookupTime = {};

    this.timeRef.on(
      'value',
      snapshot => {
        _async(() => {
          const now = moment.utc(snapshot.val().moment);
          this.publishTimeTables(now);
        })().catch(err => {
          console.error(err);
        });
      },
      errorObject => {
        console.error('The read failed: ' + errorObject.code);
      }
    );
  }

  publishTimeTables(now) {
    const panels = _await(
      this.panelConfig.map(panel => {
        return {
          left: _await(
            panel.routesGroups[0].map(route_id => {
              return this.tripsLookup(route_id, now);
            })
          ),
          right: _await(
            panel.routesGroups[1].map(route_id => {
              return this.tripsLookup(route_id, now);
            })
          )
        };
      })
    );
    this.panelsRef.set(panels);
  }

  tripLookup(trip) {
    const stop_info = _await(this.gtfs.getStopInfoForTrip(trip.trip_id));
    return {trip, stop_info};
  }

  haveDirectionsResponseCachedForTrip(trip) {
    return this.directionsResponseForTrip(trip) !== undefined;
  }

  directionsResponseForTrip(trip) {
    return this.pttForTrip[trip.trip_id];
  }

  tripsLookup(route_id, now) {
    function round_moment(m) {
      if (m.second() > 30) {
        return m.add(1, 'minute').startOf('minute');
      }
      return m.startOf('minute');
    }

    const date = now.format('YYYYMMDD');
    const time = now.format('HH:mm:ss');
    const route = _await(this.gtfs.getRouteById(route_id));
    const nextTrips = _await(
      this.gtfs.getNextThreeTripsForRoute(route_id, date, time)
    );

    nextTrips.forEach(trip => {
      this.cacheDirectionsResponseForTrip(trip);
    });

    const returnValue = {route, next_in_label: '', next_in: ''};
    if (nextTrips.length >= 1) {
      const next_trip = _await(this.tripLookup(nextTrips[0]));
      returnValue.next_trip = next_trip;
      if (
        this.haveDirectionsResponseCachedForTrip(returnValue.next_trip.trip)
      ) {
        const ptt = this.directionsResponseForTrip(returnValue.next_trip.trip);
        const time = moment.utc(
          `${next_trip.stop_info[0].date} ${next_trip.stop_info[0].departure_time}`,
          DATE_FORMAT
        );
        let index = 1;
        ptt.routes[0].legs.forEach(leg => {
          const delta = leg.duration.value;
          time.add(delta, 'seconds');
          const time_display = round_moment(time).format('HH:mm:ss');
          next_trip.stop_info[index].departure_time = time_display;
          next_trip.stop_info[index].arrival_time = time_display;
          // Assume we stop at each way point for three minutes
          time.add(3, 'minutes');
          index++;
        });
      }
      const next_trip_time_str = `${next_trip.stop_info[0].date} ${next_trip.stop_info[0].departure_time}`;
      const next_trip_time = moment.utc(next_trip_time_str, DATE_FORMAT);
      const next_trip_delta = next_trip_time.diff(now, 'minutes');
      if (next_trip_delta <= 120) {
        returnValue['leaving_in_label'] = 'Leaving in';
        returnValue['leaving_in'] = `${next_trip_delta} mins`;
        if (nextTrips.length >= 2) {
          let trip_after = _await(this.tripLookup(nextTrips[1]));

          // In the mornings we have a bunch of overlapping trips on inbound routes
          if (
            trip_after.stop_info[0].date === next_trip.stop_info[0].date &&
            trip_after.stop_info[0].departure_time ===
              next_trip.stop_info[0].departure_time
          ) {
            trip_after = _await(this.tripLookup(nextTrips[2]));
          }

          const trip_after_time = moment.utc(
            `${trip_after.stop_info[0].date} ${trip_after.stop_info[0].departure_time}`,
            DATE_FORMAT
          );
          returnValue['next_in_label'] = 'Next in';
          const trip_after_delta = trip_after_time.diff(now, 'minutes');
          if (trip_after_delta <= 120) {
            returnValue['next_in'] = `${trip_after_delta} min`;
          } else {
            returnValue[
              'next_in'
            ] = `${trip_after_time.diff(now, 'hours')} hrs`;
          }
        }
      } else {
        returnValue['leaving_in_label'] = next_trip_time.format('MMM Do');
        returnValue['leaving_in'] = next_trip_time.format('h A');
      }
    }
    return returnValue;
  }

  requestDirectionsForTrip(trip) {
    const trip_info = this.tripLookup(trip);
    const stops = [];
    trip_info.stop_info.forEach(stop => {
      stops.push({lat: stop.lat, lng: stop.lng});
    });
    const request = {origin: stops[0], destination: stops[stops.length - 1]};
    if (stops.length > 2) {
      request['waypoints'] = stops.slice(1, -1);
    }
    return request;
  }

  cacheDirectionsResponseForTrip(trip) {
    if (
      this.pttLookupTime[trip.trip_id] === undefined ||
      moment().diff(this.pttLookupTime[trip.trip_id], 'minutes') > 20 ||
      (this.pttForTrip[trip.trip_id] === undefined &&
        moment().diff(this.pttLookupTime[trip.trip_id], 'minutes') > 3)
    ) {
      this.pttLookupTime[trip.trip_id] = moment();
      const request = this.requestDirectionsForTrip(trip);
      if (
        this.pttLookupFailure == undefined ||
        moment().diff(this.pttLookupFailure, 'minutes') > 20
      ) {
        const initiatedAt = moment();
        this.googleMapsClient
          .directions(request)
          .asPromise()
          .then(response => {
            this.pttForTrip[trip.trip_id] = response.json;
          })
          .catch(err => {
            this.pttLookupFailure = moment();
            console.error(
              `Google Maps Directions API request failed, initiated at ${initiatedAt.format('hh:mm a')}: ${err}`
            );
          });
      } else {
        console.log(
          `Not looking up ${trip.trip_id}, rate limiting due to API error, at ${moment().format('hh:mm a')}`
        );
      }
    }
  }
};

The TimeTable class contains a lot of code. It's responsible for querying the GTFS database, then gathering predictive travel times from the Google Maps Directions API, caching those results, and then grouping the transformed data in accordance with the panel configuration. This is all driven from HeartBeat, via the time published in the Firebase Realtime Database. This class contains all the core business logic of the Transport Tracker, so feel free to gloss over this file and come back to it later when you have time and a need of driving your own time table driven displays.

Next, update main.js to take advantage of the new TimeTable class.

main.js

/*eslint-disable unknown-require */
const trackerConfig = require('./tracker_configuration.json');

const Promise = require('bluebird');
const admin = require('firebase-admin');
const serviceAccount = require('./serviceAccountKey.json');
const panelConfig = require('./panels_config.json');
const googleMapsClient = require('@google/maps').createClient({
  key: trackerConfig.mapsApiKey,
  Promise
});

admin.initializeApp({
  credential: admin.credential.cert(serviceAccount),
  databaseURL: trackerConfig.databaseURL
});

// Database references
const panelsRef = admin.database().ref('panels');
const timeRef = admin.database().ref('current-time');

// Library classes
const {GTFS} = require('./gtfs.js');
const {HeartBeat} = require('./heart_beat.js');
const {TimeTable} = require('./time_table.js');

const gtfs = new GTFS();
new HeartBeat(timeRef, trackerConfig.simulation);
new TimeTable(timeRef, panelsRef, gtfs, panelConfig, googleMapsClient);

This updated main.js file has a few additions since last time you saw it. First, we have added in bluebird, as we require a Promise constructor while constructing googleMapsClient for async/await coding that we have in TimeTable. Next we pull in the panels_config.json we constructed above, along with instantiating the aforementioned googleMapsClient. In the database references section, we create a second reference, this time for the panels path in our Firebase instance. This is where the time table content will be published. In the library classes section we require both GTFS and TimeTable. Finally, in the last stanza, we construct the named gtfs instance to load the GTFS data into our in-memory SQLite database, and then instantiate a TimeTable instance.

Re-run main.js, then take a look at the Database tab in your Firebase Realtime Database console. You should now see more data being published:

This time with feeling

So far we have populated two of the four paths in the Firebase Realtime Database that our front end will need to display a working simulation of the Transport Tracker. We will populate the next two paths in this step. We will add a path called bus-locations that the front end uses to plot bus locations on the map, and a path called map that drives the bounds of the map that front end displays, along with the group of route cards to show.

We construct two new classes. The first is PanelChanger, which is responsible for publishing the correct part of the panels_config.json file to make the front end display the correct portion of the South Bay, along with markers for the hotels appropriate for that panel. The second is BusSimulator, which simulates the positions of buses using the paths we generated earlier in this codelab.

Let's now create a new file called panel_changer.js, and fill it with this content:

panel_changer.js

// PanelChanger changes the focus of the front end display
// by publishing the three panels defined in `panelConfig`,
// one after the other every ten seconds. This changes the
// bounds of the displayed map, along with the featured
//  Hotels and bus stops.
exports.PanelChanger = class {
  constructor(mapRef, panelConfig) {
    this.mapRef = mapRef;
    this.panelConfig = panelConfig;
    this.panelIndex = 0;

    // Change the panel once every ten seconds
    this.timeTimerId = setInterval(() => {
      this.panelAdvance();
    }, 10000);
  }

  panelAdvance() {
    this.mapRef.set(this.panelConfig[this.panelIndex]);
    this.panelIndex = (this.panelIndex + 1) % this.panelConfig.length;
  }
};

After the intensity of TimeTable, this class is almost an anticlimax. What this class is doing isn't technically difficult, but it is important for the front end functionality. Next, let's create a file called bus_simulator.js for our BusSimulator class.

bus_simulator.js

// BusSimulator updates the simulated location of the buses
// every time `timeRef` changes. This uses a combination of the
// generated paths along with route information pulled from `gtfs`.
// The updated simulated bus locations is published to `busLocationsRef`.
exports.BusSimulator = class {
  constructor(timeRef, gtfs, busLocationsRef, generatedPaths) {
    this.timeRef = timeRef;
    this.gtfs = gtfs;
    this.busLocationsRef = busLocationsRef;
    this.paths = generatedPaths;

    this.timeRef.on(
      'value',
      snapshot => {
        _async(() => {
          const now = moment.utc(snapshot.val().moment);
          this.busAdvance(now);
        })().catch(err => {
          console.error(err);
        });
      },
      errorObject => {
        console.error('The read failed: ' + errorObject.code);
      }
    );
  }

  busAdvance(now) {
    const buses = this.getBusPositionsAt(now);
    const busLocations = {};
    buses.forEach(bus => {
      const route = _await(this.gtfs.getRouteById(bus.trip.route_id));
      busLocations[`Trip_${bus.trip.trip_id}`] = {
        route_id: bus.trip.route_id,
        route_name: route.route_long_name,
        route_color: route.route_color,
        lat: bus.location.lat,
        lng: bus.location.lng
      };
    });
    this.busLocationsRef.set(busLocations);
  }

  getBusPositionsAt(time) {
    function interpolate(before, after, proportion) {
      return {
        lat: before.lat * proportion + after.lat * (1 - proportion),
        lng: before.lng * proportion + after.lng * (1 - proportion)
      };
    }

    function search(bus, before, after) {
      if (after - before > 1) {
        const midpoint = Math.round(before + (after - before) / 2);
        const midpointTime = moment.utc(bus.points[midpoint].time, DATE_FORMAT);
        if (midpointTime.isBefore(time)) {
          return search(bus, midpoint, after);
        }
        return search(bus, before, midpoint);
      }
      const beforeTime = moment.utc(bus.points[before].time, DATE_FORMAT);
      const afterTime = moment.utc(bus.points[after].time, DATE_FORMAT);
      const proportion = time.diff(beforeTime) / afterTime.diff(beforeTime);
      return interpolate(
        bus.points[before].location,
        bus.points[after].location,
        proportion
      );
    }

    const busPositions = [];
    this.getBusesActiveAt(time).forEach(bus => {
      busPositions.push({
        trip: bus.trip,
        location: search(bus, 0, bus.points.length - 1)
      });
    });
    return busPositions;
  }

  getBusesActiveAt(time) {
    const buses = [];
    this.paths.forEach(bus => {
      const start = moment.utc(bus.points[0].time, DATE_FORMAT);
      const end = moment.utc(
        bus.points[bus.points.length - 1].time,
        DATE_FORMAT
      );
      if (start.isBefore(time) && end.isAfter(time)) {
        buses.push(bus);
      }
    });
    return buses;
  }
};

BusSimulator sits at a complexity point somewhere between TimeTable and PanelChanger. BusSimulator is responsible for putting fake buses on the map, using the time signal from HeartBeat to look up bus locations in our generated paths.json store. If you dig into BusSimulator, you will probably spot a hidden algorithm from your Algorithms and Data Structures 101 course. Finally, let us add these classes to our main.js file.

main.js

/*eslint-disable unknown-require */
const trackerConfig = require('./tracker_configuration.json');
const Promise = require('bluebird');
const admin = require('firebase-admin');
const serviceAccount = require('./serviceAccountKey.json');
const panelConfig = require('./panels_config.json');
const generatedPaths = require('./paths.json');
const googleMapsClient = require('@google/maps').createClient({
  key: trackerConfig.mapsApiKey,
  Promise
});
admin.initializeApp({
  credential: admin.credential.cert(serviceAccount),
  databaseURL: trackerConfig.databaseURL
});
// Database references
const busLocationsRef = admin.database().ref('bus-locations');
const mapRef = admin.database().ref('map');
const panelsRef = admin.database().ref('panels');
const timeRef = admin.database().ref('current-time');
// Library classes
const {BusSimulator} = require('./bus_simulator.js');
const {GTFS} = require('./gtfs.js');
const {HeartBeat} = require('./heart_beat.js');
const {PanelChanger} = require('./panel_changer.js');
const {TimeTable} = require('./time_table.js');
const gtfs = new GTFS();
new HeartBeat(timeRef, trackerConfig.simulation);
new TimeTable(timeRef, panelsRef, gtfs, panelConfig, googleMapsClient);
new PanelChanger(mapRef, panelConfig);
if (trackerConfig.simulation) {
  new BusSimulator(timeRef, gtfs, busLocationsRef, generatedPaths);
} else {
  // Exercise for the reader: integrate real bus location data
}

We have made a couple of additions to main.js to tie in the new classes we just created. First, we've added a require to load the paths.json file generated earlier in this codelab. Next, in the database references section we have added the two paths we alluded to at the start of this step. Next, in the library classes section we have loaded in the two JavaScript files we authored above. Finally, in the last section we have instantiated both PanelChanger and BusSimulator with their required parameters.

Fire up the main script as usual:

npm run main

Then, have a look at your Firebase Realtime Database, and you should see something like the following:

GPS signals can be a little unpredictable

Now that we have simulated buses running around the map, it would be nice to introduce a little reality into this picture. And by reality, I mean noise. One of the harsh realities of reality is that no matter how good your GPS signal is, it's always going to be wrong. Sometimes by a little, sometimes by a lot. The upside here is that Google Maps Roads API is here to save the day. Using the Roads API we can take a collection of estimated locations, and get the Google Maps Roads API to snap our location estimations to actual road locations. Suddenly we aren't in the corn fields no more!

The first step is to enable the Google Maps Roads API for your project. Go to the Google API Console, open the project that you created in step 4, and check the list of enabled APIs on the console Dashboard. If the Google Maps Roads API is not there, click Enable API, search for the Roads API, and enable it now.

To do this we will add a new module, appropriately called RoadSnapper. The responsibility of this module is to collect the location history of the buses, and then every ten seconds send this location history to Roads API Snap To Roads service. The results from this Web Service request are our history of bus locations, nicely snapped to roads. Create a file called road_snapper.js and put the following code in it:

road_snapper.js

/* eslint-disable no-undef-expression */
const _async = require('asyncawait/async');
const _await = require('asyncawait/await');
const TRIP_HISTORY_LENGTH = 20;
exports.RoadSnapper = class {
  constructor(
    timeRef,
    rawBusLocationsRef,
    snappedBusLocationsRef,
    googleMapsClient
  ) {
    this.snappedBusLocationsRef = snappedBusLocationsRef;
    this.googleMapsClient = googleMapsClient;
    this.history = {};
    this.snapped = {};
    this.val = null;
    this.time = null;
    timeRef.on('value', snapshot => {
      this.time = snapshot.val();
    });
    rawBusLocationsRef.on('value', snapshot => {
      const val = snapshot.val();
      this.gatherHistory(val);
    });
    // Snap to roads every ten seconds
    this.timeTimerId = setInterval(() => {
      _async(() => {
        this.snapToRoads();
      })().catch(err => {
        console.error(err);
      });
    }, 10000);
  }
  gatherHistory(val) {
    this.val = val;
    if (val && this.time) {
      const tripnames = new Set(Object.keys(val));
      Object.keys(this.history).forEach(historicTripname => {
        if (!tripnames.has(historicTripname)) {
          delete this.history[historicTripname];
        }
      });
      tripnames.forEach(tripname => {
        if (!this.history[tripname]) {
          this.history[tripname] = [];
        }
        const point = {
          lat: val[tripname].lat,
          lng: val[tripname].lng,
          moment: this.time.moment
        };
        this.history[tripname].push(point);
        if (this.history[tripname].length > TRIP_HISTORY_LENGTH) {
          this.history[tripname] = this.history[tripname].slice(
            -TRIP_HISTORY_LENGTH
          );
        }
      });
    }
  }
  snapToRoads() {
    // Work around concurrency issues by taking a snapshot of val
    const valSnapshot = this.val;
    if (valSnapshot && this.time) {
      const tripnames = new Set(Object.keys(valSnapshot));
      Object.keys(this.snapped).forEach(snappedTripname => {
        if (!tripnames.has(snappedTripname)) {
          delete this.snapped[snappedTripname];
        }
      });
      tripnames.forEach(tripname => {
        if (this.history.hasOwnProperty(tripname)) {
          const path = this.history[tripname].map(point => {
            return [point.lat, point.lng];
          });
          const result = _await(
            this.googleMapsClient.snapToRoads({path}).asPromise()
          );
          if (result.json.snappedPoints) {
            this.snapped[tripname] =
              result.json.snappedPoints[
                result.json.snappedPoints.length - 1
              ].location;
          } else {
            console.error(result);
            this.snapped[tripname] = {};
          }
        } else {
          this.snapped[tripname] = {};
        }
      });
      let snappedBusPositions = {};
      tripnames.forEach(tripname => {
        snappedBusPositions[tripname] = {
          lat: this.snapped.hasOwnProperty(tripname) &&
            this.snapped[tripname].latitude
            ? this.snapped[tripname].latitude
            : valSnapshot[tripname].lat,
          lng: this.snapped.hasOwnProperty(tripname) &&
            this.snapped[tripname].longitude
            ? this.snapped[tripname].longitude
            : valSnapshot[tripname].lng,
          route_color: valSnapshot[tripname].route_color,
          route_id: valSnapshot[tripname].route_id,
          route_name: valSnapshot[tripname].route_name
        };
      });
      this.snappedBusLocationsRef.set(snappedBusPositions);
    } else {
      // If valSnapshot is empty, we have no buses on the road.
      this.snappedBusLocationsRef.set({});
    }
  }
};

There are a couple of interesting points of note in this code. First, we are dealing with asynchronicity here in multiple forms. We guard against this by having a fair number of null/undefined checks in various places, and by caching values that we will need to stay relevent before and after a asynchronous call to the web service.

Now we can tie this module into the configuration by updating main.js as follows.

main.js

/*eslint-disable unknown-require */
const trackerConfig = require('./tracker_configuration.json');
const Promise = require('bluebird');
const admin = require('firebase-admin');
const serviceAccount = require('./serviceAccountKey.json');
const panelConfig = require('./panels_config.json');
const generatedPaths = require('./paths.json');
const googleMapsClient = require('@google/maps').createClient({
  key: trackerConfig.mapsApiKey,
  Promise
});
admin.initializeApp({
  credential: admin.credential.cert(serviceAccount),
  databaseURL: trackerConfig.databaseURL
});
// Database references
const rawBusLocationsRef = admin.database().ref('raw-bus-locations');
const snappedBusLocationsRef = admin.database().ref('bus-locations');
const mapRef = admin.database().ref('map');
const panelsRef = admin.database().ref('panels');
const timeRef = admin.database().ref('current-time');
// Library classes
const {BusSimulator} = require('./bus_simulator.js');
const {GTFS} = require('./gtfs.js');
const {HeartBeat} = require('./heart_beat.js');
const {PanelChanger} = require('./panel_changer.js');
const {RoadSnapper} = require('./road_snapper.js');
const {TimeTable} = require('./time_table.js');
const gtfs = new GTFS();
new HeartBeat(timeRef, trackerConfig.simulation);
new TimeTable(timeRef, panelsRef, gtfs, panelConfig, googleMapsClient);
new PanelChanger(mapRef, panelConfig);
if (trackerConfig.simulation) {
  new BusSimulator(timeRef, gtfs, rawBusLocationsRef, generatedPaths);
  new RoadSnapper(
    timeRef,
    rawBusLocationsRef,
    snappedBusLocationsRef,
    googleMapsClient
  );
} else {
  // Exercise for the reader: integrate real bus location data
}

Congratulations! You've built the backend for the Transport Tracker and seen the results appear in real time, in your Firebase Realtime Database. Wouldn't it be cool to see the results on a map? Follow the Transport Tracker Map codelab to build the frontend.