Facebook messenger bot to identify birds

Bird Bot is a Facebook messenger bot. Bird watchers in India shoot photos of birds. And they want to know the bird name. My good old friend from college is an expert in Machine learning. He is building a bird identification api with the help of TensorFlow and Keras. I am helping him with the front-end. Instead of developing an app with React Native, I wrote a simple bot which works with Facebook messenger.

Facebook messenger bot

There are just five interactions with the bot.

1. When the user says Hi or Hello, we intimate the user that we are a bot.
2. When the user uploads a photo, we identify the bird within the photo and reply back with the bird name.
3. If the user thanks the bot, the bot replies back with You are welcome message.
4. When the user says Bye, the bot says Bye as well.
5. If the user types any other text message, we ignore it.

These interactions are shown in the demo video.

Bot Registration

A facebook messenger bot is an api or a webhook. It is associated with a Facebook page via a Facebook app. There is a registration step for the bot in the Facebook app. The registration step requires a webhook URL and a verify token. We deploy our bot to a HTTP endpoint which becomes the webhook URL. Verify token is any arbitrary string that our bot knows. Bot registration involves the following step.

1. Facebook calls the bot endpoint (GET request) with the verify token and a challenge.
2. Our bot verifies if the token is right
3. And it returns the challenge with a 200 OK status code.

For building the bot, I have an API built on top of express. The registration code is as follows.

register(req, res) {
    const { query } = req;
    const mode = query['hub.mode'];
    const verifyToken = query['hub.verify_token'];
    const challenge = query['hub.challenge'];
    if (mode === 'subscribe' && verifyToken === 'myToken') {
        res.send(challenge);
    } else {
        res.status(403).end();
    }
}

Express API

Bot is nothing but an API. We subscribe to messages from Facebook page. Whenever a user sends a message to our page, the bot receives it via a POST call to the same endpoint. There are two server routes, one for GET and another for POST to the root endpoint.

"use strict"
const express = require('express');
const app = express();
const bodyParser = require('body-parser');
app.use(bodyParser());

const bot = require('./bot');

app.get('/', function (req, res) {
  bot.register(req, res);
});

app.post('/', function (req, res) {
    bot.handleMessage(req, res);
});

app.listen(3000, function () {
  console.log('Bird bot listening on port 3000!');
});

There is a Bot class with two methods, register and handleMessage. We have already seen the register method in the above code. The handleMessage method is where we interpret user messages.

Handling messages

Another setting that we should turn on in our app is NLP (Natural Language Processing). This enables Facebook to interpret our messages partially and provide us with NLP data. Accordingly, we get three types of message: greetings, thanks and bye. The code for handleMessage method is shown below.

handleMessage(req, res) {
    console.log('processing message');
    const data = req.body;
    if (data.object === 'page') {
        data.entry.forEach(page => {
            page.messaging.forEach(message => {
                const senderId = message.sender.id;
                const { text, nlp, attachments } = message.message;
                if (nlp) {
                    const { entities } = nlp;
                    if (entities.bye && entities.bye[0].value) {
                        this.handleBye(senderId);
                    } else if (entities.greetings && entities.greetings[0].value) {
                        this.handleGreeting(senderId);
                    } else if (entities.thanks && entities.thanks[0].value) {
                        this.handleThanks(senderId);
                    }
                }
                if (attachments && attachments.length) {
                    this.handleImage(senderId, attachments[0].payload);
                }
            });
        })
    }
    res.sendStatus(200);
}

The code is a bit complicated. But it is standard boilerplate code for bots. As you can see, the message has text, attachments and nlp properties. We use the nlp property to respond to greetings, thanks and bye. For responding to images, we use the attachments collection. The handleImage method is shown below.

handleImage(senderId, imageUrl) {
    // send the image to the prediction api and parse the results!
    // but for now, send a hard-coded message.
    this.sendText(senderId, 'The bird is Eurasian Wigeon.')
        .catch(err => console.log(err));
}

For now, we don’t have a prediction API. All images are responded with The bird is Eurasian Wigeon. But you get the idea, right? Our prediction API will tell what bird it is. And we respond back to the user with the bird name.

Sending messages

Sending a message to the user is done via another API call as follows.

sendText(id, text) {
    return new Promise((resolve, reject) => {
        request({
            uri: 'https://graph.facebook.com/v2.6/me/messages',
            qs: {
                access_token: 'page-access-token'
            },
            method: 'POST',
            json: {
                recipient: {
                    id
                },
                message: {
                    text
                }
            }
        }, (error, response) => {
            if (!error && response.statusCode === 200) {
                resolve();
            } else {
                reject(error || response.statusCode);
            }
        });
    });
}

We use the request package to make an API call. The JSON payload has both the recipient (or user-id) and the message text.

Testing

Our fully functional bird bot is ready for testing. For testing, we take the help of an utility, ngrok.

nodemon server
./ngrok http 3000

We run the server module with nodemon. Nodemon takes care of server restarts after code changes. ngrok utility provides a https URL. And we use that URL as the webhook URL.

Go to the Facebook page. And test the messenger. The bot receives any message that the page receives. You should see diagnostic messages in the console window.

If everything is OK, deploy the bot to Heroku (or Digital ocean or AWS). And submit the Facebook app for approval.