Pipe heroku logs to AWS CloudWatch

October 12, 2017
DevOps Software Engineering

Introduction

At Enhancv, we went through various way to collect the logs from our microservices in one place. When we were building the MVP, just a simple Heroku Papertrail addon was enough to keep track of all application logs. As our product was growing, it became much more complex to monitor everything. Since most of our services are hosted on AWS, CloudWatch seemed to be the best option, but our REST API was still hosted on Heroku, because of our per-branch staging system. So, we wrote a simple tool that drain all logs from heroku and pushes them to AWS CloudWatch. One of biggest pros of logging everything together is that we can monitor all of our services at one place and setup complex alarm conditions that notify us, when something goes down.

x / Dashboard of our infrastructure /

You can find the repository on https://github.com/deepsyx/node-heroku-cloudwatch-drain

It’s implemented via Heroku drains.

Heroku also supports the HTTPS protocol for sending logs to a drain service so that you can more easily write your own log processing logic and run it on a service like Heroku itself. Like with a syslog drain, you must configure the service such that Heroku can make requests to it (for example by deploying it as a Heroku app).

The flow goes like this:

x

Setup on AWS

1. Login to AWS console

https://console.aws.amazon.com

2. Create a new EC2 instance

In most cases t2.nano will do the job, it uses around 5% cpu with 50 requests / second. We’re using the Ubuntu 16.04 AMI ami-cd0f5cb6, but any with NodeJS support would work. 8GB SSD should be enough disk space to setup everything. Don’t forget to save your SSH key (you’ll need them to log into the instance later)

3. Create a security group

From the side menu choose Security groups and create a new security group with the following settings inbound settings:

Type: Custom TCP Rule
Protocol: TCP
Port Range: 3000
Source: Anywhere
4. Attach security group to instance

Go to the Instances, select your freshly created instance and click on Actions -> Networking -> Change Security Groups and attach the security group we just created.

5. Create AWS user

Click on your name in the top bar and go to My Security Credentials -> Users -> Add User and create a new AWS user with programic access, then go to the next step and choose Attach existing policies directly and add the CloudWatchFullAccess permissions. Save the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, because you’ll need them later

6. Login into the instance

SSH into the instance with the private key, that you downloaded on the second step like this:

$ ssh -i "YourPrivateKey.pem" ubuntu@INSTANCE_IP_ADDRESS
7. Add AWS credentials to environment variables

Execute sudo nano /etc/environment and add the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY keys you received when you created the users above.

8. Install NodeJS
$ curl -sL https://deb.nodesource.com/setup_7.x | sudo -E bash -

9. Install depencencies
$ npm install -g node-heroku-cloudwatch-drain forever

10. Create configuration

Create a config.js file with the following content:

module.exports = {
    // how many messages should we buffer and send to AWS at once, do not change unless necessary
    batchSize: 50,

    // at what port should our server be listening
    serverPort: 3000,

    // your cloudwatch log group
    logGroup: "heroku",

    // prefix for all stream names (a random number will be appended)
    logStreamPrefix: "dyno",

    // we strongly recommend using env variables for the credentials, however you can still add them here
    awsCredentials: {
        region: "us-east-1",
    },

    // access token, which will be sent from heroku on every request
    accessToken: "sometoken",

    // [OPTIONAL] array with regexs to avoid pushing useless logs to cloudwatch
    filters: [/getVersion/],
};
11. Start the service

Run the tool with forever (it will auto-restart on failure)

forever start -c node-heroku-cloudwatch-drain config.js
12. Test endpoint

Get your EC2 instance IP from the console and try to open the following link in your browser:

http://ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:3000/

and you should see {"message":"Invalid token"} as response.

13. Install Heroku CLI

Download Heroku CLI to your local machine, install itself and login via heroku login

14. Add drain endpoint to your heroku instance
$ heroku drains:add http://ec2-XXX-XXX-XXX-XXX.compute-1.amazonaws.com:3000/log?accessToken=sometoken --app YOUR APP

If the above command succeed, your heroku logs should start appearing in CloudWatch. If you encounter any problems/bugs, feel free to add them as issue on node-heroku-cloudwatch-drain repo.

Read more

Home automation with Raspberry Pi, Node and React
Bypassing legacy capchas combined with email confirmation

Hey! I just started blogging and I'll post regularly about tech. If you enjoy my posts, leave your email below and I'll try to send you a newsletter once a month. I won't spam you :)