Pranav Prakash

Debugging ES2017 async/await in VS Code using Source Maps

2017-10-18T14:38:54+05:30

async/await has been the most promising feature of ES2017. It allows you to write asynchronous code in more elegant way saving you from the hassles of callback. async/await is supported from Node versions >= 8.5. However if you are writing JavaScript code targeting Node versions < 8.5, you need to set up a transpiler like Babel to convert your code compatible to your version of Node. To debug in this scenario, you need to make sure that Source Maps are generated and are linked in transpiled version generated by Babel. To do so, add following parameters in your .babelrc file:

1
2
"sourceMaps": true,
"retainLines": true

For more details about .babelrc, refer this. Now add following configuration to launch.json file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
{
  "version": "0.2.0",
  "configurations": [
    {
      "name": "Launch",
      "type": "node",
      "request": "launch",
      "program": "${workspaceRoot}/dist/index.js",
      "stopOnEntry": false,
      "args": [],
      "cwd": "${workspaceRoot}",
      "preLaunchTask": null,
      "runtimeExecutable": "nodemon",
      "restart": true,
      "runtimeArgs": ["--nolazy"],
      "env": {
        "NODE_ENV": "development"
      },
      "console": "internalConsole",
      "sourceMaps": true,
      "outFiles": ["${workspaceRoot}/dist/**/*.js"],
      "skipFiles": [
        "${workspaceRoot}/node_modules/**/*.js",
        "<node_internals>/**/*.js"
      ],
      "smartStep": true
    }
  ]
}

Here in line number 8, program points to entry point of your transpiled code. We are also setting sourceMaps parameter to true in line number 20. The outFiles parameter in line number 21 contains a glob pattern for an absolute path that matches our .js files. We also don’t want to debug the Node internal programs and programs in node_modules folder and hence they are specified in skipFiles parameter in line number 22. Finally I have set the smartStep to true in line number 26 to automatically step over code that doesn’t map to source files. These codes are generally injected by compiler during async/await downcompilation. smartStep will automatically step through code not covered by a source map until it reaches a location that is covered by a source map again.

Now before starting debugging, make sure the transpiled code is availabe in directory specified in configuration file above. Also make sure source maps are generated and are present along with .js files. Once everything is in place, just press F5 and start debugging.

Building a fast text suggester using Node.js, Koa2 and Elasticsearch

2017-09-30T12:31:09+05:30

In this blog post, I’ll show you how to create a fast text suggester using Koa2 framework of Node.js backed by Elasticsearch. Koa2 is a minimalist framework developed by team behind Express.js and comes with no middlewares inbuilt. Elasticsearch is a distributed search engine which comes with handy features for doing full text search.

To start with, I’ll use the koa-api-generator to bootstrap the application. Start by issuing following commands

1
2
3
npm i -g koa-api-generator
koa-api node-search-engine && cd node-search-engine
npm install

Now I’ll install the elasticsearch package to connect to Elasticsearch.

1
npm i elasticsearch --save

I’ll assume that you have installed and configured the Elasticsearch on your machine. If not then follow the installation guide here to set up Elasticsearch on your machine. Next thing we need to do is to create an index and define mapping in Elasticsearch so that it can know how to map the data that is going to be stored in it, in order to serve results faster. To do so, I’ll use Elasticsearch’s HTTP APIs by issuing following curl request from command line:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
curl -X PUT \
  http://localhost:9200/persons/ \
  -H 'content-type: application/json' \
  -d '{
    "mappings":{
        "person":{
            "properties":{
                "name":{
                    "type":"string"
                },
                "suggest":{
                    "type":"completion"
                }
            }
        }
    }
}'

This request creates an index persons having type person with property name. The suggest field would be used for completion and hence type has been specified as completion. Now I will create a module elasticService.js which can be imported wherever I want to interact with Elasticsearch.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
import elasticsearch from "elasticsearch";

const elasticClient = new elasticsearch.Client({
    host: "localhost:9200",
    log: "info"
});

const elasticService = {};
const index = "persons";
const type = "person";

/**
 * Add a person
 */

elasticService.addPerson = name => {
    return elasticClient.index({
        index,
        type,
        body: {
            name,
            suggest: name.split(" ")
        }
    });
};

/**
 * Fetch suggestions
 */

elasticService.getSuggestions = input => {
    return elasticClient.suggest({
        index,
        body: {
            persons: {
                prefix: input,
                completion: {
                    field: "suggest",
                    fuzzy: true,
                    size: 10
                }
            }
        }
    });
};

export default elasticService;

addPerson function is just adding the name of person to index persons and populating suggest field with array of strings obtained after splitting the name. For example, if name is Pranav Prakash, suggest field will contain ["Pranav", "Prakash"]. getSuggestions uses Completion Suggester api to return suggestions. I have set the fuzzy searching true which will return results even if we have typo in search input. The size parameters simply restricts the number of suggestion which in this case has been set to 10. Now since our module is complete, I will integrate it with Koa routes. I’ve populated the routes.js with two routes as below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import Router from "koa-router";
import elasticService from "./elasticService";

const router = new Router();

/**
 * Get suggestions
 */

router.get("/suggest/:term", async ctx => {
    ctx.body = await elasticService.getSuggestions(ctx.params.term);
});

/**
 * Create person
 */

router.post("/", async ctx => {
    ctx.body = await elasticService.addPerson(ctx.request.body.name);
});

export default router;

These routes are quite simple: get("suggest/:term") route is returning the suggestions and post("/") route adds a person to Elasticsearch. Now we’ll start the server using following command:

1
npm run dev

Now I need to add some data in Elasticsearch so that I can perform searches on it. I’ll add some data by issuing following POST requests from command line:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
curl -X POST \
  http://localhost:8080/ \
  -H 'content-type: application/json' \
  -d '{
    "name" : "Ranbir Kapoor"
}'

curl -X POST \
  http://localhost:8080/ \
  -H 'content-type: application/json' \
  -d '{
    "name" : "Ranvir Singh"
}'

curl -X POST \
  http://localhost:8080/ \
  -H 'content-type: application/json' \
  -d '{
    "name" : "Ranveer Singh"
}'

Now I’ll query the application using commands below:

1
2
curl -X GET \
  http://localhost:8080/suggest/ranbir

And voila, I’m presented with results below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
{
    "_shards": {
        "total": 5,
        "successful": 5,
        "skipped": 0,
        "failed": 0
    },
    "persons": [
        {
            "text": "ranbir",
            "offset": 0,
            "length": 6,
            "options": [
                {
                    "text": "Ranbir",
                    "_index": "persons",
                    "_type": "person",
                    "_id": "AV7Sns-uOy-E67GQK4As",
                    "_score": 5,
                    "_source": {
                        "name": "Ranbir Kapoor",
                        "suggest": [
                            "Ranbir",
                            "Kapoor"
                        ]
                    }
                },
                {
                    "text": "Ranvir",
                    "_index": "persons",
                    "_type": "person",
                    "_id": "AV7SnraQOy-E67GQK4Ar",
                    "_score": 3,
                    "_source": {
                        "name": "Ranvir Singh",
                        "suggest": [
                            "Ranvir",
                            "Singh"
                        ]
                    }
                }
            ]
        }
    ]
}

Notice how fuzzy searching has returned results for both Ranbir as well as Ravir, pretty neat, eh? Only difference is in their score which Elasticsearch calculates internally based on how well the query term matches with current result.

The complete source code for this project is available on Github here. This post just scrathces the surface of what can be done with Elasticsearch. You can dive deeper by specifying weight for suggestions, trying out different analyser and search_analyzer, increasing or decreasing fuzziness or using regex queries depending on your use case.

Case insensitive exact match search in Elasticsearch

2017-09-13T16:31:32+05:30

I had a requirement where I needed to do exact match search in ElasticSearch. After searching a bit I landed to this page of ElasticSearch documentation. I was thrilled that I got the solution quickly (thanking god and ElasticSearch team in my mind). Hence I started testing by creating an index using request below:

1
2
3
4
5
6
7
8
9
10
11
12
13
PUT localhost:9200/products/
{
  "mappings": {
    "product": {
      "properties": {
        "productName": {
          "type": "string",
          "index": "not_analyzed"
        }
      }
    }
  }
}

Then I added some data using request below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
POST localhost:9200/products/
{
  "productName": "Green Ball"
}

POST localhost:9200/products/
{
  "productName": "Red Ball"
}

POST localhost:9200/products/
{
  "productName": "Yellow Ball"
}

Now it was time for testing and hence I fired following request:

1
2
3
4
5
6
7
8
9
10
11
12
POST localhost:9200/products/product/_search
{
  "query":{
    "constant_score":{
      "filter":{
        "term":{
          "productName":"green ball"
        }
      }
    }
  }
}

And I was presented with following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
{
  "took":0,
  "timed_out":false,
  "_shards":{
    "total":5,
    "successful":5,
    "skipped":0,
    "failed":0
  },
  "hits":{
    "total":0,
    "max_score":null,
    "hits":[]
  }
}

I was surprised (and frustrated at the same time) as the data was there in ElasticSearch, still it was not returning the data. Soon enough I found that it was because the search was case sensitive and searching with “Green Ball” returned result. This was unacceptable in my use case as I needed case insensitive search. After searching a little bit more, I found that I have to create an analyzer with lowercase filter and use this to analyze the field on which I want to perform search. Hence I deleted the index and recreated it as below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
DELETE localhost:9200/products/

PUT localhost:9200/products/
{
  "settings":{
    "index":{
      "analysis":{
        "analyzer":{
          "analyzer_case_insensitive":{
            "tokenizer":"keyword",
            "filter":"lowercase"
          }
        }
      }
    }
  },
  "mappings":{
    "product":{
      "properties":{
        "productName":{
          "type":"string",
          "analyzer":"analyzer_case_insensitive"
        }
      }
    }
  }
}

I needed to switch to match query as term query would return result with only lower case search text. Below is the modified query:

1
2
3
4
5
6
7
8
POST localhost:9200/products/product/_search
{
  "query":{
    "match":{
      "productName":"greeN ball"
    }
  }
}

And this resulted in following:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
  "took":0,
  "timed_out":false,
  "_shards":{
    "total":5,
    "successful":5,
    "skipped":0,
    "failed":0
  },
  "hits":{
    "total":1,
    "max_score":0.6931472,
    "hits":[
      {
        "_index":"products",
        "_type":"product",
        "_id":"AV57L6vzqKOq3BnmUAzd",
        "_score":0.6931472,
        "_source":{
          "productName":"Green Ball"
        }
      }
    ]
  }
}

I was happy with the result as it served my purpose. Let me know in comments if there is a better way to achieve this.

Automated PostgreSQL backups

2017-05-16T11:59:50+05:30

Recently we migrated one of our applications from MongoDB to PostgreSQL (which is a story for another time) at work. Similar to the script I developed for MongoDB in my last blog post, I developed one for PostgreSQL. pg_dump utility of PostgreSQL provides some nifty options to create different formats to create backup files. My script produces two types of backup files:

Plain bakup - This produces a gzipepd version of SQL script file which you can execute to restore the database.
Custom backup - This produces a compressed custom-format archive suitable for input into pg_restore.

Without further ado, below is the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
#!/bin/bash

##############################
## POSTGRESQL BACKUP CONFIG ##
##############################
 
# Optional hostname to adhere to pg_hba policies.  Will default to "localhost" if none specified.
HOSTNAME=127.0.0.1
 
# Optional username to connect to database as.  Will default to "postgres" if none specified.
USERNAME=pgusername
 
# This dir will be created if it doesn't exist.  This must be writable by the user the script is
# running as.
BACKUP_DIR=/var/backups/

# Database name to backup
DATABASE=mydb
 
# Will produce a custom-format backup if set to "yes"
ENABLE_CUSTOM_BACKUPS=yes
 
# Will produce a gzipped plain-format backup if set to "yes"
ENABLE_PLAIN_BACKUPS=yes

#### SETTINGS FOR ROTATED BACKUPS ####

# Number of days to keep daily backups
DAYS_TO_KEEP=14

 
###########################
### INITIALISE DEFAULTS ###
###########################
 
if [ ! $HOSTNAME ]; then
  HOSTNAME="localhost"
fi;
 
if [ ! $USERNAME ]; then
  USERNAME="postgres"
fi;
 
 
###########################
#### START THE BACKUPS ####
###########################
 
function perform_backups()
{
  SUFFIX=$1
  FINAL_BACKUP_DIR=$BACKUP_DIR"`date +\%Y-\%m-\%d`$SUFFIX/"
 
  echo "Making backup directory in $FINAL_BACKUP_DIR"
 
  if ! mkdir -p $FINAL_BACKUP_DIR; then
    echo "Cannot create backup directory in $FINAL_BACKUP_DIR. Go and fix it!" 1>&2
    exit 1;
  fi;
 
  echo -e "\n\nPerforming full backup of $DATABASE"
  echo -e "--------------------------------------------\n"
 
    if [ $ENABLE_PLAIN_BACKUPS = "yes" ]
    then
      echo "Plain backup of $DATABASE"
 
      if ! pg_dump -Fp -h "$HOSTNAME" -U "$USERNAME" "$DATABASE" | gzip > $FINAL_BACKUP_DIR"$DATABASE".sql.gz.in_progress; then
        echo "[!!ERROR!!] Failed to produce plain backup database $DATABASE" 1>&2
      else
        mv $FINAL_BACKUP_DIR"$DATABASE".sql.gz.in_progress $FINAL_BACKUP_DIR"$DATABASE".sql.gz
      fi
    fi
 
    if [ $ENABLE_CUSTOM_BACKUPS = "yes" ]
    then
      echo "Custom backup of $DATABASE"
 
      if ! pg_dump -Fc -h "$HOSTNAME" -U "$USERNAME" "$DATABASE" -f $FINAL_BACKUP_DIR"$DATABASE".custom.in_progress; then
        echo "[!!ERROR!!] Failed to produce custom backup database $DATABASE"
      else
        mv $FINAL_BACKUP_DIR"$DATABASE".custom.in_progress $FINAL_BACKUP_DIR"$DATABASE".custom
      fi
    fi
 
  echo -e "\nDatabase backup complete!"
}
 
# DAILY BACKUPS
 
# Delete daily backups older than DAYS_TO_KEEP
find $BACKUP_DIR -maxdepth 1 -mtime +$DAYS_TO_KEEP -name "*-daily" -exec rm -rf '{}' ';'

perform_backups "-daily"

Save this in a file named pgbackup.sh. After that you need to make the file executable by issuing following command:

1
chomd +x pgbackup.sh

You must be wondering about where to specify the database password. Well as per PostgreSQL documentation, you need to create password file .pgpass in user’s home directory and should contain lines of the following format:

1
hostname:port:database:username:password

Make sure the permissions on .pgpass must disallow any access to world or group. You can do this my issuing following command:

1
chmod 0600 ~/.pgpass

This password file is automatically referanced by pgdump utility whenever invoked. Finally you need to create a cron job to execute the pgbackup.sh script automatically. Issue following command to create a cron job:

1
crontab -e

Enter following:

1
2
# PostgreSQL Backup
00 00 * * * /path/to/script/pgbackup.sh

Save the file by pressing Escand then wq + Enter (similar to vi editor). This cron job will execute the backup script at 12 o’clock every night creating your database backups.

Automate MongoDB backups using Cron and Shell Script

2017-02-04T19:07:36+05:30

After having a bitter experience of losing data due to execution of wrong command, I decided to backup my database regularly. Hence I wrote a simple shell script which backs up the database and produces gzipped version of backup files. Below is the shell script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
#!/bin/sh

#=====================================================================
# Set the following variables as per your requirement
#=====================================================================
# Database Name to backup
MONGO_DATABASE="mydb"
# Database host name
MONGO_HOST="127.0.0.1"
# Database port
MONGO_PORT="27017"
# Backup directory
BACKUPS_DIR="/var/backups/$MONGO_DATABASE"
# Database user name
DBUSERNAME="username"
# Database password
DBPASSWORD="passw0rd"
# Authentication database name
DBAUTHDB="admin"
# Days to keep the backup
DAYSTORETAINBACKUP="14"
#=====================================================================

TIMESTAMP=`date +%F-%H%M`
BACKUP_NAME="$MONGO_DATABASE-$TIMESTAMP"

echo "Performing backup of $MONGO_DATABASE"
echo "--------------------------------------------"
# Create backup directory
if ! mkdir -p $BACKUPS_DIR; then
  echo "Can't create backup directory in $BACKUPS_DIR. Go and fix it!" 1>&2
  exit 1;
fi;
# Create dump
mongodump -d $MONGO_DATABASE --username $DBUSERNAME --password $DBPASSWORD --authenticationDatabase $DBAUTHDB
# Rename dump directory to backup name
mv dump $BACKUP_NAME
# Compress backup
tar -zcvf $BACKUPS_DIR/$BACKUP_NAME.tgz $BACKUP_NAME
# Delete uncompressed backup
rm -rf $BACKUP_NAME
# Delete backups older than retention period
find $BACKUPS_DIR -type f -mtime +$DAYSTORETAINBACKUP -exec rm {} +
echo "--------------------------------------------"
echo "Database backup complete!"

You need to update the parameters in above shell script as per your environment. Specify the name of database to be backed up, host name, port in MONGO_DATABASE, MONGO_HOST and MONGO_PORT parameters. To specify the backup directory, update the BACKUPS_DIR parameter. You need to update the database credentials in DBUSERNAME, DBPASSWORD and DBAUTHDB parameters if you are using one. Finally you need to specify the number of days you want to retain the backup file in DAYSTORETAINBACKUP parameter. Once you update all the parameters as per your environment, save the file as mongobackup.sh. After that you need to make the file executable by issuing following command:

1
chomd +x mongobackup.sh

Now you need to create a cron job to execute this script automatically. Issue following command to create a cron job:

1
crontab -e

Enter following:

1
2
# MongoDB Backup
00 00 * * * /path/to/script/mongobackup.sh

Save the file by pressing Escand then wq + Enter (similar to vi editor). This cron job will execute the backup script at 12 o’clock every night. You can head over to this site to generate a cron entry of your liking.

That’s it, sit back and relax. The script will execute every midnight, create backup files and will also delete the backup files older than specified in DAYSTORETAINBACKUP parameter.

Kill idle connections to Redis

2016-02-05T17:40:18+05:30

I was using a Amazon Elasticache cache.t2.medium Redis instance as a cache store for one of the Rails application. Over the time I observed that connections to Redis instance are increasing day by day and once it hit ~1500, the Rails application started throwing connection timeout exceptions:

1
Error connecting to Redis on redis-host:6379 (Redis::TimeoutError)

On searching over internet, I found that the default parameter group has timeout value set to 0 which means, any connection that has been initialized to redis will be kept by redis even if the connection initialized by the client is down. Well the first thing I did was to create a new parameter group and set its value to a non-zero integer. However I couldn’t apply the parameter group changes to live production application and hence started looking for alternatives.

On further investigation, I found that Redis provides CLIENT KILL command to kill the client connections. To see all client connections to your Redis instance, you can use CLIENT LIST command:

1
2
3
redis 127.0.0.1:6379> CLIENT LIST
id=9115408 addr=10.0.1.252:56348 fd=95 name= age=23 idle=23 flags=N db=14 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get
id=9115336 addr=10.0.2.151:52018 fd=37 name= age=107 idle=107 flags=N db=14 sub=0 psub=0 multi=-1 qbuf=0 qbuf-free=0 obl=0 oll=0 omem=0 events=r cmd=get

To kill a connection, you can use CLIENT KILL command as below:

1
2
redis 127.0.0.1:6379> CLIENT KILL 10.0.1.252:56348
(integer) 1

On successful completion, the command will return number of client connections killed. Well this was all good but I can’t kill ~1500 connections one by one. Hence I wrote a little python script to achieve this task.

1
2
3
4
5
6
7
8
9
10
11
12
13
import redis
import re

idle_max = 300

r = redis.Redis(host="localhost", port=6379, password=None)
cl = r.execute_command("client", "list")

pattern = r"addr=(.*?) .*? idle=(\d*)"
regex = re.compile(pattern)
for match in regex.finditer(cl):
    if int(match.group(2)) > idle_max:
        r.execute_command("client", "kill", match.group(1))

Above script will kill all redis client connections which have been idle for 300 seconds. You can adjust the value of idle_max variable as per your need. Save this script as killredisconn.py and execute as below:

1
$ python killredisconn.py

You need to install redis python package before executing this command. You can install the same by issuing following command:

1
$ sudo pip install redis

Once I executed above script, number of client connection decreased dramatically and connection timeout exceptions from our Rails application disappeared.

Sending asynchronous emails using Sidekiq on OpenShift

2015-02-09T07:38:09+05:30

If you need to send transactional emails from your Rails application, you should always send them asynchronously using a background job. If you send transactional emails synchronously, the user will see a loading page for the time rails connect to SMTP server and delivers your email which is about 2-3 seconds. This will result in a very bad user experience and it’s advisable to send emails using a background job.

Sidekiq advertises it self as simple, efficient background processing engine for Ruby. Sidekiq has dependency on Redis version 2.4 or greater. Hence to setup Sidekiq on OpenShift, we have to setup Redis first. So let’s get started.

Setting up Redis

Redis cartridge for Openshift is available here. To add this cartridge to existing Openshift application, issue following command:

1
rhc add-cartridge http://cartreflect-claytondev.rhcloud.com/reflect?github=smarterclayton/openshift-redis-cart

If this command fails, inspect the log files generated at $OPENSHIFT_REDIS_DIR/logs/redis.log.

Setting up Sidekiq

The Sidekiq community cartridge hasn’t been updated in a while and it doesn’t work with current OpenShift. Hence I forked the repository and fixed the issues. I also bumped the Sidekiq version to 3.3.1 which was latest at that time. You can see the code changes here.

To add Sidekiq cartridge to application, issue following command:

1
rhc add-cartridge http://cartreflect-claytondev.rhcloud.com/github/pranavpr/openshift-origin-cartridge-sidekiq --app myapp

Once the installation completes, you can access the monitoring interface at https://YOUR_APP-NAMESPACE.rhcloud.com/sidekiq. Sidekiq will print the full URL with authentication after successful install.

Updating Gemfile

Add following line to your Gemfile to add Sidekiq to your Rails application:

1
gem 'sidekiq'

Then do a bundle install to finish installation of sidekiq gem.

Send asynchronous emails using Sidekiq

Delayed extensions of Sidekiq allows sending emails asynchronously with Action Mailer out of the box. It provides three methods viz. delay, delay_for(interval) and delay_until(time), to send email asynchronously.

Use delay to deliver emails asynchronously, delay_for(interval) or delay_until(time) to deliver the email at some point in the future.

1
2
3
UserMailer.delay.welcome_email(@user.id)
UserMailer.delay_for(5.days).find_more_friends_email(@user.id)
UserMailer.delay_until(5.days.from_now).find_more_friends_email(@user.id)

Devise emails

devise-async gem provides the ability to send Devise emails using Sidekiq. Add following line to the Gemfile:

1
gem 'devise-async'

And then execute bundle install to finish installation of devise-async gem.

Now add :async to the devise call in your model:

1
2
3
class User < ActiveRecord::Base
  devise :database_authenticatable, :async, :confirmable # etc ...
end

Finally set your queuing backend to Sidekiq by creating config/initializers/devise_async.rb:

1
Devise::Async.backend = :sidekiq

Epilogue

I issued a pull request to merge my changes to community cartridge and it got accepted. So you can also use the community cartridge to add Sidekiq to your OpenShift application.

Active Admin and Paperclip done right

2014-12-28T06:39:07+05:30

Active Admin is an administration framework for Ruby on Rails applications which helps in creating beautiful administration panels. Paperclip is a file attachment library for Active Record which makes management of files to models really easy.

Recently while creating administration page for a model, in which Paperclip was being used for managing attachments, I oberved that Active Admin was not handling it properly. For example, the view page was showing as below:

And the edit and new page was being shown as below:

There was no way one could upload the images with above interface. Hence I scoured internet and found some nifty hacks. Let’s go through each of them:

View Page

Instead of showing attached image properties, I wanted to show the image itself. Also for other attached files types (like pdf and doc), I wanted to give an URL. Below code achieved the same:

1
2
3
4
5
6
7
8
9
10
11
12
13
ActiveAdmin.register Item do
  show do |recipe|
    attributes_table do
      row :instruction do
        item.instruction? ? link_to(item.instruction_file_name, item.instruction.url) : content_tag(:span, "No instruction file yet")
      end
      row :photo do
        item.photo? ? image_tag(item.photo.url, height: '100') : content_tag(:span, "No photo yet")
      end
    end
    active_admin_comments
  end
end

This resulted in following:

New and Edit Page

I wanted to show an upload button to upload files. Also I wanted to show current file attached on Edit page so users can verify the attached content before modifying it. The code below achieved the same:

1
2
3
4
5
6
7
8
9
ActiveAdmin.register Item do
  form :html => { :enctype => "multipart/form-data" } do |f|
    f.inputs do
      f.input :instruction, hint: f.item.instruction? ? link_to(f.item.instruction_file_name, f.item.instruction.url) : content_tag(:span, "Upload PDF file")
      f.input :photo, hint: f.item.photo? ? image_tag(f.item.photo.url, height: '100') : content_tag(:span, "Upload JPG/PNG/GIF image")
    end
    f.actions
  end
end

This resulted in following:

Note that you have to set enctype attribute in your form to multipart/form-data to enable upload of files.

And that’s it. I was pretty happy with the end result. Do let me know in comments if you happen to know any other trick.

Migrating Ruby on Rails app from Heroku to OpenShift

2014-11-08T13:31:31+05:30

OpenShift is Red Hat’s Platform-as-a-Service (PaaS) that allows developers to quickly develop, host, and scale applications in a cloud environment similar to Heroku. OpenShift also supports deploying of application with simple git push just like Heroku. OpenShift supports a variety of technologies through use of cartridges, including Java (Wildfly, JBossEAP, Tomcat), PHP, Node.js, Python, Perl, MySQL, PostgreSQL, MongoDB, Jenkins, Cron, and JBoss xPaaS Services (Fuse, BPM Suite, BRMS, Data Virtualization, Aerogear, and more).

Why OpenShift?

The free version of OpenShift, OpenShift Online, has many advantages over free tier of Heroku:

If application is not accessed for an hour on Heroku, web dyno goes to sleep mode making the next request take up to 30 seconds to serve leading users to abandon the site visit. OpenShift’s idling time is 24 hours and, as of my personal experience, it takes around 10 seconds for next request to serve.

The free database tier on Heroku allows only 10,000 rows to be stored. There is no such limitation in OpenShift.
Heroku lacks persistent storage, its file system is read only and hence it doesn’t allow file upload from application, e.g. user avatars. OpenShift provides 1GB of persistent storage with free tier which can be used to store the assets.
OpenShift provides SSH access to control the application which is lacking in Heroku.
OpenShift also provides port-forwarding feature which lets one connect to the database using locally installed client.
OpenShift provides support for deployment hooks which can be used for running custom scripts before or after deployment.

Getting Started

Sign up for a free account of OpenShift at https://www.openshift.com and set up the rhc client tool as documented here.

Create the Ruby Application

Create the Ruby application by logging in to web console or via rhc client tool by issuing following command:

1
$ rhc app create myapp ruby-2.0 postgresql-9.2

Here myapp is the name of the application, and ruby-2.0 and postgresql-9.2 are the cartridges. Once the application is created, it will output the git repository path. Take a note of this path as this would be used to push the code to OpenShift.

Setup git remote

Navigate to the existing rails application directory and add the OpenShift remote as below:

1
$ git remote add openshift ssh://...@myapp-mynamespace.rhcloud.com/~/git/myapp.git/

Now pull the remote changes by issuing following command:

1
$ git pull openshift master

It will result a conflict in config.ru. Just abandon the content created by OpenShift. Below is the default config.ru file for the reference which is generated by rails:

1
2
3
4
# This file is used by Rack-based servers to start the application.

require ::File.expand_path('../config/environment',  __FILE__)
run Rails.application

Modify Gemfile and .gitignore

OpenShift uses an old version of bundler and hence if ruby version is specified in Gemfile, as shown below, it breaks.

1
2
3
4
5
source 'https://rubygems.org'

ruby '2.0.0' # Causes problem in OpenShift
 
gem 'rails', '4.0.8'

Also, as recommended by OpenShift, using their Ruby mirror http://mirror.ops.rhcloud.com/mirror/ruby/ can quicken the installation of gems.

And at last, if application was deployed on Heroku, it’s high chance that database.yml was added to .gitignore file as Heroku creates it’s own database.yml file during deployment. If it’s the case, remove the database.yml line from the .gitignore file.

Database configuration

Add following section in database.yml file:

1
2
3
4
5
6
7
8
9
production:
  adapter: postgresql
  encoding: unicode
  pool: 5
  database: <%=ENV['OPENSHIFT_APP_NAME']%>
  host: <%=ENV['$OPENSHIFT_POSTGRESQL_DB_HOST']%>
  port: <%=ENV['$OPENSHIFT_POSTGRESQL_DB_PORT']%>
  username: <%=ENV['OPENSHIFT_POSTGRESQL_DB_USERNAME']%>
  password: <%=ENV['OPENSHIFT_POSTGRESQL_DB_PASSWORD']%>

Deployment hook

To run the database migrations, create a deployment hook by adding following lines to .openshift/action_hooks/deploy file:

1
2
3
4
5
6
#!/bin/bash

pushd ${OPENSHIFT_REPO_DIR} > /dev/null
echo "Running migrations"
bundle exec rake db:migrate RAILS_ENV="production"
popd > /dev/null

In above script, pushd will put the current working directory on the directory stack and switch to the directory path in the ${OPENSHIFT_REPO_DIR} environment variable and popd will retrieve the directory from the stack and set it again as the current one. This is because bundle should be executed in the application’s directory, and then come back to the original directory, in order not to disrupt anything that may be executed afterwards.

Publishing the application

To publish the application on OpenShift, just issue the git push command as below:

1
$ git push openshift master

Import database from Heroku

Importing and exporting PostgresSQL databases on Heroku is well documented here. Firstly, we need to add pgbackup addon to our Heroku app:

1
$ heroku addons:add pgbackups

To export the data from Heroku PostgresSQL database, create a new backup and use any number of download tools, such as curl or wget, to store the backup locally.

1
2
$ heroku pgbackups:capture
$ curl -o latest.dump `heroku pgbackups:url`

Restore database on OpenShift

To restore the database on OpenShift, we need to know the value of OPENSHIFT_POSTGRESQL_DB_USERNAME environment variable. Issue following commands to get this information:

1
2
3
$ rhc ssh myapp
$ env| grep OPENSHIFT_POSTGRESQL_DB_USERNAME
$ exit

Take a note of the output and start port forwarding as below:

1
$ rhc port-forward myapp

It will give output similar to following:

1
2
3
4
5
6
7
8
9
10
11
12
Checking available ports ... done
Forwarding ports ...

To connect to a service running on OpenShift, use the Local address

Service    Local                OpenShift
---------- --------------- ---- ----------------
httpd      127.0.0.1:8080   =>  x.x.x.x:8080
postgresql 127.0.0.1:5432   =>  x.x.x.x:5432
ruby       127.0.0.1:32036  =>  x.x.x.x:32036

Press CTRL-C to terminate port forwarding

Now issue following commang to restore the database:

1
$ pg_restore –verbose –clean –no-acl –no-owner -h localhost -U OPENSHIFT_POSTGRESQL_DB_USERNAME -d myapp latest.dump

Replace OPENSHIFT_POSTGRESQL_DB_USERNAME with the output from command at the start of this section.

And that’s it. Now visit the application and make sure everything is working as expected.

Epilogue

Heroku is a great platform for developing and deploying Rails application. I started looking for alternatives of Heroku because once I enabled rails_admin gem in my application, Heroku was not able to serve the requests. I started getting intermittent application error. Once I migrated to OpenShift, these errors went away.

Navigation menu in Ruby on Rails

2014-10-25T17:20:01+05:30

Recently while working on a Rails project, I came across following markup for navigation:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
<ul class="sidebar-menu">
  <li class="treeview active">
    <a href="#">
      <span>Category</span>
    </a>
    <ul class="treeview-menu">
      <li class="active">
	      <a href="#">
	      	<span>Add Category</span>
	      </a>
      </li>
      <li>
	      <a href="#">
	      	<span>Update Category</span>
	      </a>
      </li>
    </ul>
  </li>
</ul>

Note that in above markup, for each active menu item, there is a parent active element. For example, when “Add Category” (line number 7) is active, it’s parent element “Category” is also active (line number 2).

To incorporate this in Rails, I decided to make parent element as controller and child elements as actions for that controller. Thus for above markup, I created a categories controller and added action new, create, edit and update to it. Then I created two helper functions as below:

1
2
3
4
5
6
7
8
9
module ApplicationHelper
	def controller?(*controller)
		controller.include?(params[:controller])
	end

	def action?(*action)
		action.include?(params[:action])
	end
end

Then I created the menu layout as below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<ul class="sidebar-menu">
  <li class="treeview <%= 'active' if controller?('categories') %>">
    <a href="#">
      <span>Category</span>
    </a>
    <ul class="treeview-menu">
      <li class="<%= 'active' if controller?('categories') && action?('new', 'create') %>">
      <%= link_to(raw('<span>Add category</span>'), new_categories_path) %>
      </li>
      <li class="<%= 'active' if controller?('categories') && action?('update', 'edit') %>">
      <%= link_to(raw('<span>Modify category</span>'), edit_categories_path) %>
      </li>
    </ul>
  </li>
</ul>

This layout perfectly synchronized the navigation links with pages.

I found this as a neat way of making navigation in my rails project. You can also use current_page? (documentation) method to accomplish the same. There are many gems also available out there to build the navigation in Rails. You can have a look at them here.