Last Resort: How to Use a Backup to Start a Secondary Instance for MongoDB?

In this blog post, I’ll look at how you can use a backup to start a secondary instance for MongoDB.

Although the documentation says it is not possible to use a backup to start a secondary, sometimes this is the only possible way to start a new instance. In this blog post, we will explain how to bypass this limitation and use a backup to start a secondary instance.

The initial sync/rsync or snapshot works fine when the instances are in the same data center, but it can fail or be really slow. Much slower than moving a compressed backup between data centers.

Not every backup can be used as a source for starting a replica set. The backup must have the oplog file. This means the backup must be done in a previously existent replica set using the

--oplog flag

 point in time backup when dumping the collections. The time spent to move and restore the backup file must be less than the oplog window.

Please follow the next steps to create a secondary from a backup:

1. Create a backup using the

   --oplog

    command.
   

   mongodump --oplog -o /tmp/backup/

2. Backup the replica set collection from the local database.
   
   

   mongodump -d local -c system.replset

3. After backup finishes please confirm the oplog.rs file is in the backup folder.
4. Use

   bsondump

    to convert the oplog to JSON. We will use the last oplog entry as a starting point for the replica set.
   

   bsondump oplog.rs > oplog.rs.txt

5. Initiate the new instance without the replica set parameter configured. At this point, the instance will act as a single instance.
6. Restore the database normally using

   --oplogreplay

    to apply all the oplogs that have been recorded while the backup was running.
   

   mongorestore /tmp/backup --oplogReplay

7. Connect to this server and use the local database to create the oplog.rs collection. Please use the same value as the other members (e.g., 20 GB).
   

   mongo
   use local
   db.runCommand( { create: "oplog.rs", capped: true, size: (20 * 1024 * 1024 * 1024) } )

8. From the oplog.rs.txt generated in step 4, get the last line and copy the fields ts and h values to a MongoDB document.
   
   

   tail -n 1 opslog.rs.txt

9. Insert the JSON value to the oplog.rs collection that was created before.
   

   mongo
   use local
   db.oplog.rs.insert({ ts: Timestamp(12354454,1), h: NumberLong("-3434387384732843")})

10. Restore the replset collection to the local database.
    

    mongorestore -d local -c system.replset ./backup/repliset.bson

11. Stop the service and edit the parameter replica set name to match the existing replica set.
12. Connect to the primary and add this new host. The new host must start catching up the oplog and get in sync after a few hours/minutes, depending on the number of operations the replica set handles. It is important to consider adding this new secondary as a hidden secondary, without votes if possible, to avoid triggering an election. When the secondary is added to the replica set drivers, it will start using this host to perform reads. If you don’t add the server with

    hidden: true

    , the application will read inconsistent data (old data).
    
    

    mongo
    PRIMARY>rs.add({_id : <your id>, host: "<newhost:27017>", hidden : true, votes : 0, priority :0})

13. Please check the replication lag, and once the seconds behind master is near to zero, change the host parameters in the replica set to

    hidden: false

     and priority or

    votes

    .
    

    mongo
    PRIMARY> rs.printSlaveReplicationInfo()

14. We are considering a replica set with three members, where the new secondary has the ID 2 in the member’s array. Use the following command to unhide the secondary and make it available for reads. The priority and votes depend on your environment. Please notice you might need to change the member ID.
    

    mongo
    PRIMARY> cfg = rs.config()
    cfg.members[2].hidden = false
    cfg.members[2].votes = 1
    cfg.members[2].priority = 1
    PRIMARY> rs.reconfig(rs)

I hope this tutorial helps in an emergency situation. Please consider using initial sync, disk snapshots and hot backups before using this method.

Feel free to reach out me on twitter @AdamoTonete or @percona.