Diagnostics & Performance Computing

by Dan Crosta, Software Engineer, 10gen


  • MongoDB is a high performance database
  • But how do you know you are getting the best performance


1. mongostat

  • give it a host and port number. So we can connect to production. Woot!

  • tons of useful columns

  • mapped

  • vsize

  • res

  • faults - how many disk faults

  • locked %

    • In a given window of time, measures two things (TODO find out)
    • Rough percentage measure - not perfectly accurate

2. serverStatus

What powers mongostat

> db.serverStatus();
        "version": "2.0.1",
        "prodess": "mongod",

    // lots more stats

3. Profiler

> db.setProfilingLevel(2)
    {"was": 0, "slowms": 100, "ok": 1}

This saves the data into a collection within the MongoDB. Which is nice cause you can reference it later.

See it in operation!

> db.system.profile.find()
    "ts": ISODate,
    "op": "query",
    "ns": "docs.spreadsheet",
    "query": {"username":"dcrosta"},
    "scanned": 200001,
    "millis": 1407
    // tons more!


This is a capped collection of 1MB. So it stores only the most recent. You can change this with some hacks. TODO - find it out

4. Monitoring Service

  • MMS: 10gen.com/try-mms (Free service provided by 10gen)
  • Also check out Nagios
  • Also check out Munis

Common problems

1. Slow Operations

Check the logs! From the shell:

query docs.spreadsheets ntoreturn:100
nscanned:19976 { username: "dcrosta"}
nreturned:100 147ms

This means you need to index the username field

2. Replication Lag

Every time you do a read/write, it hits a capped collection called the oplog. Replication lag refers to the time between when a read/write is called and when it is performed.

Example: If you have a very high write rate on the Primary, your secondaries can have trouble keeping up.

3. Resident Memory

Always use 64-bit!

> db.serverStatus().mem
    "bits"64, // need 64, not 32
    "" resident: 7151
    "virtual": ???
    "???": ??
> db.stats()
// other things
"avgObjSize": 5107.02342342, // capped at 16MB
"dataSixe": 234424323423, // make sure this doesn't exceed your server space!
// other things


indexSize + dataSize <= RAM

4. Page Faults

> db.serverStatus().extra_info

    "heap_usage_buytes": 2313132,
    "page_faults": 2381

5. Write Lock Percentage

> db.serverStatus().global_lock

    "totalTime": 23234234,
    "lockTime": 134646546,
    "ration": 0.002342342


What to look for: ???

6. Reader and Writer Queues

> db.serverStatus().globallock
    "blah": "blak=h"


What to look for: Things that are eating up tons of process. To stop it, run:

7. Background Flushing

> db.serverStatus().backgroundFlushing

    "flushes": 5634,
    "total_ms": 83556,
    "average_ms": 14.832342342,
    "last_ms": 4,
    "last_finished": ISODate

In some case you should flush more frequently then MongoDB does by default

Disk Considerations

  • Raid: Use it
  • SSD: If you can get your server on a SSD, then things will go much, much faster.
  • SAN?:

8. Connections

> db.serverStatus().connections
{"current": 7, "available": 19993}
  • Make sure you have enough connections.
  • On Linux, change the number of connections that can be opened.
  • MongoDB can handle up to 20,000 open connections

9. Network Speed

> db.serverStatus()..network
// data here

Check this as one of the things that might be bottlenecking

10. Fragmentation

> db.spreadsheets.stats()
{// data here
  • When you move data around frequently, fragmentation occurs.
  • THis will cost you more memory, slowing things down
  • “2 is the magic number”. You disk should be at least twice as big as the MongoDB memory
// blocking command.
// Be careful!!!

Compacting fixes the problem, but it stops operations on that server. So run it against a secondary instead of the primary.