The modular debugger (MDB) is a vital tool to have at your disposal for debugging issues in your Node.js application, no matter if they are trivial or complex, like that of the infamous Node.js memory leak. Unfortunately, setting up an environment to run MDB is often too much of a hurdle for developers to consider even using it. This post explains how to run MDB as a service to make debugging Node.js applications easier.

Over the years, Joyent has shared information on how to use MDB to debug core dumps generated from Node.js applications. The information in those posts is still valid, but there are new dcmds that are worth a demonstration. This post demonstrates some of the new commands. Additionally, it assumes that you don't have and don't plan to setup SmartOS. The only prerequisite is that you sign up for a free Manta account and install the Manta CLI.

Copying the core dump and node binary

To use MDB with a Node.js core dump, you will need to pass in the Node.js binary used to run the process that generated the core dump. And, of course, you will also need the core dump file itself. Once you have these files saved locally, you can upload them into Manta using the mput command. Below is an example of uploading the Node.js binary and a core dump to Manta.

mput -f node ~~/stor/mput -f core.dump ~~/stor/

In addition to the above two files, you will also need the mdb_v8 binary file. Fortunately, this file is in Manta in a public location. As a result, you can use mln to create a SnapLink to the file. Below is an example of setting up a SnapLink named v8_amd64.so from the public file located at /Joyent_Dev/public/mdb_v8/v1.1.3/mdb_v8_amd64.so.

mln /Joyent_Dev/public/mdb_v8/v1.1.3/mdb_v8_amd64.so ~~/stor/v8_amd64.so

Launching a Manta session and running MDB

Now that all of the files required are in Manta, you can create a new interactive session and start debugging with MDB. To create a new session use the mlogin command specifying the two assets along with the main core file. Below is an example of running this command.

mlogin ~~/stor/core.dump -s ~~/stor/v8_amd64.so -s ~~/stor/node

mlogin connects you to Manta with the v8_amd64.so and node files available in the /assets/{username}/stor/ folder. Next, load MDB by passing in the path to the Node.js binary and the core dump file, below is what this command should look like, replacing {username} with your Manta username.

mdb /assets/{username}/stor/node $MANTA_INPUT_FILE

Load the v8_amd64.so file into MDB in order to gain access to the Node.js dcmds using the ::load command.

> ::load /assets/{username}/stor/v8_amd64.so

Everything should be loaded and ready. To verify that this is the case and to see some basic information about the process that created the core dump run the ::status command.

> ::statusdebugging core file of node (64-bit) from smartosfile: /assets/{username}/stor/nodeinitial argv: node --abort-on-uncaught-exception server.jsthreading model: raw lwpsstatus: process terminated by SIGILL (Illegal Instruction), addr=f401e9

V8 dcmds

To see a listing of all of the available dcmds, execute ::dcmds. If you want to see more information on a particular dcmd run ::help followed by the command. For example, to get the help information on the ::nodebuffer command execute the following.

> ::help nodebufferNAME  nodebuffer - print details about the given Node BufferSYNOPSIS  addr ::nodebuffer [-a]ATTRIBUTES  Target: proc  Module: v8_amd64  Interface Stability: Unstable

For the most part, ::findjsobjects is the command to become the most familiar with, as it will be what you will use to look for the JavaScript objects in memory. For example, if you want to see all of the requests to your process at the time the core dump was created you can use ::findjsobjects with the constructor flag as shown below.

> ::findjsobjects -c IncomingMessage

The ::findjsobjects command executed above, outputs the address for any object that has the constructor IncomingMessage. You can pipe this result to ::jsprint to see what the object values are.

> ::findjsobjects -c IncomingMessage | ::jsprint{    "_readableState": {        "objectMode": false,        "highWaterMark": 16384,        "buffer": [...],

To see a listing of the functions that were in memory, use the ::jsfunctions command, as shown below.

> ::jsfunctions            FUNC   #FUNCS NAME                                     FROM    378381117f51        1  (as req.once)                /server.js line 11    378381117be1        1  (as req.on)                  /server.js line 7    378381116091        1 resOnFinish                              _http_server.js position 15138

If you are curious about what the actual source code of a function is, you can use the ::jssource command. For example, to print the source of resOnFinish shown above, the command would be the following.

> 378381116091::jssourcefile: _http_server.js  491     }  492  493     // When we're finished writing the response, check if this is the last  494     // response, if so destroy the socket.  495     res.on('finish', resOnFinish);  496     function resOnFinish() {  497       // Usually the first incoming element should be our request.  it may  498       // be that in the case abortIncoming() was called that the incoming  499       // array will be empty.  500       assert(incoming.length === 0 || incoming[0] === req);

Similarly, to know what variables a function has a reference to, use the ::jsclosures command, as shown below.

> 378381116091::jsclosure    "req": 3783811141d9: {        "_readableState": 3783811142c9: [...],        "readable": 1e40ce504299: false,        "domain": 1e40ce504101: null,

When finished using MDB execute the ::quit command and exit the Manta session using exit.

Thoth

In addition to running MDB directly, there is a utility named Thoth that can be used to upload and analyze a core dump by using Manta Functions. Use npm to install manta-thoth globally (npm -g manta-thoth). Once installed, you can use Thoth to upload and issue commands into MDB. For example, the following shows how to upload a core into Manta and retrieve basic information about it, similar to running the ::status command previously.

$ thoth upload core.dumpthoth: creating 2bcc33c390eda0f5b106041f3bdb5910...$ thoth info 2bcc33c390eda0f5b106041f3bdb5910{    "name": "/{username}/stor/thoth/2bcc33c390eda0f5b106041f3bdb5910",    "dump": "/{username}/stor/thoth/2bcc33c390eda0f5b106041f3bdb5910/core.dump",    "pid": "50187",    "cmd": "node",    "psargs": "node --abort-on-uncaught-exception server.js",    "platform": "joyent_20160622T234741Z",    "node": "smartos",    "version": "1",    "time": 1473781733,    "stack": []}

Summary

It should be clear that there are many powerful commands available in MDB when using the mdb_v8 binaries. Over the years many commands have been added as well as support for debugging Linux core dumps. Additionally, there isn't even a need setup a local or hosted instance of SmartOS, as Manta is all you need for debugging purposes. Therefore, there hasn't been a better time to use MDB with Manta for post-mortem Node.js debugging.