Return

Loopback, Or: How I Learned to Stop Worrying and Read the Source Code

A lot goes into starting a new development project. Meetings, designs, user stories, etc. Inevitably, the team will face the daunting decision of “what framework are we going to use?”

Over the past several years, the number of open source JavaScript frameworks, libraries and tools has grown exponentially. The choices are endless, and at times, overwhelming. This phenomenon even has its own name: “JavaScript Fatigue”. While writing an application from scratch is a fantastic exercise, few development groups have the time, resources or budget for such luxuries, and so we turn to the world of open source to seek out the building blocks that will be assembled to form our latest and greatest projects. The finely-honed, peer-reviewed, community-maintained, commercially-backed, magic black boxes that will deliver the best algorithms, smartest implementations, and cleanest code. But sometimes, things aren’t quite that easy.

Recently, our team was tasked with building a solution that included native mobile applications (iOS and Android) as well as a web-based administrative interface. We knew that we wanted to utilize a REST API to make support and integration easier, and we were interested in using Node.js for the backend. We settled on LoopBack as our foundation.

LoopBack is an open source framework from StrongLoop, an IBM-owned company that also maintains the ubiquitous Express framework for Node. Designed for quickly constructing and deploying RESTful APIs backed by a variety of databases, it seemed the perfect fit, and, indeed, we were able to stand up a robust, authenticated, MongoDB-backed API in no time. The framework even includes an API browser using Swagger UI. It comes with a command-line tool to assist in creating models, relations, access controls, remote methods, and other components. Applications are constructed from a combination of simple JSON files, and more complex, corresponding JavaScript files. Setting up relations, adding remotely accessible custom methods, and writing boot logic is all very straightforward. Because the framework is built on top of Express, it inherits that platform’s ease of use (and its extensive middleware ecosystem) as well.

Invariably when using any new tool, however, you’re bound to come up against some limitations. One of the key benefits of open source software is not only that you get to see how the sausage is made, but you get to dig right in and, hopefully, try not to lose a few fingers making some yourself. The documentation for LoopBack is somewhat of a mixed bag. Much of it is well-written and contains useful examples to get you up and running quickly, which is commendable given the extensive functionality of the framework. Because it’s under such active development, however, the documentation is not always in sync with the code, and some sections are either lacking in information or wholly misleading. One area where we encountered this issue was with model relationships.

LoopBack offers many different kinds of model relations, all variations on the standard one-to-many/many-to-many/many-to-one paradigm most developers are familiar with. The main differences between relation types are where the relation information is stored. For our project, we picked the referencesMany type, which creates a one-to-many relationship, storing the IDs of the related models in the target model’s database record. This allows easy manipulation of relationships at the API level through create and update methods. It is a one-sided relationship, so no additional work is needed when updating or deleting, which helped keep our code simple.

{
  "name": "CustomModel",
  "base": "PersistedModel",
  "properties": {
    "name": {
      "type": "string",
      "required": true
    }
  },
  "relations": {
    "user": {
      "type": "belongsTo",
      "model": "UserProfile",
      "foreignKey": "userId"
    },
    "assets": {
      "type": "referencesMany",
      "model": "Asset"
    }
  }
}

A simple model configuration JSON file is all that’s needed to create a custom model in LoopBack. This model defines 2 relations, user and assets. The foreign keys for both relations are stored on the record of our CustomModel in the database.

While testing this new model, we discovered a problem. The model was not updating its relations properly after adding or deleting a related asset. The database was correct, but the API was returning stale data. Tracking down this error required cracking open the black box of the LoopBack framework and really exploring the source code, which ended up giving us a much greater understanding of the system as a whole.

Before getting to that level, however, we wanted to make sure our custom code wasn’t at fault. LoopBack provides several remote hooks to make adding functionality, and debugging, easier. At the API level, there are before and after hooks available to intercept any method. We were able to verify the state of our model before adding or deleting a related item, and sure enough, the after state was incorrect.

One tricky part of this sleuthing was figuring out which method to hook into. The documentation is not very clear on this, but when a relation is created, the underlying methods actually look like __{action}__{relationName}. For example, __get_assets was the method we needed to check.

Next, we wanted to check the state of our model on the server, behind the API. Again, LoopBack has hooks that make this easy. Because our model inherits from the built-in PersistedModel class, we have access to operation hooks such as before save and after delete which, unlike their remote counterparts above, operate on groups of CRUD operations instead of individual methods. This means before save will fire if a model is created or updated. Using these hooks, we again verified that the data going in was correct, but the data being returned was not. This meant that the problem likely wasn’t with our code, but instead appeared to be occurring somewhere deeper, possibly in the framework itself.

Because our model inherits from PersistedModel, we began our search there. This class acts like an abstract class, in that many of its methods are intended to be overridden, and none of them relate to the methods we are calling. We decided to look deeper down, so we dove into the base Model class instead. Here, we can see more concrete functionality, including setup of the remote methods for relations.

For our model, we created a default scope which automatically calls the relation method whenever the model is accessed.

"scope": {
    "include": {
      "relation": "assets",
      "scope": {
        "where": {
          "deleted": false
        }
      }
    }
  }

Unfortunately, this class does not have the implementations of those remote methods (i.e. __get__assets, mentioned above.) Determining where to go next was not obvious from the source code, so we turned to the debugger. Node has an incredibly useful debugger feature that lets you use Chrome’s DevTools for viewing and stepping through code. Using this feature, we set a breakpoint in our custom model and followed the code to see where it led us.

CustomModel.observe('after save', function (ctx, next) {
   ctx.instance.assets(function (err, assets) {
     console.log(err)
     console.log(assets)
   })
})

The operation hook in our custom code. We set our breakpoint on ctx.instance.assets( ... ).

Following this function call, we wound up in a file called scope.js. We won’t delve into this file much except to say that it, in turn, calls a function inside relation-definition.js, specifically the related method of the ReferencesMany class.

To discern the above, we simply used the debugger to follow the execution path of our call. The code in-between is rather complex, but DevTools makes inspecting objects easy, as shown in the screenshots below.

Scope

Relation definition

The related method has the following signature:

function(receiver, scopeParams, condOrRefresh, options, cb)

Using the debugger, we can inspect these function parameters and discern their purpose; for example, receiver is our calling object (the ctx.instance from earlier) and cb is our callback function.

That condOrRefresh parameter shows up as null, and it’s not clear from any documentation what it’s for. We know that we want our model to “refresh” its data, so perhaps this parameter is meant to do just that? Looking through relation documentation pages we see a few mentions about passing a filter to the main ‘get’ function for related models. If we step back in the debugger, to the scope.js file, we see some comments in the code that shed more light on the situation. Examining the function that our debugger passes through, we see the following:

if (typeof condOrRefresh === 'function' &&
	options === undefined && cb === undefined) {
  // customer.orders(cb)
  cb = condOrRefresh;
  options = {};
  condOrRefresh = undefined;
} else if (typeof options === 'function' && cb === undefined) {
  // customer.orders(condOrRefresh, cb);
  cb = options;
  options = {};
}
...
return definition.related(self, f._scope, condOrRefresh, options, cb);

In the first if block above, we see the method signature we used in our custom model, ctx.instance.assets(callbackFn). However, in the else block, we see a second syntax, ctx.instance.assets(condOrRefresh, callbackFn). A bit of Googling turns up a thread discussing caching on related models, which confirms that this other signature is exactly what we’re looking for. Knowing that, we update our custom model’s code:

CustomModel.observe('after save', function (ctx, next) {
   ctx.instance.assets(function (err, assets) {
     console.log(err)
     console.log(assets)
   })
})

And promptly get an error!

Uncaught TypeError: Cannot create property 'where' on boolean 'true'
    at RelationDefinition.applyScope (/node_modules/loopback-datasource-juggler/lib/relation-definition.js:208:16)
    at ReferencesMany.related (/node_modules/loopback-datasource-juggler/lib/relation-definition.js:3131:19)
    at ScopeDefinition.method [as related] (/node_modules/loopback-datasource-juggler/lib/relation-definition.js:735:33)
    ...

We’re definitely on to something, though. Heading back into the debugger, we can follow along to where we left off, in relation-definition.js at the aforementioned ReferencesMany.related method, which happens to be the second item in the error’s stacktrace above. This time, however, the condOrRefresh parameter, which was undefined before, now has a value - true - which is what we passed in. We can see the error we’re getting is coming from RelationDefinition.applyScope. That method is called at the bottom of the related method we’re currently debugging. It gets passed two parameters, modelInstance and actualCond. Inspecting those values, we see that modelInstance is the relation we’re working with (assets), and actualCond is true, which is what we passed in.

Relation definition

This looks correct.

Using the debugger, we dive into the applyScope method, which is the method throwing an error. We see right away that there is a problem: That second parameter, filter, which is currently true, is expected to be an object.

* Apply the configured scope to the filter/query object.
 * @param {Object} modelInstance
 * @param {Object} filter (where, order, limit, fields, ...)
 */
RelationDefinition.prototype.applyScope = function(modelInstance, filter) {
  filter = filter || {};
  filter.where = filter.where || {};
  ...

The code tries to add a where property to the parameter, which throws the error and leaves us worse off than we were when we started! It’s at this point in debugging where it’s generally a good idea to get up and walk around for a bit.

Taking stock of where we are, we seem to have found a bug in the LoopBack library. In an ideal world, we would fix this bug, and submit a pull request to the maintainers of LoopBack in hopes that our fix will make it into the next version, and save someone else this trouble. And ideally, we will do that, but right now, we have deadlines.

So how do we fix it and get our code shipped? Sometimes, when the clock is ticking, a hammer is the best tool for the job. Jumping all the way back in our debugger, we can inspect our custom model instance, and see two very interesting private properties, __data and __cachedRelations.

Custom model

Those assets arrays are the symptom we’re trying to cure. While inspecting the model, we see that they aren’t being updated after a save operation. By passing true into the assets() method, the framework was supposed to clear out those cached models, but because of the error we discovered above, that’s not happening. One of the beautiful, and terrible, things about JavaScript is that you, the programmer, have access to any and all properties of an object. To fix the problem, we simply clear out the __cachedRelations and __data properties by hand, using the after save hook to ensure it happens every time the model is created or saved.

// Define a custom instance method on our model
  CustomModel.prototype.reorderAssets = function reorderAssets (callback) {
    // Check if our model has any related asset references
    if (this.assetIds) {
      // Get a reference to the Asset class via the `app` object, which is the running LoopBack application
      let Asset = CustomModel.app.models.Asset
      // Look up the related assets, using the array of IDs stored on our custom model
      Asset.find({
        where: {
          id: {
            inq: this.assetIds
          }
        }
      }, (err, assets) => {
        if (err) {
          return callback(err)
        }
        // Override the properties that LoopBack doesn't propertly clear
        // This fixes the problem we were having by ensuring the underlying properties
        // match the actual state of the database
        this.__cachedRelations.assets = this.__data.assets = assets
        callback()
      })
    } else {
      // If a CustomModel has no assetIds, just make sure the cached assets are removed
      this.__cachedRelations.assets = this.__data.assets = null
      callback()
    }
  }

  // Add a hook to call our custom method whenever our model is created or updated
  CustomModel.observe('after save', function (ctx, next) {
    if (ctx.instance) {
      // Call the custom method above, passing the `next` callback which allows execution to continue
      ctx.instance.reorderAssets(next)
    } else {
      next()
    }
  })

/common/models/custom-model.js

The above is not a perfect solution, but because we went through the underlying codebase thoroughly, we can feel safe knowing that it’s “good enough” and has no undesirable side effects. We’re essentially doing the exact same thing the framework code is supposed to do.

Although this problem took quite a bit of time to resolve, the lessons we learned along the way helped us to write better, more informed code going forward. The use of 3rd party libraries and frameworks can save huge amounts of time and effort, but those savings don’t come without risk. Open source software gives you the ability, and the responsibility, to examine how it works, and to fix it when it doesn’t.

 

By: Ken Dunnington


Want to offer your customers a better experience?

Let's work together: Contact Us Today

Contributed by:
Ken Dunnington Technical Architect

Newsletter Sign Up




Mad*Pow HQ

27 Congress Street
Portsmouth, NH 03801

Office: 603.436.7177
Fax: 603.386.6608

Boston

179 Lincoln Street
Boston, MA 02111

Office: 617.426.7177

PROJECT INQUIRIES

Complete Sale Inquiry Form

EMAIL US

Sales: solutions
Press: communications



Get in Touch

CALL US

603.387.8307

CONTACT US

SALES | PR/MARKETING | EVENTS