Node.js & the Magento 1.x REST API

I most often use JetBrains Webstorm when creating Node.js projects.

Recently I was tasked with a project to create a feed of products on our website from our Magento 1.x instance. If you’re reading this, it’s assumed that you already know what Magento is, but perhaps I’ll edit this later on to cater to those who may not. The Magento 1.x REST & SOAP API reference documentation cater specifically to PHP developers. However, I wanted to utilize Node.js to accomplish my task.

The link to the repo in its current state is here on Github.

There were a couple of hurdles I had to overcome on my way to getting a functioning interface to retrieve the information I wanted from the Magento 1.x REST API.

Brackets in the URI string

This was perplexing at first as the Magento API utilizes bracket notation for filters in the URI string. For instance, if you wanted to get a list of products that had a ‘visibility’ with a value of ‘4’ you had to craft your string in this manner: http://dev.somedomain.com/api/rest/products?limit=100&page=1&filter[1][attribute]=visibility&filter[1][in]=4

Now, I’m using the request with Promises via the request-promise-native package which helped me through another hurdle I’ll mention later on. Apparently when the URI is encoded using request, brackets are not handled correctly. According to this statement at MDN web docs regarding the encodeURIComponent function, square brackets, per RFC3986, are reserved for IPv6 and not encoded.

MDN does provide a function you can plugin to remedy this, however.

From what I can tell, request does not encode them either correctly or at all. What I ended up doing was going into the vanillaoauth.js file in requestand editing a section of the code that deals with formatting the uri component. So this:

var params = qsLib.parse([].concat(query, form, qsLib.stringify(oa)).join(‘&’))

Changes to this:

var param_list = [].concat(query, form, qsLib.stringify(oa)).join('&').split('&');
    
var params = {};
   
for(var j = 0; j < param_list.length; j++) {
   var p = param_list[j].split("=");
   if (p[0] === '') {
      //do nothing with this garbage!
   } else if(p[1]){
      params[decodeURIComponent(p[0])] = decodeURIComponent(p[1]);
   } else {
      params[decodeURIComponent(p[0])] = "";
   }
}    
//var params = qsLib.parse([].concat(query, form, qsLib.stringify(oa)).join('&'), {parseArrays: false})

Where you can see the original line of code was commented out. So using the very handy request-debug I was able to observe my URI strings encoded correctly.

{ request: 
   { debugId: 34,
     uri: 'http://dev.somedomain.com/api/rest/products?limit=100&page=34&filter[1][attribute]=visibility&filter[1][in]=4&filter[2][attribute]=status&filter[2][in]=1/store/1',
     method: 'GET',

On to the next hurdle.

Result Limits & No Pagination

The Magento 1.x REST API provides us with a limited number of items per response. By default you get 10 items back. So for every call I did without any parameters I would receive a JSON object back containing 10 items. Obviously, that is not ideal when you’re dealing with over 4,000 products on your Magento site. Magento has a limit of 100 objects per response. That sounds reasonable, right? Especially if the API you’re working with provides you with information such as pagination, or how many pages of items are available based on your query, or provides a method such as the Amazon MWS NextToken feature where if there’s more data available, you just have to ask for it. Guess what the Magento 1.x REST API does not provide you? That’s right; the Magento 1.x REST API does not provide you with pagination/paging details. This is something you have to do yourself.

Below is the function I developed to help me with this. I wanted to allow my Node.js project to function asynchronously so I created an async paginated request function which returns a Promise and contains a non-async recursive function — a function that calls itself. This helps us achieve something I personally considered to be a struggle due to the abstract way of thinking required. Let’s take a look at the code in sections:

First we declare an async function called paginateResponse, and the idea is to be able to create our own paginated requests to retrieve more responses than the default set. We also want to retrieve them provided a specific endpoint such as ‘products’ or ‘orders’, and provide a filter for the URI to only return the results we really want. If you do not use the available filters, your response is liable to be very large.

async function paginateResponse(endpoint, response_filter) {
    if (validateEndpoint(endpoint)) {
        console.time('paginateResponse');
        console.log('Starting API call');
        let page = 1;

The validateEndpoint() function simply validates the string passed as the endpoint argument. I’m also timing the function just to get a feel for how long it takes to return. I set the page variable at 1 which I will pass to the non-async recursive function api_call().

The next step is to construct our Promise and our recursive function api_call that will provide the resolution requirements. I did not provide a rejection condition, however you should always try to.

let p = new Promise(resolve => {
    let response_arr = [];
    api_call(page, response_filter);

    function api_call(curr_page, filter) {
        let uri;

        if (response_filter){
            uri = apiUrl + `/${endpoint}?limit=100&page=` + curr_page + filter + '/store/1';
        }
        else {
            uri = apiUrl + `/${endpoint}?limit=100&page=` + curr_page + '/store/1';
        }

        let req = request.get({
            url: uri,
            oauth: oauth,
            json: true
        });
        req
            .then(res => {
                response_arr.push(res);
                if (Object.keys(res).length === 100) {
                    page++;
                    console.log('Requesting page ' + page);
                    api_call(page, response_filter);
                }
                else {
                    resolve(response_arr);
                }
            })
            .catch(err => {
                console.log('Promise Rejection Error: ' + err);
            });
    }
});
return await p;

So, we create our Promise p and build in the recursive function api_call and stipulate that api_callshould continue to be called until the number of items returned is less than 100, which signifies that we’ve reached the end of all possible results (pages). We also increment our page variable to alter the uri string each time.

So, those were the two main hurdles. Now we can test and make sure we get the expected results. The following code block calls our paginateResponse() function and, since it returns a Promise, we can asynchronously tell it what to do next, namely mutate our object a bit to match our desired format and then write it to a JSON file.

let product_filter = '&filter[1][attribute]=visibility&filter[1][in]=4&filter[2][attribute]=status&filter[2][in]=1';

paginateResponse('products', product_filter)
    .then(results=>{
        // can retrieve products in an array
        // or as an object indexed with top-level key being the SKU
        if (results){
            let products = mutateObjects(results, 'object');
            writeJSON(products);
            console.log(`Total products returned: ${Object.keys(products).length}`);
        }
    })
    .catch(err => {
    console.log(`Error: ${err}`)
    });

If you want to check out the repo in its current state, head on over to https://github.com/aflansburg/rc-magento-api.