How to call GoodReads API using YQL

blogentry, programming, todayilearned, es6


title: How to call GoodReads API using YQL date: "2017-06-20" banner: ./images/featured-image-1.jpg published_at: "2017-06-20T23:53:58.000Z" tags: "blogentry, programming, todayilearned, es6" author: Sung M. Kim

Featured Image - "Reading" by Sam Greenhalgh, used under CC BY 2.0

I have been trying to use GoodReads data to display my reading statistics and to find patterns using Javascript.

The problem is that GoodReads API did not enable Cross-origin resource sharing (CORS) headers thus AjAX calls were failing.

How can we get around the issue?

1st Hurdle

Here is the code for the first attempt.

export default class DemoMain {
constructor() {
this.goodReadsURL = `https://www.goodreads.com/shelf/list.xml`;
this.q = `key=${apiConfig.goodreadsKey}&user_id=${apiConfig.goodreadsUserID}&page=1`;
this.url = `${this.goodReadsURL}?${this.q}`;
logResult(url) {
axios.get(url, { params: { format: "json" } })
.then((yqlResponse) => {
let data = yqlResponse.data;
// DefiantJS XPath query for user shelf for "read" section.
let search = JSON.search(data, "//*/user_shelf[name='read']");
}).catch((error) => {
alert(`error: ${error}`);
firstAttempt() {
var demo = new DemoMain();
view raw index.js hosted with ❤ by GitHub

Above code returns following error message.

XMLHttpRequest cannot load https://www.goodreads.com/shelf/list.xml?key=SECRET_KEY&user_id=25927588&page=1&format=json. No 'Access-Control-Allow-Origin' header is present on the requested resource. Origin 'https://localhost:8080' is therefore not allowed access.

This GoodReads Developer forum post shows that other people are having the same issue.

There is a workaround found in this post that one of the workarounds is to use a proxy server.

2nd Hurdle

If you have never set up a proxy server, you'd need to search for documentations and read them and so on.

Instead of setting up your own proxy server, you can use Yahoo's YQL (Yahoo Query Language) as an external proxy.

The following code snippet wraps a GoodReads URL using [YqlAjax](https://github.com/dance2die/Blog.Javascript.GoodReadsAPI/blob/master/js/yql_ajax.js), which does an AJAX request and returns a promise. secondAttempt runs without an issue and returns a record from GoodReads.

import YqlAjax from './yql_ajax';
secondAttempt() {
let yqlAjax = new YqlAjax();
.then((yqlResponse) => {
// DefiantJS XPath query for user shelf for "read" section.
let search = JSON.search(yqlResponse.data, "//*/user_shelf[name='read']");
}).catch((error) => {
alert(`error: ${error}`);
view raw index.js hosted with ❤ by GitHub
var axios = require('axios');
var apiConfig = require('../apikey.js');
export default class YqlAjax {
ajax(url) {
const yqlUrl = "http://query.yahooapis.com/v1/public/yql";
let goodReadsURL = `${url}?key=${apiConfig.goodreadsKey}&user_id=${apiConfig.goodreadsUserID}&page=1`;
let q = `select * from xml where url="${goodReadsURL}"`;
return axios.get(yqlUrl, {
params: {
q: q,
format: "json"
view raw yql_ajax.js hosted with ❤ by GitHub

3rd Hurdle

The last problem is that it's quite a pain to generate a YQL URL and is error prone. Creating another Javascript file (yql_ajax.js) seems unnecessary.

There is an NPM package called proxify-url, which returns a new YQL URL given any URL. Now the code becomes simpler and you have one less file to create/maintain.

var proxify = require('proxify-url');
thirdAttempt() {
// GoodReads API returns result in "XML" format.
// "XML" is the "input" format fed into YQL
let proxyUrl = proxify(this.url, { inputFormat: 'xml' });
view raw index.js hosted with ❤ by GitHub

One thing to note is that GoodReads API returns data in XML format instead of JSON. So you need to pass { inputFormat: 'xml' } as an option to proxyfy method.

The code is now much more readable using existing library.


Using Yahoo Proxy with YQL, we can circumvent CORS restriction and query GoodReads API.

You can find the source code on here. Refer to the README.md file on how to set up the project and run it.