Jump to navigation

Nikola Plejić

org the ultimate API client

2020-12-03 in web, HTTP, emacs

Using emacs & org-mode as an exploratory API client.

There’s a genre of software I like to call “GUIs for curl”: Postman, PatchGirl, and others. Their aim is to make web developers’ lives easier by providing nice graphical interfaces for executing HTTP requests. I never got around to get into a habit of using them since they either had clunky UIs or were not free software. Thus, I defaulted to using curl in a shell.

With time, I did come up with a wishlist of things I wish I could do with this setup. This boils down to being able to:

  • execute HTTP requests
  • parse, analyze and extract data from API responses
  • “compose” API requests by using parts of the response of one API request in the body of another
    • this is most notable during authentication: somewhere early on in your workflow you’ll usually get an access token that you need to pass in each subsequent request
  • “bookmark” (persist) well-formulated API calls
  • annotate queries
  • tag the queries
  • share your queries with others

I tend to spend my working days in Emacs. Unlike the average Emacs person, I don’t really live in Emacs, thus I don’t mind an occasional context shift, yet a decent integrated solution is always appreciated. As it turns out, org-mode ticks all the boxes (as it usually does).

Executing HTTP requests

You could certainly use request.el to send HTTP requests out into the world. However, I just use an org-babel “shell” block and curl like a barbarian. I believe curl is the de-facto standard HTTP client, and it’s familiar to a lot of people. This approach has the extra benefit of transferring nicely into the command-line should it prove necessary.

Here’s a sample HTTP POST request:

* Example API Request: Authentication

  #+NAME: auth
  #+begin_src shell :cache no :results verbatim
    curl -X POST \
      --data "grant_type=password&username=nikola&password=hunter22&scope=all" \
      http://localhost:8080/v1/auth
  #+end_src

  #+RESULTS[...]: auth
  : {"access_token": "f744cb5c55d2946c4jzb3d7d61fb4850967.dae42e4e4d40", "expires": "..."}
  #+end_src

CTRL-c CTRL-c (or leader leader for evil folks) executes the block and outputs the result to be used further in the document. Which leads us to…

Parsing, analyzing and extracting data in API responses

Most APIs these days output JSON or XML, thus having results in the form of plain strings isn’t very useful.

Let’s say we’re only interested in the access_token property of the response to the request executed above. Again, you could very legitimately manipulate that data in elisp, but since we’re in the shell, we can pipe the data to another process:

  #+NAME: auth
  #+begin_src shell :cache no :results verbatim
    curl -X POST \
      --data "grant_type=password&username=nikola&password=hunter22&scope=all" \
      http://localhost:8080/v1/auth | jq .access_token
  #+end_src

  #+RESULTS[...]: auth
  : f744cb5c55d2946c4jzb3d7d61fb4850967.dae42e4e4d40
  #+end_src

This uses jq, a nice little tool for manipulating JSON documents on the command line. There are similar tools for XML/XPath as well.

Using API results as API parameters

Using some wonderful features of org-babel, you can trivially use results of previous commands in subsequent code blocks:

  #+begin_src shell :var t=auth :results verbatim :cache no
    curl \
        -H "Authorization: Bearer ${t}" \
        http://localhost:8080/v1/endpoint | jq .
  #+end_src

This will capture the result of our auth code block into a variable called t, which will then be available within the code block as a regular variable in your language of choice. org-babel will keep track of dependencies and offer to execute any code blocks that are necessary to figure out all of the variables used by the current one.

This has other handy use-cases as well. Need to generate some JSON for your API requests? Just do it in your favorite language:

  #+NAME: json_data
  #+begin_src python :results verbatim :python python3
    import json
    import random

    return json.dumps({"param1": random.randint(0, 100)})
  #+end_src

  #+RESULTS: json_data
  : {"param1": 49}

  #+begin_src shell :var data=json_data :var t=token :results verbatim :cache no
    curl \
        -X POST \
        -H "Authorization: Bearer ${t}" \
        -H "Content-Type: application/json" \
        --data "${data}" \
        https://httpbin.org/post | jq .
  #+end_src

Bookmarking, tagging and annotating API calls

org-mode files are text files, so each entry is automatically “persisted” when you save the file. Everything outside of code blocks is regular org, and org works well for taking notes! You can embed images, add tables, annotate, whatever. For tagging, I just use org’s built in tagging facilities at the heading level.

Sharing

org-mode files are plain text files, so anyone can open them in their prefered text editor and have a reasonably decent reading experience. But org can also export to various formats, and exporting to HTML will give you readable API documentation for free!

By default, org-babel will only export source blocks — not the results. This is good default behavior since sometimes API results involve secrets (e.g. access tokens). To override this, annotate your code block with :exports both:

  #+NAME: auth
  #+begin_src shell :cache no :results verbatim :exports both
    curl -X POST \
      --data "grant_type=password&username=nikola&password=hunter22&scope=all" \
      http://localhost:8080/v1/auth
  #+end_src

  #+RESULTS[...]: auth
  : {"access_token": "f744cb5c55d2946c4jzb3d7d61fb4850967.dae42e4e4d40", "expires": "..."}
  #+end_src

Problems

When exporting, org will export your code blocks literally — any variables will be exported verbatim and not replaced by its respective value. This may or may not be desired — I think it’s acceptable, but it can be confusing. There may be a snippet of Elisp somewhere that makes this go away.

Feedback

Feedback, corrections, comments & co. are welcome: you can write me an email or contact me on Matrix.