19 November 2015
Comparing Scala’s HTTP client libraries
Or why should probably use Play! WS
Today we’re looking at the various Scala libraries available to perform HTTP requests. It’s a very easy problem, yet there’s a bunch of projects on Github all hoping to do better than the previous one. Let’s see how they really compare to each other.
This comparison is not about low-level performance or cleverness of implementation. We will be only talking about already long-lived and popular libraries : I assume they are all correct and perform decently for an everyday usage. Instead, I’m focusing on the quality of the documentation, the ease-of-use of the API, and what the resulting code looks like.
Let’s start where a Scala newcomer would start : googling. If one types “scala http client” into Google, he’ll get in order :
Most of them give a one-line example of how to perform a very simple GET request in their docs. That’s not sufficient. In real-life, you always need something a bit more complex than what’s in the doc.
So we’re gonna do a little experiment. We will take a moderately complex request and see how each of these libraries can help us to do it. Here’s our arbitrary requirements :
- We want to perform a GET request on
- We want to add two query string parameters, i.e.
- We want to add some additional HTTP header. I randomly picked
Cache-Control : no-cache
- We want to throw an exception if the API doesn’t respond with an status code between 200 and 299
- We want to print both the response body as a string, and an arbitrary response header (I randomly picked
Nothing fancy, that should be easy.
Dispatch looks good. First to pop-up in the results, and the doc http://dispatch.databinder.net/ looks good, that’s a good sign right ? Guess again.
Sure it works, if you like hieroglyphs. Noticed the funky method names ? Here’s a peek on a few other methods defined by the API :
as_!. Good luck.
Oh, and actually this library is not maintained. The last meaningful commits seems to be from early 2013, and there’s a bunch of unanswered issues on Github.
Diagnostic : abandonware with a cryptic DSL. Don’t use it.
EDIT I stand corrected. Dispatch is still maintained, they just moved to a different repo (but it still seems that they have a bunch of unanswered issues). Also the weird method names do have English equivalents, like
setQueryParameters. Personally I would still not use this lib, the doc is too sparse for me to trust it, but it’s your call.
From the github documentation https://github.com/stackmob/newman : “Newman comes with a DSL which is inspired by Dispatch, but uses mostly english instead of symbols”. Should be good, right ? Well, maybe, but here too the project is abandoned. Last commits in 2014, and Scala 2.11 isn’t supported.
Somebody made a quick fork (https://github.com/megamsys/newman) to support Scala 2.11, but without much info available. It’s unlikely the original author wakes up and merges the fork, or that the fork gains enough traction to replace the original.
I still call it an abandonware. No point in going further.
Diagnostic : abandonware. Don’t use it.
Scala-http (https://github.com/scalaj/scalaj-http) looks clean and maintained. However it is synchronous ! No Futures there. Each HTTP request will block the thread.
In Scala it is quite idiomatic to do everything asynchronously, using synchronous code feels like a regression. It’s probably a decent library but I can’t see a use case for it. No point in going further.
Diagnostic : synchronous. Good to know this kind of stuff still exist somewhere. But if your app is anything more than a prototype, there’s really no value in using it.
A few observations there.
First, spray-client, like everything in Spray, is heavily tied to Akka. You need an actor system. If your app is not already built on Akka, it’s not ideal. Sure, you can declare an actor system just for your HTTP requests, but it feels a bit cumbersome.
Second, the API is convoluted. Take a look at the first line :
You probably thought that
sendReceive is a variable being assigned to
pipeline. Nope. The
pipeline variable is of type
sendReceive is actually a method, that builds a
SendReceive object. It would be better to add parentheses to clarify that :
But you can’t, because the method takes implicit parameters (the actor system and the execution context). Adding the parentheses would require to pass those explicitly.
OK, nevermind. What about this type,
SendReceive, what is it ? Well it’s actually a type alias for a function type :
sendReceive that you start with is actually a function that produces a function. Feeling confused ? It’s OK, it’s because it is confusing. Now I’m not saying functions that produce functions are bad inherently, it can very useful to express complex logic. What I’m saying is that if your API uses that as the building block for even just a simple HTTP call, your API is convoluted.
To be thorough, I should mention that Spray actually advocates a different writing style, based on a DSL and method composition. From the doc :
We’re back to hieroglyphs. I’m sure once you’ve mastered this DSL (which is barely documented), it will probably look very clean and functional. In the meantime, if you just want to do a simple HTTP call, and don’t want to spend a few hours deciphering spray-client’s code source to grasp the API, use something else.
Diagnostic : solid but over-engineered. Use it if you want to feel clever.
I saved the best for last. Play! is a massive Scala web framework. It provides a bunch of useful stuff, and while less modular than Spray, some stuff can be used independently. One of them is the WS API. It is oddly named (who calls HTTP APIs webservices these days ?) and not really advertised, but it’s pretty decent.
Overall, the API is easy to understand, everything is self-explanatory.
Cons : the documentation is awful for our use-case. It is exclusively targeted at people using it inside the Play! framework, even if it’s perfectly usable outside of it.
Pro : you don’t need the doc. The methods are well named, just use the auto-completion in your favorite IDE and you will find what you’re looking for quickly.
Diagnostic : good. Could really use a bit more visibility outside of the Play! world.
Today we learned a few things.
First, the Google ranking is not an accurate metric to evaluate the quality of a library :)
Second, Github should really display a big red flag over a repo when it has not been updated in over a year.
Third, that there is actually only two serious HTTP client libraries in Scala : spray-client and Play! WS.
And finally : that you probably want the latter.