Latest effort to revamp the blog. I’ll use this to share news about go-swagger and such I think.
I’m starting the process of rescueing whatever is possible from my previous blog at http://flanders.co.nz. But that will take a little while.
Just wanted to see if I could publish to this site really.
Last week we released Scalatra 2.2. This is our biggest release so far and introduces a bunch of exciting features like commands for handling input, atmosphere for websockets and comet. It has a much deeper swagger integration now and that API has been completely upgraded. In short this scalatra version fixes most of the big problems we were aware off. Probably one of the nastiest of those problems was the fact that we were using thread-locals to store the request and response, when you then use a future or something the request is no longer available. Let’s walk through some of these changes.
In a previous version we had migrated all our internal state to either the servlet context attributes or the request attributes, depending on their scope. In this release we made everything that accesses the request or response take them as implicit parameters. For people overriding our methods this is a breaking change, but easily fixed by adding the parameters to your method override. We also added an AsyncResult construct whose only purpose is to help you not close over thread-locals.
So what is exactly the problem?
def skip = params.getAsOrElse("skip", 0)
get("/things/:id") {
Future {
params("id") // throws NPE because request is not available when the future executes
}
}
post("/things") {
myThingsActor ? Post(parsedBody.extract[Things]) map { things =>
if (things.isEmpty) status = 404 // throws NPE because response is not available when the future executes
()
}
}
// assuming scentry is mixed in and user is something stored on the request or in cookies or something
get("/stuff/:id") {
val stuff: Future[Stuff] = getStuff(params("id"))
// everything is still fine
stuff map { allTheThings =>
getTrinketsForUser(allTheThings, user, skip) // throws NPE because request is not available when the future executes
}
}
And since this is something we absolutely had to fix, we had to introduce some breaking changes but they really were for the better. Currently there are 2 ways to get around it: bring request/response into your action in implicit vals or use the AsyncResult trait to do this for you.
Let’s rewrite the broken examples in terms of the first work around:
def skip(implicit request: HttpServletRequest) = params.getAsOrElse("skip", 0)
get("/things/:id") {
implicit val request = this.request
Future {
params("id") // no more NPE
}
}
post("/things") {
implicit val response = this.response
myThingsActor ? Post(parsedBody.extract[Things]) map { things =>
if (things.isEmpty) status = 404 // no more NPE
()
}
}
// assuming scentry is mixed in and user is something stored on the request or in cookies or something
get("/stuff/:id") {
implicit val request = this.request
implicit val response = this.response
val stuff: Future[Stuff] = getStuff(params("id"))
stuff map { allTheThings =>
getTrinketsForUser(allTheThings, user, skip) // no more NPE
}
}
With the AsyncResult you get another chance to add some default context to your async operations but other than that it works very similar.
def skip(implicit request: HttpServletRequest) = params.getAsOrElse("skip", 0)
get("/things/:id") {
new AsyncResult { val is =
Future {
params("id") // no more NPE
}
}
}
post("/things") {
new AsyncResult { val is =
myThingsActor ? Post(parsedBody.extract[Things]) map { things =>
if (things.isEmpty) status = 404 // no more NPE
()
}
}
}
// assuming scentry is mixed in and user is something stored on the request or in cookies or something
get("/stuff/:id") {
new AsyncResult { val is = {
val stuff: Future[Stuff] = getStuff(params("id"))
stuff map { allTheThings =>
getTrinketsForUser(allTheThings, user, skip) // no more NPE
}
} }
}
The AsyncResult has an implicit parameter of ScalatraContext and every ScalatraBase has an implicit conversion to a ScalatraContext so the request and response are now stable values and no longer stuck in thread-locals.
With that bug out of the way, and you’re a swagger user then the next examples are for you.
In the previous version of scalatra we introduced swagger support. While the API we introduced then worked it ended up being very messy and was error prone since most of it used strings. At wordnik we started using scalatra and one of my co-workers, who just started learning scalatra, remarked: Swagger makes Scalatra ugly. Clearly something had to be done about this! This release tries to fix some of that by using as much information from the context as it possibly can and defining a fluent api for describing swagger operations.
There are no more strings except for things that are notes, descriptions, names etc. It integrates with scalatra’s commands so you only define the parameters for a request once. It automatically registers models when you provide them and it converts the scalatra route matcher to a swagger path string. Let’s take a look at a before and after:
This is how it used to be:
// declare the models, and the models it uses
// case class Pet(id: Long, category: Category, name: String, urls: List[String], tags: List[Tag], status: String)
models = Map("Pet" -> classOf[Pet], "Category" -> classOf[Category], "Tag" -> classOf[Tag])
// declare the route with the swagger annotations
get("/findByStatus",
summary("Finds Pets by status"),
nickname("findPetsByStatus"),
responseClass("List[Pet]"),
endpoint("findByStatus"),
notes("Multiple status values can be provided with comma separated strings"),
parameters(
Parameter("status",
"Status values that need to be considered for filter",
DataType.String,
paramType = ParamType.Query,
defaultValue = Some("available"),
allowableValues = AllowableValues("available", "pending", "sold")))) {
data.findPetsByStatus(params("status")) // this is our actual implementaton, you might have missed it.
}
This is what it is now:
// declare the swagger operation description
val findByStatus =
(apiOperation[List[Pet]]("findPetsByStatus")
summary "Finds Pets by status"
notes "Multiple status values can be provided with comma separated strings"
parameter (queryParam[String]("status").required // required is the default value so not strictly necessary
description "Status values that need to be considered for filter"
defaultValue "available"
allowableValues ("available", "pending", "sold")))
// declare the route with the swagger annotation
get("/findByStatus", operation(findByStatus)) {
data.findPetsByStatus(params("status"))
}
So there is no more endpoint declaration necessary, you work with actual types and you don’t have to remember to register models and all their referenced models anymore. In my opinion if you simply write the swagger declaration close to where your route lives you still have the docs live with the code but it won’t obscure the application code anymore.
Let me know what you think.
It seems to me that every webframework in Scala, of which we have plenty, also insists on writing their own JSON library. But various tools rely on the AST from lift-json. This is both good and bad, there were a number of gaps in the lift-json version. The most notable being that it does not support any other type than Double for representing decimal numbers. A second problem is that because of the large number of dependencies in the lift project it typically is a bottleneck for upgrading to scala-2.10. And thirdly it just seems an odd place for a nice library like that to live. I’m hopeful that more libraries like Play, Spray etc will contribute to this project so that instead of a fragmented json landscape we get a homogeneous one. All of them have a json ast with the same types defined in it.
So I’ve set out to set lift-json free from the lift project and tried to add some improvements along the way.
The first improvement I’ve made is that you now have the choice between using BigDecimal or Double for representing decimal numbers, so you can use the library also to represent invoices etc.
The second change I made is to add several backends for parsing. The original lift-json parser is still available in the native package but you can now also use jackson as a parsing backend to the json library.
It’s really easy to add more backends so smarterjson, spray-json etc are all in the cards.
I took a look at what play2 has in their json support and it looks like their main thing is a type class based reader/writer story instead of the formats based one from lift-json, so for good measure I also added that system to this library. In general I like typeclasses for this type of stuff but in this case I actually think that lift-json has a nicer approach by assembling all the context into a single formats object instead of requiring many type classes to be in scope at any given time when you want to parse or write json.
There are a few more convenience methods added on the JValue type that allow you to camelize or underscore keys, remove the nulls etc.
I spoke with Joni Freeman about the general idea of this library and he showed me what he did on a branch for lift-json 3.0 so I incorporated his work into json4s too. it basically means that a key/value pair used to represent a JObject is no longer a valid json AST node and there are some extra methods to work with those fields. All of this is explained in the README of the json4s project.
I’ve also added support for json4s to the dispatch reboot project so you can use it there just like you can with lift-json. Furthermore Rose Toomey let me know that salat is now using json4s as json library instead of lift-json.
Some of the improvements I still want to make is for scala 2.10 I want to use the Mirror api to be able to reflect over more stuff than just case classes. For some of the use cases we have at where I work it makes sense to be able to use a few annotations (yes annotations unfortunately) to have certain keys be ignored and so on. I’ll probably steal some of that from the Salat project so that there is still some degree of consistency between our libraries.
I also want to figure out how we can possibly make the AST based approach useful with huge data structures, as it’s not inconceivable to want to send 100MB or 10GB json docs over the wire. At that moment a lazy approach actually makes a lot of sense, I’m open to suggestions on how this could be achieved efficiently without breaking the AST model.
So if you’re using json in scala consider using or contributing to the json4s project.
One of the big new features for scalatra in the next release will be databinding. We chose to use a command approach which includes validation. It is built on top of scalaz 6.0.4 at the moment, as soon as scalaz7 is released we’ll probably migrate to that.
One of the tenets of scalatra is that it’s a mutable interface, the request and response of servlets are both mutable classes and we depend on those. I also wanted for the commands to be defined with vals that you can address later. The core functionality of the bindings is actually immutable but to the consumer of the library it pretends to be a mutable interface. Perhaps those properties will make people frown, perhaps not.
It provides a way to represent input forms with validation attached to the fields, maybe a code sample will be a better explanation.
import org.json4s._
import org.json4s.jackson._
import org.scalatra.databinding._
class LoginForm extends JacksonCommand {
val login: Field[String] = asString("login").notBlank
val password: Field[String] = asString("password").notBlank
}
This interface relies on a bunch of implicit conversions to be in scope. To be able to present this mutable interface it is needed to use the Field[T] label after the val declaration otherwise things won’t work as you’d expect them to work.
Then in the scalatra application you can do
import scalaz._
import Scalaz._
class LoginApp extends ScalatraServlet with JacksonJsonSupport with JacksonParsing {
post("/login") {
val cmd = command[LoginCommand]
if (cmd.isValid) {
users.login(~cmd.login.value, ~cmd.password.value)
} else {
halt(401, "invalid username/password")
}
}
}
This will work with data provided as form params in a http post, as json or as xml. Of course the example above is a simple one and there are many other propeties you can interrogate. At this moment the databinding is a bit light on documentation as in there is none. If you’d like to take it for a spin feel free to ping us in the #scalatra irc room.
At some point in the future I’d like turn a command into a monoid so that we have a map method on it, but for now I need to move onto another part of scalatra.