Have you ever got bored writing Kotlin classes which serialise a JSON payload?
Me too. Fortunately, we're going to write a Kotlin library that automates the whole process, using Square's awesome KotlinPoet.
We're also going to create a command-line tool and Spring Boot application, then deploy both to production.
We'll start by generating an empty Kotlin project with a module named core.
This module will contain all the conversion code, so we should define an API that encapsulates most of the gory details. Firstly, we'll need a method that converts a JSON InputStream to an OutputStream:
fun convert(input: InputStream, output: OutputStream, args: ConversionArgs)
Several JSON serialisation libraries require annotations like @SerializedName. We'll provide callbacks via a delegate for each time a class or property is added, which will allow us to tack on annotations after the source has been generated:
class Kotlin2JsonConverter(
private val buildDelegate: SourceBuildDelegate = GsonBuildDelegate()
)
For the Spring Boot and command-line applications, we'll create separate modules which both depend on the core module. More on that later.
JSON is a tree, where each array and object node could contain child nodes.
Our algorithm will start at the bottom of the tree, and work its way up to the top, one level at a time. For each level, we should group similar objects together, and build a Kotlin type representation for each grouping.
That all sounds very abstract. Consider the following JSON:
{
"obj": {
"foo": "a string value"
},
"another": {
"foo": "another string value"
},
"primitive": true
}
We can clearly see that both objects near the bottom of the tree have a key of foo, and a type of String:
{
"foo": "a string value"
},
{
"foo": "another string value"
}
Therefore we'll group them together, and represent them with a single Kotlin type:
data class Obj(val foo: String)
When we go up a level, we can immediately see that there are two fields of type Obj, and one primitive field of type Boolean:
{
"obj": Obj,
"another": Obj,
"primitive": Boolean
}
And as we've reached the root node, we've finished converting our JSON to Kotlin, and can write the results to our OutputStream:
data class Example(val obj: Obj, val another: Obj, val primitive: Boolean)
data class Obj(val foo: String)
Of course, it's not always as easy as our example. JSON is deceptively complex, and there are a surprising number of scenarios which we'll need to address. For example, here are just a few to think about:
- What happens if a field is null or omitted from one of the objects?
- How should JSON keys be converted into valid Kotlin identifiers?
- What happens if two objects have matching fields, but use different types?
- How should objects be grouped, if they share around 50% of the same keys?
Sounds hard! We'd better write some tests.
We want to verify that given a JSON input, the correct Kotlin output is generated. As we're going to check many different JSON structures, this is a prime candidate for a JUnit parameterised test.
Here's a simplified version of our test, which parameterises two filenames:
@RunWith(Parameterized::class)
class JsonConverterTest(val expectedFilename: String,
val jsonFilename: String) {
companion object {
@JvmStatic
@Parameterized.Parameters
fun filenamePairs(): Collection<Array<String>> {
return listOf(arrayOf("HelloWorld.kt", "hello_world.json"))
}
}
}
JUnit will run the test once for each parameter pair. Therefore, all we need to do is compare the generated source code against the expected source code, and supply filenames:
@Test
fun testJsonToKotlinConversion() {
val outputStream = ByteArrayOutputStream()
jsonConverter.convert(json, outputStream, ConversionArgs())
val generatedSource = String(outputStream.toByteArray())
val expectedContents = fileReader.readContents(expectedFilename)
Assert.assertEquals(msg, expectedContents)
}
And then our test case will verify that a JSON input matches an output similar to this:
{"foo":"Hello World!"} // "Example.json"
data class Example(val foo: String) "Example.kt"
This assumes that a JSON and Kotlin file are present in src/test/resources, and that the file contents can be read via the ClassLoader:
val classLoader = ResourceFileReader::class.java.classLoader
val inputStream = classLoader.getResourceAsStream("HelloWorld.kt")
The full test takes this even further, by recursively detecting JSON/Kotlin files within the resources directory. This means we can just add a pair of JSON and Kotlin files, and we'll automatically have a test case.
The latest test suite is available on the Github repository.
Before we look at our approach in more detail, let's summarise the steps:
- Sanitise input and generate a JSON tree using GSON.
- Use breadth-first search to push each JSON node onto a Stack, along with some additional metadata.
- Traverse in reverse level order and generate type information, group common objects, then reduce common objects to a single type types.
- Pop the stack of generated type information and write it to an
OutputStream.
We'll start by implementing a breadth-first search that pushes non-primitive JSON nodes to a stack.
private fun buildStack(bfsStack: Stack<TypedJsonElement>,
parent: JsonElement,
key: String?) {
val queue = LinkedList<TypedJsonElement>()
queue.add(TypedJsonElement(parent, key!!, 0))
while (queue.isNotEmpty()) {
val element = queue.poll()
bfsStack.push(element)
val complexChildren = with(element) {
when {
isJsonObject -> convertParent(asJsonObject, level + 1)
isJsonArray -> convertParent(asJsonArray, jsonKey, level + 1)
else -> Collections.emptyList()
}
}
queue.addAll(complexChildren)
}
}
We'll then pop from the stack and process JSON nodes. For each level, we'll generate a TypeSpec for each distinct object grouping.
fun generateTypeSpecs(bfsStack: Stack<TypedJsonElement>): Stack<TypeSpec> {
val typeSpecs = Stack<TypeSpec>()
var level = -1
val levelQueue = LinkedList<TypedJsonElement>()
while (bfsStack.isNotEmpty()) {
val pop = bfsStack.pop()
if (level != -1 && pop.level != level) {
processTreeLevel(levelQueue, typeSpecs)
}
levelQueue.add(pop)
level = pop.level
}
processTreeLevel(levelQueue, typeSpecs)
return typeSpecs
}
Our grouping strategy will be very simple. For objects to belong to the same type, they must share 1/5 or more of the same keys. The only exception to this will be an empty class, which will be considered to share 1 key automatically.
// builds a list of common objects, implementation omitted
fun groupCommonJsonObjects
(jsonElements: MutableList<TypedJsonElement>): List<List<TypedJsonElement>>
internal fun defaultGroupingStrategy(lhs: TypedJsonElement,
rhs: TypedJsonElement): Boolean {
val lhsKeys = lhs.asJsonObject.keySet()
val rhsKeys = rhs.asJsonObject.keySet()
val lhsSize = lhsKeys.size
val rhsSize = rhsKeys.size
val emptyClasses = (lhsKeys.isEmpty() || rhsKeys.isEmpty())
val maxKeySize = if (lhsSize > rhsSize) lhsSize else rhsSize
val commonKeyCount = if (emptyClasses) {
1
} else {
lhsKeys.intersect(rhsKeys).size
}
// at least a fifth of keys must match
return (commonKeyCount * 5) >= maxKeySize
}
We'll also make it easy to alter the grouping strategy at a later date, by accepting a function reference as a constructor parameter:
// typealias used to make method signature more human-readable
typealias GroupingStrategy =
(lhs: TypedJsonElement, rhs: TypedJsonElement) -> Boolean
class JsonFieldGrouper(private val strategy: GroupingStrategy = ::myStrategy)
Now that our objects are grouped, we can generate type representations using KotlinPoet. We'll start by building a TypeSpec:
private fun buildClass(commonElements: List<TypedJsonElement>,
fields: Collection<String>): TypeSpec.Builder {
val identifier = commonElements.last().kotlinIdentifier
val classBuilder = TypeSpec.classBuilder(identifier.capitalize())
val constructor = FunSpec.constructorBuilder()
if (fields.isNotEmpty()) {
val fieldTypeMap = typeReducer.findDistinctTypes(fields,
commonElements,
jsonElementMap)
fields.forEach {
buildProperty(it, fieldTypeMap,
commonElements, classBuilder, constructor)
}
classBuilder.addModifiers(KModifier.DATA)
classBuilder.primaryConstructor(constructor.build())
}
delegate.prepareClass(classBuilder, commonElements.last())
return classBuilder
}
There's quite a lot going on here. First, we create a TypeSpec.Builder, which holds the information used to build a class.
We then add the data modifier, build a constructor, and for each JSON field, call the buildProperty method:
private fun buildProperty(fieldKey: String,
fieldTypeMap: Map<String, TypeName>,
commonElements: List<TypedJsonElement>,
classBuilder: TypeSpec.Builder,
constructor: FunSpec.Builder) {
val kotlinIdentifier = fieldKey.toKotlinIdentifier()
val typeName = fieldTypeMap[fieldKey]
val initializer = PropertySpec.builder(kotlinIdentifier, typeName!!)
.initializer(kotlinIdentifier)
delegate.prepareProperty(initializer, kotlinIdentifier,
fieldKey, commonElements)
classBuilder.addProperty(initializer.build())
constructor.addParameter(kotlinIdentifier, typeName)
}
We add a property for each JSON field by generating a PropertySpec.
For our case, all we need to include is the Kotlin identifier, and the property's Kotlin type, which is represented by TypeName. The typename may be a Standard Library type such as String?, but could also be a type generated earlier on, such as Foo.
Each generated TypeSpec will then be pushed to a Stack, where it will eventually be written to an OutputStream as a Kotlin source file.
Our conversion tool is working pretty nicely by this point, so we'll start writing a command-line app. This module will depend on the core module and the Apache Commons CLI, which does all the hard work of parsing arguments for us.
compile project(":core")
compile "commons-cli:commons-cli:1.4"
We'll start off by supporting help and input arguments:
private fun prepareOptions(): Options {
return with(Options()) {
addOption(Option.builder("input")
.desc("The JSON file input")
.numberOfArgs(1)
.build())
addOption(Option.builder("help")
.desc("Displays help on available commands")
.build())
}
}
Now we need to handle each argument in our main method. If the arguments were invalid or not present, then we'll print a message indicating that was the case, otherwise we'll attempt to convert the JSON to Kotlin:
try {
val cmd = parser.parse(prepareOptions(), args)
if (cmd.hasOption("help") || !cmd.hasOption("input")) {
printHelp(options)
} else {
val parsedOptionValue = cmd.getParsedOptionValue("input") as String
val inputFile = Paths.get(parsedOptionValue).toFile()
if (inputFile.exists()) {
val outputFile = findOutputFile(inputFile)
val input = inputFile.inputStream()
val output = outputFile.outputStream()
Kotlin2JsonConverter().convert(input, output, ConversionArgs())
println("Generated source available at '$outputFile'")
} else {
println("Failed to find file '$inputFile'")
}
}
} catch (e: ParseException) {
println("Failed to parse arguments: ${e.message}")
}
JVM languages run pretty much anywhere in a JAR, and Gradle has a task that simplifies the distribution process. Our build file needs a few modifications, such as specifying the location of our main class:
apply plugin: 'application'
mainClassName = "com.fractalwrench.json2kotlin.AppKt"
jar {
manifest {
attributes "Main-Class": "$mainClassName"
}
from {
configurations.compile.collect { it.isDirectory() ? it : zipTree(it) }
}
}
distributions {
main {
baseName = 'json2kotlin'
}
}
We will then run the following commands to distribute and test our application:
./gradlew assemble
unzip cmdline/build/distributions/json2kotlin.zip -d json2kotlin
./cmdline -input /c/Users/<User>/json2kotlin/bin/test.json
Our final step is to put the archive somewhere that people can download it, which in this case is GitHub.
Onto Spring Boot.
Spring Boot is a Java Framework that can be used to create web applications, and has recently announced first-class Kotlin support. It comes with sensible defaults, so we should be able to write a useful app in very few lines of code.
We'll start off by creating an empty Spring Boot project by following Pivotal's very handy guide. We'll create a class annotated with Controller, and setup a RequestMapping to the root endpoint:
@Controller
class ConversionController {
@GetMapping("/")
fun displayConversionForm(model: Model): String {
model.addAttribute("conversionForm", ConversionForm())
model.addAttribute("kotlin", "class Example")
return "conversion"
}
}
There's a lot of magic going on here.
Behind the scenes, Spring will detect the @Controller annotation and route HTTP requests to our displayConversionForm method.
This method then adds the generated Kotlin source as an attribute of a Model. Finally it returns a view, which corresponds to a HTML template stored under src/main/resources/static/templates. This template may look familiar to anyone who has used the Android Data Binding Library before, and the principle is the same:
<html>
<head></head>
<body>
<textarea th:text="${kotlin}"></textarea>
</body>
</html>
Thymeleaf binds model attributes to the view by evaluating any expressions in the th namespace. The generated HTML is then returned as an HTTP response to the user:
<html>
<head></head>
<body>
<textarea text="class Example"></textarea>
</body>
</html>
We're getting slightly ahead of ourselves here, as the generated Kotlin can't be displayed until a user submits their JSON input. Therefore we need to display a form to the user, which will POST the JSON to our / endpoint:
<form id="jsonForm"
th:action="@{/}"
th:object="${conversionForm}"
method="post"
onsubmit="return validateForm()">
<textarea maxlength="10000"
placeholder="Paste JSON here..."
th:field="*{json}"></textarea>
<input type="reset" value="Reset"/>
<input type="submit" value="Convert"/>
</form>
You may have noticed that the HTML form binds the conversionForm attribute in our previous method, as a form object.
When a POST request is submitted, the conversionForm method parameter will contain the text entered into our <textarea> element. We can then pass the user's input into the Kotlin2JsonConverter, and our HTML response will contain dynamically generated Kotlin:
@PostMapping("/")
fun convertToKotlin(model: Model,
@ModelAttribute conversionForm: ConversionForm): String {
val os = ByteArrayOutputStream()
Kotlin2JsonConverter().convert(conversionForm.json, os, ConversionArgs())
model.addAttribute("kotlin", String(os.first.toByteArray()))
return displayConversionForm(model)
}
Of course, this isn't quite production ready - we'd certainly want to sanitise user input before attempting to convert it.
We'll skip a few steps here such as adding validation, and making the HTML prettier with CSS. If you're interested in how this functionality works, I'd encourage you to browse through the Spring module of the project.
Now that we've completed an MVP web app, we're going to deploy to AWS using the free tier, which meets the needs of most hobby projects. Some level of familiarity with AWS is assumed from here on out, but here's a quick refresher on the services we'll use:
- Elastic Beanstalk: Controls all the AWS services required to build a scalable application.
- EC2: Provides on-demand containers which run a JVM application in the cloud.
- Elastic Load Balancer: Directs traffic between EC2 instances depending on how busy they are, and scales EC2 instances depending on load.
- Route 53: Allows us to register a domain name and point the DNS at an Elastic Beanstalk application.
The first step we'll take is to update our JAR metadata, as we did for the command-line application:
jar {
baseName = 'json2kotlin'
version = '0.1.0'
manifest {
attributes 'Main-Class': 'AppKt'
}
from { configurations.compile.collect {
it.isDirectory() ? it : zipTree(it) }
}
}
We'll also want to add Spring Actuator as a dependency, as it exposes several endpoints that provide useful information for devops.
compile("org.springframework.boot:spring-boot-starter-actuator")
Finally, we'll need to update our server port in application.properties so that it can communicate with the load balancer:
server.port=8888
The next step is to register a domain name, in this case json2kotlin.co.uk, by following the AWS guide.
After the rest of setup is completed, we'll configure a hosted zone, which routes requests towards our Elastic Beanstalk environment.
We now need to setup an Elastic Beanstalk environment, which can be achieved by following this very helpful blog from AWS. ./gradlew bootRepackage will generate a JAR of our Spring Boot application.
Our application will use a load balancer, which will automatically scale up EC2 instances in the face of heavy traffic. Depending on your anticipated traffic, it's possible that you could skip this step.
If you do use a load balancer, it is vital that a health check is setup, as otherwise the load balancer will assume that all the instances are unhealthy, and all the requests will timeout. Fortunately, Spring Actuator contains a ready-made /health endpoint, so we'll configure our environment to use this.
There are probably a few bugs lurking in our application, so our next step is to add an error-detection SDK that reports any uncaught exceptions that occur in the wild. I chose Bugsnag for this task, because:
- It supports both Kotlin and JavaScript, as well as most languages under the sun
- It's free for open-source/small projects
- My JavaScript is awful and I'm anticipating a lot of browser compatibility errors
(Full disclaimer, I work for Bugsnag, so you can blame me if anything goes wrong)
The final web application is available as a command-line tool from Github. At less than 1000 lines of code, this would be a nice project to wrap your head around if you're looking to submit pull requests, issues, or just general feedback!
I hope you've enjoyed learning about Kotlin source generation, and will sleep easy at night in the knowledge that you'll never have to write data classes by hand again.
If you have any questions, feedback, or would like to suggest a topic for me to write about, please get in touch via Twitter!