Brazillian Inflation Calculator

Github Project

I had the idea to do this project while I was reading a chat which some friends discussing real state properties prices, one of them asked to another one to do some calculations using a R project and I thought that it would be nice to have something easier to use than this R one. Althought if you plan to do some automations in your enterprise, the R package is probably more suited to you.

I’ve been studying Clojure for a while, reading some books and guides, and following examples from those, but the real challenge usually only appears once you go on your own, so I decided to use it for the back-end. I’ve also decided to use Reagent in the front-end, while I had some practice in Angular, I was also looking for something different in the front-end.

First a little thought about Reagent and Hiccup, Hiicup is pretty easy to write and understand, my code were pretty compact, and the syntax of :element#element-id.element-class is very clean. Most of the work here was just getting some data, with minor validation and sending it to the back-end, and them just rendering it back, I also decided to use a simple chart solution, D3, it works just fine and the other ones I saw were too complex for this small project, I also had some prior experience with it. And let me give you a bad experience I had in a older project, after getting the data from the back-end, the chart re-renders, the first render is fine, but if you enable hover, like I did, after the first render while you are hovering the chart it may show you an older chart with wrong values, this wil be very confunsing to your user, and the only way to avoid this bug is to destroy and recreate the chart, before rendering the new data, it will impact performance a little bit, but at least here with a single graph the effect is minium and shouldn’t impact user performance. This is the second time I had this issue, and couldn’t find a better way to solve it. After testing and checks I build the project with leiningen, deployed the static data with rsync to the Droplet and served it with nginx, of course don’t forget to generate your certificate for https, this will be needed later since you probably want to run your back-end with CORS enabled.

Now let’s go to the project back-end. I had some requirements, the inflation data should be stored in a internal database, so I won’t need to query the external APIs all the time, and since the data in the APIs should be updated once a month, I want my system to query the database once a month to get the new data automatically.

For the database I decided to use PostgreSQL, could have used Datomic, but since I’m running this project in the cheapest Digital Ocean Droplet, were there is already have another project with PostgreSQL, I decided to keep it the same, to save some droplets resources.

For the monthly updates I added an field in the database called last_update, which should be checked every time the user do a new request, and if the current month and year is different from the last_update month and year the system will query the APIs for new data, while I’m writting this post I thought that I could have just used an atom in runtime state for avoid querying the database everytime.

For Clojure, Pedestal was chosen as the main library, there are also other options like Ring, Compojure, also a full-fledged framework like Luminus, Duct and Fulcro. I choose Pedestal because it was the one I liked the most in my tutorials and also due to Lacinia-Pedestal, to sum it up, it’s GraphQL implemented using Pedestal made by the Wallmart Team, and I found it to work pretty well, while I’m not using GraphQL here it’s good to have a good knowledge of a library that can easily implement it. For the database first I used the simple JDBC connector, there is also the choice to use SQL parsers like HugSQL and JeeSQL, but I prefer to write SQL queries, soon I moved to the pooling library HikariCP, using this Clojure wrapper, I can’t say much about the performance which were already pretty good, but it’s usually a good practice to use a pooling library, and also due to Clojure instead of updating all the old queries, just create a simple function to implement it.

1
2
3
4
5
6
7
8
9

(defn- pool-query
  ([conn query]
   (jdbc/with-db-connection [pool-conn conn]
                            (let [result (jdbc/query pool-conn query)]
                              result)))
  ([conn query args]
   (jdbc/with-db-connection [pool-conn conn]
                            (let [result (jdbc/query pool-conn [query args])]
                              result))))

So let’s go back to the chain of events, after creating the database with some SQL, the Clojure would check that there is no last_update and query the externals APIs, provided freely but some institutions here in Brazil, the data was parsed with the Chesire library, after the parse, all the data is than saved in the internal database. But wait, when the APIs are queried they return all the data every time, and I don’t want to have duplicates, so before adding the new data to the database I check if that date already is present, for a huge database this wouldn’t be ideal, checking millions of rows before adding a single new record isn’t great, but here I will only have 12 new records a year and the update will happen only once a month, so no big deal. If it was a big database I would have approached it differently, like using the last_update and inserting every new record more recent than the last_update, and after all the records are checked and inserted, update the last_update for the latest record added. I guess you could also get the MAX(inflation_date) and save it in a atom, compare if the value is smaller than the queried value and them insert it, just keep in mind to not update the atom after all the values returned in the query were checked, if you update the atom for every new insert and your queried data is not ordered from older to new you will end up missing something.

Next we need our chart generator endpoint using Pedestal, oh remember Ring, you can use just some parts of it, their response util is very nice and make your endpoints very readable.

1
2
3
4
5
6
7
8

(defn graph-generator
  [{{:keys [valor inicio fins]} :json-params
    {:keys [storage]}           :components}]
  (ring/content-type
    (ring/response
      (cs/generate-string
        (deflate/generate-graph valor inicio fins (:database storage))))
    "application/json"))

Below was the version without Ring, was also the first iteration, without Component, I’ll talk about it later.

1
2
3
4
5
6
7
8
9

(defn graph-generator
  [request]
    (let [json-params  (:json-params request)
         valor  (:valor json-params)
         inicio  (:inicio json-params)
         fins  (:fins json-params)
         graph-table (deflate/generate-graph valor inicio fins)]
    {:status 200
     :body   graph-table}))

So we take some json from the front-end parse it and do the operations using those values, the job is pretty much get the inflation values by dates, which were sent by the client and calculate the inflation percentage by different indexes and return all those indexes with values by date to generate the table and the chart to the front-end. DONE!

Well kinda, when I made this project I decided to do it “normal” calling namespaces directly, but after I decided to move it to use Components! As I said I had done some books tutorials, some which already had components, and I remember reading that if you don’t start a project with components, you won’t want to add components after it’s done, well, I find it to be an excellent tip! Mount is another famous choice that you may find interesting, that is easier to adapt to your existing project.

This project is pretty small, I had 5 namespaces before using Components, it worked fine, but I still wanted to use it, the main issue using components is that you must pass it to most of your functions, breaking you old arity, since there is no more state in your namespace. The graph-generator endpoint I posted above is a good example, the version without component has no :database passed to the generate-graph function. The component version has it, and since generate-graph calls other functions wich need to access the database I needed to rewrite most of the functions ariety and parameters passed, and test everything again, as I said, this is a small project, but a huge service with hundreds of functions would be a pain to adapt.

But after moving to components I started to apreciate it. Your system now needs a Datomic database besides the SQL you were using? Add a datomic component. Your system needs to run with different configs on production ? Just pass a different map to your pedestal component. Your system needs to run different routes on development ? Just have two routes options, check if the component :config is :prod or :dev and use the right routes. No more risk of deploying without CORS.

But it took a little while to understand it well, and while there are some tutorials, even a good one in the official docs, I find them to be pretty simple, if you are interested I recommend to check this Nubank’s microservices example, they use a lot of components, take your time to navigate the code.

And finally the last library that I used and want to recommend is Tufte, it’s a profiler/monitor that can easily wrap existing functions, used it on development to check for the performance of my functions, and see if I could improve something, but the most interesting function is the one were you could send your logs in another channel using core.async or go blocks that could them send it to another tool or database for some data analytics, if you have at least a half-decent server.

Finally it’s deploy time, lein uberjar, done. Of course I could use Docker, but again, cheapest Droplet, running other projects, Nginx and so on.

If you want to use Docker there are two main choices that I’m aware, running a Java or Clojure image, the Java image will use your .jar that you should had already compiled, and with the Clojure one you can build it directly in the container, while the Java one require less memory and has faster startup the Clojure makes it simple to compile and run. So maybe just create a bash file that will compile you Clojure project and call your docker after it:

1
2
3
4
5

#!/bin/sh
#Let's lein uberjar the project and call dockerfile
lein uberjar
docker build -t project-name .
docker run ...

Well to deploy it, I’m using only rsync to a specific folder in the Droplet, I’m running Supervisor to keep it running, but it doesn’t update the file automatically after I deploy on server, so I still need to access it and restart Supervisor. I’m not serving it directly to the web, I have Nginx in front, the back-end has CORS enabled to only allow the front-end to access it

This is pretty much it, hope you learned something interesting for your next project.