27 Apr 2020

Covid Api

4 minutes

Hello everybody, right now the world is dealing with the COVID-19, hope that when someone is reading this in the future it’s remembered just as a bad time long gone. :)

The inspiration for this project came from a user in the Sciloj Covid Hackaton, someone said that would be nice to have a data source in JSOn or/and easy to query for the Data Science analysis. The first idea was to use the coronadatascrapper data, but while I was finishing the first version of this project I saw that a lot of the data format changed… I probably wouldn’t be able to keep with all the updates, so I decided to move for a more “stable” source. The one from the John Hopkins University, specifically their daily reports, data there is almost synced with their map project visualizator.

Tools used were really basic, Pedestal with Ring endpoints, Components, PostgreSQL and HikariCP, data.csv to parse the data to maps.

The flow is getting the date from the last update in the database, or the base date if the last update is nill, create a vector of dates until today, using the slurp function to get the raw data from every github page, parsing the csv to clojure maps, find the correct keys of every csv column, and adding it to the database. Find the correct key was one of the most time consuming issue, because I missed that they also changed the name of some csv columns, as you can see bellow…

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


(defn- find-correct-key [k-name]
  (let [k (cstr/trim (str k-name))
        tested (case k
                 "Province/State" :province_state
                 "Province_State" :province_state
                 "Country/Region" :country_region
                 "Country_Region" :country_region
                 "Last Update" :last_update
                 "Last_Update" :last_update
                 "Lat" :lat
                 "Long_" :long
                 "Confirmed" :confirmed
                 "Deaths" :deaths
                 "Recovered" :recovered
                 "Active" :active
                 "FIPS" :fips
                 "Admin2" :admin2
                 "Tested" :tested
                 (keyword k))]
    (if (nil? tested)
      (prn (str "PARSED INTO NIL" k)))
    [tested]))

And also the date of the last_update field changed, first I tried to do a normal java time parse, on error I checked if the date has a ‘-’ or ‘/’ in the string to parse the date.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22


(defn- mapper-to-date [val-to-date k]
  (if (= :last_update k)
    (do
      (try
        (do
          (jt/to-sql-date (jt/local-date-time val-to-date)))
        (catch Exception e
          (if (cstr/includes? val-to-date "/")
            (let [splitted (cstr/split val-to-date #"/")
                  day (Integer/parseInt (second splitted))
                  month (Integer/parseInt (first splitted))
                  un-year (first (cstr/split (last splitted) #" "))
                  year (Integer/parseInt (if (= 4 (count un-year))
                                           un-year
                                           (str "20" un-year)))]
                (jt/to-sql-date (jt/local-date year month day)))
            (let [splitted (cstr/split val-to-date #"-")
                  day (Integer/parseInt (first (cstr/split (last splitted) #" ")))
                  month (Integer/parseInt (second splitted))
                  year (Integer/parseInt (first splitted))]
                (jt/to-sql-date (jt/local-date year month day)))))))
    val-to-date))

I also saw that sometimes they would update some old data in Github, due to some user complain about a wrong format or missing data, so I needed to add a ‘retroactive’ route to delete every data in the system until a specified date, to them get all the data again to add it to the database. Since I would keep an open api in my little droplet I decided to add a super insecure password system to avoid people testing it.

1
2
3
4
5
6
7


(defn check-if-pwd-valid [pwd]
  (let [correct-pwd (System/getenv "COVID_PWD")]
    (if (nil? correct-pwd)
      "No password (COVID-PWD) on the environment."
      (if (= correct-pwd pwd)
        true
        "Password don't match."))))

Only check for a registered field in the environment, the password is passed as text in a path parameter, super insecure, but okay for the needs of this project. Well I guess those are the most interesting parts of this project, feel free to check the Github, and message me in Twitter if you need anything.

Be safe and well.