11 Jan 2016
I started work on my CSV converter that runs on Github Pages back in 2013, while I was working in the federal government. I recently updated the look and feel of it, but I also wanted to simplify if, and add some features in support of my Adopta.Agency project.
I was getting complaints that the tool was too complicated, with all the settings, so I limited to purely a simple CSV to JSON conversion. Then I made another tool that will take any simple JSON file, and convert it back into a CSV file. Of course it will only work on simple JSON files, but it does the job for simple CSV to JSON, and back again.
As a programmer, I want to get everything into JSON, but I understand that there are many data stewards out there who would rather have their data in a CSV, so they can work with in Excel or Google Spreadsheets. It makes sense to be able to go back and forth, as you need to, working with the data in whichever format is more suitable.
I think maybe next, I will add a simple editor, for editing the values of any file uploaded, whether it is JSON or CSV.
12 Nov 2013
Once any data steward has converted CSV files to either XML or JSON, they now need somewhere to put these open data files. With an entirely client-side tool like CSV Converter than runs 100% on Github, the only option is to save files to Github. There is no other backend.
To be able to write files to the Github repository you have to be a collaborator on the project, and be logged in using Github oAuth. To do this I used oAuth.io, which allows me to login to CSV Converter with my Github account, but if you are not on the collaborator list for the repo you won’t be given access.
Once authenticated I use Github.js to be able to actually create, edit and delete files within the Github repository. Once authenticated I allow any converted CSV to be saved, entirely with JavaScript. I’ve chosen to put all data in the master branch, since the conversion tool runs on Github Pages with the gh-pages branch.
I was pretty happy to find out I could give data stewards a place to put their open data files using oAuth.io and Github.js, while using Github as a backend.
All of this reduces the barrier to entry for any government data steward, giving them a completely open source, free way to manage their open data assets.
12 Nov 2013
When you ask many of people in government to open up data, the chances it will come as a Microsoft Excel spreadsheet is pretty high. People just speak Microsoft Excel. It is the closest thing they know to being machine readable by default.
If you want users to be able to make their “open data” machine readable, you have to request data in a CSV format. If a user cannot learn how to export an Excel worksheet as CSV, they probably aren’t your target audience. If they do, you need to help them understand the difference between CSV, XML and JSON.
To help accomplish this process I created CSV Converter. I found the really cool open source project Mr. Data Converter, and it seemed like the perfect tool for educating users about the difference between CSV, XML and JSON. Using the entirely JavaScript tool you could paste in a CSV file and easily convert to HTML, XML and JSON, quickly seeing the differences.
Of course, to truly understand the difference you need to be able to put each of these formats to use. This is whats next, allow users to not just convert, but manage their open data in any of these formats.
12 Nov 2013
I used Mr. Data Converter to handle the CSV to XML and JSON conversion, and while it had great features for copy / pasting from Microsoft Excel or CSV files, I wanted to provide the ability for uploading a CSV directly into the conversion tool.
Since my CSV Converter would run entirely client-side I needed to provide “upload like” features, but actually grab the contents of the CSV file on upload and just insert into the CSV Converter text area.
The HTML5 File API is exactly what I needed. While this will not work in all browsers, it will in Chrome and Firefox allowing a pretty wide audience. The HTML5 File API allowed me to provide the upload feature, but grab the contents upon click of upload button without actually uploading any file to the server.
Once the CSV is inserted into the CSV Converter, the user can choose a conversion format type such as HTML, XML or JSON, and if they are logged in with Github using oAuth they can save the converted files to the master branch of the repository.
