Comments on: Open government and parsable data formats

By: Adam Kennedy

Adam Kennedy — Mon, 02 Nov 2009 02:43:09 +0000

After talking to a number of government agencies at the GovHack event over the weekend, they were quite positive about the idea of producing a tool that could be pointed at a geo data MID/MIF file and validate as “export grade”, making sure it has a default project, appropriate datum, ST_IsValid-passing polygons, and so on and so forth.

I’m hoping to produce it (or something like it) shortly.

By: Skud

Skud — Sun, 01 Nov 2009 21:23:59 +0000

Yeah, we mostly use JSON at Freebase.com and although I was skeptical at first (probably a hangover from early Javascript trauma) I am starting to really love working with it. XML does have some benefits though, like being able to specify that certain fields are required, or whatever.

By: J Chris Anderson

J Chris Anderson — Sun, 01 Nov 2009 20:56:26 +0000

I know we can’t ask old govt data to get future-perfect right away, but for greenfield projects, maybe it makes sense to sell JSON as the simplest option. JSON definitely simplifies the parse-and-go workflow compared to XML, and it isn’t subject to the frame issues of CSV.

By: Skud

Skud — Sun, 01 Nov 2009 20:42:39 +0000

Sure, but any kind of CSV format, once inspected, is still easier to write a parser for than PDF. And, arguably, better than XML for most applications. Perl’s Text::CSV module, which comes with the distribution by default, handles all the issues you describe above except those that are Excel’s fault. And quoting fields that aren’t really numeric (eg. zipcodes, phone numbers) will fix most of those.

It’s not that you can’t make some godawful messes with CSV, but they’re *still* more parsable than PDF.

By: Philip Newton

Philip Newton — Sun, 01 Nov 2009 19:53:49 +0000

CSV is a great format for open data [...]. It’s [...] easy to parse [...].

Um. Only if you restrict yourself to a subset of CSV, especially if you expect anyone to import the stuff into Excel at some point.

Excel is a bit too smart at times, and you’ll get weird things like ZIP codes being recognised as numeric (so leading zeroes disappear), numbers being recognised as dates or vice versa, etc.

Numbers with decimal places might also cause problems for people in locales where the decimal separator is not a dot – for example, in Germany, exporting to “CSV” from Excel will result in a *semicolon*-separated file with decimal *commas*.

Some applications support line breaks inside a field (some only if the field is surrounded by quotation marks), while others mess up completely in such a situation.

Tab-separated is a bit better, since it avoids the problem of the separator being embedded in the contents (most data is less likely to contain a tab than a comma, semicolon, or quotation mark). But even that will fall prey to the heuristics Excel and friends use to determine data type based on content.