(originally posted at http://linkedgov.org/?p=237)

The first LinkedGov hack day (kindly arranged by Geeks of London) will be taking place over the weekend of 9th and 10th April. Headlined as "Keep Calm and Hack On!", registration is currently open.  

But hurry-- the first 30 tickets went in about 20 seconds, such is the interest! The next release of tickets will be on Friday 18th March at 12:00 so keep an eye on the hackcamp site and grab ‘em while you can.

How is this different to other hack days?  Well, LinkedGov need your help.  One of our steps to clean the mountains of public data requires us to engage people all over the public sector (those valuable, wonderful people who can tell us what a budget code means!).  To help them make sense of LinkedGov (after all, not everyone is as naturally as excited about data as we are), we need to show them fresh, compelling apps that will make their eyes light up.  (Apps are sexier than spreadsheets.  It's a sad truth!)

We'll be encouraging business or commercial uses for the data, so get your thinking caps on.  Anything you build at the LinkedGov hack camp is yours to keep, but we would be grateful if we could borrow it to help explain what we're up to.  It'll help us get better data to clean and link, which should then help you make better apps in the future.

See you there!

Hi all,

Here at LinkedGov HQ we are looking at the data to load to LinkedGov that will be useful and interesting to people. 

Please have a look at where we've begun, and feed your thoughts back to us! We'd love to incorporate your input.

Cheers,

    Hadley

The Government Data Mind Map

http://linkedgov.org/news/?p=209

This is just a quick catch-up and an invitation to join us for a techie workshop this Sunday in London.  See below for details. 

Update

We've been hard at work, getting the open government data project (now called LinkedGov) ready for everyone to pile in and help build it together. 

What we've been up to:

• Building a spec, based on the previous blog posts, of what we're trying to do and how it can be built.  (We're hoping you can help us make this strong and communicate it effectively.  Watch this space-- a strategy-and-comms workshop will be coming soon!)

• Defining the framework (the bare minimum) so that all of you can drop in your brilliance in the easiest way possible.  That has meant:

- working out how to communicate, at a very high level, what each section of LinkedGov needs to accomplish

- defining the APIs between sections

- designing the bits that are common to all the sections (where the data is stored, the queueing system that handles the "what does this code mean?" questions, etc.)

- laying the groundwork for a business case and the engagement strategy to make this successful

• Working on securing a bit of funding and setting out the structure necessary to support this open source project.

The details of all this will be up soon at http://linkedgov.org.  We've been madly typing away, getting all this ready; we would love to get it out into the community for thoughts and feedback.  And, more than anything, we would love to get started building!

Developers' workshop - Sun 5 Dec

To that end, we will be hosting a kick-off workshop for developers in London on Sunday, 5 December.  Sign-up is necessary: http://linkedgov-kickoff.eventbrite.com.  It will be a fairly techie workshop and hackday; we'll be going through the overall architecture we are proposing and starting people on coding the fun bits.  We hope you can join us there!  (We're also trying to have it webcast, to incorporate those of you who can't make it to London this weekend.)

Rest assured we'll be coming to other towns soon...  And we will be setting that strategic-and-comms focused workshop early in the new year.  More info will be coming ot you soon.

Thanks again for all your support and encouragement thus far.  We are thrilled to have the chance to build this and are keen to get started!

----------------------

Update:  The venue for the workshop has now been confirmed!  Details are on the Eventbrite page:  http://linkedgov-kickoff.eventbrite.com

The last week has been exciting, with new ideas and suggestions at every corner.  So many of you have sent comments, emails, we've had face-to-face chats and meetings... It's been amazing. Thank you for all your input.  (And keep it coming!)

The biggest question is, "How are we going to do this?"  To begin to answer that, I thought we should get a bit more into "What do we need to do?  Where are those pesky gaps in the opendata process we need to fill?"

The diagrams below are ones I've been scrawling in my notebook at least once a day for the past few weeks.  Because I like you all, I thought I'd spare you my handwriting and give it to you in a prettier format.

The opendata vision

Diagram of the government opendata vision (click to enlarge)

This diagram shows you the vision of government opendata; the way it's supposed to run.  Various parts of government (local and central, plus all the non-departmental public bodies) publish their datasets, either on their own websites or on data.gov.uk.  Developers come along and code neat apps and visualisations to do fun things with the data, and users/citizens are able to be more informed about their government's activities, more closely connected to the resources in their neighbourhoods, and able to mash up the data with activities in their own lives (or other sectors).

The problem we have in realising the opendata vision

As the previous post outlined, this process isn't exactly flowing the way it is meant to.  Differences in formatting and, most importantly, undefined codes and values mean that much of the published data can't be analysed or compared to any other data.  This frustrates us.

The diagram showing the problem (click to enlarge)

The solution we are working towards

In order to fix these problems, we have to build a few things:

1. A process or mechanism to reformat the problem datasets into something we can work with.  (RDF? I know a lot of you aren't thrilled with that.  We'll sort this out at the planning workshop.)
2. A site/module by which to crowdsource the missing metadata
3. Tools for querying, filtering and searching across all the government datasets (even the ones we haven't had to tidy) and APIs to release the data to developers
4. A web site with a simple search box, to provide basic answers to questions like "How many toilet rolls has my organisation purchased in the past year?"  (This part will be primarily aimed at government users, to encourage them to see value in the project and help us with the metadata -- but will be open to all.

The diagram now looks like this:

What this project should do for opendata (click to enlarge)

The things we need to build are in green.

And what's next?

As always, comments very welcome!  I'll be getting further into the "How do we do this?" bits in posts to come over the next few days-- I have a lot more information and structure which should make it much easier to see how we'll break down the tasks and actually build this thing. 

Also more info on the planning workshop.  Stay tuned!

Cheers, all!

This post explains a project that has been incubating for a few weeks now. It's a community effort to fill in the holes in the government opendata process, and to make an interface to search across all government datasets.  This project is not for profit, is for the common good-- and we need your help.

The history

Fresh from my last project at the Department of Health, I found myself chatting over coffee to frustrated friends in the civil service who want to help with the opendata activities.  "We keep getting questions about what various codes mean in COINS, for example," they said.  "We have the answers, and we want to help, but they aren't always asking the right people. We need need an easy way to get them the information -- and to do it once, for everyone." 

I sympathised with them, listened carefully, and made a mental note: there's a gap between the government opendata vision and the reality.  The datasets are often released full of unintelligible codes, information that the developers outside government (building apps and visualisations) would love to have.  This makes sense to me: I've seen budget codes, cost centre codes, programme codes in my various government roles...  I can imagine that analysis would be complicated if you didn't have a legend or translations for them.

I've also seen datasets that need explanations. "We undercounted people using this service last year. We figured out why and we fixed it, but some of the numbers, historically, will be misleading. We need to make that clear to anyone using the data."  Hmm... so the people in government who work with this data know valuable information about it.  Metadata, if you will.  Interesting quandary.

Moving on, I then found myself having coffee with developers who have been valiantly wrangling with the government datasets. (I get to drink a lot of coffee when I'm catching up, between projects. It's quite fun.)  "We are having a horrible time finding out what each code means," they said. "We think other developers are having this problem too, and it would be great if we could join up efforts."

Now you're talking.  I'm starting to see a solution here -- and I think we, as a community, can help.

Crowdsourcing metadata

The first thing we need is a tool to crowdsource metadata about government data. This should allow those who know something about the data (civil servants, local government officers, etc.) to easily mark it in such a way that everyone can see and use their knowledge. 

Essentially, we will be adding to the datasets as they come out of government, so that everyone who wants to use them will have better data to work with.

Looking across all of government

Here's the project's next leap in logic:  If we will make this metadata crowdsourcing possible for all government datasets (why not dream big, right?), shouldn't we be able to analyse them all together?  I'd love to know how my street's rubbish compares to national averages, or how an NHS trust's spend on cleaning correllates to their MRSA infection rates. The fun is in comparing across datasets.

(These datasets should include: spending data, geographic data, headcounts, street-level crime, house prices, etc. You name it, it should be in here.)

So...  we really should make all datasets comparable. It would be fantastic if they will all play nicely with each other.  (RDF anyone?)  Looks like we'll have some work to do there, as well.

How will all this be used?

We'll build an API for apps and visualisations.  (This is both in the spirit of openness, and as an incentive to people who want to use it.  If you need tidier data, come help us make it happen!)

We will also make a web interface: a very simple search that will let you ask basic questions of the data.  "How many children are in school in Kirklees?" or "How much money does the entire government spend on toilet rolls?"  This site should be an everyday tool for grannies, researchers, school children, public servants... a very diverse group of people.  (This presents us with a pretty sizeable UI challenge, which should be fun.)

Who is involved?

Pretty much everyone you can think of.  As a community, open-source project, this should be the brainchild of a lot of people, all of whom have answers for some small part of the puzzle. (My job, thus far, is as organiser.  I'm happy to pull everyone together and create a structure for this to happen, but I'm no more important than anyone else here!)

For this to work, we need quite a lot of people to

• help design and build the pieces (developers, programmers, data structure experts, RDF enthusiasts, UX designers)
• make sure it fits with the way government handles data, and the ways they think about exceptions and comments on the data (civil servants, local government officers, maybe suppliers)
• help the community around this project (developers, users, government, volunteers) stay engaged and excited (community managers, open-source project managers, potential users)
• raise the profile of this work across the country: with government, schools, universities, the volunteer/community organisation sector, the media, private sector companies, etc. (comms and marketing professionals, journalists, civil servants, local government officers)
• keep the project plugged into and fitting the needs of the rest of the opendata/developer community (anyone involved in existing projects, programmes, and foundations)
• keep an eye on the wider, democratic issues around publishing government data.  Are we making things better? How can we use this work to improve government? (policy thinkers, democracy enthusiasts, community builders and leaders)

There are probably others whose help we could use too... If you're not on this list and would like to join us, please let me know (in the comments or by email). Whatever your idea, we probably need you!

How can I help?

You can help this project in many ways! 

1. Right now, it would be great if you could spread the word about this work.  We need as many people as possible to be aware of it.  We are gathering ideas; the more the merrier.
2. We will have a day-long kick-off workshop soon to hash out the details of the project.  Please comment on this post to register your interest in coming along, or talk to me (@hadleybeeman). 
3. Also, this project desperately needs a name.  Thoughts? Suggestions?  Please put name suggestions in the comments, and we'll all vote.

Thank you!!