From VisibleGovernment

Introduction

Developing open data and open government

Forward

This document was crowdsourced — open to the public for discussion and editing prior to submission. It can be found at http://visiblegovernment.ca/w/Ethics_Standing_Committee

About Visible Government: VisibleGovernment.ca is a Canadian non-profit that promotes online tools for government transparency. One of our main focuses is providing relevant information and resources to support the healthy development of Open Data and Open Government projects and culture. Our objectives include finding ways to encourage more diverse participation, adding information about existing projects and creating easy to follow information around standards, language and accessibility.

About the presenter: Now serving as Executive Director of Visible Government, David H. Mason helped start Canada's first consumer Internet Service Provider in 1993. He participated as Lead Developer of the world's first online consultation on foreign policy, and in a rigorous, people focused hospital environment developed a prototypical Virtual Clinic, which provided online access to lab results, analysis, peer and patient-provider communications. David studied Community Economic Development and now works in science, innovation and human rights projects, with a particular focus on structured open knowledge transfer using wikis.

Open data offers many advantages within and outside the government. This document is about open data, open government, and growing a more representative democracy.

Within the government

Open data lets us integrate information from many sources and organize it in more share-able ways.

But this takes more than putting existing documents on the web. They have to be findable and useful to persons beyond the immediate experts who create them or specialize in their subject.

The United States government went from 47 datasets online in May 2009 to over 300 thousand as of early 2011. The US approach runs under the motto "transparency, participation, collaboration." The UK efforts, under Tim Berners-Lee's guidance (developer of the Web and semantic systems) emphasizes well-structured and organized information. The huge volumes of government data and the many views that can be taken of it necessitate easy and flexible naming: tagging, schemas, taxonomies and ontologies. Open data with supporting methods and culture makes it easier to organize vast databases.

A related risk is that proprietary software (which usually accompanies closed data) has historically had security and capability problems that go unresolved for longer than open source or free software alternatives. This is due in part to "lock-in" when vendors believe government has no choice but to continue using its software or proprietary data formats. Vendor choice gives government more leverage to resolve such problems.

Supporting many views of the same information lets government support more communication between agencies, and between agencies and citizens. This leads to better sharing, fewer disconnected silos, more efficiency and effectiveness. It supports a critical mass of individuals that can scrutinize, organizing and linking data policies competently, enabling very many people to have a new interest in openness.

Internal government culture change may be required to encourage sharing. Already, GCPedia, a cross department open communication tool, has been called a "key medium". The government geomatics community has been learning about publishing data under open licenses for over a decade.

The United States began compiling intelligence and diplomatic reports using the Mediawiki software in the Bush Administration (Intellipedia and Diplopedia). Wikileaks doesn't seem to have gained access to this material. Instead, they've published classified material stored in an old-fashioned "silo" database — providing a caution that organizations that have legacy tools and attitudes often cannot adapt or easily carry systems forward, compared to systems designed for full integration within social systems.

Similarly, hospital health care records often contain comments and jargon that means they can never be released to the individual. Modern health care records will not be able to include a patient's history. In the United States, these records may never become available because they would become exposed to massive automated lawsuits.

These issues are all because of a closed attitude towards data. Is this a decent legacy to create, one that poisons the future?

GCPedia in contrast instilled the idea that one day the system will be open, leading to more thoughtfulness in what's published. Because it uses the same software as Wikipedia, GCPedia can constantly benefit from the wisdom, recognition and evolving functionality of one of the world's most tremendous free, open, reusable innovations.

Consider the effects of a generation of global mainstream Internet. The world is online, talking, dreaming, creating and connecting. Developments we thought astounding ten years ago continue to evolve. Government will serve itself, and all members of the public by quickly adapting to open data and government approaches.

Science, education and business

Scientists need more access to better organized data, not just published papers but the data of failed experiments, with more ability to easily compare formats of raw data and intermediate results. Scientific experiments also cannot be easily reproduced without more standard apparatus or control group descriptions. Open dialogue between scientific and other technical disciplines require at least agreement that government-sponsored research will follow such standards to speed research. In his article, "Wikiscience", Kevin Kelly, editor of Wired, outlines many more such opportunities.

In the 2010 Canadian National Consultation on Access to Scientific Research Data, in addition to the benefits of enabling a Knowledge Society, the following major benefits were outlined:

• Efficient research (eg barrier-free research; less re-invention, redundancy, red tape)
• Network effects (eg the effect that one user of a good or service has on the value of that product to other people)
• New discoveries (eg. new data out of old data)
• Data stewardship practices
• Capacity building for data archiving
• Removal of discipline boundaries (eg researcher as citizen)
• New methodologies (eg faster science, publishing negative results)
• International leadership in data management (eg accepted Canadian standards; international teams)
• Enhanced education (eg full discovery process as a learning tool)

These benefits apply generally to open data.

In education, institutions like MIT lead the way in making their course material available online for free around the world. This supports a culture of constant learning that's available to anyone who is interested. It also guarantees that brilliant students abroad will think of attending MIT before they think of any Canadian institution - in part because they can contribute to these works.

Corporations who adopt government-sponsored data standards will soon find them equally useful in their own operations, to streamline supply chains and even to open up their own operations — openness as a competitive advantage. At least one Canadian gold mining company released some of its assay data to prospectors around the world, and in doing so improved its yield for those properties beyond any reasonable expectation.

Opening data lets institutions and business focus on their real goals, focused on the highest integrity meshing with every person's goals.

Culture of problem solving

Today most information is still not open and contains no external links. This despite the fact that every new desktop computer is approaching a teraflop of processing power, which was considered a "supercomputer" a generation ago. We are also raising generations of digitally literate people: People who have mastered the Internet as pre-teenagers, or whose day job it is to organize vast pools of information, link and publish it. Some come home and solve immense problems using widely distributed "volunteer computing" networks. With access to the right data, they can process and publish it and open it to additional perspective and insight from every perspective.

Today, thanks to fragments of open data and volunteer or supported projects, people can more easily find a skating rink, better use public transportation, know where crime is taking place, where government or charity spends their money, and make better purchasing decisions. They will naturally demand much more in the future.

A recent project of mine had huge potential to improve health care and reduce its costs. Using data available online from Montreal-area hospitals, I built an application that lets anyone search for the nearest emergency clinics and see the wait times of each. Thus, people needing medical attention can immediately find the closest clinics and average wait times. It's easy to imagine how this application could be augmented to focus on specialties of hospitals and clinics and include real-time views, to help someone determine what facility to go to in an emergency situation, ensuring no lineups build at one while staff are overloaded at another — a very inexpensive way to reduce the average wait times, which is today a massive, unnecessary and potentially dangerous waste of many people's time.

This consolidated view doesn't exist officially today, and probably won't exist officially anytime soon. But with free access to the data required, the project was completed in one afternoon. There is so much that could be built on an application like this, that will benefit patients, providers and the system, yet today we're stalled by a serious lack of imagination and will. A lack in this case that leads to thousands of people wasting their time and suffering unduly daily.

But it's much more than just application-building. Half of Canada is on Facebook. Some uses are silly, but serious debates also occur, and sometimes life-critical guidance is shared. Wikipedia teaches us that we can find answers quickly, but also that we often must dig deeper for the full story and check original sources. Newspapers and media become richer, better linked, and more inclusive of readers. As we get past simple discussion systems, into rich networked fact-based debating, we stimulate natural curiosity and desire to participate. As Wikipedia proves, even a small percentage actively editing information under scrutiny from many others has a transformative effect that is hard to underestimate.

Benefits

Beyond obvious economic benefits from new enterprises like Google and Facebook, Wikitravel and RIM, we all gain from a society where people can expect real answers to hard questions. Not simplistic answers, but serious inquiries citizens can take part in understanding and solving, broadly improving quality of life or challenging assumptions that have been held as dogma but which fail on examination in the present.

We gain by better appreciating the complexities of our systems, with more faith that officials who participate are trying to solve their problems. We can take more care when purchasing or donating. Things become more real and engaged. Instead of pre-digested reports, an eleven year old writing a school report, or an eighty year old nursing home resident, can constantly learn to understand and take part in rigorous, fascinating, progressive layers of community, government, and researcher developed information. Moderated by world-scale practices, and emulating, in easy to use systems, rigorous methods similar to the development of law and science.

The recent "Usage Based Billing" controversy has had over 400,000 responses for one position. Compared to traditional petitions, they're largely backed by easily reachable individuals in a rich, verifiable network.

Individuals will organize information and create new applications more easily than using a typewriter. Basic reports will soon write themselves on a few keywords, linking rich media with the interested people who challenge, annotate or improve situations. Automated inference will illustrate hidden problems — why exactly does this hospital get better funding than this one? Before this decade is over, not exposing and addressing such problems will seem suspicious. Some careers may be lost to that suspicion, or discoveries of actual corruption. But many more will be made developing sensitive, creative self-correcting feedback models that radically improve all institutions.

Open data steps

Tim Berners-Lee's five levels of re-usable open data:

1. simply making your data available on the web with an open license, about equivalent to a fax and other nearly non reusable information;
2. make it available as structured data, where data can be re-used with the right software;
3. release it in non-proprietary formats;
4. map it to [persistent public] web locations so it can be reliably re-used, and;
5. rich linking between data sources.

All open data must be available under a suitable license that favours easy re-use.

Most projects today do only the first two, but there's a streamlined path to linked data people can easily learn. These can be encouraged first by those in government, then socially by each other in interest groups, each adding connections and depth and filtering for their own purposes. We will need to better understand our institutions and the way they reflect (or suppress) representative social systems. We must avoid pretending that we can address each other as a mass, “official” to “public." We must avoid creating additional crusts, technocracies or one-speaker broadcast media. We must transfer and enable more responsibility to individuals and corporations in responsive networks to meet these standards.

Ultimately, government can offer efficient services, supporting ethical norms through a connecting fabric instead of bulky compartmentalized institutions, moderating and shaping decisions according to the developing will of constituents.

Licensing

The important question of copyright status is absolutely central to all open data projects. Most such projects employ the Creative Commons licenses, which can be recognized at a glance, are legally robust and designed by a team of legal scholars to encourage maximum re-use of material and clear rules for its attribution and improvement. Using these, a government can require merely attribution (the CC-by license), or that other users share all improvements (the CC-by-sa license) or that commercial use be paid for or explicitly authorized in advance (the CC-by-nc-sa license). All three are considered "open content" licenses as are the "free software" licenses which also operate on share-alike principles. In general, as a powerful public force, government projects should systematically favour Creative Commons for content, and AGPL, GPL or LGPL for software.

A LEGO-like problem

The pieces fit together in many ways. A digitally-enabled "open Canada" needs many hands and eyes. Computers only augment people's real senses, needs and ethics, so this process needs to be inclusive and social; Connectors and enablers retain the vital importance they always had, though they become more visible, human, and also easier to hold to account. A diverse culture demands sensitivity to priorities, professional blinders and plain old prejudices. If these are at least expressed in a common linked language, they become easier to understand and evolve.

Enthusiasm in government and outside for a digital nervous system motivates us to learn new things. Re-structuring information for a receptive, aware culture, will require new roles and new job descriptions.

Selected issues and challenges

Inclusion

Not everyone is connected; about a third of Canadians may never be connected due to poverty, illness, etc. This "digital divide" should not delay sharing of data, but care must be taken to enable this group's full participation in emerging information realities. "Hacker" and organized municipal open data and academic projects are becoming more diverse in their constituents, but turning the situation around by supporting easy to use electronic systems to augment social services, and especially using social networks to support those not online must be emphasized.

Procurement, hosting

Today official open data projects are usually hosted on inexpensive servers around the world – the “cloud.” This is suitable at least for experimentation. Some projects may never have a revenue model, but they can be part of improving quality of life; fortunately most maintenance costs are negligible.

Many developers consider it an obligation of citizenship, or a chance to enhance their reputation, to provide projects to the government for official hosting. This can be done exploiting open principles of “many eyes” (open review with high levels of re-use).

Canadians may “import” externally developed projects available under open licenses. MySociety in the UK has built many advanced, free open-source applications. By improving software and schemas and sharing our improvements under free software or open content licenses, we gain access not only to innovations from other countries, but their feedback as well.

The right to describe public players

Many believe that data open to many users and editors creates liability that is not inherent in more closed systems. In practice, institutions that pay persons to edit information and distribute it with an official stamp of approval accrue much more liability than those that merely facilitate an open discussion or mass editing process. The vast Wikipedia attracts fewer lawsuits per year than some newspapers.

There is probably no better response to claims of defamation or copyright than version where the problem quickly disappeared and were explicitly noted as unwelcome, incorrect, or withdrawn. Far more effective responses are available in online media: right to reply, or even to "refactor" (completely rewrite and re-order) biased, slanted or incorrect material. While English Canada's libel and defamation laws are primitive and contradict treaties Canada has signed on freedom of political comment, recent Supreme Court of Canada rulings have leaned heavily towards US, AU, NZ and other Commonwealth models in which public issue comment provides broad protection from lawsuits.

Another common claim is that without versions produced in inflexible forms at regular intervals by official sources, no one knows what to rely on. In practice, approved versions can be marked very easily after an actual review, one responding to broad feedback on any data's apparent shortfalls. This makes far better use of scarce expert review time, directing it to what ordinary users cannot resolve. Large closed databases contain more, more insidious, less obvious errors than open data. A culture of looking closer — more sophisticated citizens, whether as a constructive hobby or professionally — needs to be enabled and perpetuated.

The vested interests

Government purchasers and standards-setters face a lot of pressures in writing RFPs and otherwise specifying information systems and policies. Much of the pressure comes from corporations that constantly spread fear, uncertainty and doubt (FUD) along the lines that there is no alternative to their market-dominating product or format, or that terms of the RFP must require inordinate levels of compatibility with proprietary products that in practice the corporation can change on demand.

Systems need to focus on data formats rather than the software which rapidly changes around it; there is no reason whatsoever that multiple products or vendors cannot support one data format with an open "reference implementation".

Social and political

In democracy, the solution to social problems is often more participants, providing a wider pool in which one can almost certainly find someone with the ethics, expertise and and motivation to resolve conflicts creatively. Bernard Crick, writing "In Defense of Politics", proposed a few "political virtues" including:

• prudence - take one step, then see its results before taking another
• conciliation - making friends with people you have argued with
• compromise - giving up some things you want to get those things that are most important to you
• variety - people want to have a number of choices that are different from each other
• adaptability - meet the needs of changing times
• liveliness - never be boring

Others have since proposed humour, empathy, initiative and compassion.

Today, five of the top six (all but the third) of the hits on the search phrase "political virtues" were wikis open for anyone to edit, or text "mirrored" over from Wikipedia. It is not unreasonable to imagine that wikis themselves require and thus engender these same virtues.

The actual text describing the virtues above, and the list of new ones, is from the Simple English Wikipedia, which is intended for primary students, those learning English as a second language, or with learning disabilities.

When we are learning from the same sources we are writing, we are participants in our own culture. When we are merely copying or obeying, we are not, regardless of how often we may vote or complain.

There are no shortcuts to solve ethical problems, real rights must be balanced against real rights, and real people's needs must be balanced against those of others. The push back against the inefficiency and rigidity of bureaucracy must be countered with a more social, considerate and open network of people, institutions and business.

Culture

A more open world demands a new mindset: One that values usefully shaped challenge, dissent and criticism, rewards correct inference and considerate sharing, carefully measures credentials and regards firsthand knowledge as most important.

We have many examples of sub-cultures with such values. One of these is our independent judiciary:

• Judges must explain in detail, with extensive references, the reasons for every single decision.
• Appeals (to a panel of three provincial or nine federal judges) are even more explicit, with the minority often authoring an extensive account of what is agreed on, so the majority disagrees on only a very few carefully marked points. By no means is the minority ignored, suppressed or feared: The minority view remains on the record for posterity and may justify a reversal someday.
• Frequency of reversal in this process is a major career success indicator for judges and lawyers.
• Where values converge or treaties apply, or where two jurisdictions have a common history, any of the authorities, experiences and precedents of these other jurisdictions can be used in an argument.
• Rules of evidence prevent rumour, hearsay, allegation or suspicion from causing most bad decisions.
• Resources are often available to equalize access to the court system (legal aid or pro bono help).
• Participants are trained to recognize logical fallacy and strongly discouraged from engaging in it.
• Agents with special knowledge or access ("officers of the court") are bound to stricter rules than ordinary participants (defendants and plaintiffs), and often required to assist the opposing side.

One way to imagine our new culture is one where we all have a few skills of our best judges or most ethical lawyers. If this seems unlikely, consider how television's "Law and Order Effect" trained many criminal defendants in fine points of the law, while many forensic shows taught victims to demand deeper investigations of crimes. And again, who knew that pre-teens could competently edit encyclopedias?

Despite our differences, we all submit to our judiciary and most of us watch the same TV shows. And no matter how obscure we can probably read (and edit) Wikipedia in our native tongue, and participate in social networks.