sdecoret - Fotolia
ODI report reveals key data publishing challenges
Lack of standards, vision and adequate skills among key reasons organisations are struggling to publish quality data, according to Open Data Institute
The Open Data Institute (ODI) has published a report that highlights key data challenges and sets out how organisations can improve data quality and output.
The report is part of a series of projects the ODI is undertaking to improve the country’s data work and ensure the UK is at the “forefront of data innovation”.
One of the issues organisations have is making data a priority. Data publishing is “often an additional, temporary assignment”, the report said, adding that this could also lead to issues around prioritising which data to release.
“As different audiences do not have the same needs, levels of expertise or maturity, data published incorrectly or not reaching the right users renders the effort of publication worthless,” the report said.
The ODI report also found that people often fear making data publicly available because they’re worried there could be errors in the datasets.
“A lot of publishers seem to think every publication must be perfect: they set themselves impossible goals that keep them from publishing. This fear can be crippling in bigger projects where no data gets published unless a hard deadline is set by government officials,” the report said.
Commenting on the report, ODI’s head of technology, Olivier Thereaux, said there was often a “tendency to want to create the highest quality of data”.
“This perfectionism can be wrong: quality is in the eye of the beholder, and often it is unclear what ‘good’ means. Sometimes you can start small, meeting some people’s needs with seemingly low-quality data, and then iteratively discovering what data quality is actually needed and improving it by expanding the range of people who re-use the data,” he said.
Call for standards
While there are many different publishing tools out there, they don’t often integrate with each other. Thereaux pointed out that there are many free or open source community-driven tools, but perhaps not a lot of good commercially maintained tools. “This might be a sign that the open data ecosystem is still maturing, especially when it comes to private sector adoption,” he said.
Standards continue to present an issue, meaning collaboration and avoiding duplication is difficult. The report found this to be particularly true in local government.
“One of the most problematic issues publishers face is trusting the quality of the data they publish and use. There is a need for data standards to be adopted more and for data and metadata to have more consistent structure,” the ODI report said, calling on government to do more to promote common standards and increase awareness.
“Standards adoption is a real problem, partly because standards are seen as too complex and difficult to comply with. Sometimes the problem is simply that the standards are not well known and are hard to find. There is a need for a government standards to be better exposed, along with what they should be used for and how.”
It added that often, local councils publishing the same types of data used different approaches, even going as far as inventing a new system with every publication, and that datasets were often not interoperable, meaning it would be hard to collaborate and share data.
Thereaux said standards can make life easier for both publishers and consumers of data.
“One thing that can be improved is to make sure that when organisations need to develop a new data standard, they can do so by building on the past experience of others, proven standards-building processes and useful tools, rather than building standards from scratch and reinventing the wheel,” he said.
The ODI has been working with several organisations to document their standards-making journey, aiming to publish a “guidebook” to help the creation of open standards for data.
Increase data skills
A lack of skills or confidence in data publishing is also a key issue. People often have “great intentions and want to publish, but often lack the proper skills and technical expertise”, the report said.
Learning is often done through premium platforms that will only teach people how to use those specific tools or workflows.
A centralised learning hub could break down those silos and teach people general knowledge that can be used across different products and projects, the report said.
Thereaux pointed out that there was a need for building a community, or peer networks. There is already work going on – the ODI runs a “broad network of nodes, members and working with like-minded partner organisations” across several sectors.
“This support network for practitioners feels well-established in places, but it is worth asking ourselves how accessible we are to people who do not yet know the lingo or have the expertise,” he said.
“There are still barriers to overcome if we want to make open data publishing easy and safe and natural. It is good that more and more businesses and public sector organisations are publishing data openly, but they need support and to work together to learn how to do it right.”
Last year, the ODI was awarded £6m in funding from InnovateUK for a series of projects over the next three years, aiming to improve the UK’s data expertise.
