Top 10 information management stories of 2018

In the field of information management, 2018 brought to the fore artificial intelligence (AI) and the ethics of data exploitation, both together and separately. This was dramatically made manifest in the notorious Cambridge Analytica controversy, where that company used Facebook data to, allegedly, influence the 2016 US presidential election and, possibly, the UK EU referendum.

Advanced analytics is increasingly melding with that machine learning sub-set of AI, with the rise, and now consolidation, of big data environments, such as data lakes, in organisations. Often, when people say “AI”, they really mean machine learning, which then feeds into the analytical work done by data scientists, using a plethora of data preparation and visualisation tools, as well as statistical analysis tools from SAS, the R programming language, Python, and so on.

The Cambridge Analytica case raised an ethical issue of particularly keen concern to early-career data scientists. Should they take a job with any company that seems to be capitalising on personal data in areas that regulation did not yet reach? “Just because you can does not mean you should” was a commonly expressed idea in the data science community. But what does it mean in practical terms? And can it win out over massive pay cheques?

That the UK could find an ethics niche in the global AI economy was one prominent idea that emerged from a House of Lords select committee report. Nigel Shadbolt, principal of Jesus College, Oxford and co-founder of the Open Data Institute, sat down with Computer Weekly in the summer to discuss this and other data matters.

Earlier in the year, Computer Weekly also looked at the work of the Data Lab in Scotland, which is supported by the national government there. And we found Dominic Raab, the then secretary of state for exiting the European Union, telling an audience of technology business people, in November, that the UK AI sector would be positively boosted by Brexit “done right”.

On the industry side of big data, we saw Hadoop (the open source distributed processing framework that manages data processing and storage) continue to diminish as two of the companies to which it gave birth – Cloudera and Hortonworks – decided to sink their differences and overcome their rivalry by becoming one entity. That will play out in 2019.

The gaining of business value from all the technology and people investments companies and organisations have made in big data over the past decade was a big theme in 2018 – as it was in 2016 and 2017. Some Silicon Valley startups and early-stage companies homed in on that, on the supply side. But we also saw, on the user side, practical demonstrations of value from data – including at Rome’s Fiumicino Airport, and at Kone and ThyssenKrupp.

But it was the ethics of data analytics and AI that was the leading information management theme of 2018, as Computer Weekly’s interview at the end of the year with Roger Taylor, chair of the new Centre for Data Ethics and Innovation, demonstrated.