Programmes and projects in an age of data applications
During the Future Decoded conference in London, Microsoft discussed a paradigm shift in application development. But it was not setting out its plans for a new software developer’s kit, programming language or the next big thing after cloud native applications and microservices.
Instead, the company wanted delegates to appreciate the importance of data. Why the paradigm shift? Industry pundits describe data as the new oil. The biggest companies in the world are data-driven.
Microsoft believes that artificial intelligence will change the way data-centric applications are developed and deployed. Rather than hard-code software to make use of the vast amounts of data being generated, the application being developed will evolve out of data models that feed AI-based inference engines. Programming becomes supervised machine learning; quality assurance is the feedback loop that applies the data model to real world data and continually optimises it to improve its accuracy; the application is the predictions the data model makes.
If Microsoft’s assertions are true, and data becomes the new programming paradigm, business and IT need a different approach to how they identify valuable data. It will often prove almost impossible to spot something useful in the reams of data an organisation has collected.
Is there a needle in this haystack?
According to Scott Crawford, head of data science enablement at 84.51°, the technology business at retail giant Kroger, the future for many companies will be determined by their ability to collaborate, using diverse teams to help them to look for a needle in a haystack of data. Crawford believes that as data science expands in an organisation, that diversity of capabilities becomes critical.
IT leaders will also need to assess when a request from the business is a defined project with design, build, test and deploy phases – or when it is more of a programme of continuous development and improvement.
As Crawford points out, there are plenty of use cases that solve a particular problem which means that the data team can crunch the data, create the data model and move on. But there are also cases which build evolution into a data science programme to provide better estimates, using the most recent set of data. One example in retailing may be when the data team is asked to build a better model for forecasting instore product demand. While it may indeed be possible for the team to develop a superior data model, Crawford said: “Very little thought goes into what happens if the new model wins.” Potentially, the business may change they way it does something. This change in business could invalidate the data model, or at the very least, mean that it requires further refinement, leading to further adaption of the business process ad infinitum.
In the age of the AI-driven data applications, such questions will need to be raised every time the business wants something new done.