IT chiefs fear Kubernetes data log overload

Over half of IT leaders polled by Coleman Parker in a global survey for Dynatrace believe their technology stacks will grow in complexity.

In a survey of 1,300 chief information officers, chief technology officers (CTOs) and other senior technology leaders involved in IT operations and DevOps management in large enterprises, 88% say the complexity of their tech stack has increased in the past 12 months. The full survey, which is published in Dynatrace’s The state of observability 2024 report, found that 51% of those polled say the complexity of their tech stacks will continue to increase.

The study reported that large enterprises with a thousand or more staff will deploy around 10 observability or monitoring tools. These are used to manage applications, infrastructure and user experience.

A Kubernetes architecture makes it easier to scale services to new users quickly and drive efficiency gains through dynamic resource provisioning. However, the study reported that 76% of technology leaders believe such an IT architecture makes it more difficult for IT and security teams to maintain visibility of their overall IT environment. This is because it is a constantly changing, dynamic cloud environment with many dependencies.

According to Dynatrace, it’s impossible for these teams to manually configure an array of traditional monitoring tools to provide observability into their Kubernetes environments.

Dynatrace warned in the report that IT security and operations teams are often unable to access the real-time insights they need to improve user experience, optimise costs and strengthen security. “This visibility challenge is a growing concern, and is delaying many organisations from migrating more of their mission-critical services, such as core banking applications and ERP [enterprise resource planning] systems, to Kubernetes,” Dynatrace said.

The situation is likely to get much worse. Among the IT chiefs polled, 13% of mission-critical applications are currently being run on Kubernetes. This is set to almost double over the next 12 months to 21% of enterprise applications. In five years, the survey forecasts that 35% of mission-critical applications will be running on Kubernetes.

According to Dynatrace, these modern environments generate data at a rate that is impossible for teams to cost-effectively capture and analyse using traditional practices and fragmented monitoring tools. The survey reported that 86% of technology leaders say cloud-native technology stacks produce an explosion of data that is beyond humans’ ability to manage. The survey also found that 85% of technology leaders are concerned that the rising costs of storing and analysing this data outweigh the benefits.

“Teams simply cannot manually query all data, from all sources, in context, to access precise insights in a timely manner,” Dynatrace warned.

“Cloud-native architectures have become mandatory for modern organisations, bringing the speed, scale and agility they need to deliver innovation,” said Bernd Greifeneder, CTO at Dynatrace.

“These architectures reflect a growing array of cloud platforms and services to support even the simplest digital transaction. The huge amount of data they produce makes it increasingly difficult to monitor and secure applications. As a result, critical business outcomes like customer experience are suffering, and it is becoming more difficult to protect against advanced cyber threats.”