Data engineering - Nutanix: All engineering starts with infrastructure
This is a guest post for the Computer Weekly Developer Network written by James Sturrock, in his capacity as director of systems engineering for the UK & I region at Nutanix.
Nutanix is of course known for its leadership position in hybrid multicloud computing and its rich heritage in cloud infrastructure services and tools.
Sturrock writes in full as follows…
In this world, all ‘constructs’ start with infrastructure.
It’s an immutable fact of life and technology, we can not build upward effectively until we understand the core platform requirements needed to make any given environment function effectively once it is created.
As we now embrace the next phase of cloud-native development across enterprise application and data services, we need to think about the degree to which intelligent data engineering can help us to build truly functional IT platforms that have enough agility to serve modern business requirements.
Think about LLMs, last year saw us extend our Nutanix GPT-in-a-Box technology to include new integrations with Nvidia NIM inference microservices and the Hugging Face LLMs library. Stepping outside of our platform, let’s ask why we made that move and what engineering lesson it teaches us.
The development happened because it enabled access to community-built AI models via Application Programming Interfaces (APIs), so the opportunity to optimise, accelerate and subsequently innovate could be made if the infrastructural data engineering could be executed at a commensurate level…which in this case it very much could.
Every day’s a school day!
There are lessons that teach us more on this everywhere. We know that environmental, social and governance (ESG) concerns are now paramount inside enterprises striving to operate on what they hope will be a carbon-neutral basis in the near and immediate future. We view AI skills development as a key part of this process as it acts as a data engineering facilitator to create more efficient systems that organisations can operate on effectively, profitably… but above all on a cleaner, keener and leaner basis.
Data engineering – like engineering – has a lot of grease in it, but hands-free operation is sure to require less handwashing at the end of the day.
Looking further afield, our software engineering teams worked hard to make sure we were able to launch Nutanix Kubernetes Platform (NKP) last year, a technology designed to simplify the management of container-based applications using Kubernetes in CNCF-compliant cloud-native stacks.
Daily data engineering directives
This move was directly taken to ensure platform engineering teams who engage in daily data engineering tasks and workloads are able to use a consistent operating model for securely managing Kubernetes clusters across on-premises, hybrid and multi-cloud environments. Again, it’s all about what kind of data engineering tools we can bring forward in the face of major modern platform evolutions, such as the widespread popularisation and standardisation of Kubernetes in this case.
The lesson here is… when data engineering controls spanning multicloud environments can be accessed from a single pane of glass, organisatio
ns have the chance to reduce not only operating costs, but also the complexity of the stack they see before them at any one time.
Directing disparate incongruity
The message from our team internally (and from a customer-facing perspective) is abundantly clear i.e. we want organisations to be able to run any application at any scale and have the freedom to span multiple cloud services (and multiple clouds on any given hyperscaler) and – where needed – to operate a cloud-like (cloud-native-like even) infrastructure on-premises by unified operations across all of your IT sites and clouds.
While we’re busy doing this for customers, we must also remain constantly aware of the disparate incongruity that exists within working, living data streams inside organisations. We know that not all data is created equal and that information channels and repositories inside (for example) a CRM system will not necessarily exist in anything like the same format as the same organisation’s manufacturing inventory system (for want of a suitably disconnected example) in working practice.
These inconvenient truths mean that we need to arm the data engineering team with as much platform flexibility as possible as they work to bring about data management practices that will unify storage formats, unify application access and – importantly – also look to unify governance across the organisation’s IT stack, which itself may be in the process of scaling and morphing at any time.
Cloud, on any terms
All of this is why we inside Nutanix talk about “cloud on your terms” when we talk to data engineering teams and their DevOps counterparts throughout the operations function. We know that every working business wants to avoid the daily grind of outages, maintenance tasks and integration headaches so that it can centralise its attention on thoughtful design, future-proof implementations and optimal performance for business applications at every level.
This story is really only just getting started, we will also need to work with any cloud on any set of terms in any deployment scenario… and that means on-premises, cloud-native and remote Internet of Things edge computing deployments.
Why do we need to pay so much attention to data engineering right now? I could give you a shopping list of reasons, but we might consider a) just how much disruption there is in the virtualisation market right now (we don’t need to mention any names to make the point) and b) the looming (although largely positive) spectre of Artificial Intelligence and its impact upon the cloud stack and… let’s add c) a need to also straddle major developments in core technology practices across databases, end user computing, networking and security and the wider architectural configuration challenges of working with hyperscalers with business-critical applications.
It’s good to talk
Nutanix’s Sturrock: At Nutanix, we see the strength that customers can build into live production systems every day to increase user experience satisfaction if they take a grassroots approach to multicloud enablement.
As eminently technical as data engineering discussions will always be, we want to champion the need to create an open discourse between the business function and the IT function when we consider just how data engineering will impact everyday commercial operations.
What I mean is, let’s say our cloud team is busy working on a customer job with a technology like Nutanix Files. This is a software-defined scale-out file server that can be used as the repository for unstructured data like home directories, user profiles, departmental shares and application logs.
Or, let’s say the data engineering team is busy deploying our Nutanix Cluster Check (NCC) technology to offer a suite of functions designed to ensure an organization’s clusters operate at peak performance.
All of these advancements need to be contextualised and qualified for the business function to understand why they are being applied – even if it’s just in one line of plain English – so that the business can maximise the opportunity to thrive, innovate, expand and improve.
We said that all data engineering starts with infrastructure and (as an infrastructure specialist) we see the strength that customers can build into live production systems every day to increase user experience satisfaction if they take a grassroots approach to multicloud enablement.
As a back-end discipline, in our eyes, data engineering is at the fore, all power to the team say we.
