Better user acceptance testing through log management

Logging season is open

"Theoretically, in the UAT phase of software development, the new code should run perfectly. Sadly, pure theory is rarely reflected in real applied life," says Sumo Logic co-founder and chief technology officer (CTO) Christian Beedgen. "What you can always be sure of is that there’s an error somewhere – and if you don’t find it during UAT, your users will point it out to you later when it’s in production."

Beedgen asserts that logs hold the key to this problem for development teams. He says that as a dataset, logs are an opportunity to run queries and analysis to identify errors and exceptions, as well as model behaviour and alert deviations.

"So beyond UAT, once in production, logs will continue to capture the information that enables you to conduct root cause analysis and troubleshooting, so you can remediate issues that you find. For the testing, development and production environments, collecting and analysing logs will support your mission to find and eliminate anything standing in the way of uptime and quality user experience," says Beedgen.

What are we looking for?

But what can we learn from logs and how should they connect to UAT? Essentially we are looking for any activity which we would deem to be abnormal in respect of the anticipated normal execution of the application runtime. From this we hope to be able to deduce which elements of the software development lifecycle need to be escalated as problems that need to
be resolved.

But we need to be careful here; one could say that user acceptance testing and log management are concepts and worlds that are by nature always far apart, argues Balázs Scheidler, CTO of log management infrastructure company Balabit.

Why should this be so? Scheidler says that it is because system and application logs are usually managed by operations and security operations teams, to ensure the continuous operation or the continued security of an IT system. UAT on the other hand is performed by quality assurance (QA) people at the end of the deployment/delivery process.

"User acceptance testing tends to be black-box testing, using a set of predefined test scenarios or just freestyle testing without too much concern about the internal structure – or under‑the‑hood behaviour – of the application. It is simply assumed that this under‑the‑hood behaviour was properly validated during the earlier stages of the QA process using unit tests, integration tests and system tests," says Scheidler.

Wasted insight opportunity

However, this can be a great waste of insight. This is because application logs tend to include a lot of information about internal behaviour as they are the primary means of troubleshooting problems encountered in production. With a few techniques in place, log analysis can uncover the rough edges of the application while performing acceptance testing.

"UAT is generally wide in scope and shallow in depth. That is to say it tries to cover most of the functionality without trying all the combinations, whereas earlier testing stages usually take care of the depth but concentrate on units or components at a time," says Scheidler.

"During UAT, the application needs to work end to end, however, in most cases, only a handful of transactions are performed. It’s often the case that the application functions properly for a few requests but a bug leaves a trap somewhere in the application’s state which can trip the next user. This is exactly what we intend to uncover before production," he says.

Weapons in the UAT arsenal

Balabit’s Scheidler lists the application awareness opportunities available to us if we include logs into our UAT arsenal. He says logs will give us the chance to:

• Look at various known bad patterns – error, failure, warning and so on;
• Track known good messages and find the exceptions – a technique also called artificial ignorance;
• Use clustering and other machine learning techniques to find differences between known good data.

Putting these steps into practice is done via a process. Trevor Pott is an IT consultant and network administrator in Edmonton, Alberta, Canada. He stipulates that we need to correlate user activities with actual events, but we must remember that access logs and error logs are usually separate. If we can narrow down an individual user’s access and timeframe, we can see what might have caused their errors.

Pott urges us to remember that system administrators are users too. Logs need to be usable, he says. "They can’t just spew forth so much white noise that nobody ever checks them. Thus services such as BigPanda, which scans alert emails to make sure that you are paying attention to the logs you actually need, are an important part of UAT as well."

It is a bigger gameplay all round if we do address logs in this way. This means UAT logging should be pretty intensive – more so than the sorts of logs we might collect during regular operation.

"If companies are using a single logging server infrastructure for UAT, testing, development and production, log management will be required to separate one class of logging – and the insane levels of alerts it will generate – from the other," says Pott.

Virtuous circle

So what happens if you get the connection from log management to UAT right?

The log management-UAT state of nirvana brings us to a place where we achieve wider overall domain knowledge in terms of the entire scope and breadth of the software we are trying to develop and deploy. This in turn leads us into a virtuous circle of intelligence where we can potentially learn more about the actual functionality that manifests itself in the software that has
been produced.

Onward from here, we get less errors in live production, better performing software and happier users with fewer risks associated with security, compliance, governance and licensing.

The concepts here are simple, or at least logical enough, in terms of the way they look on paper. Working these conceptual methods through into live application production takes a little more blood, sweat and tears. Early signs are that it is worth pushing through the pain barrier.