Payday lender cuts troubleshooting by 30% with Virtual Instruments

Financial services organisation Curo Financial has cut staff time troubleshooting storage performance and network issues by upwards of 30% following deployment of Virtual Instruments monitoring products.

Virtual Instruments’ Virtual Wisdom allowed Curo’s IT team to pinpoint storage, network and virtual machine (VM) issues, which also meant it knew which supplier to talk to about performance issues and counter supplier finger-pointing.

Curo specialises in quick cash loans and cheque cashing and has around 400 stores in the US and Canada, plus a large online presence that extends to the Wageday Advance and Juo Loans brands in the UK.

It runs out of two datacentres in Wichita, Kansas, and is an EMC shop, with about 1PB of storage. Its application stack majors on Microsoft, with SQL, Exchange and file servers, plus VMware and Citrix virtualisation.

Its web operations are mission critical with as many as 50 transactions per customer interaction, said director of enterprise systems Mitch Roberson.

“The key challenge we have is that if anything is delayed it can affect the end user,” said Roberson. “On average, if there is 20ms of latency coming from the storage, the customer notices. We’re always trying to monitor it to ensure that doesn’t happen.”

Curo uses EMC’s Vplex to synchronously replicate to its secondary datacentre. It requires a full copy to be written before an acknowledgement is sent back.

If any delays occur, they can become compounded and multiply exponentially, and this was a key problem Roberson’s team wanted more insight into. “We wanted to be more definite about what was happening,” he added.

The company deployed Virtual Instruments physical taps into its Fibre Channel storage fabrics and Ethernet networks plus its software Virtual Wisdom platform.

Virtual Instruments has two key product areas. The first is its VirtualWisdom storage monitoring products that enable customers to measure SAN, NAS and software-defined storage input/output (I/O) for hundreds of metrics for troubleshooting and performance optimisation purposes. Second is its WorkloadWisdom platform, which can generate near-realistic test workloads based on existing traffic on customers’ systems.

“We can see every packet between one host and another,” said Roberson. “We started looking for delays between the two Vplex replication sites, but we didn’t have the problems we thought we had with storage.

“We were also able to see we weren’t having latency issues between the datacentres. Everything was happening in 1ms to 3ms.”

So, what was causing the problems? “A lot of it was tuning VMware,” said Roberson. “Network turning – buffers that needed adjusting, for example – and memory issues.”

Roberson said his team is now able to correlate central processing unit (CPU) and memory usage to identify their busiest times. They can also track down root causes to issues.

For example, when trying to diagnose a 30% spike in CPU usage in Curo’s Ctirix environment, his team could only guess at when it started and what the issue was. By using Virtual Instruments, it was possible to discover that the issue commenced following some Microsoft patching.

Elsewhere, in storage, Curo’s IT team have been able to spot SANs that were stuggling in terms of I/O and fabric issues such as ports that were over- and under-provisioned. Roberson said it has “changed the game significantly in terms of vendors and troubleshooting.”

“VMware would say, ‘It’s a storage issue’, and we’d be able to show it wasn’t. We were very reactive, but now we’re move towards being proactive. We’re better able to predict issues than ever before.”