thodonal - stock.adobe.com
French baby store modernises backup as Veeam handles 150TB a night
France-based maternity, baby and children’s store Orchestra needed to share IT resources between 700 points of sale across multiple time zones, and backup for 100 applications
French mother and baby store chain Orchestra has made the switch to Veeam backup software, in a move that has seen its backup window shrink while making dramatic savings in IT staff time.
“Our priority is to meet the demands of the business. To permanently record the activity of 700 points of sale, of which 400 are in France, our IT department must respond to the needs of our designers, our logistics chain, the need for reporting and BI,” said Bastien Pino, head of technical and infrastructure projects at Orchestra.
“We’ve got to look after some extremely complex data streams and we can’t allow backup to impact the availability of our infrastructure or our experts,” he said.
Orchestra runs around 500 virtual machines and several physical servers, with about 150TB of data that needs to be protected every day. To minimise performance impacts on applications, backups are made overnight. The backup window runs, in theory, from 9pm to 10am the next day at its France HQ.
But, backup must be carried out without dominating IT resources during this period because there are points of sale internationally that need to function.
“Backups are simply critical,” said Pino. “We are obliged by regulations to archive certain data, in particular financial data. We also have in place a disaster recovery plan that consists of restoring backups to continuity servers.”
“Obviously, we often also have to deal with human error, such as restoring documents or emails that have been erroneously deleted.”
Un-recoverable data
In 2016 Orchestra became aware that its existing backup software had become dated.
“On the one hand, we had moved to Office 365 for emails and needed a compatible backup tool,” said Pino. “On the other hand, our software assets had become increasingly complex and we were obliged to backup all our production stack. We couldn’t do this in the time allowed and this led to incomplete restores.”
Pino’s team tested several products on the market. Evaluation criteria included integration with a virtualised architecture, simplicity of the console, speed of backups to allow completion in the nightly backup window, cost of the solution, human support in case problems arise, and automated recovery testing.
“Every time our team had to restore something it was in crisis management mode,” said Pino. “It’s always necessary to work quickly, and you don’t want to find out at times like that data is not recoverable.”
Of all the products tested, Veeam Backup and Replication Enterprise Edition fitted the needs of Orchestra the best and the software was deployed in summer of 2017.
Backup for 30 days, then archive
Veeam provided some key benefits over the existing backup software at Orchestra. The first is that integration with the ESX hypervisor avoids having to install an agent on every VM.
ESX generates – on request from Veeam – a global snapshot image of all its virtual machines. That puts all the burden on the hypervisor where before it was necessary to perform the job at VM level.
That snapshot is then held on the Veeam server where its contents are indexed, duplicates and previously backed up data removed and staged off to shared storage.
“This approach is so efficient we can save an entire month’s worth of backups on one 400TB array,” said Pino. “As it happens, we make a full backup every Sunday of 150TB, with incrementals during the week, which in theory amount to more than 400TB.”
After a month, backups are archived to tape and deleted from disk.
Beware Windows licensing
Another advantage is that the backup server’s duties can be offloaded to several proxy servers to allow them to execute a backup job simultaneously.
“Veeam’s servers are virtual machines that run on our ESX server and use only one core each to carry out their work,” said Pino. “Because our servers are used minimally at night we have deployed 13 proxy servers in order to use those free cores to carry out 13 backup jobs at once.”
Pino notes that he doesn’t choose to deploy more proxies to shorten the backup window because that would require the purchase of a Windows server licence each time. He points out, however, that the Veeam licence allows the deployment of as many proxies as he wants without paying more.
Servers to be backed up (ESX and physical servers) and the proxies, all communicate via 10Gbps Ethernet with the backup master server that runs on a physical machine that has SAS-connected JBOD disk.
A year to test backup with 100 applications
The switch to uniquely Veeam backup took a year to carry out.
“The migration was long because we tested each of our 100 applications one after the other,” said Pino.
That’s because Pino and his team had to define rules for nearly every application.
“Some applications are susceptible to rewriting all their data every day. In those cases it’s just as quick to carry out full backups as incrementals so it’s a case of working out which type of operation is optimal,” said Pino. “We have also wanted to test restoration of data because we have got burnt by failures so many times in the past.”
Read more about backup
- Backup methods pros and cons: Full, incremental, differential, synthetic full, incremental forever, and reverse incremental. We set out the advantages and drawbacks of each.
- A data protection and backup strategy has to take account of the expansion of the sphere of IT, from the datacentre to the public cloud to devices and locations at the edge.
Each month in the early stages of the migration, the IT team observed the behaviour of the backups for about a dozen applications. In this period, two backups were made; one on Veeam, the other on the old product. After that testing period the old software was decommissioned.
After a year had passed, Orchestra had still not been able to dispense with the existing system.
“The problem was that the format of the old backups is incompatible with Veeam,” said Pino. “But, with regulatory requirements that we are able to restore data up to 10 years old we are obliged to pay for support for our old solution for all data backed up before summer 2017.”
Pino considers it a price worth paying. He said that to convert the old backups to Veeam’s format would be very costly.
Liberating expertise
Key among the benefits for Pino is the simplicity of Veeam.
“The interface allows the granularity for us to restore a single email, VM or file from among a lot of backup data,” said Pino. “That ease of use has a very practical consequence in terms of time gained. Previously we had to devote the equivalent of half a full time expert-level staff member to restoring data. Now it costs us one-eighth of the time of a junior member of the team.
“Thanks to the Veeam deployment our experts are more available to solve the problems of the business. And that allows us to move more quickly with other project, because, as I always say, it’s always better to work with one who knows rather than three who have to look for the answers.”
Looking to the future, Pino intends to look at Availability Orchestrator, which is Veeam software that automatically builds an environment to fail over to in case the DR plan is invoked.
“As well as improving our DR planning provision, this software can help with building test platforms with real data so we can eliminate processes that are essentially manual today, and in that way free up even more expert time,” said Pino.
