Seek ditches pagers for cloud service in IT operations

At Seek, one of Australia’s biggest online marketplaces for jobs, keeping track of IT issues did not always correspond with its adoption of DevOps.

Although the Australian Securities Exchange-listed company had gone full-on with DevOps to build the applications that power its services, it was using a manually configured monitoring system to manage and surface alerts on IT issues.

These alerts would be sent to an email box, which would then sound off physical pagers that were issued to members of the support team who were on call at a given time.

As its business grew, Seek started to move from a centralised IT operational model to a decentralised one, so it could respond more quickly to changing demands. That meant moving development, deployment and support capabilities into platform engineering teams.

“The people writing the code are generally the best people to diagnose problems,” said Andrew Hatch, head of platform engineering at Seek. “With our platform engineering teams doing their own core support and building their own systems and services, we have been able to build things faster with a lot better quality as well.”

The faster pace of software development meant it could have deployed and managed more pagers, including replacing batteries and instituting processes to handle lost devices.

It’s up to users of the system to ensure alerts are properly configured so they don’t fall into the trap of tracking noisy vanity metrics

In about 2016, Seek started exploring options that would allow it to manage IT incidents and receive alerts through a mobile app powered by a cloud-based service. After a thorough assessment of the offerings in the market, it settled for PagerDuty, a digital operations management platform.

“We found that PagerDuty’s features suited our needs, and it integrates well with our ways of working,” said Hatch. “It delivers the value we need and supports our team to do on-call support.”

With PagerDuty, Seek is now able to plan on-call schedules ahead of time, including overriding duty schedules if a team member falls sick. “In the past, that was all done manually, so it helps with those kinds of management tasks,” said Hatch.

Hatch said PagerDuty is also highly configurable to suit the needs and business outcomes of each platform engineering team.

“But it’s up to users of the system to ensure alerts are properly configured so they don’t fall into the trap of tracking noisy vanity metrics,” he added. “It’s not like you install the tool and it solves all your problems – it’s really about the practices around the tool.”

Hatch said it all boils down to engineering teams having deep knowledge of their systems, their service-level agreements, objectives and level of fault tolerance. “And that’s really done on a case-by-case or team-by-team basis, because teams take full responsibility for what they build and deploy,” he said.

Configuring and monitoring the right performance indicators is crucial, especially since the applications that Seek has been building over the last five to six years have become more complex and diverse.

“We need to constantly take proactive, autonomous measures, so tools like PagerDuty help a lot in coping with the complexity of identifying issues when things go wrong – and, more importantly, alerting teams before they impact customers,” said Hatch.

Likening PagerDuty to a life support system that monitors the vital signs of computer systems and powers them down if they are overheating, Hatch said engineering teams can also pause alerts and monitors when deploying software changes, thanks to the use of application programming interfaces (APIs).

With its expansion into Asian markets, including China and Southeast Asia, Seek has extended the use of PagerDuty to engineering teams outside Australia.

“As we integrate our businesses in Asia with our Australian business, we want to leverage our existing investments with our vendors as much as possible, not just PagerDuty, but other cloud vendors as well,” said Hatch.

“We want to make sure we don’t have too many things in the mix because that becomes too complex to manage. We have teams working on the same projects in Australia and Asia at the same time, and we don’t want them to be using different management tools.

“We already have a lot of people in Australia with expertise in PagerDuty, so whenever team members are integrating with what’s going on in Asia, they’re bringing that experience with them as well.”