Monitoring tools and network operation centers (NOCs) continue to improve, but every organization still experiences issues that affect business. Regardless of the size and scope of an incident, the impact is often determined by the speed and effectiveness of an organization’s response. Recent research shows that an hour of downtime often costs up to $300,000. Reducing MTTR should therefore be an ongoing goal for any company and this puts continued pressures on NOC’s and incident response capabilities.
Incident response is not an easy problem to solve though. In a 2015 survey conducted by xMatters, 60% of respondents indicated that it takes between 15 minutes and 6 hours just to engage the right person so that the issue resolution processes can begin. However, nearly half told us their business starts to feel the effects of the incident within 15 minutes.
In late August 2017, we talked to Ken Wood, Senior Manager of Enterprise Tools at inContact. InContact is the global leader in cloud contact center software, helping organizations transform their customer experience by enabling contact centers to win every customer interaction, deliver great customer experiences, and achieve their business goals. The company supports more than 200,000 contact center agents for customer organizations world-wide. So resolving issues before they affect customers is crucial. Ken reflected on how far his organization has come in incident response and what the future holds.
The company’s incident resolution processes used to be very people-intensive, which lowered the ROI on incident management, reduced efficiency, and created risk for the company and its customers. The company also suffered from too much email, confusion over the right groups to notify, and the challenges of manning a NOC 24 hours a day. The NOC staff often had to make their best guess as to which group to contact.
“Unfortunately, at that point in time, all of our groups didn’t even use a consistent method to maintain their own call lists,” Ken told us. “Maybe they had a spreadsheet, maybe it was in SharePoint, maybe it was on a website, whatever the case may be. Then, for the operation center, they would have to know, OK, the database group has their own application; we have to go to this application; and hopefully it’s up-to-date.”
Then, the operation center would start at the top of the list and start calling people in the database group. Meanwhile, another problem would often occur that also needed attention. “Now, they’ve kind of got their attention diverted to the new hottest issue,” Ken continued. “They start engaging some other group. No one answers. They call back. Someone finally answers and says, ‘Oh no, this isn’t a database issue. This is actually a system issue.’ Now, where’s their on-call list maintained?”
To create a more efficient and collaborative business environment, inContact sought some new business tools. “We needed more automation for our teams,” Ken told us. “Manual processes were putting too much strain on our workers, and it was threatening to affect our business.”
It’s all about using the right tools
inContact started using BMC Remedyforce, a cloud-based IT service management tool designed to deliver leading IT and business service management to its users.
BMC Remedyforce enables organizations to respond quickly and efficiently to conditions that disrupt business services and minimize downtime. It also allows you to make changes in a controlled way, reduce the risk to timely delivery of services, and align with business objectives. Remedyforce helps speed response time and improves productivity by incorporating IT Infrastructure Library (ITIL) best practices.
inContact automated notifications from BMC Remedyforce, enabling the inContact operations staff to discover suspicious events earlier and thus setting the company up to fix issues faster. Built on the Salesforce platform, BMC Remedyforce puts information in the hands of front-line personnel.
“We wanted a cloud-based notification solution usable with a mobile application,” Ken explained. “Being able to accept incident assignment via a mobile notification is a great time saver.”
Finding and recognizing the issue is one thing. Fixing it requires a whole other level of communication and collaboration.
Collaborating on resolution
xMatters, an integration-driven collaboration platform, assembled a toolchain with its built-in integration to BMC. Now, the information in BMC tickets could automatically be moved into notifications to incident resolution experts, IT managers, operations managers, developers, and anyone else who could help resolve the issue.
The time for inContact to resolve issues is now often less than the time that was previously consumed just making the right person aware of the problem. “The fact that both of these solutions are SaaS-based is a real positive,” Ken said. “It provides availability to our people at any time. It also provides expandability and scalability in the future, and it’s just not as human intensive.”
xMatters has provided the solution. Now any P1 or P2, with the Remedyforce incident, is automatically notifying the appropriate group. It’s going through the on-calls, and it has already been defined within xMatters.
“The best part is that there was a very minimal learning curve,” Ken added. “Some team members were resistant to change at first because we completely changed our whole processes, but they’ve all come on board now. It’s been an educational opportunity.”
Using the mobile app from xMatters
Ken also recommends having the teams use the xMatters Mobile app as an additional cost savings for notifications, and for convenience to the users. “There are a few reasons for this,” Ken explained. “Number one, it’s free. You can send an unlimited number of notifications to the app and there’s no charge. You don’t have to worry how many of these you are sending.”
inContact has built notification templates so that you can quickly notify the right people when an incident occurs – regardless of the nature of the ticket, how many groups need to be notified, or whether the ticket involves a conference bridge.
Looking to the future
- In September 2017, inContact extended its notifications to include SLAs and OLAs. In the future, inContact plans to include escalation notifications.
- Although the initial focus of inContact’s BMC and xMatters integration was on 10 operational groups, that number is likely to increase as the toolchain expands.
- inContact is already expanding its use of both xMatters and BMC Remedyforce. Ken told us that inContact plans to start using its new service desk processes for DevOps and monitoring to improve application development, delivery, and enhancement processes.
To find out more about inContact’s service processes, listen to the on-demand webinar: “Implementing an Automated Service Desk Magic Button, with inContact, BMC Remedyforce, and xMatters.”
These postings are my own and do not necessarily represent BMC's position, strategies, or opinion.
See an error or have a suggestion? Please let us know by emailing email@example.com.