SignifAI Uses Machine Learning to Monitor the Full Stack
Picture this: You’re sitting in front of a bank of monitors, watching dashboards from eight different systems. You allow yourself a smile because everything is green, across all systems. And Bam! The production database goes down.
The familiar crisis routine gets launched and after several hours of trouble shooting and angry clients/managers/CEOs, you discover that a recent update to PagerDuty is not completely compatible with a slightly old version of NewRelic which is scheduled to be upgraded next week. You want to write up a postmortem with the fix you so painstakingly discovered but a new crisis takes your attention.
This is the exact scenario that led to the development of SignifAI, a new startup that bounced out of stealth mode at the Monitorama conferencein Portland.
There has to be a better way, thought Guy Fighel, chief technology officer and co-founder of SignifAI. A year ago, he gathered together a team of TechOps professionals who got tired of their jobs managing multiple systems across multiple monitors, each system with it’s own alerts, tracking different things, and spending most of their time putting out fires.
There’s so much they wanted to do to improve the system, wrote Fighel in a blog post, but they could never get there because they were constantly in crisis mode.
As Capital Picard says, if you’re on red alert every day, then red alert means nothing.
Fighel looked at the rise in the cloud and machine learning and thought, ‘What if we create a monitoring system on top of all these existing monitoring systems? What if this system ties all of them together into one huge data set so we can track downtime across systems and collate the event data, the log data and the metric data to give us ways to predict outages or possible outages. And then use machine learning and machine intelligence to capture and store the post-mortem information?’
So Marcos and Fighel and SignifAI CEO JP Marcos cherry-picked a team of TechOps engineers, all of whom had experience with monitoring in complex stack environments, and began to build the system they wanted. The goal was to free their time from the drudgery of monitoring systems and the nightmare of crisis management, on-call nights and weekends. They wanted to be able to do fun stuff like fine-tune the environments or make them run like a super-tuned sports car. And maybe keep up with all the system updates on a sane schedule.
“We essentially built a tool for ourselves,” said Marcos. “How can I cut through the data to what’s important? How can I reduce MTTR [mean time to repair], how can I understand what happened right away in a language I can understand, and once I figure out what it is and fix it, how can I capture that knowledge in a streamlined way in a direct language that I will actually do.”
The first rule, said Marcos, was to make it easy. Abstract away as much of the complexity of running multiple systems as possible. The second rule was to make it easy. Capture post-mortems in four screens. The last rule was to make it easy. Run all of the systems off one screen, showing only the most high-level data, with drill-downs into individual monitoring systems. And so it is.
SignifAI pulls data through APIs to the IT assets. The initial set-up is wizard-driven, a simple set of steps where the customer selects which systems they want to add. The entire process, including the data download, takes about 20 minutes, the company claims.
The software currently support over 60 integrations including Slack, NewRelic, PagerDuty, GitHub, AppDynamics, Amazon Web Services, and Datadog.
There are two ways the system connects to the data, Marcos explained. One function, the Active Inspector, looks inside the monitoring data from all the systems; Another one, called Web Collector, listens to alerts.
