The internal infrastructure (datasets, backtests, scorecards) we built to create our AI investigation system

I’ve been part of the AI team at incident.io from its inception, working on an AI product to automate the investigations into incidents - we’re nearing customer release.

As part of this, and particularly at the beginning, we had an immensely frustrating time working to get this thing built - we’d make a change one place, and performance would fall off a cliff in another.

We then rearchitected everything from the ground up around our backtesting system - which I’d like to demo.

This became not just a tool to check our working - but an active accelerant for development, something the team has come to trust, I’d like to show it off.