When evaluating Stack Overflow for Teams, people often ask two questions:
- How big of a problem are repeat questions? In other words, do I really have a problem that Teams can fix?
- I know I have a problem, but how can I identify which questions should be in Teams to start?
This tool uses AI to read Slack transcripts and pulls out the questions contained within them, then groups together the ones that are duplicates or similar. This helps to address both concerns above. If you have a lot of duplicate questions, you'll better understand how much time and effort could be saved. Second, with both unique and duplicate questions, you'll be able to more easily pull them into a central repository to search and evaluate versus reviewing a random set of Slack messages. Customers can use the updated list to pre-seed questions and answers on Teams. Making the case internally of the amount of time wasted on answering the same set of questions in Slack or call-out content that hadn’t been socialized or distributed broadly enough throughout the company to enable teams to mitigate these knowledge distribution challenges quickly.
What we learned from the alpha
Every customer we spoke with saw the potential value in the tool. Knowledge silos in Slack are everywhere, and people are approaching this problem using manual efforts. Even the slightest improvement over a manual process is positive.
Running the app locally
Decipher was an app that runs locally via a Docker container. Getting this to run on a customer's machine was much more seamless than we suspected. Within one 30-minute meeting, we were able to download/install/run the app.
The tool did a great job of actually detecting questions in Slack messages. It was not pulling in generic Yes/No questions; it was capturing the more complex ones.
Question summary statistics
Related to the above, the handful of summary statistics for questions was very interesting to the customers we tested with. Simply seeing the trend and volume of questions help them quantify the problem they are trying to solve with Stack Overflow for Teams.
By far, the biggest challenge was having the customers get access to the raw Slack exports - it usually required them to go through their IT department. We believe that streamlining the export process of getting this data into the tool should be a top priority.
The other main challenge was another foreseeable one. After all the questions were detected, they are passed into a series of unsupervised machine learning algorithms. Unsupervised means the model does not actually have labels, so it does not know what is the ground truth about what questions are actually related to each other.
When testing with customer data, the clusters were not as usable as we would have liked. Luckily the customers were technical enough to feel comfortable adjusting some of the parameters. But they would have to do additional curation before handing them over to people.