Machine Intelligence Capstone Project
Motivation
The University of Toronto Business Intelligence (UTBI) team works with institutional data, which includes information on admissions, enrollment, and funding in the university. The team maintains the tools for data reporting and analysis, as well as helps end-users make better data-driven decisions. In particular, the team provides extensive support with data analytics and visualizations using the Tableau platform. As the university’s Tableau users grow, the UTBI support team receives hundreds of user requests and inquiries each month. The staff is overwhelmed by administrative and technical questions, which they spend on average 10 hours per week answering, translating into a labor cost of over $20,000 annually. This process can be partly automated via a solution involving text-based information retrieval and natural language processing. To this end, we build a support chatbot that can provide users with the information they need without assistance from a live support agent.
Solution
We use an information retrieval (IR) model to approach the tasks of user intent identification and question-answering. As shown in the flowchart, our IR model has a modularized and cascaded design, instead of attempting to identify all user intents in a single step as a classification task. This model consists of four steps and addresses administrative and technical requests using different strategies. For detailed description of each module as well as example usage, please refer to the technical report.
Impact
In order to evaluate our model quantitatively, we create a test set that consists of typical chat style questions supplied by our client, and a selected subset of email requests that are manually rephrased into chat-style sentence structures. The chatbot is able to handle 63% of requests in the test set, which can translate to significant time savings for the UTBI staff.
UTBI is considering the deployment of the chatbot to different platforms, with a priority placed on Microsoft Teams. The chatbot will be added as a channel on Teams where UofT employees can send messages to the chatbot in the same way they chat with their colleagues. After deployment, our chatbot will continuously measure user satisfaction through the
response ratings, allowing the UTBI staff to make changes to the chatbot accordingly. The knowledge base can be easily maintained and updated.
Chenxuan (Chen) Cui 