BotSIM: An End-to-End Bot Simulation Toolkit for Commercial Task-Oriented Dialog Systems

by   Guangsen Wang, et al.

We introduce BotSIM, a modular, open-source Bot SIMulation environment with dialog generation, user simulation and conversation analytics capabilities. BotSIM aims to serve as a one-stop solution for large-scale data-efficient end-to-end evaluation, diagnosis and remediation of commercial task-oriented dialog (TOD) systems to significantly accelerate commercial bot development and evaluation, reduce cost and time-to-market. BotSIM adopts a layered design comprising the infrastructure layer, the adaptor layer and the application layer. The infrastructure layer hosts key models and components to support BotSIM's major functionalities via a streamlined "generation-simulation-remediation" pipeline. The adaptor layer is used to extend BotSIM to accommodate new bot platforms. The application layer provides a suite of command line tools and a Web App to significantly lower the entry barrier for BotSIM users such as bot admins or practitioners. In this report, we focus on the technical designs of various system components. A detailed case study using Einstein BotBuilder is also presented to show how to apply BotSIM pipeline for bot evaluation and remediation. The detailed system descriptions can be found in our system demo paper. The toolkit is available at: .


page 2

page 11

page 14


ConvLab: Multi-Domain End-to-End Dialog System Platform

We present ConvLab, an open-source multi-domain end-to-end dialog system...

Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation

There is a growing interest in developing goal-oriented dialog systems w...

Dialog System Technology Challenge 7

This paper introduces the Seventh Dialog System Technology Challenges (D...

Personalization in Goal-Oriented Dialog

The main goal of modeling human conversation is to create agents which c...

Multi-Grained Knowledge Retrieval for End-to-End Task-Oriented Dialog

Retrieving proper domain knowledge from an external database lies at the...

Joint System-Wise Optimization for Pipeline Goal-Oriented Dialog System

Recent work (Takanobu et al., 2020) proposed the system-wise evaluation ...

SimBricks: End-to-End Network System Evaluation with Modular Simulation

Full system "end-to-end" measurements in physical testbeds are the gold ...

Please sign up or login with your details

Forgot password? Click here to reset