Simulation-Driven Automated End-to-End Test and Oracle Inference

by   Shreshth Tuli, et al.

This is the first work to report on inferential testing at scale in industry. Specifically, it reports the experience of automated testing of integrity systems at Meta. We built an internal tool called ALPACAS for automated inference of end-to-end integrity tests. Integrity tests are designed to keep users safe online by checking that interventions take place when harmful behaviour occurs on a platform. ALPACAS infers not only the test input, but also the oracle, by observing production interventions to prevent harmful behaviour. This approach allows Meta to automate the process of generating integrity tests for its platforms, such as Facebook and Instagram, which consist of hundreds of millions of lines of production code. We outline the design and deployment of ALPACAS, and report results for its coverage, number of tests produced at each stage of the test inference process, and their pass rates. Specifically, we demonstrate that using ALPACAS significantly improves coverage from a manual test design for the particular aspect of integrity end-to-end testing it was applied to. Further, from a pool of 3 million data points, ALPACAS automatically yields 39 production-ready end-to-end integrity tests. We also report that the ALPACAS-inferred test suite enjoys exceptionally low flakiness for end-to-end testing with its average in-production pass rate of 99.84


End-to-End Test Coverage Metrics in Microservice Systems: An Automated Approach

Microservice architecture gains momentum by fueling systems with cloud-n...

Production Monitoring to Improve Test Suites

Software testing ensures that a software system behaves as intended. In ...

Automated System Performance Testing at MongoDB

Distributed Systems Infrastructure (DSI) is MongoDB's framework for runn...

Benchmarks for End-to-End Microservices Testing

Testing microservice systems involves a large amount of planning and pro...

Bunk8s: Enabling Easy Integration Testing of Microservices in Kubernetes

Microservice architecture is the common choice for cloud applications th...

Gap-Measure Tests with Applications to Data Integrity Verification

In this paper we propose and examine gap statistics for assessing unifor...

Please sign up or login with your details

Forgot password? Click here to reset