Recording and recreating interface navigation processes based on a whitelist generated by a machine-learned model

摘要

A computer system records and recreates an interface navigation process performed by a user with a host system. The computer system observes the user's interactions with the various UI elements during an interface navigation process by using a browser extension for the browser application in which the user is performing the interface navigation process. The browser extension then stores information about the interactions the user is performing and the UI elements that they are being performed on. The browser extension sends the stored interactions to the computer system, which processes the interactions to generate steps of the interface navigation recording. In some embodiments, the computer system also identifies one or more UI elements present in the final UI state of the interface navigation process as “verifiers” that can be used to determine whether the recreation of the interface navigation process is successful.