image2emmet: Automatic code generation from web user interface image

摘要

Web development usually follows with analyzing the functionality, designing the userinterface (UI) prototype, implementing the UI by front-end (FE) developers andimplementing the REpresentational State Transfer (RESTful) application programminginterface (API) by back-end (BE) programmers. Unfortunately, web development is atedious, cumbersome, and time-consuming task, which makes it a challenge for theFE programmers to work in an efficient way. In this paper, we propose an approach,image2emmet, to assist FE programmers in implementing the UI. First, we collectHyperText Markup Language, Cascading Style Sheets (HTML-CSS) dataset in anautomatic and efficient way. The HTML-CSS dataset used for model training consistsof HTML-CSS code and its display images. Second, the faster region-basedconvolutional neural network (CNN) (R-CNN) is utilized to detect the UI component.Finally, we build a model combining CNN and long short-term memory (LSTM) totransform the UI component into the HTML-CSS code. The empirical studydemonstrates that image2emmet can achieve a precision of 80% on the UIcomponent detection and 60% on the transformation of UI component into HTMLCSScode.