CN112379878A - Multi-label learning-based UI element Web code generation method - Google Patents
Multi-label learning-based UI element Web code generation method Download PDFInfo
- Publication number
- CN112379878A CN112379878A CN202011131833.3A CN202011131833A CN112379878A CN 112379878 A CN112379878 A CN 112379878A CN 202011131833 A CN202011131833 A CN 202011131833A CN 112379878 A CN112379878 A CN 112379878A
- Authority
- CN
- China
- Prior art keywords
- tag
- picture
- attrb
- classifier
- attra
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 45
- 239000013598 vector Substances 0.000 claims abstract description 78
- 238000012549 training Methods 0.000 claims abstract description 18
- 238000007781 pre-processing Methods 0.000 claims abstract description 7
- 230000000694 effects Effects 0.000 claims description 7
- 238000010586 diagram Methods 0.000 claims description 6
- 238000004140 cleaning Methods 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 3
- 230000009193 crawling Effects 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- 238000010606 normalization Methods 0.000 claims description 3
- 238000012545 processing Methods 0.000 claims description 3
- 238000000605 extraction Methods 0.000 claims description 2
- 238000011161 development Methods 0.000 abstract description 8
- 238000013461 design Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- BUGBHKTXTAQXES-UHFFFAOYSA-N Selenium Chemical compound [Se] BUGBHKTXTAQXES-UHFFFAOYSA-N 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000006243 chemical reaction Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 239000002360 explosive Substances 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000010801 machine learning Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
- 229910052711 selenium Inorganic materials 0.000 description 1
- 239000011669 selenium Substances 0.000 description 1
- 238000004513 sizing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/38—Creation or generation of source code for implementing user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/958—Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/33—Intelligent editors
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Image Analysis (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The invention discloses a Web code generation method of UI elements based on multi-label learning, which comprises the following steps: acquiring a data set of UI element pictures and codes thereof; preprocessing each UI element picture to obtain a processed picture; extracting the features of the preprocessed picture to obtain a feature vector v; training an HTML label classifier, an HTML attribute classifier, n 'classifiers corresponding to discrete CSS attribute values and m' continuous CSS attribute value regressors based on the vector v, predicting new UI element pictures by using the classifiers and the regressors, and carrying out code assembly on predicted results to obtain codes corresponding to the pictures. The method can obtain the generation model M from the UI element picture to the Web code, can generate the corresponding Web code by inputting the UI element picture to be processed to the M, has stronger universality and universality, can replace part of links in actual development, and ensures that the actual use cost is lower and the application field is wider.
Description
Technical Field
The invention belongs to the field of software development, and particularly relates to a multi-tag learning-based UI element Web code generation method.
Background
Due to the explosive growth of the scale and complexity of software products, it becomes more and more challenging to develop software quickly, and especially in the early stage of software development, designers design prototype drawings and implement the prototype drawings with codes, which is very heavy and inefficient. By researching the automatic generation of the software codes, developers can accelerate the development progress of the developers, realize the software functions more quickly and finally push out the software products of the developers quickly in the development process. Therefore, research into automatic software code generation is becoming increasingly important.
At present, there are many technologies for automatic code generation, including freemaker, XSLT, velocity based on templates, MDA and MDD based on model driving, ORM and MVC based on object relationship mapping, Annotation and xdocket based on document Annotation, and AOP, PROXY and ASM based on PROXY dynamic classes, and these automatic code generation methods need some code work in the early stage, and only play a very local auxiliary role in the software development process, cannot replace a certain link of software development, and have a very limited effect on improving the software development speed. In addition, the automatic code generation methods need to be learned in the related fields first and then can be applied to actual development after mastering the methods, and the methods have high use threshold and are not beneficial to most developers, so the method has poor universality and cannot be applied to the actual development in a large scale.
Disclosure of Invention
The invention aims to provide a method for generating a Web code by using a UI element, which has the advantages of low cost, high accuracy and wide application range, aiming at the problems in the prior art.
The technical solution for realizing the purpose of the invention is as follows: a method for generating Web code of UI elements based on multi-tag learning, the method comprising the steps of:
step 1, acquiring a data set D of UI element pictures and codes thereof, wherein the data set D is { IMG, TAG, ATTRA, VALA, ATTRB and VALB }, wherein the IMG represents a picture set of UI elements, the TAG represents an HTML label set of UI element Web codes, the ATTRA is a discrete attribute set in which values in CSS codes in the UI element Web codes are discrete, the ATTRA and the VALA are in one-to-one correspondence, the ATTRB is a continuous attribute set in which the values in the CSS codes in the UI element Web codes are continuous, the VALB is a continuous attribute set in which the values in the CSS codes in the UI element Web codes are continuous, and the ATTRB and the VALB are in one-to-one correspondence;
step 2, IMG each UI element picture in IMG0Preprocessing to obtain a processed picture img;
step 3, extracting the characteristics of the preprocessed UI element picture img to obtain a characteristic vector v;
step 4, training an HTML label classifier tagclf based on the feature vector v; then, predicting an HTML label tag of a new UI element picture Web code by using the classifier;
step 5, training a classifier attrclf with HTML attributes based on the feature vector v, ATTRA and ATTRB; then, predicting a discrete attribute set ATTRA 'containing n' attributes and a continuous attribute set ATTRB 'containing m' attributes of the new UI element picture Web code by using the classifier;
step 6, aiming at n ' + m ' attributes obtained by the classifier attrclf, training n ' classifiers VALCLF (VALCLF) corresponding to the discrete CSS attribute values by using the feature vector v and ATTRA, VALA, ATTRB and VALB1,valclf2,…,valclfn'And m' continuous CSS attribute value regressors VALREG ═ reg [ reg ]1,reg2,…,regm'Predicting discrete attribute values VALA 'of n' CSS attributes and continuous attribute values VALB 'of m' CSS attributes of the new UI element picture Web code by using a classifier VALCLF and a classifier VALREG respectively;
and 7, carrying out code assembly on the tag, ATTRA ', ATTRB', VALA 'and VALB' to obtain a code corresponding to the new UI element picture.
Compared with the prior art, the invention has the following remarkable advantages: 1) fully excavating the representation of the original code, and learning a potential characteristic relation by using a machine learning algorithm; 2) the development process of front-end developers is met, and a great assisting effect can be provided for the developers; the mapping relation between the HTML element display effect and the codes is obtained and can be continuously expanded, the model effect of the method is finally strengthened, the accuracy of generating the codes by the UI element pictures is improved, and the application range is wider; 3) a conversion model from the UI element picture to the Web code is obtained, when the method is actually used, the final code result can be obtained only by providing corresponding UI element picture input, partial links in actual development can be replaced, and the method has the advantages of lower actual use cost and wider application field.
The present invention is described in further detail below with reference to the attached drawing figures.
Drawings
FIG. 1 is a flow chart of a method for Web code generation based on UI elements for multi-tag learning in accordance with the present invention.
Png is a Web design drawing demo.png of a UI element picture input in one embodiment.
Png is a UI element picture scaled for demo in one embodiment.
Png is a UI element picture normalized for demo in one embodiment.
Detailed Description
In order to make the objects, technical solutions and advantages of the present application more apparent, the present application is described in further detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
In one embodiment, in conjunction with fig. 1, there is provided a Web code generation method for UI elements based on multi-tag learning, the method comprising the steps of:
step 1, acquiring a data set D of UI element pictures and codes thereof, wherein the data set D is { IMG, TAG, ATTRA, VALA, ATTRB and VALB }, wherein the IMG represents a picture set of UI elements, the TAG represents an HTML label set of UI element Web codes, the ATTRA is a discrete attribute set in which values in CSS codes in the UI element Web codes are discrete, the ATTRA and the VALA are in one-to-one correspondence, the ATTRB is a continuous attribute set in which the values in the CSS codes in the UI element Web codes are continuous, the VALB is a continuous attribute set in which the values in the CSS codes in the UI element Web codes are continuous, and the ATTRB and the VALB are in one-to-one correspondence;
step 2, IMG each UI element picture in IMG0Preprocessing to obtain a processed picture img;
step 3, extracting the characteristics of the preprocessed UI element picture img to obtain a characteristic vector v;
step 4, training an HTML label classifier tagclf based on the feature vector v; then, predicting an HTML label tag of a new UI element picture Web code by using the classifier;
step 5, training a classifier attrclf with HTML attributes based on the feature vector v, ATTRA and ATTRB; then, predicting a discrete attribute set ATTRA 'containing n' attributes and a continuous attribute set ATTRB 'containing m' attributes of the new UI element picture Web code by using the classifier;
step 6, aiming at n ' + m ' attributes obtained by the classifier attrclf, training n ' classifiers VALCLF (VALCLF) corresponding to the discrete CSS attribute values by using the feature vector v and ATTRA, VALA, ATTRB and VALB1,valclf2,…,valclfn'And m' continuous CSS attribute value regressors VALREG ═ reg [ reg ]1,reg2,…,regm'Predicting discrete attribute values VALA 'of n' CSS attributes and continuous attribute values VALB 'of m' CSS attributes of the new UI element picture Web code by using a classifier VALCLF and a classifier VALREG respectively;
and 7, carrying out code assembly on the tag, ATTRA ', ATTRB', VALA 'and VALB' to obtain a code corresponding to the new UI element picture.
Further, in one embodiment, the step 1 of acquiring the data set D of the UI element picture and the code thereof includes:
step 1-1, crawling CSS codes of UI elements in a webpage and screenshots of display effects of the UI elements by using a Web automation tool;
and step 1-2, data cleaning is carried out on the data crawled in the step 1-1, and pictures with only one color and corresponding codes are filtered.
Further, in one embodiment, the step 2 of the IMG is performed for each UI element in the IMG0Preprocessing is carried out to obtain a processed picture img, and the specific process comprises the following steps:
step 2-1, img picture UI element0Zooming is carried out; suppose img0Is of width0×height0，width0、height0Respectively indicate width and height, the scaled size is width × height, and the scaling ratio is
Step 2-2, carrying out standardization processing on the zoomed picture, wherein the formula is as follows:
where x represents each pixel value in the scaled picture,
represents the mean of all pixels of the scaled picture, σ represents the standard deviation of all pixels of the scaled picture, and x' represents the pixel value after x normalization.
Further, in one embodiment, the step 3 of extracting features of the preprocessed UI element picture img to obtain a feature vector v includes:
step 3-1, calculating a histogram of the UI element picture img to obtain a feature vector vhist(each element in the vector represents the number of a certain pixel value);
step 3-2, expressing the vector of the UI element picture img RGB three channels as vr,vg,vb；
Step 3-3, calculating a convolution characteristic diagram of the UI element picture img by utilizing convolution operation, and converting the characteristic diagram into a one-dimensional vector vconv；
Step 3-4, constructing a scaling vector vscale＝[scalewidth,scaleheight]T；
And 3-5, splicing all vectors from the step 3-1 to the step 3-4 to obtain a final characteristic vector v.
Further, in one embodiment, in step 4, training an HTML tag classifier tagclf based on the feature vector v, and then predicting an HTML tag of a new UI element picture Web code by using the classifier includes:
step 4-1, TAG for each element in TAGiEncoding, i.e. on tagiNumbering to make the coded value at [0, ntag-1]Wherein n istagIs the number of elements in TAG, followed by ntagDigital construction vector labeltag；
Step 4-2, utilizing the feature vector v and the vector labeltagLearning the SVM to obtain an HTML tag classifier tagclf;
and 4-3, executing the step 2 and the step 3 on the new UI element picture, inputting the obtained feature vector into an HTML tag classifier tagclf, and obtaining an HTML tag of the Web code of the new UI element picture.
Further, in one embodiment, the training step 5 includes training a classifier attrclf of HTML attributes based on the feature vector v, atra, and ATTRB, and then predicting a discrete attribute set ATTRA 'containing n' attributes and a continuous attribute set ATTRB 'containing m' attributes of the new UI element picture Web code by using the classifier, where the discrete attribute set attrclf includes:
step 5-1, carrying out ONE-HOT coding on each element in ATTRA and ATTRB to obtain a vector labelattriBuilding a sparse matrix Labelattr；
Step 5-2, adopting an OneVsRest strategy to apply the feature vector v and the LabelattrInputting the HTML attribute into an SVM for learning, and obtaining a classifier attrclf of the HTML attribute;
step 5-3, executing step 2 and step 3 on the new UI element picture, and inputting the obtained feature vector into HTML attributeA classifier attrclf obtains a discrete attribute set ATTRA ' of n ' attributes { ATTRA '1,attra'2,…,attra'i,…,attra'n'And a consecutive attribute set ATTRB ' containing m ' attributes '1,attrb'2,…,attrb'i,…,attrb'm'}。
Further, in one embodiment, the process of step 6 specifically includes:
step 6-1, training and obtaining n' classifiers VALCLF (VALCLF) corresponding to discrete CSS attribute values by using the feature vector v, ATTRA, VALA, ATTRB and VALB learning SVM1,valclf2,…,valclfn'}；
Step 6-2, learning SVM by using the feature vector v and ATTRA, VALA, ATTRB, VALB, m' continuous CSS attribute value regressors VALREG ═ reg1,reg2,…,regm'}；
Step 6-3, executing step 2 and step 3 on the new UI element picture, inputting the obtained feature vector to the classifier VALCLF, and obtaining the discrete attribute value VALA '═ VALA' of the n 'CSS attributes of the new UI element picture Web code'1,vala'2,…,vala'n'}; the obtained feature vector is input to the classifier VALREG, and the continuous attribute value VALB '({ VALB'1,valb'2,…,valb'm'}。
Further, in one embodiment, in step 7, the code assembly is performed on the tag, atra ', ATTRB', VALA ', and VALB', and the obtained code corresponding to the new UI element picture is:
as a specific example, in one of the embodiments, the invention is further described. The method for generating the Web code of the UI element based on the multi-tag learning comprises the following steps:
1. acquiring a data set D of a UI element picture and a code thereof, wherein the data set D is { IMG, TAG, ATTRA, VALA, ATTRB and VALB }, wherein the IMG represents a picture set of a UI element, the TAG represents an HTML label set of a UI element Web code, the ATTRA is a discrete attribute set in which values in CSS codes in the UI element Web code are discrete attribute sets, the ATTRA and the VALA are in one-to-one correspondence, the ATTRB is a continuous attribute set in which values in CSS codes in the UI element Web code are continuous attribute sets, the VALB is a continuous attribute set in which values in CSS codes in the UI element Web code are continuous attribute sets, and the ATTRB and the VALB are in one-to one correspondence; specifically, the method comprises the following steps:
1-1, crawling CSS codes of UI elements in webpages and screenshots of display effects of the UI elements on some mainstream websites by using a Web automation tool (Selenium);
and 1-2, performing data cleaning on the data crawled in the step 1-1, and filtering out pictures with only one color and corresponding codes.
Currently known UI element tags are: a. abrr, acronym, address, applet, area, article, side, audio, b, base, basefont, bdi, bdo, big, blockquot, body, br, button, cans, capture, center, cite, code, col, collegroup, data, datalist, dd, del, detail, dfn, dialog, dir, div, dl, dt, em, bed, set, firmware, facility, font, fomer, form, frame, h1, h2, h3, h4, h5, h6, head, header, html, script, input, mark.
In this embodiment, TAG is a, abbr, acronym, address, applet, area, feature, side, audio, b, base, basefont, bdi, bdo, big, blockquote, body, br, button, cans, title, center, cite, code, col, conglomeratoup, data, datalist, dd, del, detail, dfn, dialog, dir, div, dl, dt, embed, fielder, firmware, font, fomer, form, frame, h1, h2, h3, h4, h5, h6, head, header, html, flag.
2. Picture IMG for each UI element in IMG0Preprocessing is carried out to obtain a processed picture img, and the method specifically comprises the following steps:
2-1, picture img for UI element0Zooming is carried out; suppose img0Is of width0×height0，width0、height0Respectively indicate width and height, the scaled size is width × height, and the scaling ratio is
2-2, carrying out standardization processing on the zoomed picture, wherein the formula is as follows:
where x represents each pixel value in the scaled picture,
represents the mean of all pixels of the scaled picture, σ represents the standard deviation of all pixels of the scaled picture, and x' represents the pixel value after x normalization.
In this embodiment, a sample d is selected from the IMG0FIG. 2 is a UI element picture img0Tag is button. Here width0＝108，height0＝44，width＝64，height＝64，
Fig. 3 shows the zoomed picture, and fig. 4 shows the normalized picture img.
3. Performing feature extraction on the preprocessed UI element picture img to obtain a feature vector v, which specifically comprises the following steps:
step 3-1, calculating a histogram of the UI element picture img to obtain a feature vector vhist；
Step 3-2, expressing the vector of the UI element picture img RGB three channels as vr,vg,vb；
Step 3-3, calculating a convolution characteristic diagram of the UI element picture img by utilizing convolution operation, and converting the characteristic diagram into a one-dimensional vector vconv；
Step 3-4, constructing a scaling vector vscale＝[scalewidth,scaleheight]T；
And 3-5, splicing all vectors from the step 3-1 to the step 3-4 to obtain a final characteristic vector v.
In this embodiment, each feature vector is:
vhist＝[4,0,1,…,0,0,4096]T
vr＝[112,112,112,…,114,112,112]T
vg＝[114,114,114,…,110,114,114]T
vb＝[112,112,112,…,88,112,112]T
vconv＝[38.008476,30.570044,46.95743,…,62.427425,51.65786,56.05508]T
vscale＝[1.6875,0.6875]T
finally obtaining a characteristic vector v ═ vhist|vr|vg|vb|vconv|vscale]。
4. Based on the feature vector v, training an HTML tag classifier tagclf, and then predicting an HTML tag of a new UI element picture Web code by using the classifier, wherein the specific process comprises the following steps:
4-1, TAG for each element in TAGiEncoding, i.e. on tagiNumbering to make the coded value at [0, ntag-1]Wherein n istagIs the number of elements in TAG, followed by ntagDigital construction vector labeltag；
4-2, using feature vector v and vector labeltagLearning the SVM to obtain an HTML tag classifier tagclf;
and 4-3, executing the step 2 and the step 3 on the new UI element picture, inputting the obtained feature vector into an HTML tag classifier tagclf, and obtaining an HTML tag of the Web code of the new UI element picture.
In this example, tagi＝i-1，labeltag＝[0,1,…,121]Tagclf can be derived from sklean.
5. Training a classifier attrclf of HTML attributes based on the feature vector v, ATTRA and ATTRB, and then predicting a discrete attribute set ATTRA 'containing n' attributes and a continuous attribute set ATTRB 'containing m' attributes of a new UI element picture Web code by using the classifier, wherein the specific process comprises the following steps:
5-1, carrying out ONE-HOT coding on each element in ATTRA and ATTRB to obtain a vector labelattriBuilding a sparse matrix Labelattr；
5-2, adopting OneVsRest strategy to combine the above feature vector v and LabelattrInputting the HTML attribute into an SVM for learning, and obtaining a classifier attrclf of the HTML attribute;
5-3, executing step 2 and step 3 on the new UI element picture, inputting the obtained feature vector into a classifier attrclf of HTML attributes, and obtaining a discrete attribute set ATTRA ' containing n ' attributes, { ATTRA '1,attra'2,…,attra'i,…,attra'n'And a consecutive attribute set ATTRB ' containing m ' attributes '1,attrb'2,…,attrb'i,…,attrb'm'}。
In this embodiment, ATTRA and ATTRB are align-items、appearance、background-attachment、background-clip、background-color、background-image、background-origin、background-position-x、background-position-y、background-repeat-x、background-repeat-y、background-size、border-bottom-color、border-bottom-left-radius、border-bottom-right-radius、border-bottom-style、border-bottom-width、border-image-outset、border-image-repeat、border-image-slice、border-image-source、border-image-width、border-left-color、border-left-style、border-left-width、border-right-color、border-right-style、border-right-width、border-top-color、border-top-left-radius、border-top-right-radius、border-top-style、border-top-width、box-shadow、box-sizing、color、cursor、display、font-family、font-size、font-stretch、font-style、font-variant-caps、font-variant-east-asian、font-variant-ligatures、font-variant-numeric、font-weight、height、letter-spacing、line-height、margin-bottom、margin-left、margin-right、margin-top、outline-color、outline-style、outline-width、padding-bottom、padding-left、padding-right、padding-top、text-align、text-indent、text-rendering、text-shadow、text-transform、user-select、white-space、width、word-spacing、writing-mode，
attrclf can be derived from sklern multiclass onevsrestclassfier.
6. For n ' + m ' attributes obtained by the classifier attrclf, training n ' classifiers VALCLF corresponding to discrete CSS attribute values by using the feature vector v and ATTRA, VALA, ATTRB, VALB1,valclf2,…,valclfn'And m' continuous CSS attribute value regressors VALREG ═ reg [ reg ]1,reg2,…,regm'Predicting discrete attribute values VALA 'of n' CSS attributes and continuous attribute values VALB 'of m' CSS attributes of the new UI element picture Web code by using a classifier VALCLF and a classifier VALREG respectively; the method specifically comprises the following steps:
step 6-1, benefitTraining the classifier VALCLF (VALCLF) for obtaining n' corresponding discrete CSS attribute values by using the feature vector v and ATTRA, VALA, ATTRB and VALB learning SVM1,valclf2,…,valclfn'}；
Step 6-2, learning SVM by using the feature vector v and ATTRA, VALA, ATTRB, VALB, m' continuous CSS attribute value regressors VALREG ═ reg1,reg2,…,regm'}；
Step 6-3, executing step 2 and step 3 on the new UI element picture, inputting the obtained feature vector to the classifier VALCLF, and obtaining the discrete attribute value VALA '═ VALA' of the n 'CSS attributes of the new UI element picture Web code'1,vala'2,…,vala'n'}; the obtained feature vector is input to the classifier VALREG, and the continuous attribute value VALB '({ VALB'1,valb'2,…,valb'm'}。
In this embodiment, n' classifiers can be derived from sklean.
7. Performing code assembly on the tag, ATTRA ', ATTRB', VALA 'and VALB', and obtaining a code corresponding to the new UI element picture, wherein the code is as follows:
in this embodiment, the specific splicing code is:
the method can obtain the generation model M from the UI element picture to the Web code, can generate the corresponding Web code by inputting the UI element picture to be processed to the M, has stronger universality and universality, can replace part of links in actual development, and ensures that the actual use cost is lower and the application field is wider.
The foregoing illustrates and describes the principles, general features, and advantages of the present invention. It will be understood by those skilled in the art that the present invention is not limited to the embodiments described above, which are described in the specification and illustrated only to illustrate the principle of the present invention, but that various changes and modifications may be made therein without departing from the spirit and scope of the present invention, which fall within the scope of the invention as claimed. The scope of the invention is defined by the appended claims and equivalents thereof.
Claims (8)
1. A method for generating Web code of UI elements based on multi-tag learning, the method comprising the steps of:
step 1, acquiring a data set D of UI element pictures and codes thereof, wherein the data set D is { IMG, TAG, ATTRA, VALA, ATTRB and VALB }, wherein the IMG represents a picture set of UI elements, the TAG represents an HTML label set of UI element Web codes, the ATTRA is a discrete attribute set in which values in CSS codes in the UI element Web codes are discrete, the ATTRA and the VALA are in one-to-one correspondence, the ATTRB is a continuous attribute set in which the values in the CSS codes in the UI element Web codes are continuous, the VALB is a continuous attribute set in which the values in the CSS codes in the UI element Web codes are continuous, and the ATTRB and the VALB are in one-to-one correspondence;
step 2, IMG each UI element picture in IMG0Preprocessing to obtain a processed picture img;
step 3, extracting the characteristics of the preprocessed UI element picture img to obtain a characteristic vector v;
step 4, training an HTML label classifier tagclf based on the feature vector v; then, predicting an HTML label tag of a new UI element picture Web code by using the classifier;
step 5, training a classifier attrclf with HTML attributes based on the feature vector v, ATTRA and ATTRB; then, predicting a discrete attribute set ATTRA 'containing n' attributes and a continuous attribute set ATTRB 'containing m' attributes of the new UI element picture Web code by using the classifier;
step 6, aiming at n ' + m ' attributes obtained by the classifier attrclf, training n ' classifiers VALCLF (VALCLF) corresponding to the discrete CSS attribute values by using the feature vector v and ATTRA, VALA, ATTRB and VALB1,valclf2,…,valclfn'And m' continuous CSS attribute value regressors VALREG ═ reg [ reg ]1,reg2,…,regm'Predicting discrete attribute values VALA 'of n' CSS attributes and continuous attribute values VALB 'of m' CSS attributes of the new UI element picture Web code by using a classifier VALCLF and a regressor VALREG respectively;
and 7, carrying out code assembly on the tag, ATTRA ', ATTRB', VALA 'and VALB' to obtain a code corresponding to the new UI element picture.
2. The method for generating the Web code of the UI element based on multi-tag learning according to claim 1, wherein the step 1 of obtaining the data set D of the UI element picture and the code thereof comprises the following specific steps:
step 1-1, crawling CSS codes of UI elements in a webpage and screenshots of display effects of the UI elements by using a Web automation tool;
and step 1-2, data cleaning is carried out on the data crawled in the step 1-1, and pictures with only one color and corresponding codes are filtered.
3. The method for generating Web code of UI element based on multi-tag learning according to claim 2 wherein step 2 is that IMG is photographed for each UI element in IMG0Preprocessing is carried out to obtain a processed picture img, and the specific process comprises the following steps:
step 2-1, img picture UI element0Zooming is carried out; suppose img0Is of width0×height0，width0、height0Respectively representing width, height, zoomThe latter size is width × height, then the scaling is
Step 2-2, carrying out standardization processing on the zoomed picture, wherein the formula is as follows:
where x represents each pixel value in the scaled picture,
represents the mean of all pixels of the scaled picture, σ represents the standard deviation of all pixels of the scaled picture, and x' represents the pixel value after x normalization.
4. The method for generating the Web code of the UI element based on the multi-tag learning of claim 3, wherein the step 3 is to perform feature extraction on the preprocessed UI element picture img to obtain a feature vector v, and the specific process includes:
step 3-1, calculating a histogram of the UI element picture img to obtain a feature vector vhist；
Step 3-2, expressing the vector of the UI element picture img RGB three channels as vr,vg,vb；
Step 3-3, calculating a convolution characteristic diagram of the UI element picture img by utilizing convolution operation, and converting the characteristic diagram into a one-dimensional vector vconv；
Step 3-4, constructing a scaling vector vscale＝[scalewidth,scaleheight]T；
And 3-5, splicing all vectors from the step 3-1 to the step 3-4 to obtain a final characteristic vector v.
5. The method for generating a Web code of a UI element based on multi-tag learning according to claim 4, wherein the step 4 trains an HTML tag classifier tagclf based on the feature vector v, and then predicts an HTML tag of a new UI element picture Web code by using the classifier, and the specific process includes:
step 4-1, TAG for each element in TAGiEncoding, i.e. on tagiNumbering to make the coded value at [0, ntag-1]Wherein n istagIs the number of elements in TAG, followed by ntagDigital construction vector labeltag；
Step 4-2, utilizing the feature vector v and the vector labeltagLearning the SVM to obtain an HTML tag classifier tagclf;
and 4-3, executing the step 2 and the step 3 on the new UI element picture, inputting the obtained feature vector into an HTML tag classifier tagclf, and obtaining an HTML tag of the Web code of the new UI element picture.
6. The method for generating a Web code of a UI element based on multi-tag learning according to claim 5, wherein the step 5 trains a classifier attrclf of HTML attributes based on the feature vector v, atra, ATTRB, and then predicts a discrete attribute set ATTRA 'containing n' attributes and a continuous attribute set ATTRB 'containing m' attributes of a new UI element picture Web code by using the classifier, and the specific process includes:
step 5-1, carrying out ONE-HOT coding on each element in ATTRA and ATTRB to obtain a vector labelattriBuilding a sparse matrix Labelattr；
Step 5-2, adopting an OneVsRest strategy to apply the feature vector v and the LabelattrInputting the HTML attribute into an SVM for learning, and obtaining a classifier attrclf of the HTML attribute;
and 5-3, executing the step 2 and the step 3 on the new UI element picture, inputting the obtained feature vector into a classifier attrclf of HTML attributes, and obtaining a discrete attribute set ATTRA ' containing n ' attributes, { ATTRA '1,attra'2,…,attra'i,…,attra'n'And a contiguous attribute set ATTRB containing m' attributes'＝{attrb'1,attrb'2,…,attrb'i,…,attrb'm'}。
7. The method for generating a Web code of a UI element based on multi-tag learning according to claim 6, wherein the process of step 6 specifically comprises:
step 6-1, training and obtaining n' classifiers VALCLF (VALCLF) corresponding to discrete CSS attribute values by using the feature vector v, ATTRA, VALA, ATTRB and VALB learning SVM1,valclf2,…,valclfn'}；
Step 6-2, learning SVM by using the feature vector v and ATTRA, VALA, ATTRB, VALB, m' continuous CSS attribute value regressors VALREG ═ reg1,reg2,…,regm'}；
Step 6-3, executing step 2 and step 3 on the new UI element picture, inputting the obtained feature vector to the classifier VALCLF, and obtaining the discrete attribute value VALA '═ VALA' of the n 'CSS attributes of the new UI element picture Web code'1,vala'2,…,vala'n'}; the obtained feature vector is input to the classifier VALREG, and the continuous attribute value VALB '({ VALB'1,valb'2,…,valb'm'}。
8. The method for generating a Web code of a UI element based on multi-tag learning according to claim 7, wherein the step 7 is to assemble the code of tag, atra ', ATTRB', VALA 'and VALB', and obtain the code corresponding to the new UI element picture as follows:
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011131833.3A CN112379878B (en) | 2020-10-21 | 2020-10-21 | Web code generation method of UI element based on multi-label learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011131833.3A CN112379878B (en) | 2020-10-21 | 2020-10-21 | Web code generation method of UI element based on multi-label learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112379878A true CN112379878A (en) | 2021-02-19 |
| CN112379878B CN112379878B (en) | 2024-03-26 |
Family
ID=74580430
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011131833.3A Active CN112379878B (en) | 2020-10-21 | 2020-10-21 | Web code generation method of UI element based on multi-label learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112379878B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342416A (en) * | 2021-06-09 | 2021-09-03 | 上海万物新生环保科技集团有限公司 | Method and equipment for displaying H5 graphic data in applet |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130151951A1 (en) * | 2011-12-08 | 2013-06-13 | Microsoft Corporation | Generating css shorthand properties |
| US20160140451A1 (en) * | 2014-11-17 | 2016-05-19 | Yahoo! Inc. | System and method for large-scale multi-label learning using incomplete label assignments |
| CN110377282A (en) * | 2019-06-26 | 2019-10-25 | 扬州大学 | The method for generating Web code based on the UI for generating confrontation and convolutional neural networks |
| CN110968299A (en) * | 2019-11-20 | 2020-04-07 | 北京工业大学 | Front-end engineering code generation method based on hand-drawn webpage image |
-
2020
- 2020-10-21 CN CN202011131833.3A patent/CN112379878B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20130151951A1 (en) * | 2011-12-08 | 2013-06-13 | Microsoft Corporation | Generating css shorthand properties |
| US20160140451A1 (en) * | 2014-11-17 | 2016-05-19 | Yahoo! Inc. | System and method for large-scale multi-label learning using incomplete label assignments |
| CN110377282A (en) * | 2019-06-26 | 2019-10-25 | 扬州大学 | The method for generating Web code based on the UI for generating confrontation and convolutional neural networks |
| CN110968299A (en) * | 2019-11-20 | 2020-04-07 | 北京工业大学 | Front-end engineering code generation method based on hand-drawn webpage image |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113342416A (en) * | 2021-06-09 | 2021-09-03 | 上海万物新生环保科技集团有限公司 | Method and equipment for displaying H5 graphic data in applet |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112379878B (en) | 2024-03-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108256562B (en) | Salient target detection method and system based on weak supervision time-space cascade neural network | |
| CN111666994A (en) | Sample image data enhancement method and device, electronic equipment and storage medium | |
| CN106777011A (en) | A kind of file classification method based on depth multi-task learning | |
| US11599727B2 (en) | Intelligent text cleaning method and apparatus, and computer-readable storage medium | |
| WO2021098689A1 (en) | Text recognition method for natural scene, storage apparatus, and computer device | |
| CN109408058B (en) | Front-end auxiliary development method and device based on machine learning | |
| Li et al. | Attention guided global enhancement and local refinement network for semantic segmentation | |
| CN111208998A (en) | Method and device for automatically laying out data visualization large screen and storage medium | |
| CN116311279A (en) | Sample image generation, model training and character recognition methods, equipment and media | |
| Xu et al. | RGB-T salient object detection via CNN feature and result saliency map fusion | |
| CN112379878A (en) | Multi-label learning-based UI element Web code generation method | |
| Lu et al. | DCACNet: Dual context aggregation and attention-guided cross deconvolution network for medical image segmentation | |
| CN112037239A (en) | Text guidance image segmentation method based on multi-level explicit relation selection | |
| CN110458162B (en) | Method for intelligently extracting image text information | |
| CN111488953A (en) | Method for rapidly classifying webpage topics based on HTM L source code characteristics | |
| CN114663665A (en) | Gradient-based confrontation sample generation method and system | |
| CN113139544A (en) | Saliency target detection method based on multi-scale feature dynamic fusion | |
| CN116977624A (en) | Target identification method, system, electronic equipment and medium based on YOLOv7 model | |
| Wang et al. | Multi‐level feature fusion network for crowd counting | |
| Memon et al. | Glyph identification and character recognition for Sindhi OCR | |
| CN112732259B (en) | Front-end interactive page conversion method, device and medium based on artificial intelligence | |
| Li et al. | MFEAFN: Multi-scale feature enhanced adaptive fusion network for image semantic segmentation | |
| Wang | Packaging style design based on visual semantic segmentation technology and intelligent cyber physical system | |
| CN115147850B (en) | Training method of character generation model, character generation method and device thereof | |
| WO2024103997A1 (en) | Handwriting recognition method and handwriting recognition model training method and apparatus |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |