CN112732259A - Front-end interactive page conversion method, device and medium based on artificial intelligence - Google Patents
Front-end interactive page conversion method, device and medium based on artificial intelligence Download PDFInfo
- Publication number
- CN112732259A CN112732259A CN202110034032.3A CN202110034032A CN112732259A CN 112732259 A CN112732259 A CN 112732259A CN 202110034032 A CN202110034032 A CN 202110034032A CN 112732259 A CN112732259 A CN 112732259A
- Authority
- CN
- China
- Prior art keywords
- model
- file
- training
- component
- html5
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000006243 chemical reaction Methods 0.000 title claims abstract description 32
- 230000002452 interceptive effect Effects 0.000 title claims abstract description 29
- 238000013473 artificial intelligence Methods 0.000 title claims abstract description 26
- 238000013145 classification model Methods 0.000 claims abstract description 20
- 238000013461 design Methods 0.000 claims abstract description 14
- 238000010801 machine learning Methods 0.000 claims abstract description 7
- 238000012549 training Methods 0.000 claims description 56
- 238000010586 diagram Methods 0.000 claims description 13
- 238000007781 pre-processing Methods 0.000 claims description 13
- 238000002372 labelling Methods 0.000 claims description 10
- 238000004590 computer program Methods 0.000 claims description 8
- 238000011156 evaluation Methods 0.000 claims description 4
- 230000003993 interaction Effects 0.000 claims description 3
- 230000001105 regulatory effect Effects 0.000 claims description 3
- 238000010276 construction Methods 0.000 claims description 2
- 230000002708 enhancing effect Effects 0.000 claims description 2
- 230000009466 transformation Effects 0.000 claims description 2
- 230000004807 localization Effects 0.000 claims 2
- 238000011161 development Methods 0.000 abstract description 16
- 238000005516 engineering process Methods 0.000 abstract description 3
- 230000009286 beneficial effect Effects 0.000 abstract description 2
- 230000006870 function Effects 0.000 description 6
- 238000001514 detection method Methods 0.000 description 4
- 238000013528 artificial neural network Methods 0.000 description 2
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000002035 prolonged effect Effects 0.000 description 1
- 238000013526 transfer learning Methods 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F8/00—Arrangements for software engineering
- G06F8/30—Creation or generation of source code
- G06F8/38—Creation or generation of source code for implementing user interfaces
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/22—Image preprocessing by selection of a specific region containing or referencing a pattern; Locating or processing of specific regions to guide the detection or recognition
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V30/00—Character recognition; Recognising digital ink; Document-oriented image-based pattern recognition
- G06V30/10—Character recognition
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Software Systems (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Biomedical Technology (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- Multimedia (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Character Discrimination (AREA)
Abstract
The invention relates to a front-end interactive page conversion method, a device and a medium based on artificial intelligence, which comprises the following steps: performing machine learning on a plurality of component elements of HTML5 to obtain a classification model of an HTML5 component; performing machine learning on the position of the component and the printing font to obtain an OCR character recognition model; acquiring parameter information of a vector graphic to be converted, converting the vector graphic into an HTML5 definition file according to the parameter information, and further generating a plurality of basic components of HTML5 according to the HTML5 definition file; and selecting the basic component with the highest similarity for filling, carrying out recognition based on the OCR character recognition model on the filled image area, and adding recognition character information of the OCR character recognition model in the basic component. The invention has the beneficial effects that: the artificial intelligence image classification technology is introduced into the front-end interface development, so that the coding work of developers can be reduced, the conversion efficiency of the front-end interface from the UI design drawing to the final finished product HTML5 file is improved, and the front-end development becomes efficient.
Description
Technical Field
The invention relates to the field of computers, in particular to a front-end interactive page conversion method, a front-end interactive page conversion device and a front-end interactive page conversion medium based on artificial intelligence.
Background
At present, various services of banks all adopt HTML5 as a main human-computer interaction interface, including a counter system used inside, a monitoring system, various management applications, a mobile phone bank used outside and self-service equipment, so that the development demand of the HTML5 is increased sharply at present. Meanwhile, the requirement on user experience is higher and higher, a UI designer is often required to perform a large amount of design work first and then is realized by a technician, so that the development period is prolonged.
The traditional HTML5 front-end development schemes are all that UI designers use tools such as Sketch and the like to perform prototype design first, and then technicians perform manual coding development by contrasting design drawings or perform repeated engraving development by a visualization tool; the existing development modes require considerable development cost for converting prototype designs into practically usable HTML5 files, and the work is often repeated mechanical labor with low technical difficulty, so that the development cost of the whole project is increased.
Disclosure of Invention
The invention aims to solve at least one of the technical problems in the prior art, and provides a front-end interactive page conversion method, a device and a medium based on artificial intelligence, which aim to reduce the development cost and improve the development efficiency by an image classification technology of neural network deep learning.
The technical scheme of the invention comprises a front-end interactive page conversion method based on artificial intelligence, which is characterized by comprising the following steps: training a classification model, namely performing machine learning on a plurality of components of HTML5 based on image classification of a MobileNet model to obtain a classification model of an HTML5 component; training a character positioning model, and obtaining an OCR character positioning model based on the position of the CTPN model learning component; training a character recognition model, learning printed characters based on a DenseNet model, and obtaining an OCR character recognition model; a vector graphic file first conversion step, namely acquiring parameter information of a vector graphic to be converted, converting the vector graphic into an HTML5 definition file according to the parameter information, and further generating a plurality of first components of HTML5 according to the HTML5 definition file, wherein the first components comprise texts, pictures and vector diagrams; a second conversion step of the vector graphic file, namely calling a classification model for recognition according to the front-end interactive page image information to be displayed, matching a corresponding first component according to a recognition result, selecting the first component with the highest similarity for filling, recognizing a filled image area based on an OCR character recognition model, adding recognition character information of the OCR character recognition model to the first component, and obtaining a second component; wherein the vector graphics file is a Sketch file.
According to the artificial intelligence-based front-end interactive page conversion method, the training of the classification model comprises the following steps: data preprocessing, pre-training model training, data labeling, model fine tuning and model publishing; the data preprocessing comprises the steps of classifying ReDraw data sets according to component types, and uniformly scaling the picture size to set pixels in an equal proportion mode; wherein the pre-training model training comprises training the model using a plurality of ReDraw data sets based on a MobileNet model; the data labeling comprises correspondingly labeling a plurality of component data of HTML page data; the model fine adjustment comprises the steps of using a MobileNet model and loading a pre-training model for fine adjustment; model publishing includes publishing the model and providing HTTP services and providing a calling interface.
According to the front-end interactive page conversion method based on artificial intelligence, OCR model training comprises character positioning model training and character recognition model training, wherein the character positioning model training is realized through a CTPN algorithm, and the character recognition model training is realized through a DenseNet algorithm.
According to the artificial intelligence-based front-end interactive page conversion method, the training of the character positioning model comprises the following steps: the method comprises the steps of character positioning picture marking, image data preprocessing and data enhancing, CTPN model building, CTPN model training, CTPN model evaluation and CTPN model storage.
According to the artificial intelligence-based front-end interactive page conversion method, the training of the character recognition model comprises the following steps: the method comprises the steps of character recognition picture labeling, image data preprocessing and data enhancement, construction of a DenseNet model, training of the DenseNet model, evaluation of the DenseNet model and storage of the DenseNet model.
According to the method for converting the front-end interactive page based on the artificial intelligence, the first conversion step of the vector graphic file comprises the following steps: acquiring each layer information from the Sketch design file, specifically, reading the Sketch file and converting the Sketch file into a JSON file through a Sketch-file plug-in, wherein field picture information, document information and page information in the JSON file are used; converting the picture information, the document information and the page information of the JSON file into a Def file, wherein the Def file is a definition file; and converting the Def file into a VUE page file in an HTML5 standard format, wherein the VUE page comprises a text label, a picture label and a vector diagram label, and the module structure of the VUE page is kept consistent with that of the Sketch file.
According to the method for converting the front-end interactive page based on the artificial intelligence, the second conversion step of the vector graphic file comprises the following steps: s410, screenshot is conducted on the front-end interaction page, an interface area for obtaining the first component is selected, and an interface area image is obtained by combining the screenshot; s420, performing Base64 coding on the interface region image, calling a classification model to perform coding on the image, arranging the recognition results from high to low according to the similarity, and returning a plurality of candidate components; s430, replacing the corresponding basic component with the best matching component according to the best matching component selected by the user; s440, calling an OCR recognition model to perform character recognition on the Base64 code of the interface region image to obtain a character recognition result; and S450, adding a character recognition result to the assembly regulated in the S430 to obtain the second assembly.
According to the artificial intelligence based front-end interactive page conversion method, an HTML5 definition file is used for WEBIDE.
The technical scheme of the invention also comprises a front-end interactive page conversion device based on artificial intelligence, which comprises a memory, a processor and a computer program which is stored in the memory and can run on the processor, and is characterized in that any one of the method steps is realized when the processor executes the computer program.
The present invention also includes a computer-readable storage medium, in which a computer program is stored, wherein the computer program, when executed by a processor, implements any of the method steps.
The invention has the beneficial effects that: the artificial intelligence image classification technology is introduced into the front-end interface development, compared with the traditional front-end interface development process, the coding work of developers can be greatly reduced, and the conversion efficiency of the front-end interface from a UI design drawing to a final finished product HTML5 file is greatly improved through artificial intelligence auxiliary rib development, so that the front-end development becomes very efficient.
Drawings
The invention is further described below with reference to the accompanying drawings and examples;
FIG. 1 illustrates an overall flow diagram according to an embodiment of the invention;
FIG. 2 is a flow chart of a classification model training process according to an embodiment of the present invention;
FIG. 3 is a text positioning training process according to an embodiment of the present invention;
FIG. 4 is a flow diagram of OCR text recognition reasoning according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an interface definition file in JSON format according to an embodiment of the present invention;
FIG. 6 is a flow diagram of a transformation of a base component according to an embodiment of the present invention;
fig. 7 shows a schematic view of an apparatus according to an embodiment of the invention.
Detailed Description
Reference will now be made in detail to the present preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings, wherein like reference numerals refer to like elements throughout.
In the description of the present invention, the meaning of a plurality of means is one or more, the meaning of a plurality of means is two or more, and larger, smaller, larger, etc. are understood as excluding the number, and larger, smaller, inner, etc. are understood as including the number.
In the description of the present invention, the consecutive reference numbers of the method steps are for convenience of examination and understanding, and the implementation order between the steps is adjusted without affecting the technical effect achieved by the technical solution of the present invention by combining the whole technical solution of the present invention and the logical relationship between the steps.
In the description of the present invention, unless otherwise explicitly defined, terms such as set, etc. should be broadly construed, and those skilled in the art can reasonably determine the specific meanings of the above terms in the present invention in combination with the detailed contents of the technical solutions.
FIG. 1 shows a general flow diagram according to an embodiment of the invention, comprising the following steps: training a classification model, namely performing machine learning on a plurality of components of HTML5 based on image classification of a MobileNet model to obtain a classification model of an HTML5 component; training a character positioning model, and obtaining an OCR character positioning model based on the position of the CTPN model learning component; training a character recognition model, learning printed characters based on a DenseNet model, and obtaining an OCR character recognition model; a vector graphic file first conversion step, namely acquiring parameter information of a vector graphic to be converted, converting the vector graphic into an HTML5 definition file according to the parameter information, and further generating a plurality of first components of HTML5 according to the HTML5 definition file, wherein the first components comprise texts, pictures and vector diagrams; a second conversion step of the vector graphic file, namely calling a classification model for recognition according to the image information of the front-end interactive page to be displayed, matching a corresponding first component according to a recognition result, selecting the first component with the highest similarity for filling, performing recognition based on an OCR character recognition model on the filled image area, and adding the recognition character information of the OCR character recognition model to the first component to obtain a second component, wherein the second component is generally a functional component such as a button; wherein the vector graphics file is a Sketch file.
Specifically, a structuring preprocessing is carried out by importing the Sketch design file, the Sketch file is preliminarily converted into an HTML5 file, and the Sketch primitive is preliminarily converted into an HTML5 primitive label. A user selects and replaces the HTML5 primitive labels as corresponding HTML5 component labels according to the user requirements by image classification of the HTML5 primitive labels, and then tag texts are automatically added through OCR recognition. Thus forming a basic usable HTML5 page file.
FIG. 2 is a classification model training flow according to an embodiment of the present invention, which performs machine learning on various component elements of HTML5 based on image classification of a MobileNet model to form a classification model for an HTML5 component. MobileNet is a lightweight convolutional neural network based on deep separable convolution, compared with a standard convolutional network, the number of convolution parameters and the calculated amount can be greatly reduced, so that the inference speed is improved, and the precision is only lost by about 1%.
Aiming at the training process of various components of HTML5, the method comprises the following steps of data preprocessing, pre-training model training, data labeling, model fine tuning, model publishing and the like, and the specific flow is as follows:
ReDraw data preprocessing: the ReDraw data set is categorized by component category and the picture size is uniformly scaled to 224 pixels.
Training a pre-training model: because the data volume of HTML page scenes is small, the generalization capability of the model is improved by using a transfer learning idea, the model is trained by using a large-scale ReDraw data set based on a MobileNet model, and then the internal parameters of the model are transferred to a downstream task.
And (3) HTML page data labeling: the HTML page data is labeled, the current version is labeled with 29 types of components, and the data volume of each type is not less than 300.
HTML page data preprocessing: the picture size is uniformly scaled to 224 pixels.
Fine adjustment of a model: and (3) fine tuning by using a MobileNet model and loading a pre-training model, wherein after fine tuning, the precision of the model is as high as 99%.
Model release: publish the model and provide HTTP services to other module calls.
Model release: publish the model and provide HTTP services to other module calls.
Fig. 3 is a text positioning training process according to an embodiment of the present invention, a text positioning model is formed based on the position of the CTPN model learning component, and a text recognition model DenseNet training process is identical to the CTPN training process. The OCR character recognition model includes a character locating function and a character recognition function. The text positioning function is realized based on a CTPN algorithm, the algorithm is improved based on an image target detection algorithm, the text positioning is different from the object positioning, the text is composed of a series of characters, the detection target is not a closed area, an interval exists between every two characters, the text positioning has higher requirements on the precision, and the whole detection area needs to be covered. The CTPN improves the area suggestion network part in the target detection algorithm and is added with the context information of the circular neural network structure extracted text.
The character recognition function is realized based on a DenseNet algorithm, the basic idea of the DenseNet is consistent with that of a deep residual error network, but the establishment of the character recognition function is the dense connection of all the layers in the front and the layers behind, and the other characteristic is that the characteristic reuse is realized through the connection of the characteristic on a channel, and the specific formula is as follows:
wherein [ x ]0,xl,....,xl-1]Respectively representing the output of the convolutional network from layer 0 to layer l-1, HlRepresenting a characteristic combination function, xlRepresenting the newly generated convolution signature. These features may allow DenseNet to achieve better performance than the deep residual network with less parameter and computation cost.
FIG. 4 is a flow chart of OCR character recognition reasoning process according to the embodiment of the invention, wherein the OCR character recognition reasoning process comprises image data preprocessing, CTPN model loading and DenseNet model loading, component position recognition and component character recognition.
Fig. 5 is a schematic diagram of an interface definition file in a JSON format according to an embodiment of the present invention, and each item of information of the JSON file is defined in fig. 5.
Fig. 6 is a flowchart of the basic components for conversion according to the embodiment of the present invention, where the Sketch file is primarily converted into the HTML5 file, and the steps are as follows:
obtaining useful information of each Layer (i.e. elements in a Layer, i.e. a design drawing, such as text, pictures and the like) from the Sketch design file, such as the size of a drawing board (i.e. the size of a page), the position and width and height of the Layer, the type of the Layer and the like, so that the prototype conversion tool firstly reads the Sketch file into JSON format content through a Sketch-file plug-in, wherein the fields used in JSON mainly include (1) images: picture id and picture file data used in the Sketch file; (2) document: the whole document information comprises a current file, and contents such as a public layer, a text style, a layer style and the like introduced into an external resource library; (3) pages: all pages in the design drawing are JSON, and the JSON contains detailed information of each layer of the page, such as the width and the height of the layer, the size of the layer, a text style, a layer style and the like. Combining the information of the three parts, the useful information of each layer can be obtained for converting the useful information into the attribute of the component in the Def; for example, a picture assembly in a picture layer conversion editor in Sketch needs to take picture data from an image to write into a png picture file, refer to the picture in src of the picture assembly, find the picture layer from pages, obtain the width, height, top and left of the picture layer, and fill in width, height, top and left of the picture assembly.
Preliminarily converting each page in the Sketch design file into an interface definition file in a JSON format, wherein the JSON text format is as follows: the Def file is converted into a VUE page file in an HTML5 standard format, the VUE page is composed of basic components such as basic text labels, picture labels, vector diagram labels and the like, and the page module structure is consistent with the Sketch design module structure. Through the steps, the HTML file of the Sketch file preliminary conversion is obtained, and at the moment, all elements of the HTML are basic text, pictures, icons and the like without being signed, and do not have any components and attributes thereof.
Converting the basic components in the page file generated by the primary conversion, which comprises the following specific steps
S410, screenshot is conducted on an interface displayed by the html, the selected basic component is clicked to obtain an interface area (Xmin, Ymin, Xmax and Ymax) of the component, and an image of the area is obtained by combining the screenshot.
S420, performing Base64 coding on the regional image, calling a component model to recognize the image, arranging recognition results from high to low according to the similarity, and returning a plurality of candidate components.
S430, the user selects the best matching component, and the corresponding basic component replaces the selected module as the component
S440, calling an OCR recognition model, performing character recognition on the regional image Base64 codes, and returning the result
And S450, adding character attributes to the components regulated in the S430 to obtain a return result of the S440.
Fig. 7 shows a schematic view of an apparatus according to an embodiment of the invention. The apparatus comprises a memory 100 and a processor 200, wherein the processor 200 stores a computer program for performing: training a classification model, namely performing machine learning on a plurality of components of HTML5 based on image classification of a MobileNet model to obtain a classification model of an HTML5 component; training a character positioning model, and obtaining an OCR character positioning model based on the position of the CTPN model learning component; training a character recognition model, learning printed characters based on a DenseNet model, and obtaining an OCR character recognition model; a vector graphic file first conversion step, namely acquiring parameter information of a vector graphic to be converted, converting the vector graphic into an HTML5 definition file according to the parameter information, and further generating a plurality of first components of HTML5 according to the HTML5 definition file, wherein the first components comprise texts, pictures and vector diagrams; a second conversion step of the vector graphic file, namely calling a classification model for recognition according to the image information of the front-end interactive page to be displayed, matching a corresponding first component according to a recognition result, selecting the first component with the highest similarity for filling, performing recognition based on an OCR character recognition model on the filled image area, and adding the recognition character information of the OCR character recognition model to the first component to obtain a second component, wherein the second component is generally a functional component such as a button; wherein the vector graphics file is a Sketch file.
The embodiments of the present invention have been described in detail with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the gist of the present invention.
Claims (10)
1. A front-end interactive page conversion method based on artificial intelligence is characterized by comprising the following steps:
training a classification model, namely performing machine learning on a plurality of components of HTML5 based on image classification of a MobileNet model to obtain a classification model of an HTML5 component;
training a character positioning model, and obtaining an OCR character positioning model based on the position of the CTPN model learning component;
training a character recognition model, learning printed characters based on a DenseNet model, and obtaining an OCR character recognition model;
a vector graphic file first conversion step, namely acquiring parameter information of a vector graphic to be converted, converting the vector graphic into an HTML5 definition file according to the parameter information, and further generating a plurality of first components of HTML5 according to the HTML5 definition file, wherein the first components comprise texts, pictures and vector diagrams;
a second conversion step of the vector graphic file, namely calling a classification model for recognition according to the front-end interactive page image information to be displayed, matching a corresponding first component according to a recognition result, selecting the first component with the highest similarity for filling, recognizing a filled image area based on an OCR character recognition model, adding recognition character information of the OCR character recognition model to the first component, and obtaining a second component;
wherein the vector graphics file is a Sketch file.
2. The artificial intelligence based front-end interactive page turning method of claim 1, wherein the classification model training comprises: data preprocessing, pre-training model training, data labeling, model fine tuning and model publishing; the data preprocessing comprises the steps of classifying ReDraw data sets according to component types, and uniformly scaling the picture size to set pixels in an equal proportion mode; wherein the pre-training model training comprises training the model using a plurality of ReDraw data sets based on a MobileNet model; the data labeling comprises correspondingly labeling a plurality of component data of HTML page data; the model fine adjustment comprises the steps of using a MobileNet model and loading a pre-training model for fine adjustment; model publishing includes publishing the model and providing HTTP services and providing a calling interface.
3. The artificial intelligence based front-end interactive page turning method according to claim 1, wherein said OCR model training comprises a character localization model training and a character recognition model training, wherein the character localization model training is implemented by CTPN algorithm, wherein the character recognition model training is implemented by DenseNet algorithm.
4. The artificial intelligence based front-end interactive page turning method of claim 3, wherein the text-positioning model training comprises: the method comprises the steps of character positioning picture marking, image data preprocessing and data enhancing, CTPN model building, CTPN model training, CTPN model evaluation and CTPN model storage.
5. The artificial intelligence based front-end interactive page turning method of claim 3, wherein the text recognition model training comprises: the method comprises the steps of character recognition picture labeling, image data preprocessing and data enhancement, construction of a DenseNet model, training of the DenseNet model, evaluation of the DenseNet model and storage of the DenseNet model.
6. The artificial intelligence based front-end interactive page turning method according to claim 1, wherein said vector graphics file first turning step comprises:
acquiring each layer information from the Sketch design file, specifically, reading the Sketch file and converting the Sketch file into a JSON file through a Sketch-file plug-in, wherein field picture information, document information and page information in the JSON file are used;
converting the picture information, the document information and the page information of the JSON file into a Def file, wherein the Def file is a definition file;
and converting the Def file into a VUE page file in an HTML5 standard format, wherein the VUE page comprises a text label, a picture label and a vector diagram label, and the module structure of the VUE page is kept consistent with that of the Sketch file.
7. The artificial intelligence based front-end interactive page turning method according to claim 1, wherein said vector graphics file second turning step comprises:
s410, screenshot is conducted on the front-end interaction page, an interface area for obtaining the first component is selected, and an interface area image is obtained by combining the screenshot;
s420, performing Base64 coding on the interface region image, calling a classification model to perform coding on the image, arranging the recognition results from high to low according to the similarity, and returning a plurality of candidate components;
s430, replacing the corresponding basic component with the best matching component according to the best matching component selected by the user;
s440, calling an OCR recognition model to perform character recognition on the Base64 code of the interface region image to obtain a character recognition result;
and S450, adding a character recognition result to the assembly regulated in the S430 to obtain the second assembly.
8. The artificial intelligence based front-end interactive page turning method of claim 1, wherein the HTML5 definition file is for WEBIDE.
9. An artificial intelligence based front-end interactive page transformation apparatus comprising a memory, a processor and a computer program stored in said memory and executable on said processor, wherein said processor implements the method steps of any of claims 1 to 8 when executing said computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method steps of any one of claims 1 to 8.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110034032.3A CN112732259B (en) | 2021-01-11 | 2021-01-11 | Front-end interactive page conversion method, device and medium based on artificial intelligence |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110034032.3A CN112732259B (en) | 2021-01-11 | 2021-01-11 | Front-end interactive page conversion method, device and medium based on artificial intelligence |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN112732259A true CN112732259A (en) | 2021-04-30 |
| CN112732259B CN112732259B (en) | 2024-05-24 |
Family
ID=75591445
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110034032.3A Active CN112732259B (en) | 2021-01-11 | 2021-01-11 | Front-end interactive page conversion method, device and medium based on artificial intelligence |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112732259B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113674387A (en) * | 2021-08-26 | 2021-11-19 | 广东中星电子有限公司 | Video processing method and device for non-natural scene video |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102385580A (en) * | 2010-08-30 | 2012-03-21 | 北大方正集团有限公司 | Method and device for customizing webpage contents |
| CN105930159A (en) * | 2016-04-20 | 2016-09-07 | 中山大学 | Image-based interface code generation method and system |
| CN107025676A (en) * | 2016-01-25 | 2017-08-08 | 阿里巴巴集团控股有限公司 | The generation method and relevant apparatus of a kind of graphic template and picture |
| CN107391153A (en) * | 2017-07-31 | 2017-11-24 | 深圳乐信软件技术有限公司 | A kind of code generating method and device based on Spring Yu MyBatis framework integrations |
| CN108228183A (en) * | 2018-01-12 | 2018-06-29 | 北京三快在线科技有限公司 | Front-end interface code generating method, device, electronic equipment and storage medium |
| CN109783094A (en) * | 2018-12-15 | 2019-05-21 | 深圳壹账通智能科技有限公司 | Front end page generation method, device, computer equipment and storage medium |
| CN110244940A (en) * | 2019-06-12 | 2019-09-17 | 四川长虹电器股份有限公司 | Optimize the method and web front-end project structure of web application system development |
| US20200133645A1 (en) * | 2018-10-30 | 2020-04-30 | Jpmorgan Chase Bank, N.A. | User interface and front end application automatic generation |
| CN111460355A (en) * | 2020-04-17 | 2020-07-28 | 支付宝(杭州)信息技术有限公司 | Page parsing method and device |
| CN112114803A (en) * | 2020-08-25 | 2020-12-22 | 济南浪潮高新科技投资发展有限公司 | Deep learning-based front-end code generation method, equipment and medium for UI (user interface) |
-
2021
- 2021-01-11 CN CN202110034032.3A patent/CN112732259B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102385580A (en) * | 2010-08-30 | 2012-03-21 | 北大方正集团有限公司 | Method and device for customizing webpage contents |
| CN107025676A (en) * | 2016-01-25 | 2017-08-08 | 阿里巴巴集团控股有限公司 | The generation method and relevant apparatus of a kind of graphic template and picture |
| CN105930159A (en) * | 2016-04-20 | 2016-09-07 | 中山大学 | Image-based interface code generation method and system |
| CN107391153A (en) * | 2017-07-31 | 2017-11-24 | 深圳乐信软件技术有限公司 | A kind of code generating method and device based on Spring Yu MyBatis framework integrations |
| CN108228183A (en) * | 2018-01-12 | 2018-06-29 | 北京三快在线科技有限公司 | Front-end interface code generating method, device, electronic equipment and storage medium |
| US20200133645A1 (en) * | 2018-10-30 | 2020-04-30 | Jpmorgan Chase Bank, N.A. | User interface and front end application automatic generation |
| CN109783094A (en) * | 2018-12-15 | 2019-05-21 | 深圳壹账通智能科技有限公司 | Front end page generation method, device, computer equipment and storage medium |
| CN110244940A (en) * | 2019-06-12 | 2019-09-17 | 四川长虹电器股份有限公司 | Optimize the method and web front-end project structure of web application system development |
| CN111460355A (en) * | 2020-04-17 | 2020-07-28 | 支付宝(杭州)信息技术有限公司 | Page parsing method and device |
| CN112114803A (en) * | 2020-08-25 | 2020-12-22 | 济南浪潮高新科技投资发展有限公司 | Deep learning-based front-end code generation method, equipment and medium for UI (user interface) |
Non-Patent Citations (4)
| Title |
|---|
| BATUHAN AŞIROĞLU等: "Automatic HTML Code Generation from Mock-Up Images Using Machine Learning Techniques", 《2019 SCIENTIFIC MEETING ON ELECTRICAL-ELECTRONICS & BIOMEDICAL ENGINEERING AND COMPUTER SCIENCE (EBBT)》, pages 1 - 4 * |
| VINCENT: "是否有可靠的方案把 Sketch 生成 HTML?", Retrieved from the Internet <URL:https://www.zhihu.com/question/380687009> * |
| 朱勇 伏海旭: "基于RESTful的在线请假管理系统设计与实现", 《现代计算机(专业版)》, vol. 36, pages 96 - 100 * |
| 邱怡琳: "基于可视化建模技术的组合分析推荐前端应用的设计与实现", 《中国优秀硕士学位论文全文数据库 信息科技辑》, pages 138 - 187 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113674387A (en) * | 2021-08-26 | 2021-11-19 | 广东中星电子有限公司 | Video processing method and device for non-natural scene video |
| CN113674387B (en) * | 2021-08-26 | 2024-04-16 | 广东中星电子有限公司 | Video processing method and device for unnatural scene video |
Also Published As
| Publication number | Publication date |
|---|---|
| CN112732259B (en) | 2024-05-24 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Chen et al. | Towards automated infographic design: Deep learning-based auto-extraction of extensible timeline | |
| Clausner et al. | Aletheia-an advanced document layout and text ground-truthing system for production environments | |
| CN112232149A (en) | Document multi-mode information and relation extraction method and system | |
| CN114596566B (en) | Text recognition method and related device | |
| WO2022028313A1 (en) | Method and device for image generation and colorization | |
| Zhou et al. | Reverse-engineering bar charts using neural networks | |
| CN114419642A (en) | Method, device and system for extracting key value pair information in document image | |
| CN112069900A (en) | Bill character recognition method and system based on convolutional neural network | |
| CN112037239B (en) | Text guidance image segmentation method based on multi-level explicit relation selection | |
| Biswas et al. | Docsynth: a layout guided approach for controllable document image synthesis | |
| CN111460782A (en) | Information processing method, device and equipment | |
| de Souza Baulé et al. | Recent Progress in Automated Code Generation from GUI Images Using Machine Learning Techniques. | |
| Zhang et al. | SSNet: Structure-Semantic Net for Chinese typography generation based on image translation | |
| Ge et al. | Exploring local detail perception for scene sketch semantic segmentation | |
| CN112732259B (en) | Front-end interactive page conversion method, device and medium based on artificial intelligence | |
| Chen et al. | Code generation from a graphical user interface via attention-based encoder–decoder model | |
| Zeng et al. | An unsupervised font style transfer model based on generative adversarial networks | |
| Feng et al. | Guis2code: A computer vision tool to generate code automatically from graphical user interface sketches | |
| CN116610304B (en) | Page code generation method, device, equipment and storage medium | |
| CN116433468A (en) | Data processing method and device for image generation | |
| Hu et al. | Mathematical formula detection in document images: A new dataset and a new approach | |
| CN115034177A (en) | Presentation file conversion method, device, equipment and storage medium | |
| CN112836467B (en) | Image processing method and device | |
| CN112329389B (en) | Chinese character stroke automatic extraction method based on semantic segmentation and tabu search | |
| Lian et al. | Cvfont: Synthesizing chinese vector fonts via deep layout inferring |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |