CN113377962A - Intelligent process simulation method based on image recognition and natural language processing - Google Patents
Intelligent process simulation method based on image recognition and natural language processing Download PDFInfo
- Publication number
- CN113377962A CN113377962A CN202110679802.XA CN202110679802A CN113377962A CN 113377962 A CN113377962 A CN 113377962A CN 202110679802 A CN202110679802 A CN 202110679802A CN 113377962 A CN113377962 A CN 113377962A
- Authority
- CN
- China
- Prior art keywords
- simulation
- software
- image recognition
- natural language
- language processing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/36—Creation of semantic tools, e.g. ontology or thesauri
- G06F16/367—Ontology
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/21—Design, administration or maintenance of databases
- G06F16/211—Schema design and management
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/27—Design optimisation, verification or simulation using machine learning, e.g. artificial intelligence, neural networks, support vector machines [SVM] or training a model
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/28—Design optimisation, verification or simulation using fluid dynamics, e.g. using Navier-Stokes equations or computational fluid dynamics [CFD]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F40/00—Handling natural language data
- G06F40/20—Natural language analysis
- G06F40/279—Recognition of textual entities
- G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
- G06F40/295—Named entity recognition
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/30—Computing systems specially adapted for manufacturing
Abstract
An intelligent process simulation method based on image recognition and natural language processing includes setting up technological thought of software operation knowledge map containing all operation objects (named entities) and operation method (relationship) between them in said process simulation software to provide professional knowledge background for effective extraction of operation event so as to obtain complete and correct simulation scheme from document data or completed working condition file.
Description
Technical Field
The invention belongs to the technical field of process simulation, and particularly relates to an intelligent process simulation method based on image recognition and natural language processing.
Background
The process simulation can provide an effective way for solving the major and major warfare problems of energy, resources, environment, materials and the like from multiple levels of materials (molecules), reactors, factories (processes) and the like, so the process simulation becomes a research hotspot of subjects such as chemistry, chemical engineering, metallurgy, energy, materials, computational science and technology and the like. For a given process problem, simulation software and computing equipment, most of the time of a technician is mainly used for learning and using simulation software such as software setting, condition debugging, result processing and the like, so that the time investment in software learning and use becomes a key factor influencing the working efficiency of the technician.
Artificial intelligence is an important means for human beings to improve work efficiency. Especially, with the rapid development of deep learning methods such as convolutional neural network, cyclic neural network, attention mechanism, etc. and the massive growth of data resources in recent years, artificial intelligence has made a great breakthrough in image recognition and natural language processing, and can more accurately recognize characters and icons in images and extract important information of interest in natural languages, but has not yet been widely and deeply applied in the field of process simulation, and there are main problems:
(1) at present, no relevant report for realizing process simulation intellectualization by using image recognition and natural language processing technologies exists, and a feasible overall technical route is lacked;
(2) a professional database related to process simulation and a corresponding construction method are lacked, and an effective simulation scheme cannot be provided for the process problem;
(3) the simulation scheme is an ordered combination of various operation events (executing a certain operation object by a certain operation method), and technicians can generally learn and obtain the operation events from document materials or finished working condition files, so that artificial intelligence technologies such as image recognition and natural language processing are expected to replace human beings to learn and understand the resources. However, the artificial intelligence technology in the prior general field lacks the professional knowledge background of process simulation, and is difficult to extract all operation events from document data and completed working condition files comprehensively and accurately, so that a complete and correct simulation scheme cannot be obtained.
Disclosure of Invention
In order to realize the intellectualization of process simulation and improve the working efficiency of technicians, the invention aims to provide an intelligent process simulation method based on image recognition and natural language processing
The invention aims at the problem that the existing artificial intelligence technology lacks the process simulation professional knowledge background, introduces artificial intelligence technologies such as image recognition, natural language processing, software automation operation and the like into the process simulation fields such as molecular simulation, computational fluid mechanics simulation, flow simulation and the like, and realizes intelligent process simulation.
In order to achieve the purpose, the technical scheme of the invention is as follows:
an intelligent process simulation method based on image recognition and natural language processing comprises the following steps:
step (1), constructing a software operation knowledge graph;
step (2), establishing a process simulation professional database based on document data;
step (3), establishing a process simulation professional database based on the working condition file;
and (4) intelligently simulating the process problem.
The step (1) is specifically as follows:
(1) determining characters, icons and positions of the characters and the icons in the current software interface through image recognition;
(2) automatically operating the operable object through the program code to obtain an operated software interface;
(3) carrying out image recognition on the operated software interface, and determining characters, icons and positions of the characters and the icons;
(4) comparing the change of the software interface before and after running, and forming a triple by taking the characters and the icons as named entities and the operation method as a relation;
(5) and sequentially executing the processes item by item, and storing and establishing the software operation knowledge graph after all the triple data are obtained.
The step (2) is specifically as follows:
1) converting non-text type process simulation document data into text data through image recognition;
2) constructing a domain dictionary according to named entities in a software operation knowledge graph, and integrating the domain dictionary with the existing deep learning technology to establish a natural language processing model;
3) testing the extracted operation event by using a software automatic operation technology, judging whether the obtained operation result is consistent with the operation result in the text data or not through image recognition, if so, determining that the operation event is correct, otherwise, correcting and perfecting the operation event through the relation between corresponding entities in a software operation map;
4) and combining all the operation events in the text data in sequence to form a simulation scheme of the problem, and establishing a process simulation professional database by associating and storing the simulation scheme with the problem description data.
The step (3) is specifically as follows:
1) opening the finished simulation working condition by using image recognition and software automatic operation technologies, carrying out screenshot on an operation interface, creating a brand-new simulation working condition which is not set, and carrying out screenshot on the same operation interface;
2) judging whether the screenshots of the operation interface of the finished simulation working condition and the brand-new working condition are consistent by adopting an image recognition technology, if so, indicating that the current interface does not need to be set, and if not, recording the object required to be operated by the current interface;
3) inquiring and retrieving the object to be operated in the software operation knowledge graph obtained in the step (1), extracting the shortest path from the initial object or the initial software interface to the object to be operated or the current interface, namely determining all the operation events in sequence for finishing the effective operation events to be executed, and summarizing and integrating to obtain simulation scheme data;
4) according to the process problem solved by the completed simulation working condition, the completing personnel provides a corresponding text description, and natural language processing is carried out on the text description to obtain problem description data;
5) and (4) associating, storing and establishing the process simulation professional database by the problem description and the simulation scheme.
The step (4) is specifically as follows:
1) performing natural language processing on the text description of the process problem to obtain problem description data;
2) searching and inquiring in the process simulation professional database to obtain a simulation scheme corresponding to the same or the closest problem description;
3) and executing a simulation scheme by a software automation operation technology to obtain a simulation result.
Compared with the prior art, the invention has the following advantages:
(1) and extracting problem description and simulation schemes from the existing resources through artificial intelligence and establishing a professional database, so that the simulation schemes can be automatically proposed for the process problems and operated.
(2) The invention can accelerate the solving speed of the process problem, reduce the entrance threshold of the process simulation, realize the effective accumulation of the process simulation knowledge and obviously improve the working efficiency of technical personnel.
(3) The process simulation method of the present invention can be applied to molecular simulation, computational fluid dynamics simulation, flow simulation, and the like.
Drawings
FIG. 1 is a flow of construction of a software operation knowledge graph.
FIG. 2 is a process flow for simulating professional database creation based on documentation.
FIG. 3 is a process simulation professional database building flow based on condition files.
FIG. 4 is a flow of an intelligent simulation of a process problem.
Detailed Description
The invention is described in further detail with reference to the application of intelligent process simulation to computational fluid dynamics simulation as a specific example.
1. Automatic construction of software operation knowledge graph
A software operation knowledge graph is constructed for certain computational fluid dynamics simulation software, and as shown in fig. 1, the method mainly comprises the following steps:
(1) screenshot is conducted on a current software interface, a multi-mode image recognition model is established based on a convolutional neural network, information of two different modes of a text and an icon is extracted and processed respectively, characters such as Display, Scale and the like and icons such as an input box and a radio box in the software operation interface are accurately recognized and positioned, and operable objects in the software operation interface include the characters and the icons and operation methods such as clicking, double clicking and the like are preliminarily judged;
(2) operating an operable object in a software interface through a program code in a software automatic operation library to obtain an operated software interface;
(3) performing image recognition on the operated software interface, and recognizing and positioning characters and icons in the software operation interface;
(4) and comparing and analyzing whether the software interface changes before and after operation, and if the software interface changes, forming a triple group to establish a software operation knowledge graph by taking the characters and the icons of the software interface as named entities and an operation method as a relation. For example, when a new interface including Colors appears after a single click on Display, Display and Colors are entities, and the relationship between the two is a single click.
(5) And sequentially executing the processes item by item, and storing and establishing the software operation knowledge graph after all the triple data are obtained.
2. Creating process simulation professional database according to document data
Aiming at relevant document data of certain computational fluid dynamics simulation software, a process simulation professional database is established, and as shown in FIG. 2, the method mainly comprises the following steps:
(1) converting process simulation document data of a non-text type such as pdf format into text data through image recognition;
(2) the method comprises the steps of constructing a domain dictionary according to named entities in a software operation knowledge graph, integrating the domain dictionary with the existing deep learning technology to establish a natural language processing model, and extracting operation events consisting of operation methods and the named entities in text data comprehensively and accurately due to the fact that the model has all the named entities, and meanwhile obtaining problem description and operation result data;
a natural language processing model is established based on a cyclic neural network and a conditional random field, a domain dictionary formed by named entities in a knowledge map is operated by combining software, knowledge extraction is carried out on text data, and problem description, operation events and operation results are extracted. Typical problem descriptions include parameters such as materials, temperature, pressure, and structure of the simulation system. Typical operational events are clicking Scale, selecting Convert Units, etc. Typical running results such as software appearance of new interfaces or prompts and the like.
(3) And automatically testing the extracted operation events according to the automatic software operation library, and comparing whether the operation results are consistent or not. If the operation events are consistent, the operation events are output, and if the operation events are inconsistent, the operation events are corrected and perfected through the relation provided by the software operation knowledge graph. The operation events are combined in sequence to form the simulation scheme.
(4) Storing and managing the problem description and the simulation scheme data of each case, and establishing a process simulation professional database.
3. Simulating a professional database according to the process of establishing a working condition file
Aiming at the simulated working condition file, a process simulation professional database is established, and as shown in fig. 3, the method mainly comprises the following steps:
(1) and performing screenshot on the operation interface which finishes the simulation working condition based on the image recognition deep learning model and the automatic software running library, and simultaneously performing screenshot on a brand new same operation interface which is not set with the simulation working condition.
(2) And comparing and analyzing whether the screenshots of the operation interface of the finished simulation working condition and the brand-new working condition are consistent by adopting an image recognition technology, if so, indicating that the current interface does not need to be set, and if not, recording objects required by the operation of the current interface, such as selection and setting of materials and models.
(3) And inquiring and retrieving the object needing to be operated in the software operation knowledge graph, and extracting the shortest path from the initial object (initial software interface) to the object needing to be operated (current interface), namely the effective operation event needing to be executed for completing the setting. And sequentially determining all operation events, and summarizing and integrating to obtain simulation scheme data.
(4) And providing related text descriptions by the completion personnel under the working condition, and extracting knowledge of the completion personnel by adopting a natural language processing model to obtain problem description data.
(5) Storing and managing the problem description and the simulation scheme data of each working condition, and establishing a process simulation professional database.
Step (4), performing intelligent simulation on the process problem, taking intelligent computational fluid dynamics simulation as an example, as shown in fig. 4, the method mainly comprises the following steps:
(1) the text description of the process problem is processed in natural language to obtain problem description data including parameters such as material, temperature, pressure, and structure.
(2) And searching data of each parameter in the process simulation professional database, carrying out comprehensive scoring on the problem description data existing in the database according to the weight of each parameter, and extracting corresponding simulation scheme data when the highest scoring is the problem description closest to the process problem.
(3) All operation events in the obtained simulation scheme are sequentially executed through an image recognition and software automation operation technology, intelligent computational fluid dynamics simulation is achieved, and detailed distribution data of components, temperature, speed, pressure and the like in a simulation system are obtained.
The foregoing is a more detailed description of the invention in connection with specific preferred embodiments and it is not intended that the invention be limited to these specific details. For those skilled in the art to which the invention pertains, several simple deductions or substitutions can be made without departing from the spirit of the invention, and all shall be considered as belonging to the protection scope of the invention.
Claims (5)
1. An intelligent process simulation method based on image recognition and natural language processing is characterized by comprising the following steps:
step (1), constructing a software operation knowledge graph;
step (2), establishing a process simulation professional database based on document data;
step (3), establishing a process simulation professional database based on the working condition file;
and (4) intelligently simulating the process problem.
2. The intelligent process simulation method based on image recognition and natural language processing according to claim 1, wherein the step (1) is specifically as follows:
(1) determining characters, icons and positions of the characters and the icons in the current software interface through image recognition;
(2) automatically operating the operable object through the program code to obtain an operated software interface;
(3) carrying out image recognition on the operated software interface, and determining characters, icons and positions of the characters and the icons;
(4) comparing the change of the software interface before and after running, and forming a triple by taking the characters and the icons as named entities and the operation method as a relation;
(5) and sequentially executing the processes item by item, and storing and establishing the software operation knowledge graph after all the triple data are obtained.
3. The intelligent process simulation method based on image recognition and natural language processing according to claim 1, wherein the step (2) is specifically as follows:
1) converting non-text type process simulation document data into text data through image recognition;
2) constructing a domain dictionary according to named entities in a software operation knowledge graph, and integrating the domain dictionary with the existing deep learning technology to establish a natural language processing model;
3) testing the extracted operation event by using a software automatic operation technology, judging whether the obtained operation result is consistent with the operation result in the text data or not through image recognition, if so, determining that the operation event is correct, otherwise, correcting and perfecting the operation event through the relation between corresponding entities in a software operation map;
4) and combining all the operation events in the text data in sequence to form a simulation scheme of the problem, and establishing a process simulation professional database by associating and storing the simulation scheme with the problem description data.
4. The intelligent process simulation method based on image recognition and natural language processing according to claim 1, wherein the step (3) is specifically as follows:
1) opening the finished simulation working condition by using image recognition and software automatic operation technologies, carrying out screenshot on an operation interface, creating a brand-new simulation working condition which is not set, and carrying out screenshot on the same operation interface;
2) judging whether the screenshots of the operation interface of the finished simulation working condition and the brand-new working condition are consistent by adopting an image recognition technology, if so, indicating that the current interface does not need to be set, and if not, recording the object required to be operated by the current interface;
3) inquiring and retrieving the object to be operated in the software operation knowledge graph obtained in the step (1), extracting the shortest path from the initial object or the initial software interface to the object to be operated or the current interface, namely determining all the operation events in sequence for finishing the effective operation events to be executed, and summarizing and integrating to obtain simulation scheme data;
4) according to the process problem solved by the completed simulation working condition, the completing personnel provides a corresponding text description, and natural language processing is carried out on the text description to obtain problem description data;
5) and (4) associating, storing and establishing the process simulation professional database by the problem description and the simulation scheme.
5. The intelligent process simulation method based on image recognition and natural language processing according to claim 1, wherein the step (4) is specifically as follows:
1) performing natural language processing on the text description of the process problem to obtain problem description data;
2) searching and inquiring in the process simulation professional database to obtain a simulation scheme corresponding to the same or the closest problem description;
3) and executing a simulation scheme by a software automation operation technology to obtain a simulation result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110679802.XA CN113377962B (en) | 2021-06-18 | 2021-06-18 | Intelligent process simulation method based on image recognition and natural language processing |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110679802.XA CN113377962B (en) | 2021-06-18 | 2021-06-18 | Intelligent process simulation method based on image recognition and natural language processing |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113377962A true CN113377962A (en) | 2021-09-10 |
| CN113377962B CN113377962B (en) | 2023-04-07 |
Family
ID=77577848
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110679802.XA Active CN113377962B (en) | 2021-06-18 | 2021-06-18 | Intelligent process simulation method based on image recognition and natural language processing |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113377962B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116664078A (en) * | 2023-07-24 | 2023-08-29 | 杭州所思互连科技有限公司 | RPA object identification method based on semantic feature vector |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1760902A (en) * | 2005-11-11 | 2006-04-19 | 杭州电子科技大学 | Method for optimizing scheduling system by technical imitaitng real flow |
| CN102063550A (en) * | 2011-01-07 | 2011-05-18 | 浙江工业大学 | Intelligent design system of cold extrusion piece with machine intelligence involved design decision |
| JP2013097560A (en) * | 2011-10-31 | 2013-05-20 | Mitsubishi Heavy Ind Ltd | System for optimizing process of construction work and method for optimizing process of construction work |
| CN110019594A (en) * | 2017-09-27 | 2019-07-16 | 中国石油化工股份有限公司 | A kind of geologic datebase method for building up based on numerical simulation |
| CN111461439A (en) * | 2020-04-02 | 2020-07-28 | 浙江大学 | Urban atmospheric diffusion process-based simulation prediction multi-heat-source heat supply system load scheduling distribution method and system |
-
2021
- 2021-06-18 CN CN202110679802.XA patent/CN113377962B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1760902A (en) * | 2005-11-11 | 2006-04-19 | 杭州电子科技大学 | Method for optimizing scheduling system by technical imitaitng real flow |
| CN102063550A (en) * | 2011-01-07 | 2011-05-18 | 浙江工业大学 | Intelligent design system of cold extrusion piece with machine intelligence involved design decision |
| JP2013097560A (en) * | 2011-10-31 | 2013-05-20 | Mitsubishi Heavy Ind Ltd | System for optimizing process of construction work and method for optimizing process of construction work |
| CN110019594A (en) * | 2017-09-27 | 2019-07-16 | 中国石油化工股份有限公司 | A kind of geologic datebase method for building up based on numerical simulation |
| CN111461439A (en) * | 2020-04-02 | 2020-07-28 | 浙江大学 | Urban atmospheric diffusion process-based simulation prediction multi-heat-source heat supply system load scheduling distribution method and system |
Non-Patent Citations (2)
| Title |
|---|
| EZUTAH UDONCY OLUGU ET AL.: ""Simulation Study on Lens Manufacturing Process Flow"", 《ASIA INTERNATIONAL CONFERENCE ON MODELING & SIMULATION》 * |
| 陈仲昌: ""某型无人机发动机缸套铸造过程模拟与分析"", 《万方数据知识服务平台》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116664078A (en) * | 2023-07-24 | 2023-08-29 | 杭州所思互连科技有限公司 | RPA object identification method based on semantic feature vector |
| CN116664078B (en) * | 2023-07-24 | 2023-10-10 | 杭州所思互连科技有限公司 | RPA object identification method based on semantic feature vector |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113377962B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111459799B (en) | Software defect detection model establishing and detecting method and system based on Github | |
| Boring et al. | GOMS-HRA: A method for treating subtasks in dynamic human reliability analysis | |
| CN109471793A (en) | A kind of webpage automatic test defect positioning method based on deep learning | |
| CN113254507B (en) | Intelligent construction and inventory method for data asset directory | |
| CN112463424A (en) | End-to-end program repair method based on graph | |
| CN115563627B (en) | Binary program vulnerability static analysis method based on man-machine cooperation | |
| CN113377962B (en) | Intelligent process simulation method based on image recognition and natural language processing | |
| CN117236677A (en) | RPA process mining method and device based on event extraction | |
| Ilkhani et al. | Extraction test cases by using data mining; reducing the cost of testing | |
| CN113312924A (en) | Risk rule classification method and device based on NLP high-precision analysis label | |
| CN113239698A (en) | Information extraction method, device, equipment and medium based on RPA and AI | |
| CN117093260A (en) | Fusion model website structure analysis method based on decision tree classification algorithm | |
| CN112417852A (en) | Method and device for judging importance of code segment | |
| CN110442858B (en) | Question entity identification method and device, computer equipment and storage medium | |
| Ulrich et al. | Operator timing of task level primitives for use in computation-based human reliability analysis | |
| CN115470854A (en) | Information system fault classification method and classification system | |
| CN111474894B (en) | Variable target PLC simulation debugging method, storage medium and functional module | |
| CN114492460A (en) | Event causal relationship extraction method based on derivative prompt learning | |
| CN109754159B (en) | Method and system for extracting information of power grid operation log | |
| CN113591484B (en) | Expert system rule base establishment method based on semantic analysis | |
| CN117151247B (en) | Method, apparatus, computer device and storage medium for modeling machine learning task | |
| CN108959588A (en) | Text customer service robot intelligence learning method based on big data | |
| CN117390090B (en) | RPA process mining method, storage medium and electronic equipment | |
| Somogyi et al. | Software Modernization Using Machine Learning Techniques | |
| CN117349186B (en) | Program language defect positioning method, system and medium based on semantic flowsheet |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |