CN113688835A - System based on object detection and OCR and method for extracting information of application fault elastic picture - Google Patents
System based on object detection and OCR and method for extracting information of application fault elastic picture Download PDFInfo
- Publication number
- CN113688835A CN113688835A CN202110925758.6A CN202110925758A CN113688835A CN 113688835 A CN113688835 A CN 113688835A CN 202110925758 A CN202110925758 A CN 202110925758A CN 113688835 A CN113688835 A CN 113688835A
- Authority
- CN
- China
- Prior art keywords
- fault
- ocr
- object detection
- screenshots
- detection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 51
- 238000004590 computer program Methods 0.000 claims description 10
- 238000007781 pre-processing Methods 0.000 claims description 3
- 238000005516 engineering process Methods 0.000 abstract description 8
- 238000000034 method Methods 0.000 description 7
- 230000019771 cognition Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000000605 extraction Methods 0.000 description 1
- 238000003384 imaging method Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000006011 modification reaction Methods 0.000 description 1
- 230000001537 neural Effects 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Abstract
The invention discloses a system based on object detection and OCR and a method for extracting information of a fault bullet frame picture, which comprises the steps of collecting screenshots of each fault bullet window of a target system, at least collecting 3 screenshots of different backgrounds and 3 screenshots of different positions for each fault, then training a model, and applying the obtained model to the detection of the fault bullet frame picture. By combining the object detection technology and the OCR image recognition technology, the method can perform OCR recognition only aiming at the content of the bullet frame, thereby avoiding the text information output by the OCR from containing much noisy or distorted information and improving the robustness of the system.
Description
Technical Field
The invention relates to the technical field of image recognition, in particular to a system based on object detection and OCR and an extraction method of information of an applied fault elastic picture.
Background
The large-scale information system applied to enterprises is large in structure and complex in assembly, functions are cooperatively provided by a plurality of subsystems and processes, iteration is frequent, and bug is easy to occur. The most popular implementation of the current information system is B/S or C/S architecture, and the system failure or bug is often represented by a wrong box mode of a client, which becomes a maximum recording and showing source of the system failure.
The related art in the prior art is OCR technology, which is a mature technology, but the related technology does not perfectly match the problem solved by the invention. Firstly, the error of the user feedback is usually a screenshot of the whole screen including the popup, because the users of the software system are usually business personnel and non-professional IT people, so that the users cannot request and mostly cannot accurately intercept the popup feedback error, and usually, the user can intercept the whole screen by using a screenshot key and the like for cognition. OCR does not sense the structure of a picture, so that the OCR scans the picture line by line from top to bottom and outputs texts from left to right, and because the picture has a structure and the like relationship, the problem that characters in the same direction are not accurately arranged in a line often occurs, so that OCR faults are caused, so that text information output by the OCR usually contains a lot of noises or is distorted, and the information is difficult to be input into a subsequent customer service and question-answering system without manual processing.
Disclosure of Invention
In view of the above, the first aspect of the present invention is to provide a system based on object detection and OCR and a method for extracting information of application fault elastic sheet. The problems in the background art can be solved.
The purpose of the first aspect of the invention is realized by the following technical scheme:
the invention relates to a system based on object detection and OCR and a method for extracting information of a fault bullet frame picture, which comprises the steps of collecting a screenshot of each fault bullet window of a target system, at least collecting 3 screenshots of different backgrounds and 3 screenshots of different positions of each fault, then training a model, and applying the obtained model to the detection of the fault bullet frame picture.
Further, the screenshots of each fault popup are to appear in various different backgrounds and locations.
Further, the detection of applying the obtained model to the faulty bullet frame picture comprises the following steps:
step S101, inputting a picture to be detected;
step S102, preprocessing the picture;
step S103, calling the model to perform frame object detection, performing frame OCR processing if the object detection is successful, and entering step S104 if the frame object detection is unsuccessful;
if the detection of the bullet frame OCR processing is successful after the bullet frame OCR processing, the step S105 is carried out, and if the detection of the bullet frame OCR processing is unsuccessful after the bullet frame OCR processing, the step S104 is carried out;
step S104, performing full-image OCR processing, entering step S105 if the full-image OCR processing detection is successful, and displaying call failure if the detection is failed, and ending;
and step S105, outputting the picture character information.
Further, the YOLO4 object detection model is used for popup position identification;
it is an object of a second aspect of the invention to provide a computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method as described above when executing the computer program.
It is an object of a third aspect of the invention to provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the method as previously described.
The invention has the beneficial effects that:
by combining the object detection technology and the OCR image recognition technology, the method can perform OCR recognition only on the content of the bullet frame without performing OCR recognition on the whole picture, thereby preventing text information output by the OCR from containing much noisy or distorted information and improving the robustness of the system.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the present invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings, in which:
FIG. 1 is a schematic structural view of the present invention;
FIG. 2 is an original display picture including a pop-up window;
fig. 3 is a picture of a bullet frame after the identification by the method of the present invention.
Detailed Description
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments are illustrative of the invention only and are not limiting upon the scope of the invention.
As shown in fig. 1, the system based on object detection and OCR and the method for extracting the information of the application fault elastic sheet of the present invention are characterized in that: the method comprises the steps of collecting screenshots of all fault popup windows of a target system, collecting at least 3 screenshots of different backgrounds and 3 screenshots of different positions for each fault, then training a model, and applying the obtained model to detection of fault popup frame pictures. In this embodiment, 5 screenshots of different backgrounds and 5 screenshots of different positions are selected and collected.
Wherein each shot of the fault popup is to appear in a variety of different backgrounds and locations. For example, screenshots of different positions in the upper, lower, left, right and middle are collected in the same background, and screenshots of different backgrounds can be collected in the same position, so that the collection of position and background information of more situations is facilitated, the constructed model can cover various situations, and the recognition efficiency is improved. In this embodiment, the YOLO4 object detection model is used to perform the popup position recognition. YOLO redefines object detection as a regression problem. It applies a single Convolutional Neural Network (CNN) to the entire image, divides the image into meshes, and predicts class probabilities and bounding boxes for each mesh. Since the detection problem is a regression problem, no complex piping is required. It is 1000 times faster than "R-CNN" and 100 times faster than "Fast R-CNN". Meanwhile, the technology can process real-time video streams, and the delay is less than 25 milliseconds. The accuracy is more than twice that of the previous real-time system.
In this embodiment, applying the obtained model to the detection of the faulty elastic frame picture includes the following steps:
step S101, inputting a picture to be detected;
step S102, preprocessing the picture;
step S103, calling the model to perform frame object detection, performing frame OCR processing if the object detection is successful, and entering step S104 if the frame object detection is unsuccessful;
if the detection of the bullet frame OCR processing is successful after the bullet frame OCR processing, the step S105 is carried out, and if the detection is unsuccessful after the bullet frame OCR processing, the step S104 is carried out; the unsuccessful detection of the bullet box OCR processing means that the picture character information cannot be extracted and recognized;
step S104, performing full-image OCR processing, entering step S105 if the full-image OCR processing detection is successful, and displaying call failure if the detection is failed, and ending; the fact that the detection of the full-image OCR processing is unsuccessful means that image character information cannot be extracted and recognized;
and step S105, outputting the picture character information.
In an embodiment of the application of the present invention, as shown in fig. 2, the error of user feedback is usually a screenshot of the whole screen containing a pop-up window, OCR does not sense the structure of the picture, so it scans the picture output text line by line from top to bottom and from left to right, and because the picture has a structure and the like, the problem that the characters in the same direction are not arranged exactly in one line often occurs, resulting in OCR failure, so the text information output by OCR usually contains much noise or is distorted, and such information is difficult to be used as input for subsequent customer service and question and answer systems without manual processing. The idea of applying the method of the invention to solve the problem is that: before OCR, an object detection process is performed. The position of the fault popup is located from the picture, the fault popup is extracted, as shown in fig. 3, and then the fault popup is used as an input to enter OCR processing, so that complete fault information without noise can be output: "receiving distribution network transfer order failure downloading and storing distribution network transfer order failure The operation has timed out".
It should be recognized that embodiments of the present invention can be realized and implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The methods may be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with the computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner, according to the methods and figures described in the detailed description. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose.
Further, the operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor.
Finally, the above embodiments are only intended to illustrate the technical solutions of the present invention and not to limit the present invention, and although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions, and all of them should be covered by the claims of the present invention.
Claims (8)
1. A system based on object detection and OCR and a method for extracting information of an applied fault elastic picture are characterized in that: the method comprises the steps of collecting screenshots of all fault popup windows of a target system, collecting at least 3 screenshots of different backgrounds and 3 screenshots of different positions for each fault, then training a model, and applying the obtained model to detection of fault popup frame pictures.
2. The system based on object detection and OCR and the method for extracting the flip chip information by applying the fault according to claim 1, wherein: the screenshots of each fault popup appear in a variety of different backgrounds and locations.
3. The system based on object detection and OCR and the method for extracting the flip chip information by applying the fault according to claim 1, wherein: the detection of applying the obtained model to the fault bullet frame picture comprises the following steps:
step S101, inputting a picture to be detected;
step S102, preprocessing the picture;
step S103, calling the model to perform frame object detection, performing frame OCR processing if the object detection is successful, and entering step S104 if the frame object detection is unsuccessful;
if the detection of the popup window OCR processing is successful after the popup frame OCR processing, the step S105 is entered, and if the detection is unsuccessful after the popup frame OCR processing, the step S104 is entered;
step S104, performing full-image OCR processing, entering step S105 if the full-image OCR processing detection is successful, and displaying that the calling is failed if the full-image OCR processing detection is failed, and ending;
and step S105, outputting the picture character information.
4. The system based on object detection and OCR and the method for extracting the flip chip information by applying the fault according to claim 3, wherein: the YOLO4 object detection model was used for pop-up position identification.
5. The system based on object detection and OCR and the method for extracting the flip chip information by applying the fault according to claim 1, wherein: each fault is collected with 3-5 screenshots of different backgrounds and 3-5 screenshots of different positions.
6. The system based on object detection and OCR and the method for extracting the flip chip information by applying the fault according to claim 5, wherein: for each fault, 5 different background screenshots and 5 screenshots from different positions were collected.
7. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory and capable of running on the processor, wherein: the processor, when executing the computer program, implements the method of any of claims 1-6.
8. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when executed by a processor, implements the method of any one of claims 1-6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110925758.6A CN113688835A (en) | 2021-08-12 | 2021-08-12 | System based on object detection and OCR and method for extracting information of application fault elastic picture |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110925758.6A CN113688835A (en) | 2021-08-12 | 2021-08-12 | System based on object detection and OCR and method for extracting information of application fault elastic picture |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN113688835A true CN113688835A (en) | 2021-11-23 |
Family
ID=78579638
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110925758.6A Withdrawn CN113688835A (en) | 2021-08-12 | 2021-08-12 | System based on object detection and OCR and method for extracting information of application fault elastic picture |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113688835A (en) |
-
2021
- 2021-08-12 CN CN202110925758.6A patent/CN113688835A/en not_active Withdrawn
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN107666987A (en) | Robotic process automates | |
| KR102220174B1 (en) | Learning-data enhancement device for machine learning model and method for learning-data enhancement | |
| CN107223246A (en) | Image labeling method, device and electronic equipment | |
| US20220391228A1 (en) | Method and system for accessing table content in a digital image of the table | |
| CN109934227A (en) | System for recognizing characters from image and method | |
| US20220130160A1 (en) | Object recognition method and apparatus, and electronic device and storage medium | |
| CN107169340A (en) | A kind of behavior formula identifying code processing method and processing device | |
| US20200241900A1 (en) | Automation tool | |
| CN110598686A (en) | Invoice identification method, system, electronic equipment and medium | |
| CN111310155A (en) | System architecture for automatic identification of slider verification code and implementation method | |
| US20200183663A1 (en) | System for image segmentation, transformation and user interface component construction | |
| CN111310156B (en) | Automatic identification method and system for slider verification code | |
| TWI678614B (en) | Test system | |
| US20090158090A1 (en) | Data entry retrieval | |
| CN109815100B (en) | Behavior monitoring method for CABAO software by utilizing image contrast analysis | |
| CN113688835A (en) | System based on object detection and OCR and method for extracting information of application fault elastic picture | |
| CN111143188A (en) | Method and equipment for automatically testing application | |
| CN110827261B (en) | Image quality detection method and device, storage medium and electronic equipment | |
| CN113256583A (en) | Image quality detection method and apparatus, computer device, and medium | |
| CN106066979A (en) | One operates in specialty bar code scanning method and system on cell phone platform | |
| CN112036522A (en) | Calligraphy individual character evaluation method, system and terminal based on machine learning | |
| CN112650796A (en) | Automatic application data collection and storage management system | |
| CN110727595A (en) | Application login interface identification method, intelligent terminal and storage medium | |
| CN110275834A (en) | User interface automatization test system and method | |
| CN110659193B (en) | Test system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20211123 |
|
| WW01 | Invention patent application withdrawn after publication |