CN113688835A - System based on object detection and OCR and method for extracting information of application fault elastic picture - Google Patents

System based on object detection and OCR and method for extracting information of application fault elastic picture Download PDF

Info

Publication number
CN113688835A
CN113688835A CN202110925758.6A CN202110925758A CN113688835A CN 113688835 A CN113688835 A CN 113688835A CN 202110925758 A CN202110925758 A CN 202110925758A CN 113688835 A CN113688835 A CN 113688835A
Authority
CN
China
Prior art keywords
fault
ocr
object detection
screenshots
detection
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
CN202110925758.6A
Other languages
Chinese (zh)
Inventor
杨箴
马小琴
吴漾
张克贤
周玲
纪元
张其静
田钺
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Guizhou Power Grid Co Ltd
Original Assignee
Guizhou Power Grid Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Guizhou Power Grid Co Ltd filed Critical Guizhou Power Grid Co Ltd
Priority to CN202110925758.6A priority Critical patent/CN113688835A/en
Publication of CN113688835A publication Critical patent/CN113688835A/en
Withdrawn legal-status Critical Current

Links

Abstract

The invention discloses a system based on object detection and OCR and a method for extracting information of a fault bullet frame picture, which comprises the steps of collecting screenshots of each fault bullet window of a target system, at least collecting 3 screenshots of different backgrounds and 3 screenshots of different positions for each fault, then training a model, and applying the obtained model to the detection of the fault bullet frame picture. By combining the object detection technology and the OCR image recognition technology, the method can perform OCR recognition only aiming at the content of the bullet frame, thereby avoiding the text information output by the OCR from containing much noisy or distorted information and improving the robustness of the system.

Description

System based on object detection and OCR and method for extracting information of application fault elastic picture
Technical Field
The invention relates to the technical field of image recognition, in particular to a system based on object detection and OCR and an extraction method of information of an applied fault elastic picture.
Background
The large-scale information system applied to enterprises is large in structure and complex in assembly, functions are cooperatively provided by a plurality of subsystems and processes, iteration is frequent, and bug is easy to occur. The most popular implementation of the current information system is B/S or C/S architecture, and the system failure or bug is often represented by a wrong box mode of a client, which becomes a maximum recording and showing source of the system failure.
The related art in the prior art is OCR technology, which is a mature technology, but the related technology does not perfectly match the problem solved by the invention. Firstly, the error of the user feedback is usually a screenshot of the whole screen including the popup, because the users of the software system are usually business personnel and non-professional IT people, so that the users cannot request and mostly cannot accurately intercept the popup feedback error, and usually, the user can intercept the whole screen by using a screenshot key and the like for cognition. OCR does not sense the structure of a picture, so that the OCR scans the picture line by line from top to bottom and outputs texts from left to right, and because the picture has a structure and the like relationship, the problem that characters in the same direction are not accurately arranged in a line often occurs, so that OCR faults are caused, so that text information output by the OCR usually contains a lot of noises or is distorted, and the information is difficult to be input into a subsequent customer service and question-answering system without manual processing.
Disclosure of Invention
In view of the above, the first aspect of the present invention is to provide a system based on object detection and OCR and a method for extracting information of application fault elastic sheet. The problems in the background art can be solved.
The purpose of the first aspect of the invention is realized by the following technical scheme:
the invention relates to a system based on object detection and OCR and a method for extracting information of a fault bullet frame picture, which comprises the steps of collecting a screenshot of each fault bullet window of a target system, at least collecting 3 screenshots of different backgrounds and 3 screenshots of different positions of each fault, then training a model, and applying the obtained model to the detection of the fault bullet frame picture.
Further, the screenshots of each fault popup are to appear in various different backgrounds and locations.
Further, the detection of applying the obtained model to the faulty bullet frame picture comprises the following steps:
step S101, inputting a picture to be detected;
step S102, preprocessing the picture;
step S103, calling the model to perform frame object detection, performing frame OCR processing if the object detection is successful, and entering step S104 if the frame object detection is unsuccessful;
if the detection of the bullet frame OCR processing is successful after the bullet frame OCR processing, the step S105 is carried out, and if the detection of the bullet frame OCR processing is unsuccessful after the bullet frame OCR processing, the step S104 is carried out;
step S104, performing full-image OCR processing, entering step S105 if the full-image OCR processing detection is successful, and displaying call failure if the detection is failed, and ending;
and step S105, outputting the picture character information.
Further, the YOLO4 object detection model is used for popup position identification;
it is an object of a second aspect of the invention to provide a computer arrangement comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing the method as described above when executing the computer program.
It is an object of a third aspect of the invention to provide a computer readable storage medium having stored thereon a computer program which, when executed by a processor, performs the method as previously described.
The invention has the beneficial effects that:
by combining the object detection technology and the OCR image recognition technology, the method can perform OCR recognition only on the content of the bullet frame without performing OCR recognition on the whole picture, thereby preventing text information output by the OCR from containing much noisy or distorted information and improving the robustness of the system.
Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of the invention. The objectives and other advantages of the present invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
Drawings
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention will be further described in detail with reference to the accompanying drawings, in which:
FIG. 1 is a schematic structural view of the present invention;
FIG. 2 is an original display picture including a pop-up window;
fig. 3 is a picture of a bullet frame after the identification by the method of the present invention.
Detailed Description
Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the preferred embodiments are illustrative of the invention only and are not limiting upon the scope of the invention.
As shown in fig. 1, the system based on object detection and OCR and the method for extracting the information of the application fault elastic sheet of the present invention are characterized in that: the method comprises the steps of collecting screenshots of all fault popup windows of a target system, collecting at least 3 screenshots of different backgrounds and 3 screenshots of different positions for each fault, then training a model, and applying the obtained model to detection of fault popup frame pictures. In this embodiment, 5 screenshots of different backgrounds and 5 screenshots of different positions are selected and collected.
Wherein each shot of the fault popup is to appear in a variety of different backgrounds and locations. For example, screenshots of different positions in the upper, lower, left, right and middle are collected in the same background, and screenshots of different backgrounds can be collected in the same position, so that the collection of position and background information of more situations is facilitated, the constructed model can cover various situations, and the recognition efficiency is improved. In this embodiment, the YOLO4 object detection model is used to perform the popup position recognition. YOLO redefines object detection as a regression problem. It applies a single Convolutional Neural Network (CNN) to the entire image, divides the image into meshes, and predicts class probabilities and bounding boxes for each mesh. Since the detection problem is a regression problem, no complex piping is required. It is 1000 times faster than "R-CNN" and 100 times faster than "Fast R-CNN". Meanwhile, the technology can process real-time video streams, and the delay is less than 25 milliseconds. The accuracy is more than twice that of the previous real-time system.
In this embodiment, applying the obtained model to the detection of the faulty elastic frame picture includes the following steps:
step S101, inputting a picture to be detected;
step S102, preprocessing the picture;
step S103, calling the model to perform frame object detection, performing frame OCR processing if the object detection is successful, and entering step S104 if the frame object detection is unsuccessful;
if the detection of the bullet frame OCR processing is successful after the bullet frame OCR processing, the step S105 is carried out, and if the detection is unsuccessful after the bullet frame OCR processing, the step S104 is carried out; the unsuccessful detection of the bullet box OCR processing means that the picture character information cannot be extracted and recognized;
step S104, performing full-image OCR processing, entering step S105 if the full-image OCR processing detection is successful, and displaying call failure if the detection is failed, and ending; the fact that the detection of the full-image OCR processing is unsuccessful means that image character information cannot be extracted and recognized;
and step S105, outputting the picture character information.
In an embodiment of the application of the present invention, as shown in fig. 2, the error of user feedback is usually a screenshot of the whole screen containing a pop-up window, OCR does not sense the structure of the picture, so it scans the picture output text line by line from top to bottom and from left to right, and because the picture has a structure and the like, the problem that the characters in the same direction are not arranged exactly in one line often occurs, resulting in OCR failure, so the text information output by OCR usually contains much noise or is distorted, and such information is difficult to be used as input for subsequent customer service and question and answer systems without manual processing. The idea of applying the method of the invention to solve the problem is that: before OCR, an object detection process is performed. The position of the fault popup is located from the picture, the fault popup is extracted, as shown in fig. 3, and then the fault popup is used as an input to enter OCR processing, so that complete fault information without noise can be output: "receiving distribution network transfer order failure downloading and storing distribution network transfer order failure The operation has timed out".
It should be recognized that embodiments of the present invention can be realized and implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The methods may be implemented in a computer program using standard programming techniques, including a non-transitory computer-readable storage medium configured with the computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner, according to the methods and figures described in the detailed description. Each program may be implemented in a high level procedural or object oriented programming language to communicate with a computer system. However, the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Furthermore, the program can be run on a programmed application specific integrated circuit for this purpose.
Further, the operations of processes described herein can be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and/or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) collectively executed on one or more processors, by hardware, or combinations thereof. The computer program includes a plurality of instructions executable by one or more processors.
Further, the method may be implemented in any type of computing platform operatively connected to a suitable interface, including but not limited to a personal computer, mini computer, mainframe, workstation, networked or distributed computing environment, separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, and the like. Aspects of the invention may be embodied in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, optically read and/or write storage medium, RAM, ROM, or the like, such that it may be read by a programmable computer, which when read by the storage medium or device, is operative to configure and operate the computer to perform the procedures described herein. Further, the machine-readable code, or portions thereof, may be transmitted over a wired or wireless network. The invention described herein includes these and other different types of non-transitory computer-readable storage media when such media include instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor.
Finally, the above embodiments are only intended to illustrate the technical solutions of the present invention and not to limit the present invention, and although the present invention has been described in detail with reference to the preferred embodiments, it will be understood by those skilled in the art that modifications or equivalent substitutions may be made on the technical solutions of the present invention without departing from the spirit and scope of the technical solutions, and all of them should be covered by the claims of the present invention.

Claims (8)

1. A system based on object detection and OCR and a method for extracting information of an applied fault elastic picture are characterized in that: the method comprises the steps of collecting screenshots of all fault popup windows of a target system, collecting at least 3 screenshots of different backgrounds and 3 screenshots of different positions for each fault, then training a model, and applying the obtained model to detection of fault popup frame pictures.
2. The system based on object detection and OCR and the method for extracting the flip chip information by applying the fault according to claim 1, wherein: the screenshots of each fault popup appear in a variety of different backgrounds and locations.
3. The system based on object detection and OCR and the method for extracting the flip chip information by applying the fault according to claim 1, wherein: the detection of applying the obtained model to the fault bullet frame picture comprises the following steps:
step S101, inputting a picture to be detected;
step S102, preprocessing the picture;
step S103, calling the model to perform frame object detection, performing frame OCR processing if the object detection is successful, and entering step S104 if the frame object detection is unsuccessful;
if the detection of the popup window OCR processing is successful after the popup frame OCR processing, the step S105 is entered, and if the detection is unsuccessful after the popup frame OCR processing, the step S104 is entered;
step S104, performing full-image OCR processing, entering step S105 if the full-image OCR processing detection is successful, and displaying that the calling is failed if the full-image OCR processing detection is failed, and ending;
and step S105, outputting the picture character information.
4. The system based on object detection and OCR and the method for extracting the flip chip information by applying the fault according to claim 3, wherein: the YOLO4 object detection model was used for pop-up position identification.
5. The system based on object detection and OCR and the method for extracting the flip chip information by applying the fault according to claim 1, wherein: each fault is collected with 3-5 screenshots of different backgrounds and 3-5 screenshots of different positions.
6. The system based on object detection and OCR and the method for extracting the flip chip information by applying the fault according to claim 5, wherein: for each fault, 5 different background screenshots and 5 screenshots from different positions were collected.
7. A computer apparatus comprising a memory, a processor, and a computer program stored on the memory and capable of running on the processor, wherein: the processor, when executing the computer program, implements the method of any of claims 1-6.
8. A computer-readable storage medium having stored thereon a computer program, characterized in that: the computer program, when executed by a processor, implements the method of any one of claims 1-6.
CN202110925758.6A 2021-08-12 2021-08-12 System based on object detection and OCR and method for extracting information of application fault elastic picture Withdrawn CN113688835A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202110925758.6A CN113688835A (en) 2021-08-12 2021-08-12 System based on object detection and OCR and method for extracting information of application fault elastic picture

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202110925758.6A CN113688835A (en) 2021-08-12 2021-08-12 System based on object detection and OCR and method for extracting information of application fault elastic picture

Publications (1)

Publication Number Publication Date
CN113688835A true CN113688835A (en) 2021-11-23

Family

ID=78579638

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202110925758.6A Withdrawn CN113688835A (en) 2021-08-12 2021-08-12 System based on object detection and OCR and method for extracting information of application fault elastic picture

Country Status (1)

Country Link
CN (1) CN113688835A (en)

Similar Documents

Publication Publication Date Title
CN107666987A (en) Robotic process automates
KR102220174B1 (en) Learning-data enhancement device for machine learning model and method for learning-data enhancement
CN107223246A (en) Image labeling method, device and electronic equipment
US20220391228A1 (en) Method and system for accessing table content in a digital image of the table
CN109934227A (en) System for recognizing characters from image and method
US20220130160A1 (en) Object recognition method and apparatus, and electronic device and storage medium
CN107169340A (en) A kind of behavior formula identifying code processing method and processing device
US20200241900A1 (en) Automation tool
CN110598686A (en) Invoice identification method, system, electronic equipment and medium
CN111310155A (en) System architecture for automatic identification of slider verification code and implementation method
US20200183663A1 (en) System for image segmentation, transformation and user interface component construction
CN111310156B (en) Automatic identification method and system for slider verification code
TWI678614B (en) Test system
US20090158090A1 (en) Data entry retrieval
CN109815100B (en) Behavior monitoring method for CABAO software by utilizing image contrast analysis
CN113688835A (en) System based on object detection and OCR and method for extracting information of application fault elastic picture
CN111143188A (en) Method and equipment for automatically testing application
CN110827261B (en) Image quality detection method and device, storage medium and electronic equipment
CN113256583A (en) Image quality detection method and apparatus, computer device, and medium
CN106066979A (en) One operates in specialty bar code scanning method and system on cell phone platform
CN112036522A (en) Calligraphy individual character evaluation method, system and terminal based on machine learning
CN112650796A (en) Automatic application data collection and storage management system
CN110727595A (en) Application login interface identification method, intelligent terminal and storage medium
CN110275834A (en) User interface automatization test system and method
CN110659193B (en) Test system

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
WW01 Invention patent application withdrawn after publication

Application publication date: 20211123

WW01 Invention patent application withdrawn after publication