CN114862824A - Intelligent device for screen data capture analysis of industrial equipment - Google Patents
Intelligent device for screen data capture analysis of industrial equipment Download PDFInfo
- Publication number
- CN114862824A CN114862824A CN202210591131.6A CN202210591131A CN114862824A CN 114862824 A CN114862824 A CN 114862824A CN 202210591131 A CN202210591131 A CN 202210591131A CN 114862824 A CN114862824 A CN 114862824A
- Authority
- CN
- China
- Prior art keywords
- camera
- data
- display
- image
- display content
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000013481 data capture Methods 0.000 title claims abstract description 25
- 238000004458 analytical method Methods 0.000 title claims description 22
- 238000012545 processing Methods 0.000 claims abstract description 35
- 238000007405 data analysis Methods 0.000 claims abstract description 24
- 238000010191 image analysis Methods 0.000 claims abstract description 13
- 230000009466 transformation Effects 0.000 claims description 18
- 238000012937 correction Methods 0.000 claims description 13
- 230000015572 biosynthetic process Effects 0.000 claims description 12
- 238000004519 manufacturing process Methods 0.000 claims description 12
- 238000003786 synthesis reaction Methods 0.000 claims description 12
- 238000005457 optimization Methods 0.000 claims description 11
- 230000009471 action Effects 0.000 claims description 7
- 230000011218 segmentation Effects 0.000 claims description 5
- 238000012163 sequencing technique Methods 0.000 claims description 3
- 230000001131 transforming effect Effects 0.000 claims description 3
- 238000000034 method Methods 0.000 abstract description 28
- 239000000203 mixture Substances 0.000 abstract description 5
- 238000004891 communication Methods 0.000 abstract description 4
- 238000009434 installation Methods 0.000 description 21
- 230000008569 process Effects 0.000 description 20
- 230000008859 change Effects 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 230000000694 effects Effects 0.000 description 6
- 238000012544 monitoring process Methods 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 230000004927 fusion Effects 0.000 description 5
- 238000012015 optical character recognition Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 239000011159 matrix material Substances 0.000 description 4
- 239000000047 product Substances 0.000 description 4
- 238000013461 design Methods 0.000 description 3
- 238000013075 data extraction Methods 0.000 description 2
- 238000005034 decoration Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 238000009826 distribution Methods 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000006855 networking Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 239000011265 semifinished product Substances 0.000 description 2
- 238000003860 storage Methods 0.000 description 2
- 230000002194 synthesizing effect Effects 0.000 description 2
- 238000011144 upstream manufacturing Methods 0.000 description 2
- 238000009825 accumulation Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 239000002131 composite material Substances 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000011031 large-scale manufacturing process Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000002688 persistence Effects 0.000 description 1
- 230000002085 persistent effect Effects 0.000 description 1
- 238000002360 preparation method Methods 0.000 description 1
- 238000003825 pressing Methods 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 238000012800 visualization Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
- G06T7/0004—Industrial image inspection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/80—Geometric correction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30244—Camera pose
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Quality & Reliability (AREA)
- General Factory Administration (AREA)
Abstract
The invention discloses intelligent equipment for capturing and analyzing screen data of industrial equipment, which comprises a server and at least one camera; each camera is respectively arranged on a display outer frame of the industrial equipment through a camera telescopic bracket; each camera is in wired connection or wireless connection with the server; each camera is used for shooting display content on a display of the industrial equipment from the outside to obtain a display content image and then transmitting the display content image to the server; and the server is used for carrying out image analysis processing on the display content images transmitted by the plurality of cameras to obtain and store image analysis results. The method can solve the problem that the capturing and analyzing of the operation data displayed on the display screen of the industrial equipment, the existing data of which do not support external connection, cannot be carried out. The invention has the capability of data capture and analysis under the conditions of not carrying out signal communication with the original industrial equipment and not modifying the software and hardware composition of the original industrial equipment.
Description
Technical Field
The invention relates to the technical field of automation control, in particular to intelligent equipment for screen data capture and analysis of industrial equipment.
Background
Computer image processing algorithms and data analysis techniques are of great significance in the field of machine vision. At present, the industrialization and the informatization are deeply integrated and developed, a large amount of automatic production equipment is put into use, and a large amount of industrial equipment is provided with a display and a control panel. The display provides an IO interface (input/output interface) between the device and a user, and the control panel provides an operation interface.
In order to improve the industrial intelligence and automation level, the demands for collecting and analyzing data generated by the operation of industrial equipment are gradually increased. For example, monitoring the operation of equipment in a production-type enterprise of a production line: on one hand, data of upstream and downstream production equipment of the same assembly line are summarized, analyzed and displayed, the working loads of the upstream and downstream equipment can be monitored and coordinated in a centralized manner, the situation of semi-finished product accumulation is avoided when local processing steps are blocked, and the operating efficiency and stability of the production line are improved; on the other hand, data of a plurality of assembly line devices are summarized, semi-finished product circulation among production lines is coordinated, and the overall intelligent control level of an enterprise can be improved.
Currently, some modern devices provide data interaction with an external data interface. However, there are also a lot of industrial equipments that do not support external data expansion due to the influence of objective factors such as equipment type, equipment producing country, deployment time, and limited factory technical level, or the original equipment cannot be modified due to the serious influence possibly generated by modifying the original equipment.
For industrial equipment of which the data do not support external connection, the data are generally acquired by adopting a manual meter reading mode at present. The working personnel regularly copy the data on the industrial equipment at fixed points to acquire the state of the equipment, and manually adjust the operating parameters of the equipment and other associated equipment according to the state of the equipment to further control the operation of the associated equipment. This approach is not only costly, but also inefficient. And in many industrial equipment in high-risk areas, the data are not easy to collect in a manual meter reading mode, and the stable operation of the equipment is seriously influenced.
Therefore, there is an urgent need to develop a technology capable of solving the above technical problems.
Disclosure of Invention
The invention aims to provide intelligent equipment for capturing and analyzing screen data of industrial equipment, aiming at the technical defects in the prior art.
To this end, the invention provides an intelligent device for screen data capture analysis of an industrial device, comprising a server and at least one camera;
each camera is respectively arranged on an outer frame of a display (31) of the industrial equipment through a camera telescopic bracket;
each camera is in wired connection or wireless connection with the server;
each camera is used for shooting display content on a display of the industrial equipment from the outside to obtain a display content image and then transmitting the display content image to the server);
and the server is used for carrying out image analysis processing on the display content images transmitted by the plurality of cameras to obtain and store image analysis results.
Compared with the prior art, the intelligent equipment for capturing and analyzing the screen data of the industrial equipment is scientific in design and can solve the problem that the capturing and analyzing of the display operation data of the display screen on the industrial equipment, of which the existing data does not support external connection, cannot be performed. The equipment has the capability of data capture and analysis under the conditions of not carrying out signal communication with the original industrial equipment and not modifying the software and hardware composition of the original industrial equipment, and has great practical significance.
Drawings
Fig. 1 is a schematic configuration diagram of an intelligent device for screen data capture analysis of an industrial device according to the present invention, which shows a kit and a packaging method under a current configuration by taking an example of a three-camera configuration. The method comprises the steps of digging grooves in an aircraft, wherein the grooves respectively correspond to a portable server 100, a wiring placement area 101, an accessory expansion area 102, a camera placement area 103 and a camera expansion area 104, and the specific digging grooves are specifically adjusted by the configuration of a kit.
FIG. 2 is a schematic diagram of a portion of an assembly of an industrial plant. The input devices of the industrial equipment mainly comprise an operation panel and an operation button area, and the display devices of the equipment comprise an LED display, an LCD display, an ink screen and the like.
FIG. 3 is a schematic illustration of a single camera installation on an industrial facility;
FIG. 4 is a schematic diagram of a wired connection scheme adopted by an intelligent device for screen data capture analysis of an industrial device according to the present invention;
FIG. 5 is a schematic diagram of a wireless connection scheme adopted by an intelligent device for screen data capture analysis of an industrial device according to the present invention;
FIG. 6 is a schematic view of a manipulator mounting arrangement;
FIG. 7 is a schematic view of an inter-line robot assembly scheme; the installation of the device can be expanded to a medium-large scale production line. The camera is mounted to a display for data capture analysis and the robotic automation components are mounted to the flow line.
FIG. 8 is a flow chart of the operation of the apparatus of the present invention;
fig. 9 is a flowchart of data optimization after image capture according to the present invention.
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the following embodiments of the present invention, and it should be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
In the description of the present invention, it is to be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", and the like, indicate orientations or positional relationships based on those shown in the drawings, and are used only for convenience in describing the present invention and for simplicity in description, and do not indicate or imply that the referenced devices or elements must have a particular orientation, be constructed and operated in a particular orientation, and thus, are not to be construed as limiting the present invention.
In the description of the present invention, it should be noted that, unless otherwise explicitly specified or limited, the terms "mounted," "connected," and "connected" are to be construed broadly, e.g., as meaning either a fixed connection, a removable connection, or an integral connection; can be mechanically or electrically connected; they may be connected directly or indirectly through intervening media, or they may be interconnected between two elements. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art through specific situations.
The present invention will be described in detail below with reference to the embodiments with reference to the attached drawings.
Referring to fig. 1 to 9, the present invention provides an intelligent device for screen data capture analysis of an industrial device, comprising a server 1 and at least one camera 2;
each camera 2 is respectively arranged on the outer frame of the display 31 of the industrial equipment 3 through a camera telescopic bracket 4;
each camera 2 is connected to the server 1 by wire or wirelessly.
In the present invention, the server 1 is used for data analysis and installation application.
It should be noted that, in the present invention, the number of cameras is determined by the screen specification of the display 31 of the industrial equipment 3, the total amount of data analysis, and the image capturing environment.
In the present invention, the server 1 includes a fixed server and a portable server.
It should be noted that, in the present invention, the server 1 is a server mainly used for data analysis and providing control logic. The portable server is generally used as the main part during the installation and debugging period of the equipment. The portable server can flexibly adapt to the installation environment, demonstrates the effectiveness of the system and is easy to change the configuration of the server. After deployment and debugging are completed, the server 1 can be replaced by a fixed server according to actual conditions. In the debugging stage, a display carried by the portable server is adopted to capture data, analyze and debug,
in actual use, the display is not generally required to display the current status. The user can select to access the display signal to the monitoring room for centralized monitoring.
In the present invention, the camera 2 is connected to the server 1 through the network cable 7.
Fig. 4 is a diagram of a wired connection scheme. The wired connection scheme installs several cameras around the display, 3 cameras are taken as an example in the figure. In actual installation, the limitation of installation environment is required to be referred to, and a plurality of installation modes such as left and right installation, upper left installation, upper right installation and the like are commonly adopted. The camera faces the display screen to capture images, and the included angle between the sight line of the image capturing device and the center perpendicular line of the display screen is as small as possible during installation. The camera and the display are adjusted from the reference camera lens. Referring to fig. 4, in this scheme, the portable server may be equipped with a 4-port network card or a USB card, and the camera is directly connected to the portable server for data analysis.
In the invention, in a concrete implementation, the camera 2 is connected with the wireless switch;
the wireless switch is connected with the server 1 through a network cable.
In the present invention, in a specific implementation, the camera telescopic bracket 4 is used as a supporting device, and is mainly used for providing position adjustment and support for the camera. And when the system is installed, capturing and analyzing the effect according to the system data to adjust.
In particular, referring to fig. 3, when only one camera 2 is provided, the camera telescopic bracket 4 includes a fixing member 41 and three pairs of telescopic rods 40;
each pair of extendable rods 40 is articulated by a rotatable connection 42;
an outer frame above the display 31 of the industrial equipment 3 is connected to one ends of a pair of extendable rods 40 by a fixing member 41, and the other ends of the pair of extendable rods 40 are hinged (i.e., rotatably connected) to the rear side of the camera 2 on which no lens is mounted;
the outer frames on the left and right sides of the display 31 of the industrial equipment 3 are respectively hinged (i.e., rotatably connected) to the rear side of the camera 2 on which no lens is mounted through a pair of extendable rods 40.
It should be noted that, referring to fig. 3, a single-camera installation mode may be adopted in the case where the display screen has a limited area, the display surface is free from obstacles, and the production line is not affected. The fixing piece and the connecting piece are tightened by fixing the camera to the front face of the display and adjusting the angle of the camera through a plurality of brackets (the number is based on the effective fixing of the camera).
In a specific implementation, referring to fig. 4, when a plurality of cameras 2 are included, each camera telescopic bracket 4 includes a fixed member 41 and a pair of telescopic rods 40;
each pair of extendable rods 40 is articulated by a rotatable connection 42;
each camera 2 is hinged to one end of a pair of extendable rods 40, and the other end of the pair of extendable rods 40 is fixedly connected to one side outer frame of the display 31 of the industrial equipment 3.
In the present invention, in terms of specific implementation, the data connection line related to the intelligent device of the present invention will be configured according to actual situations. Typical cases include data transmission lines, power lines, robot control lines, etc.
It should be noted that, in the present invention, the basic signal is obtained by capturing and analyzing the display data on the display screen. Simple data in the basic signal comprise a signal lamp state display signal, a digital panel data display signal, an equipment state code display signal and the like. The complex data in the basic signal may be complex display and control software matched with the device, and an interface of such software may include a plurality of controls, such as a button control, a state display control, a data display control, a table display control, and the like. The display and control software has various display contents and complex types, and is a main target of the equipment for data analysis.
In the present invention, in a specific implementation, each camera 2 is respectively configured to externally capture display content on the display 31 of the industrial device 3, obtain a display content image, and then transmit the display content image to the server 1;
and the server 1 is used for carrying out image analysis processing on the display content images transmitted by the plurality of cameras 2, obtaining and storing image analysis results. Further, automation control functions may be customized as desired.
In the present invention, in a specific implementation, the image analysis processing of the server 1 is based on a two-dimensional image processing library and a character recognition library, and an image processing and data analysis program is developed to form an image processing architecture.
Fig. 8 is a flowchart of the device according to the present invention, in which the device starts with the camera image capture, and a stage is shown when the screen captures data and analyzes the captured data to obtain "data", and the data can be subsequently used to control the mechanical device or be used by other data applications.
Referring to fig. 8, the image analysis processing operation of the server 1 specifically includes the following processing steps:
step S1, performing data optimization on the display content image: grouping the display content images transmitted by the cameras 2 according to different cameras (namely, dividing the images of each camera into a group for processing), judging a plurality of display content images which are acquired by each camera and in the same group and meet the approaching condition of the image-taking time, listing the display content images in each camera group at the same time (namely, judging stable images), sequencing and numbering the display content images in each camera group at the same time according to the image-taking time of each display content image, and then performing optimization processing to acquire a plurality of groups of display content images after data optimization;
in step S1, the image capturing time approaching condition includes: the time interval of the image capturing time of any two display content images in each camera set is <2 × Threshold _ 2;
the value of Threshold _2 is determined by the image capturing frequency fps of the camera 2;
Threshold_2＝1/fps。
in particular implementation, in step S1, the optimization processing operation for multiple display content images in each camera group at the same time is specifically as follows:
performing image and data identification on a plurality of continuous display content images in each camera group at the same moment (the same data refers to the same content of two images to be compared), using the existing image subtraction algorithm during the identification, because a plurality of display content images in each camera group are taken by the same camera in a rapid and continuous manner currently, if the display content on the display 31 of the industrial equipment 3 is not changed (i.e. the screen data is not changed), the plurality of continuous display content images are the same (the same refers to the same data composition and distribution of the two images), and after the identification, if the plurality of continuous display content images are not changed (for example, the similarity is greater than or equal to a preset percentage, such as 99.5%), only one display content image at the moment is retained, and the other display content images at the moment are deleted;
it should be noted that the image subtraction algorithm is a well-known image processing algorithm that is disclosed in the prior art, and is not described herein again.
It should be noted that the single-camera image capturing does not require a systematic (i.e., time-uniform) step. And the time system step also carries out the same-genus judgment on the captured data, directly carries out the image and same-data judgment under the condition of no correction, and deletes the repeated image if the image is not changed.
Step S2, distortion correction processing: the plurality of sets of display content images optimized by the data obtained in step S1 are subjected to a perspective transformation operation based on the perspective transformation parameters, and transformed into display content images in the forward direction.
In step S2, the perspective transformation parameters used in the distortion correction process are obtained when the camera is mounted and debugged, and are input to the server 1, and the perspective transformation parameters are directly called in the stable operation process of the industrial equipment to correct the image.
It should be noted that the perspective transformation is a process of transforming according to the object imaging projection rule, i.e. re-projecting the object to a new imaging plane. Because the inclination angle of the camera field of view is different from that of the object, the object image is distorted differently, and the correction of the object image is usually realized through perspective transformation. In the perspective transformation, the transformation relationship between the image before perspective and the image after perspective can be represented by a 3 × 3 matrix transformation matrix, which can be obtained by coordinates of four corresponding points in two images, and therefore the perspective transformation is also referred to as "four-point transformation". This 3 × 3 matrix transformation matrix is a perspective transformation parameter.
It should be noted that, the process of correcting the image distortion employs a publicly known processing algorithm, and it is common that: when the camera is installed and debugged, the perspective transformation parameters are solved by using a camera calibration board (standard product) and the existing known camera parameter calibration method, and after the parameters are solved, the distortion correction processing is carried out on the image data of the camera by using the perspective transformation parameters.
In the specific implementation, a screen of the conventional industrial equipment is a rectangular display, and the forward display of the screen is performed on the rectangular display, wherein the content image is a two-dimensional image with rectangular distribution, the outside of the rectangle is taken as the horizontal corresponding to the display range of the rectangular display, and the characters in the image are forward displayed (the upper side of the display screen display range is horizontal to the upper side of the image, the lower side of the display screen display range is horizontal to the lower side of the image, the left side of the display screen display range is horizontal to the left side of the image, and the right side of the display screen display range is horizontal to the right side of the image). When the OCR recognition is called, the text in the image is required to be distributed in a forward direction, otherwise, an accurate recognition result is difficult to obtain (for example, two characters, namely 'A' and 'B', are recognized in different directions, and errors are directly recognized). It is necessary to process the image using a perspective transformation.
It should be noted that the distortion of the image is introduced by the manufacturing accuracy of the camera and the deviation of the assembly process, and the correction process is a process of transforming the perspective of the current image into the forward image capturing effect. And after the data analysis is completed in the server, persistence and visualization processing of background data are performed to provide equipment data analysis for use. The data analysis can also be used to derive the signal quantity for the automatic control of the automatic mechanical components.
Step S3, multi-angle image capture synthesis: performing full-time image synthesis on the multiple groups of display content images which are subjected to distortion correction processing in the step S2 and are sorted according to the image capturing time according to the sequence of the image capturing time to obtain synthesized full-time images;
it should be noted that, during multi-path image capture, multiple groups of display screen screenshots after distortion correction are acquired by the multi-path cameras, and multi-angle image capture synthesis is used to improve the quality and improve the accuracy of data analysis. And during synthesis, performing full-time image synthesis according to the data distributed by the optimized multi-path data in the step 3) at time.
It should be noted that the full-time image refers to an image obtained by multiple cameras and optimized by multiple sets of images with image capturing time being collected into one set of image data, all images of the multiple sets of images with image capturing time and image capturing time parameters thereof are included, preparation is made for full-time synthesis, in the full-time image, image capturing data of one camera is manually designated as main data, and logic is designated as a camera with distortion correction image closest to the display effect of the device screen.
In the concrete implementation, the full-time image synthesis takes the main data as the basis, and the images after the distortion correction of other cameras are fused with the main data. Common image fusion algorithms for full-time image synthesis include the existing known algorithms such as a weighted average method, a feathering algorithm, a laplacian pyramid fusion algorithm and the like, and through a contrast effect, the image fusion of the invention adopts the laplacian fusion algorithm. The purpose of image fusion is to fuse redundant information and complementary information in a plurality of original images and strengthen the information in the images so as to facilitate further analysis of the images.
It should be noted that, in the present invention, referring to fig. 9, in step S1, the process of performing data optimization on the display content image mainly includes two aspects, on one hand, the same data image is to be removed; on the other hand, after the same data image is removed, the multi-path data matching still can match the image data at the same time according to time, and then the matching calculation is carried out. Therefore, when the device of the invention optimizes the data, the initial position image in the current same data image is stored and numbered, and when the initial position image is changed, a new image is stored to the system and numbered.
Fig. 9 is a flowchart of data optimization after image capture according to the present invention. Taking a dual-camera as an example, the dual-camera respectively and continuously captures images. For the camera 1, images are continuously taken from an image at an arbitrary time T10, the next image data is marked as T11, the T11 and the T10 image are compared, the same data is determined, the same data image is removed, and then the T12 and the T13 are sequentially processed until the last same data image T1n is obtained. And (4) judging that the newly acquired data are different, and repeating the process. Thereby obtaining optimized data of the camera 1. Thereby similarly acquiring the optimized data of the camera 2. Under the drive of a clock in the system, the subsequent image synthesis process is carried out according to the judgment of time and time.
Obviously, in step S3, the composite image is captured at multiple angles at the same time, so that a more ideal data analysis effect can be achieved. However, under the constraint of objective conditions of the equipment, the image capturing time may not be completely matched. When the server processes the image, the time of the camera is adjusted by taking the local time of the current server as a reference. And when the data is captured into the memory, the time is accurately marked, and the plurality of cameras are subjected to time sequencing. And whether the change occurs is judged by adopting the difference between continuous frames of the single camera to mark the change time, and the time system process of the system is realized by synthesizing data at the same time of the multiple cameras.
Step S4, matching the display screen segmentation template: segmenting a region with characters in a display software interface in the synthesized full-time image;
in step S4, for different types of display software interfaces, the regions having characters therein are determined in advance by performing human-assisted segmentation in advance.
In the present invention, a manual assistance division function is provided for a complicated display software interface, and OCR recognition (optical character recognition) and data classification are performed for each divided region.
Step S5, OCR recognition: the region having the character in the display software interface obtained by the division in step S4 is subjected to character recognition, and the character displayed in the display software interface is extracted.
Note that, in step S5, a user identification classification function and a data definition function are provided. And combining the identification definition of the characters, and continuously analyzing and recording the concerned data.
In the present invention, the OCR recognition algorithm may adopt the existing well-known common algorithm, which is the existing mature technology and is not described herein again.
In the invention, for a complex display software interface, the server 1 can store a plurality of interface templates in advance, thereby providing higher flexibility.
Step S6, data analysis processing operation: a plurality of key characters are stored in advance, the characters displayed in the display software interface obtained in step S5 are analyzed, the characters displayed in the display software interface are compared with the plurality of key characters stored in advance, the key characters and data corresponding to the key characters are extracted, and the extracted key characters and data are used as final data for performing data capture analysis on the display content on the display 31 of the industrial equipment 3, that is, the image analysis processing result.
It should be noted that the characters displayed in the display software interface can be analyzed, the key characters are extracted, and the final data captured and analyzed by the device of the present invention is obtained. For example, a software interface of a production device in a midstream of a production line has a device load monitoring display value X, the reasonable range is 20-80%, and the device is adopted to monitor the display value. The device will analyze the load condition stably to obtain the value and change time of the load condition.
In step S6, after extracting the key characters and the data corresponding to the key characters, the method further includes step S7:
step S7, data analysis control step: for a plurality of key characters stored in advance, the corresponding relation between a plurality of different data threshold value ranges of each key character and a plurality of different control signals is respectively preset, and then, for all the extracted key characters, when the data value of one key character is within a preset data threshold value range of the key character, the corresponding control signal is read and then sent to the external manipulator 5;
an external manipulator 5, configured to perform a touch operation on a corresponding function key (e.g., a function-off case, a function-pause key) on the operation button area 32 provided on the industrial device 3 according to a control signal (e.g., a function-off control signal, a function-pause control signal) transmitted from the server 1 when installed above the operation button area 32 provided on the industrial device 3, thereby implementing an operation control on the industrial device 3;
note that, for the industrial device 3, an operation panel area 30 for a user to perform a touch operation is further provided beside the operation button area 32.
Or, the external manipulator 5 is configured to, when installed at the flow line 6 where the industrial equipment 3 is located, perform a corresponding preset action on the product on the flow line according to a control signal (e.g., a grabbing control signal) sent by the server 1 (e.g., perform a grabbing action according to the grabbing control signal, and grab away a product on the flow line whose product detection data displayed by the display of the industrial equipment does not meet the requirement).
Referring to fig. 6, fig. 6 is a schematic view of a belt manipulator fixing assembly scheme. On the basis of data capture and analysis, the manipulator device can be configured to automatically upgrade the system. Generally, the portable installation mode is not adopted any more at this time, and the fixed server is replaced. The installation of the manipulator is carried out according to specific requirements and specific conditions on a production line.
In a specific implementation, a threshold may be set on the detected character data to change the continuously changing data amount into a semaphore. For example, the monitoring value is 0 to 100, and the signal S1 can be determined by closing 0 to 30 and opening the left and right; 30-60 left closed and right opened, and determining as a signal S2; 60-100 left closed and right closed, which are defined as signal S3. The purpose of converting continuous numerical quantity into discrete signal state is achieved, and the operation of industrial equipment is monitored. And the multi-state of the text description can be converted into semaphore to assist in centralized monitoring of the operation of the industrial equipment.
In a specific implementation, the server 1 may convert the semaphore into a control signal to be input to an external automation device (e.g., an external manipulator) for use. External automation devices (e.g., external robots) provide functionality that corresponds actions to control signals.
In a specific implementation, the server 1 may further be provided with a control logic writing module. The mechanical assembly (e.g., an external robot) of the present invention provides a continuous motion function of fixed points in an effective three-dimensional space. For example, a simple application is to acquire a signal on the current display screen, and control the external manipulator 4 to perform a button pressing action on a corresponding function key on the operation panel 32 provided on the industrial equipment 3. The invention can control the manipulator to carry out continuous multi-point actions by using the control logic compiling module, and respectively edit the execution sequence and the execution relative time for the multiple points, thereby improving the execution capacity of complex actions.
In addition, the data analyzed and extracted by the server 1 can be used for data classification, persistent storage, data analysis and application by users.
Based on the technical scheme, the equipment provided by the invention is intelligent equipment which can acquire data, analyze the data and expand automatic peripheral equipment by installing a camera outside mechanical equipment with a display screen.
It should be noted that, the intelligent device provided by the present invention is used in a situation where the existing device industrial device does not provide an external data interface and the user needs to acquire and analyze the screen display data. On the basis of analyzing data, the capability of a data control machine can be provided, so that a user can connect the result of analyzing data with peripheral control, the effects of integrated detection and integrated control of multiple devices are achieved, and the intelligent level of the devices is improved.
The intelligent equipment provided by the invention can solve the problem that the display operation data of the display screen on the industrial equipment, the conventional data of which does not support external connection, cannot be captured and analyzed. The equipment has the capability of data capture and analysis under the conditions of not carrying out signal communication with the original industrial equipment and not modifying the software and hardware composition of the original industrial equipment.
The invention provides intelligent (automatic) equipment which can capture and analyze data of a display screen of industrial equipment and can be matched with a manipulator for automatic control under the condition of not changing the structure of the original equipment.
The invention relates to intelligent upgrading equipment which can carry out display screen data capture analysis and data networking collection on target equipment under the condition of not changing the structure of the original equipment. On the basis, the function of controlling and associating various devices under the condition of no signal intercommunication is realized by adding an automatic control component. The invention is suitable for the condition that the target equipment displays the operation data through the display screen but does not provide an external interface, and the equipment can be used for capturing and analyzing the screen data. Meanwhile, the equipment is suitable for automatic expansion when the state change of the target equipment needs to be linked with other equipment.
The invention relates to intelligent (automatic) equipment which can be matched with a manipulator to carry out automatic control.
In the invention, the camera telescopic support 4 is used for adapting to a complex installation environment, and a plurality of cameras are installed for synthesizing a plurality of images to improve the image quality and further improve the data capturing capability. The portable server is used for receiving the camera return data and developing a complex display interface data segmentation and classification control system based on an image processing algorithm, so that the equipment can be used for assembling the target industrial equipment without limitation. The server of the invention provides an external data interface, and a user can monitor the operation of the target industrial equipment by using the export data.
In order to more clearly understand the technical solution of the present invention, the following describes the working procedure of the present invention.
Referring to fig. 4, the device of the present invention is wired to a target industrial device. The working flow of the equipment is as follows:
1) the cameras are installed on the outer frame of the display, the number of the installed cameras is determined by the complexity of the display content of the display screen of the device and the data analysis quality, and the angle of the support and the installation number are adjusted according to the actual situation.
2) And connecting the camera to a portable server, and obtaining data which is classified, collected and analyzed through image processing and character recognition. And checking whether the data are consistent with expectations or not, and if not, improving the image capturing quality by adjusting the position of the camera and the angle of the bracket until the purpose of extracting the data is achieved.
3) And developing and installing an equipment monitoring program and an equipment control program according to the screen display data of the target equipment obtained by analysis.
4) And the mechanical arm can be expanded and installed to a specified position according to needs, and a control program is developed according to needs to control the mechanical arm to operate an operation panel of the target device. After the installation, the portable server is modified into a fixed server.
5) In order to facilitate installation, the wireless installation scheme provided by the invention can adopt a wireless camera to be connected with a wireless switch, and the wireless switch is connected with a server through a network cable.
The screen data of the target equipment is captured, analyzed and automatically operated by the process, and intelligent upgrading of the equipment is completed.
Compared with the prior art, the intelligent equipment for screen data capture and analysis of the industrial equipment, provided by the invention, has the following beneficial effects:
1. the invention relates to a set of equipment which can capture data of a display screen of the equipment by a flexibly-installed image capturing structure and intelligently upgrade the outside of the equipment by combining an image processing technology and an automation technology.
2. The use of the equipment of the invention provides a set of solution for the automatic upgrade of the equipment which is not easy to change and expand. The data extraction, analysis, classification, automatic control extension and other applications are provided for the equipment.
3. The data extraction process of the equipment optimizes the extracted data, reduces the complexity of the system and further reduces the equipment cost of the system. In the data identification process, data are converted into discrete signals and further correspond to logic signals, and a continuous application process from screen capture analysis of data to automatic control is provided.
4. The equipment is transparent to the equipment to be analyzed, and does not need to communicate circuits, signals and data with the equipment to be analyzed, so that the integrity of the original equipment is ensured. The support design and the image distortion correction process of the equipment ensure that the equipment is flexibly installed and can adapt to various complex installation environments, thereby improving the application range of the invention.
5. In the data processing process, the invention adopts the processes of image optimization of the same data, multi-angle synthesis of image data at the same time and the like, thereby greatly improving the total amount of processing data which can be born by the equipment and leading the equipment to have the capability of capturing and analyzing large-scale screen data.
6. By adopting the equipment disclosed by the invention, an enterprise can automatically upgrade history storage equipment, equipment with insufficient automation degree and expansion equipment which is difficult to change. The coverage of multiple devices in the internet can be expanded through the networking automation process.
In summary, compared with the prior art, the intelligent device for capturing and analyzing the screen data of the industrial equipment, which is provided by the invention, has a scientific design, and can solve the problem that the capturing and analyzing of the display operation data of the display screen on the industrial equipment, which does not support external connection, cannot be performed on the existing data. The equipment has the capability of data capture and analysis under the conditions of not carrying out signal communication with the original industrial equipment and not modifying the software and hardware composition of the original industrial equipment, and has great practical significance.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.
Claims (8)
1. An intelligent device for screen data capture analysis of industrial devices, characterized by comprising a server (1) and at least one camera (2);
each camera (2) is respectively arranged on the outer frame of a display (31) of the industrial equipment (3) through a camera telescopic bracket (4);
each camera (2) is in wired connection or wireless connection with the server (1);
each camera (2) is used for shooting display content on a display (31) of the industrial equipment (3) from the outside, obtaining a display content image and then transmitting the display content image to the server (1);
and the server (1) is used for carrying out image analysis processing on the display content images transmitted by the plurality of cameras (2) to obtain and store image analysis results.
2. The intelligent device for screen data capture analysis of industrial devices according to claim 1, wherein the camera (2) is connected to the server (1) via a network cable (7);
or the camera (2) is connected with a wireless switch, and the wireless switch is connected with the server (1) through a network cable.
3. The intelligent device for screen data capture analysis of industrial devices according to claim 1, wherein the camera telescoping support (4) comprises one fixed part (41) and three pairs of telescoping rods (40) when having only one camera (2);
each pair of telescopic rods (40) is hinged through a rotatable connecting piece (42);
an outer frame above a display (31) of the industrial equipment (3) is connected with one ends of a pair of telescopic rods (40) through a fixing piece (41), and the other ends of the pair of telescopic rods (40) are hinged with the rear side surface of the camera (2) without a lens;
the outer frames on the left and right sides of the display (31) of the industrial equipment (3) are respectively hinged with the rear side surface of the camera (2) without a lens through a pair of telescopic rods (40).
4. The intelligent device for screen data capture analysis of industrial devices according to claim 1, wherein when comprising a plurality of cameras (2), each camera telescoping support (4) comprises a fixed member (41) and a pair of telescoping rods (40);
each pair of telescopic rods (40) is hinged through a rotatable connecting piece (42);
each camera (2) is hinged with one end of a pair of telescopic rods (40), and the other end of the pair of telescopic rods (40) is fixedly connected with an outer frame at one side of a display (31) of the industrial equipment (3).
5. The intelligent device for screen data capture analysis of industrial equipment according to claim 1, wherein the image analysis processing operation of the server 1 specifically comprises the following processing steps:
step S1, performing data optimization on the display content image: grouping the display content images transmitted by the cameras (2) according to different cameras, judging a plurality of display content images which are acquired by each camera and meet the approaching condition of the image capturing time in the same group as a plurality of display content images in each camera group which are listed as the same time, sequencing and numbering the plurality of display content images in each camera group at the same time according to the image capturing time of each display content image, and optimizing to acquire a plurality of groups of display content images after data optimization;
step S2, distortion correction processing: performing perspective transformation operation on the plurality of sets of display content images optimized by the data obtained in step S1 according to the perspective transformation parameters, and transforming the display content images into forward display content images;
step S3, multi-angle image capture synthesis: performing full-time image synthesis on the multiple groups of display content images which are subjected to distortion correction processing in the step S2 and are sorted according to the image capturing time according to the sequence of the image capturing time to obtain synthesized full-time images;
step S4, matching the display screen segmentation template: for a display software interface in the synthesized all-time image, segmenting an area with characters in the display software interface;
step S5, OCR recognition: performing character recognition on the region with the characters in the display software interface obtained by the segmentation in the step S4, and extracting the characters displayed in the display software interface;
step S6, data analysis processing operation: and pre-storing a plurality of key characters, analyzing the characters displayed in the display software interface obtained in the step S5, comparing the characters displayed in the display software interface with the pre-stored key characters, extracting the key characters and data corresponding to the key characters, and taking the key characters and the data as final data for performing data capture analysis on the display content on the display (31) of the industrial equipment (3), namely an image analysis processing result.
6. The intelligent device for screen data capture analysis of industrial equipment according to claim 5, wherein in step S1, the image capture time is close to a condition, specifically: the time interval of the image capturing time of any two display content images in each camera set is <2 × Threshold _ 2;
Threshold_2＝1/fps；
fps is the image capture frequency of the camera.
7. The intelligent device for screen data capture analysis of industrial equipment according to claim 1, wherein in step S1, the optimization processing operation for multiple display content images within each camera group at the same time is as follows:
and judging the same image and data of a plurality of continuous display content images in each camera group at the same time, if the plurality of continuous display content images are not changed, only keeping one display content image at the time, and deleting other display content images at the time.
8. The intelligent device for screen data capture analysis of industrial equipment according to claim 1, wherein in step S6, after extracting key characters and data corresponding to the key characters therein, further comprising step S7:
step S7, data analysis control step: for a plurality of key characters stored in advance, the corresponding relation between a plurality of different data threshold value ranges of each key character and a plurality of different control signals is respectively preset, and then, for all the extracted key characters, when the data value of one key character is within a preset data threshold value range of the key character, the corresponding control signal is read and then sent to an external manipulator (5);
the external manipulator (5) is used for performing touch operation on corresponding function keys on an operation button area (32) arranged on the industrial equipment (3) according to a control signal sent by the server (1) when the external manipulator is arranged above the operation button area (32) arranged on the industrial equipment (3) so as to realize the operation control on the industrial equipment (3);
or the external manipulator (5) is used for executing corresponding preset actions on products on the flow production line according to control signals sent by the server (1) when the external manipulator is installed beside the flow production line (6) where the industrial equipment (3) is located.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210591131.6A CN114862824A (en) | 2022-05-27 | 2022-05-27 | Intelligent device for screen data capture analysis of industrial equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210591131.6A CN114862824A (en) | 2022-05-27 | 2022-05-27 | Intelligent device for screen data capture analysis of industrial equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114862824A true CN114862824A (en) | 2022-08-05 |
Family
ID=82641547
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210591131.6A Pending CN114862824A (en) | 2022-05-27 | 2022-05-27 | Intelligent device for screen data capture analysis of industrial equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114862824A (en) |
-
2022
- 2022-05-27 CN CN202210591131.6A patent/CN114862824A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| JP6867555B2 (en) | Automated optical inspection system based on CPU + GPU + FPGA architecture | |
| US10930037B2 (en) | Image processing device for displaying object detected from input picture image | |
| US8896535B2 (en) | Image processing apparatus and method, and program therefor | |
| US20030120714A1 (en) | Human/machine interface for a machine vision sensor and method for installing and operating the same | |
| CN103973969A (en) | Electronic device and image composition method thereof | |
| WO2017057780A1 (en) | Data collection device, method, and program for display panel or control panel | |
| US20210364996A1 (en) | Programmable logic controller | |
| CN111225143B (en) | Image processing apparatus, control method thereof, and program storage medium | |
| KR101825725B1 (en) | Method and device for diagnosing deterioration of solar cell module | |
| TW201931228A (en) | Automatic equipment management system and method thereof | |
| TW201913355A (en) | System module and method structured on customized display frame of Non-invasive data extraction system providing the basic functions of a plurality of KVM switches by utilizing a signal receiving device and an HID (Human Interface Device) simulation device | |
| JP2020144660A (en) | Information processing apparatus, information processing method and program | |
| CN110536074A (en) | A kind of intelligent inspection system, method for inspecting | |
| CN111830039A (en) | Intelligent product quality detection method and device | |
| CN115880296A (en) | Machine vision-based prefabricated part quality detection method and device | |
| WO2022193640A1 (en) | Robot calibration method and apparatus, and robot and storage medium | |
| CN108972556B (en) | Wire grabbing system and method in complex illumination environment on micro special motor production line | |
| CN114862824A (en) | Intelligent device for screen data capture analysis of industrial equipment | |
| CN114264659A (en) | Image processing apparatus and method, inspection apparatus, and computer-readable storage medium | |
| CN113870432A (en) | Bus-based non-cooperative ship target full-automatic three-dimensional reconstruction system | |
| CN111158669A (en) | Graphical programming visual system and industrial control method | |
| CN114138458A (en) | Intelligent vision processing system | |
| CN115841520A (en) | Camera internal reference calibration method and device, electronic equipment and medium | |
| CN112040131A (en) | Intelligent camera based on EtherCAT industrial Ethernet bus and control method thereof | |
| US20210129318A1 (en) | Annotation device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |