CN110222575B - Equipment comprehensive efficiency analysis method, device, system, equipment and storage medium based on attention mechanism target detection - Google Patents
Equipment comprehensive efficiency analysis method, device, system, equipment and storage medium based on attention mechanism target detection Download PDFInfo
- Publication number
- CN110222575B CN110222575B CN201910376669.3A CN201910376669A CN110222575B CN 110222575 B CN110222575 B CN 110222575B CN 201910376669 A CN201910376669 A CN 201910376669A CN 110222575 B CN110222575 B CN 110222575B
- Authority
- CN
- China
- Prior art keywords
- analyzed
- equipment
- frame
- working state
- inter
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/20—Image preprocessing
- G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/46—Descriptors for shape, contour or point-related descriptors, e.g. scale invariant feature transform [SIFT] or bags of words [BoW]; Salient regional features
- G06V10/462—Salient features, e.g. scale invariant feature transforms [SIFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
Abstract
The invention discloses a method, a device, a system, equipment and a storage medium for analyzing the comprehensive efficiency of equipment based on attention mechanism target detection, wherein the analysis method comprises the following steps: firstly, identifying effective equipment and worker behaviors in a production environment by using an attention-based mechanism target detection method, and removing irrelevant factory environment and ineffective equipment; secondly, different identified devices and worker behaviors are analyzed in a targeted manner by using a proper video analysis method, so that the analysis speed and the accuracy rate are improved; and finally, efficiency calculation and abnormal event analysis are carried out by using the analyzed result, so that the intelligent analysis of the comprehensive efficiency of the equipment applied to the factory is realized, and the intelligent analysis system has strong self-adaptive capacity, good precision and high efficiency.
Description
Technical Field
The invention relates to the technical field of video image analysis, in particular to a method, a device, a system, equipment and a storage medium for analyzing the comprehensive efficiency of equipment based on attention mechanism target detection.
Background
The intelligent manufacturing is a general name of an advanced manufacturing process, system and mode with functions of information self-perception, self-decision, self-execution and the like, and is specifically embodied in that each link of the manufacturing process is deeply fused with a new generation of information technology, wherein the new generation of information technology comprises the Internet of things, big data, artificial intelligence and the like. Smart manufacturing generally has four major features: the intelligent factory is used as a carrier, the intellectualization of a key manufacturing link is used as a core, an end-to-end data flow is used as a basis, and the internet communication is used as a support.
With the development of intelligent manufacturing, factory workshops tend to be unmanned and fully automatically managed, how to manage the factory workshops in an unmanned environment and ensure the normal operation and production efficiency of the workshops become an urgent problem. With the rapid development of artificial intelligence in recent years, video analysis technology is rapidly developed and applied, the video analysis technology is accessed to various cameras and various video devices such as a DVR (digital video recorder), a DVS (digital video service system) and a streaming media server, the workshop state is monitored through an intelligent image recognition processing technology, and analysis data are conducted to a comprehensive monitoring platform and a client through real-time analysis.
The conditions of the production plants of a factory are complex and are usually at least subject to the following problems when carrying out intelligent monitoring:
firstly, various machines, transport vehicles, auxiliary instruments and other objects in a production workshop are more and often shielded from each other, the analysis and identification of videos and images are challenged by the similarity of different process operations and frequent light intensity changes in the workshop, and the traditional video image analysis method is poor in adaptability and poor in application effect directly in a factory environment because of poor adaptability due to the fact that a whole frame is processed.
Secondly, the characteristics of all procedures in the production and manufacturing process are different, the defects of undefined working time, complex workpieces and the like are overcome, and the traditional single-model video and image analysis model has weak robustness and lower efficiency when being directly applied.
In a workshop environment of a factory, due to the fact that a plurality of problems exist in intelligent monitoring, when comprehensive efficiency analysis of equipment is conducted, a manual recording mode is usually adopted, efficiency is low, and reliability is not high.
Disclosure of Invention
Based on the method, aiming at the problem of analyzing the comprehensive efficiency of the equipment in the production workshop, the comprehensive efficiency analyzing method based on the attention mechanism target detection is provided, and the efficiency and the reliability of the comprehensive efficiency analysis of the equipment are improved.
The comprehensive efficiency analysis method of the equipment based on attention mechanism target detection comprises the following steps:
acquiring a monitoring video of a production environment, and identifying at least three continuous frames in the monitoring video by using a target identification model to obtain an interested area containing an object to be analyzed in each frame;
according to different objects to be analyzed, one of the following processes is carried out on each region of interest:
identifying the working state of an object to be analyzed by using an interframe difference method;
identifying the working state of an object to be analyzed by using an HSV space-based interframe difference method;
identifying the working state of an object to be analyzed by using a characteristic matching method based on ORB characteristics;
and calculating the comprehensive efficiency of the equipment according to the working state of the object to be analyzed.
When video images under intelligent manufacturing are analyzed, the target identification model added with the attention mechanism is used, and the working state of each object to be analyzed related to the comprehensive efficiency (oee for short) of Equipment in a production workshop can be quickly and accurately identified and judged, so that the abnormal state in the workshop is intelligently monitored in real time, and the full-automatic production and manufacturing process is optimized.
The present invention may be implemented in a plant in a variety of production environments, such as a thick plate manufacturing line plant.
Several alternatives are provided below, but not as an additional limitation to the above general solution, but merely as a further addition or preference, each alternative being combinable individually for the above general solution or among several alternatives without technical or logical contradictions.
Optionally, identifying the working state of the object to be analyzed by using an interframe difference method aiming at equipment or workpieces with large component motion amplitude and workshop workers with fuzzy motion modes;
aiming at equipment or workpieces with remarkable color characteristics, identifying the working state of an object to be analyzed by using an HSV space-based interframe difference method;
and aiming at the equipment or the workpiece with single action, identifying the working state of the object to be analyzed by utilizing a characteristic matching method based on the ORB characteristics.
Optionally, identifying the working state of the object to be analyzed by using an inter-frame difference method specifically includes:
performing interframe difference operation on a first frame interesting region and a second frame interesting region in three continuous frames to obtain a first interframe difference image; performing interframe difference operation on the second frame of interested area and the third frame of interested area to obtain a second interframe difference image;
respectively carrying out binarization operation on the first inter-frame difference image and the second inter-frame difference image according to a set threshold value to obtain a binarized first inter-frame difference image and a binarized second inter-frame difference image;
and performing logic and operation on the binarized first inter-frame difference image and the binarized second inter-frame difference image, and obtaining the working state of the object to be analyzed according to the operation result.
Optionally, identifying the working state of the object to be analyzed by using an HSV space-based interframe difference method specifically includes:
converting the region of interest into an image in an HSV color space, and performing binarization processing on the image;
performing inter-frame difference operation on the binarized first frame region of interest and the binarized second frame region of interest in three continuous frames to obtain a third inter-frame difference image; performing inter-frame difference operation on the binarized second frame region of interest and the binarized third frame region of interest to obtain a fourth inter-frame difference image;
and performing logic and operation on the third inter-frame difference image and the fourth inter-frame difference image, and obtaining the working state of the object to be analyzed according to the operation result.
Optionally, identifying the working state of the object to be analyzed by using a feature matching method based on ORB features specifically includes:
and calculating an ORB characteristic operator corresponding to the region of interest, matching the ORB characteristic operator with the ORB characteristic operator of the daily state diagram of the corresponding region, and obtaining the working state of the object to be analyzed according to the matching state.
Optionally, the training process of the target recognition model includes:
acquiring a prior data set for training, wherein the prior data set comprises a plurality of characteristic pictures, each characteristic picture comprises an interested area of an object to be analyzed, and each object to be analyzed at least corresponds to 1000 characteristic pictures;
and inputting the prior data set into a neural network model for training to obtain the target recognition model.
The invention also provides an equipment comprehensive efficiency analysis device based on attention mechanism target detection, which comprises:
the system comprises a first module, a second module and a third module, wherein the first module is used for acquiring a monitoring video of a production environment, and identifying at least three continuous frames in the monitoring video by using a target identification model to obtain an interested area containing an object to be analyzed in each frame;
the second module is used for carrying out one of the following treatments on each region of interest according to different objects to be analyzed:
identifying the working state of an object to be analyzed by using an interframe difference method;
identifying the working state of an object to be analyzed by using an HSV space-based interframe difference method;
identifying the working state of an object to be analyzed by using a characteristic matching method based on ORB characteristics;
and the third module is used for calculating the comprehensive efficiency of the equipment according to the working state of the object to be analyzed.
The invention also provides computer equipment which comprises a memory and a processor, wherein the memory stores a computer program, and the processor realizes the equipment comprehensive efficiency analysis method based on the attention mechanism target detection when executing the computer program.
The invention also provides an equipment comprehensive efficiency analysis system based on the attention mechanism target detection, which comprises an image acquisition device and a server, wherein the server comprises a memory and a processor, the memory stores a computer program, and the server acquires a monitoring video of a production environment from the image acquisition device; and when the processor executes the computer program, the method for analyzing the comprehensive efficiency of the equipment based on the attention mechanism target detection is realized.
The invention also provides a computer readable storage medium on which a computer program is stored, which, when executed by a processor, implements the method for analyzing the comprehensive efficiency of a device based on attention mechanism target detection.
The comprehensive efficiency analysis method of the equipment based on the attention mechanism target detection can improve the efficiency and reliability of the comprehensive efficiency analysis of the equipment and realize intelligent monitoring and analysis of the production process.
Drawings
FIG. 1 is a flow chart of the method for analyzing the comprehensive efficiency of the equipment based on the attention mechanism target detection of the present invention;
FIG. 2 is a training process of a target model in the method for analyzing the comprehensive efficiency of the equipment for detecting the target based on the attention mechanism;
FIG. 3 is a schematic diagram of a calculation result of the comprehensive efficiency of the equipment of the workshop;
FIG. 4 is a diagram of a computer device in one embodiment.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
For a better description and illustration of embodiments of the application, reference may be made to one or more of the drawings, but additional details or examples used in describing the drawings should not be construed as limiting the scope of any of the inventive concepts of the present application, the presently described embodiments, or the preferred versions.
It will be understood that when an element is referred to as being "connected" to another element, it can be directly connected to the other element or intervening elements may also be present. When a component is referred to as being "disposed on" another component, it can be directly on the other component or intervening components may also be present.
Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The terminology used in the description of the invention herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.
In the following embodiments, the conceptual definitions and symbolic descriptions involved are summarized below:
p: representing a prior data set for training;
It-1、Itand It+1: representing three continuous frames of images in a monitoring video, wherein t is the moment;
dk-1and dk: an inter-frame difference map representing three consecutive frames of images;
bk-1and bk: denotes dk-1And dkThe binary image of (1);
i (x): representing the gray value of any point on the circumference;
i (P): representing the gray value of the center of a circle;
εd: a threshold value representing a gray value difference;
meanpool: pooling of mean values, the formula is as follows:
in the formula: t denotes the ordinal threshold of the activation values involved in pooling, RjA pooling field represented in the jth feature map, i represents an index value of an activation value in this pooling field, riAnd aiRespectively representing the sequence of the activation value i and the activation value;
leaky ReLU: the tape leakage rectification function has the following formula:
wherein: x is an input vector of the upper layer of neural network, and lambda is a parameter;
ILSVRC CLS-LOC dataset: pre-trained open source data sets.
The present invention will be described in detail below with reference to the accompanying drawings.
As shown in fig. 1, the method for analyzing the comprehensive efficiency of the device based on attention mechanism target detection includes the following steps:
acquiring a monitoring video of a production environment, and identifying at least three continuous frames in the monitoring video by using a target identification model to obtain an interested area containing an object to be analyzed in each frame;
according to different objects to be analyzed, one of the following processes is carried out on each region of interest:
identifying the working state of an object to be analyzed by using an interframe difference method;
identifying the working state of an object to be analyzed by using an HSV space-based interframe difference method;
identifying the working state of an object to be analyzed by using a characteristic matching method based on ORB characteristics;
and calculating the comprehensive efficiency of the equipment according to the working state of the object to be analyzed.
Aiming at the complex situation of the production environment, the method adopts a target recognition model with an attention mechanism to intelligently recognize the interested areas in each image under the production environment, each interested area contains an object to be analyzed, and the object to be analyzed can be production equipment, production workpieces or workers. After the interested region containing the object to be analyzed is pointed out, different image processing strategies are respectively adopted for the image of the interested region according to the difference of the object to be analyzed, so that the accuracy and the efficiency of image identification are considered, and the adaptability is improved.
After the working state of the object to be analyzed is obtained by image recognition, the comprehensive efficiency of the equipment is calculated according to the working state of the object to be analyzed, the real-time calculation and monitoring of the workshop benefit are realized, and then the intelligent management is realized.
Before the working state of the object to be analyzed is identified, the interested region containing the object to be analyzed is identified in each frame of video, and image identification operation is carried out on the interested region instead of the whole video frame when video identification is carried out subsequently on each object to be analyzed, so that irrelevant parts and background parts in the video frame can be prevented from being repeatedly operated and/or unnecessarily calculated, the image processing efficiency is greatly improved, and the accuracy of the result is ensured.
In a workshop of a production environment, on the premise that the visual angle of the image acquisition device is not changed, each device and staff have relatively fixed areas, the areas correspond to the interested areas on the image, and the working state of the corresponding object to be analyzed can be obtained by analyzing the position information and the characteristic information of the object to be analyzed in the interested areas.
The monitoring videos of the production environment are obtained from a plurality of visual angles of a production workshop, and when image recognition is carried out, image processing recognition is carried out on each visual angle. Before the target detection model is used for identifying the image, preprocessing such as graying, noise reduction and the like is sequentially carried out on the originally acquired monitoring video image, and the specific region of interest identification comprises the following steps:
step 1-a, aiming at a monitoring video of a certain production workshop, acquiring three continuous video frames It-1、ItAnd It+1；
Step 1-b, adding It-1、ItAnd It+1Respectively and sequentially carrying out graying and Gaussian smoothing to obtain
And
step 1-c, mixing
And
and respectively inputting the target recognition models to obtain a region of interest (namely the feature box region where the object to be analyzed is located) containing the object to be analyzed.
According to different objects to be analyzed, different image processing is carried out on each region of interest to obtain the working state of each object to be analyzed, the working time and the downtime of each device are recorded, the state of workers in a workshop is analyzed to obtain the specific reasons of workshop downtime (for example, workshop downtime is caused by actions of the workers such as maintenance actions and cleaning actions which can interfere machine operation) and the reasons are recorded as workshop abnormal events.
Judging whether a worker moves in a workshop or not according to the recognition result of the target recognition model on the object to be analyzed in the region of interest, recording the shutdown caused by unknown reasons if the worker does not move, judging the possible events according to the position information of the worker if the worker moves, and further judging the shutdown reasons. For example, the location of the worker in the machine maintenance is identified based on the monitoring, and the ORB feature matching method is further used to determine whether the worker is performing the maintenance work, so as to obtain the reason for the machine shutdown.
According to the working time and daily working time of each device in the workshop, the comprehensive efficiency of the devices in the workshop is calculated, mathematical statistics is carried out on the comprehensive efficiency of the devices in each time period, and the comprehensive efficiency of the devices in the workshop can be displayed in real time by adopting an intuitive statistical chart (such as a line chart, a histogram and the like), as shown in fig. 3.
It should be understood that, although the steps in the flowchart of fig. 1 are shown in order as indicated by the arrows, the steps are not necessarily performed in order as indicated by the arrows. The steps are not performed in the exact order shown and described, and may be performed in other orders, unless explicitly stated otherwise. Moreover, at least a portion of the steps in fig. 1 may include multiple sub-steps or multiple stages that are not necessarily performed at the same time, but may be performed at different times, and the order of performance of the sub-steps or stages is not necessarily sequential, but may be performed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
Different objects to be analyzed have different visual characteristics and process characteristics, so that different image processing modes are adopted for different objects to be analyzed, and the accuracy and the adaptability of identification are improved.
Aiming at equipment or workpieces with large component motion amplitude and workshop workers with fuzzy action modes, identifying the working state of an object to be analyzed by utilizing an interframe difference method;
aiming at equipment or workpieces with remarkable color characteristics, identifying the working state of an object to be analyzed by using an HSV space-based interframe difference method;
and aiming at the equipment or the workpiece with single action, identifying the working state of the object to be analyzed by utilizing a characteristic matching method based on the ORB characteristics.
For equipment or workpieces with large motion amplitude, the working state of the object to be analyzed is identified by utilizing an interframe difference method because the workshop environment is static.
For equipment or workpieces with single action, the working state of the object to be analyzed is identified by using a characteristic matching method based on ORB characteristics, the calculated amount is reduced, and the speed and the precision of real-time analysis are improved.
In one embodiment, the identifying the working state of the object to be analyzed by using an inter-frame difference method, as shown in fig. 1, specifically includes:
performing interframe difference operation on a first frame interesting region and a second frame interesting region in three continuous frames to obtain a first interframe difference image; performing interframe difference operation on the second frame of interested area and the third frame of interested area to obtain a second interframe difference image;
respectively carrying out binarization operation on the first inter-frame difference image and the second inter-frame difference image according to a set threshold value to obtain a binarized first inter-frame difference image and a binarized second inter-frame difference image;
and performing logic and operation on the binarized first inter-frame difference image and the binarized second inter-frame difference image, and obtaining the working state of the object to be analyzed according to the operation result.
The set threshold is set according to a specific analysis task, for example, when analyzing the movement of the punch in the workshop, because the movement areas of different punches are different, a section of 10 seconds of video is intercepted first to observe the numerical change condition of the difference graph in the working process of the punch, and the numerical value on the node with the numerical value mutation is used as the threshold. Comparing the first inter-frame difference image and the second inter-frame difference image with a set threshold respectively, if the inter-frame difference images are larger than the set threshold, the equipment normally works, and real-time is recorded; if the difference image between the frames is less than or equal to the set threshold value, the equipment works normally and the real-time is recorded.
Specifically, the method for identifying the working state of the object to be analyzed by utilizing an interframe difference method comprises the following steps:
step 2-a, for
And
the region of interest containing the object to be analyzed is subjected to interframe difference operation on the first two frames and the second two frames respectively to obtain a first interframe difference image dk-1And a second inter-frame difference map dkThe specific calculation formula is as follows:
step 2-b, differentiating the first frameFIG. dk-1And a second inter-frame difference map dkRespectively comparing with a set threshold value T to obtain a first binary interframe difference image bk-1And binarized second inter-frame difference image bk；
Step 2-c, mixing bk-1And bkPerforming logical AND operation to obtain an inter-frame binary image Bt(x,y)；
Step 2-d, the binary image B between framest(x, y) is compared with a threshold (not the same threshold as the set threshold T) and if the frame binary image B is not the same as the set threshold Tt(x, y) is greater than the threshold value, the equipment works normally, and the real-time is recorded; if the frame is binary image BtAnd (x, y) if the value is less than the threshold value, stopping the equipment, outputting an abnormal event record, and recording the real-time.
In one embodiment, the method for identifying the working state of the object to be analyzed by using an HSV space-based interframe difference method includes, as shown in fig. 1:
converting the region of interest into an image in an HSV color space, and performing binarization processing on the image;
performing inter-frame difference operation on the binarized first frame region of interest and the binarized second frame region of interest in three continuous frames to obtain a third inter-frame difference image; performing inter-frame difference operation on the binarized second frame region of interest and the binarized third frame region of interest to obtain a fourth inter-frame difference image;
and performing logic and operation on the third inter-frame difference image and the fourth inter-frame difference image, and obtaining the working state of the object to be analyzed according to the operation result.
Aiming at an object to be analyzed with obvious color characteristics, the method utilizes HSV color space to perform auxiliary judgment, and specifically comprises the following steps:
step 3-a, converting the image of the region of interest containing the object to be analyzed from an RGB image into an image in HSV color space, wherein the RGB image is a measure of basic three primary colors of red, green and blue (r, g and b), the value of the RGB image is a real number between 0 and 1, max is set to be equal to the maximum value of r, g and b, min is set to be equal to the minimum value of r, g and b, and then the RGB image is converted into HSV in the following process:
v＝max。
step 3-b, performing binarization processing on the obtained HSV image, setting the pixel point where the characteristic color is located as 1, and setting other pixel points as 0;
step 3-c, performing interframe difference operation on the binarized first frame interesting region and the binarized second frame interesting region in three continuous frames to obtain a third interframe difference image; performing inter-frame difference operation on the binarized second frame region of interest and the binarized third frame region of interest to obtain a fourth inter-frame difference image;
and 3-d, performing logic and operation on the third inter-frame difference image and the fourth inter-frame difference image, obtaining the working state of the object to be analyzed according to the operation result, executing the step by referring to the step 2-c, and setting the workshop to be in a shutdown state and entering abnormal event analysis when all the objects to be analyzed stop working.
In one embodiment, the identifying the working state of the object to be analyzed by using a feature matching method based on ORB features, as shown in fig. 1, specifically includes:
and calculating an ORB characteristic operator corresponding to the region of interest, matching the ORB characteristic operator with the ORB characteristic operator of the daily state diagram of the corresponding region, and obtaining the working state of the object to be analyzed according to the matching state.
If the ORB characteristic operator of the region of interest is matched with the ORB characteristic operator of the daily state diagram of the corresponding region, no abnormal event occurs, if the ORB characteristic operator of the region of interest is not matched with the ORB characteristic operator of the daily state diagram of the corresponding region, the abnormal event occurs, and the abnormal event is recorded so as to occur time.
The calculation formula of the ORB feature operator is as follows:
and detecting pixel values around the candidate characteristic point, and if the gray value difference between enough pixel points in the region around the candidate point and the candidate point is large enough, considering the candidate point as a characteristic point.
In one embodiment, the training process of the target recognition model includes:
acquiring a prior data set for training, wherein the prior data set comprises a plurality of characteristic pictures, each characteristic picture comprises an interested area of an object to be analyzed, and each object to be analyzed at least corresponds to 1000 characteristic pictures;
and inputting the prior data set into a neural network model for training to obtain the target recognition model.
Before the target recognition model is utilized, the target recognition model needs to be trained to improve the recognition accuracy. The prior data set for training needs to be collected firstly before training, and the prior data set needs to be established firstly because deep learning has high requirements on the prior data set P for training, and the factory environment is different from daily life.
When the prior data set is established, a certain amount of videos in the production environment are collected, video frames are preprocessed according to the process and equipment characteristics of the production environment, and as shown in fig. 2, regions of interest where objects to be analyzed (including equipment and personnel in the production environment, A, B, X in fig. 2 represent different objects to be analyzed) are located in each frame of videos are analyzed, positioned and segmented, so that the prior data set for training is obtained.
The video frame used for training and the video frame used for subsequent real-time monitoring are collected from the same visual angle, and after the video frame used for training is sequentially subjected to graying, Gaussian smoothing and the like, a characteristic region containing an object to be analyzed is intercepted by using specific coordinates and is used as a characteristic picture. Each feature picture comprises an interested area of an object to be analyzed, each object to be analyzed at least corresponds to 1000 feature pictures, and the feature pictures of the same object to be analyzed are stored in corresponding data sets.
Based on a neural network model (the invention uses a VGG16 model) pre-trained on an ILSVRCCLS-LOC data set, training is performed by using a prior data set P to obtain a target detection model with high robustness, and the structure of the neural network model is shown in FIG. 2 and specifically as follows:
the first six layers of the neural network model structure are VGG16 model structures: the first convolution layer is conv3-64 (convolution kernel size is 3 x 3, 64 features are output), the first pooling layer is meanpool, window size is 2, and step size is 2; the second and third convolution layers are conv3-128, the second pooling layer is meanpool, the window size is 2, and the step length is 2; the fourth, fifth and sixth convolution layers are conv3-256, the third pooling layer is meanpool, the window size is 2, and the step length is 2;
converting a seventh fully-connected layer of the VGG16 model into a convolution layer conv6 with a convolution kernel size of 3 x 3, wherein an activation function is a leak ReLU, a fourth pooling layer is a meanpool, a window size is 3, and a step size is 2;
the eighth fully-connected layer of the VGG16 model was converted to convolutional layer conv7 with convolution kernel size of 1 × 1, activation function was leaky ReLU, the fifth pooling layer was meanpool, window size was 3, and step size was 2.
In one embodiment, an apparatus for analyzing integrated efficiency of a device based on attention mechanism target detection is provided, including:
the system comprises a first module, a second module and a third module, wherein the first module is used for acquiring a monitoring video of a production environment, and identifying at least three continuous frames in the monitoring video by using a target identification model to obtain an interested area containing an object to be analyzed in each frame;
the second module is used for carrying out one of the following treatments on each region of interest according to different objects to be analyzed:
identifying the working state of an object to be analyzed by using an interframe difference method;
identifying the working state of an object to be analyzed by using an HSV space-based interframe difference method;
identifying the working state of an object to be analyzed by using a characteristic matching method based on ORB characteristics;
and the third module is used for calculating the comprehensive efficiency of the equipment according to the working state of the object to be analyzed.
For specific limitations of the device comprehensive efficiency analysis apparatus based on attention mechanism target detection, reference may be made to the above limitations of the device comprehensive efficiency analysis method based on attention mechanism target detection, which are not described herein again. The modules in the device comprehensive efficiency analysis device based on attention mechanism target detection can be wholly or partially realized by software, hardware and a combination thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
In one embodiment, a computer device is provided, which includes a memory and a processor, the memory stores a computer program, and the processor implements the method for analyzing device comprehensive efficiency based on attention mechanism target detection when executing the computer program.
The computer device may be a terminal, and its internal structure diagram may be as shown in fig. 4. The computer device includes a processor, a memory, a network interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement the method for analyzing the comprehensive efficiency of the equipment based on the attention mechanism target detection. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, a key, a track ball or a touch pad arranged on the shell of the computer equipment, an external keyboard, a touch pad or a mouse and the like.
Those skilled in the art will appreciate that the architecture shown in fig. 4 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
In one embodiment, the system for analyzing the comprehensive efficiency of the equipment based on the attention mechanism target detection comprises an image acquisition device and a server, wherein the server comprises a memory and a processor, the memory stores a computer program, and the server acquires a monitoring video of a production environment from the image acquisition device; and when the processor executes the computer program, the method for analyzing the comprehensive efficiency of the equipment based on the attention mechanism target detection is realized.
The image acquisition device can adopt various video monitoring equipment as long as continuous images reflecting the production state in a workshop can be directly or indirectly obtained.
In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the method for device integrated efficiency analysis based on attention mechanism target detection.
It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a non-volatile computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. Any reference to memory, storage, database, or other medium used in the embodiments provided herein may include non-volatile and/or volatile memory, among others. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).
The technical features of the embodiments described above may be arbitrarily combined, and for the sake of brevity, all possible combinations of the technical features in the embodiments described above are not described, but should be considered as being within the scope of the present specification as long as there is no contradiction between the combinations of the technical features.
The above-mentioned embodiments only express several embodiments of the present invention, and the description thereof is more specific and detailed, but not construed as limiting the scope of the invention. It should be noted that, for a person skilled in the art, several variations and modifications can be made without departing from the inventive concept, which falls within the scope of the present invention. Therefore, the protection scope of the present patent shall be subject to the appended claims.
Claims (9)
1. The comprehensive efficiency analysis method of the equipment based on attention mechanism target detection is characterized by comprising the following steps:
acquiring a monitoring video of a production environment, and identifying at least three continuous frames in the monitoring video by using a target identification model to obtain an interested area containing an object to be analyzed in each frame;
according to different objects to be analyzed, one of the following processes is carried out on each region of interest:
aiming at equipment or workpieces with large component motion amplitude and workshop workers with fuzzy action modes, identifying the working state of an object to be analyzed by utilizing an interframe difference method;
aiming at equipment or workpieces with remarkable color characteristics, identifying the working state of an object to be analyzed by using an HSV space-based interframe difference method;
aiming at equipment or workpieces with fixed action modes, identifying the working state of an object to be analyzed by using a characteristic matching method based on ORB characteristics;
and calculating the comprehensive efficiency of the equipment according to the working state of the object to be analyzed.
2. The method for analyzing the comprehensive efficiency of the equipment based on the attention mechanism target detection as claimed in claim 1, wherein the step of identifying the working state of the object to be analyzed by using the inter-frame difference method specifically comprises the following steps:
performing interframe difference operation on a first frame interesting region and a second frame interesting region in three continuous frames to obtain a first interframe difference image; performing interframe difference operation on the second frame of interested area and the third frame of interested area to obtain a second interframe difference image;
respectively carrying out binarization operation on the first inter-frame difference image and the second inter-frame difference image according to a set threshold value to obtain a binarized first inter-frame difference image and a binarized second inter-frame difference image;
and performing logic and operation on the binarized first inter-frame difference image and the binarized second inter-frame difference image, and obtaining the working state of the object to be analyzed according to the operation result.
3. The method for analyzing the comprehensive efficiency of the equipment based on the attention mechanism target detection as claimed in claim 1, wherein the step of identifying the working state of the object to be analyzed by using an HSV space-based interframe difference method specifically comprises the following steps:
converting the region of interest into an image in an HSV color space, and performing binarization processing on the image;
performing inter-frame difference operation on the binarized first frame region of interest and the binarized second frame region of interest in three continuous frames to obtain a third inter-frame difference image; performing inter-frame difference operation on the binarized second frame region of interest and the binarized third frame region of interest to obtain a fourth inter-frame difference image;
and performing logic and operation on the third inter-frame difference image and the fourth inter-frame difference image, and obtaining the working state of the object to be analyzed according to the operation result.
4. The method for analyzing the comprehensive efficiency of the equipment based on the attention mechanism target detection as claimed in claim 1, wherein the identifying the working state of the object to be analyzed by using the feature matching method based on the ORB features specifically comprises:
and calculating an ORB characteristic operator corresponding to the region of interest, matching the ORB characteristic operator with the ORB characteristic operator of the daily state diagram of the corresponding region, and obtaining the working state of the object to be analyzed according to the matching state.
5. The method for analyzing the comprehensive efficiency of the equipment based on the attention mechanism target detection as claimed in claim 1, wherein the training process of the target recognition model comprises the following steps:
acquiring a prior data set for training, wherein the prior data set comprises a plurality of characteristic pictures, each characteristic picture comprises an interested area of an object to be analyzed, and each object to be analyzed at least corresponds to 1000 characteristic pictures;
and inputting the prior data set into a neural network model for training to obtain the target recognition model.
6. Equipment comprehensive efficiency analytical equipment based on attention mechanism target detection, its characterized in that includes:
the system comprises a first module, a second module and a third module, wherein the first module is used for acquiring a monitoring video of a production environment, and identifying at least three continuous frames in the monitoring video by using a target identification model to obtain an interested area containing an object to be analyzed in each frame;
the second module is used for carrying out one of the following treatments on each region of interest according to different objects to be analyzed: aiming at equipment or workpieces with large component motion amplitude and workshop workers with fuzzy action modes, identifying the working state of an object to be analyzed by utilizing an interframe difference method;
aiming at equipment or workpieces with remarkable color characteristics, identifying the working state of an object to be analyzed by using an HSV space-based interframe difference method;
aiming at equipment or workpieces with fixed action modes, identifying the working state of an object to be analyzed by using a characteristic matching method based on ORB characteristics;
and the third module is used for calculating the comprehensive efficiency of the equipment according to the working state of the object to be analyzed.
7. A computer device comprising a memory and a processor, the memory storing a computer program, wherein the processor when executing the computer program implements the method for analyzing integrated efficiency of a device based on attention mechanism target detection according to any one of claims 1 to 5.
8. An equipment comprehensive efficiency analysis system based on attention mechanism target detection comprises an image acquisition device and a server, wherein the server comprises a memory and a processor, and a computer program is stored in the memory; when the computer program is executed by the processor, the method for analyzing the comprehensive efficiency of the equipment based on the attention mechanism target detection is realized according to any one of claims 1 to 5.
9. A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the method for analyzing integrated efficiency of a device based on attention mechanism target detection according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910376669.3A CN110222575B (en) | 2019-05-07 | 2019-05-07 | Equipment comprehensive efficiency analysis method, device, system, equipment and storage medium based on attention mechanism target detection |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910376669.3A CN110222575B (en) | 2019-05-07 | 2019-05-07 | Equipment comprehensive efficiency analysis method, device, system, equipment and storage medium based on attention mechanism target detection |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN110222575A CN110222575A (en) | 2019-09-10 |
| CN110222575B true CN110222575B (en) | 2021-06-29 |
Family
ID=67820819
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910376669.3A Active CN110222575B (en) | 2019-05-07 | 2019-05-07 | Equipment comprehensive efficiency analysis method, device, system, equipment and storage medium based on attention mechanism target detection |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110222575B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112016400B (en) * | 2020-08-04 | 2021-06-29 | 香港理工大学深圳研究院 | Single-class target detection method and device based on deep learning and storage medium |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1801930A (en) * | 2005-12-06 | 2006-07-12 | 南望信息产业集团有限公司 | Dubious static object detecting method based on video content analysis |
| CN202600730U (en) * | 2012-04-28 | 2012-12-12 | 上海海事大学 | Image processing device for container terminal bridge crane efficiency analysis |
| CN104700430A (en) * | 2014-10-05 | 2015-06-10 | 安徽工程大学 | Method for detecting movement of airborne displays |
| CN109086779A (en) * | 2018-07-28 | 2018-12-25 | 天津大学 | A kind of attention target identification method based on convolutional neural networks |
| CN109255739A (en) * | 2018-09-12 | 2019-01-22 | 三峡大学 | A kind of classroom learning efficiency analysis algorithm identified based on human face expression and human action |
| US10223617B1 (en) * | 2014-06-06 | 2019-03-05 | Deepmind Technologies Limited | Image processing with recurrent attention |
-
2019
- 2019-05-07 CN CN201910376669.3A patent/CN110222575B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1801930A (en) * | 2005-12-06 | 2006-07-12 | 南望信息产业集团有限公司 | Dubious static object detecting method based on video content analysis |
| CN202600730U (en) * | 2012-04-28 | 2012-12-12 | 上海海事大学 | Image processing device for container terminal bridge crane efficiency analysis |
| US10223617B1 (en) * | 2014-06-06 | 2019-03-05 | Deepmind Technologies Limited | Image processing with recurrent attention |
| CN104700430A (en) * | 2014-10-05 | 2015-06-10 | 安徽工程大学 | Method for detecting movement of airborne displays |
| CN109086779A (en) * | 2018-07-28 | 2018-12-25 | 天津大学 | A kind of attention target identification method based on convolutional neural networks |
| CN109255739A (en) * | 2018-09-12 | 2019-01-22 | 三峡大学 | A kind of classroom learning efficiency analysis algorithm identified based on human face expression and human action |
Also Published As
| Publication number | Publication date |
|---|---|
| CN110222575A (en) | 2019-09-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11380232B2 (en) | Display screen quality detection method, apparatus, electronic device and storage medium | |
| CN107194559B (en) | Workflow identification method based on three-dimensional convolutional neural network | |
| CN111754498B (en) | Conveyor belt carrier roller detection method based on YOLOv3 | |
| Golparvar-Fard et al. | Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers | |
| CN107123131B (en) | Moving target detection method based on deep learning | |
| CN107463888B (en) | Face emotion analysis method and system based on multi-task learning and deep learning | |
| US10621717B2 (en) | System and method for image-based target object inspection | |
| CN101167085A (en) | Three-dimensional road layout estimation from video sequences by tracking pedestrians | |
| CN108734109B (en) | Visual target tracking method and system for image sequence | |
| Yan et al. | YOLO V3+ VGG16-based automatic operations monitoring and analysis in a manufacturing workshop under Industry 4.0 | |
| CN109615904A (en) | Parking management method, device, computer equipment and storage medium | |
| CN112613454A (en) | Electric power infrastructure construction site violation identification method and system | |
| CN111401131A (en) | Image processing method and device for tunnel pipe gallery, computer equipment and storage medium | |
| CN111896540A (en) | Water quality on-line monitoring system based on block chain | |
| Mazzetto et al. | Deep learning models for visual inspection on automotive assembling line | |
| CN110222575B (en) | Equipment comprehensive efficiency analysis method, device, system, equipment and storage medium based on attention mechanism target detection | |
| Ji et al. | A Computer Vision-Based Model for Automatic Motion Time Study. | |
| CN115953731A (en) | Intelligent coal flow monitoring data analysis method for improving CNN algorithm model | |
| CN115512098A (en) | Electronic bridge inspection system and inspection method | |
| CN114119531A (en) | Fire detection method and device applied to campus smart platform and computer equipment | |
| Mysiuk et al. | Detection of Surface Defects Inside Concrete Pipelines Using Trained Model on JetRacer Kit | |
| Shi et al. | Human visual perception based image quality assessment for video prediction | |
| Zhang et al. | Prediction of human actions in assembly process by a spatial-temporal end-to-end learning model | |
| CN117115783B (en) | Assembly line work behavior recognition method based on machine vision | |
| CN113177452B (en) | Sample sealing method and device based on image processing and radio frequency technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |