CN114565870A - Production line control method, device and system based on vision, and electronic equipment - Google Patents
Production line control method, device and system based on vision, and electronic equipment Download PDFInfo
- Publication number
- CN114565870A CN114565870A CN202210161702.2A CN202210161702A CN114565870A CN 114565870 A CN114565870 A CN 114565870A CN 202210161702 A CN202210161702 A CN 202210161702A CN 114565870 A CN114565870 A CN 114565870A
- Authority
- CN
- China
- Prior art keywords
- event
- production line
- neural network
- camera
- time
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02P—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN THE PRODUCTION OR PROCESSING OF GOODS
- Y02P90/00—Enabling technologies with a potential contribution to greenhouse gas [GHG] emissions mitigation
- Y02P90/02—Total factory control, e.g. smart factories, flexible manufacturing systems [FMS] or integrated manufacturing systems [IMS]
Abstract
The invention discloses a production line control method, a production line control device, a production line control system and electronic equipment based on vision, and relates to the technical field of machine vision. In an industrial production workshop, some important production line processes need strict flow management and control, and need take out articles and put in articles illegally to be identified. Meanwhile, some dangerous actions of related personnel around the production line also need to be identified in time, and the risks of injury of the personnel and damage of equipment are reduced. The invention uses an event camera to obtain pictures around a production line, and arranges a network camera in an open area of the production line. And performing processing analysis by using a processor based on the image processing correlation algorithm and the deep learning correlation algorithm. And when more events are output by the event camera in a short time, illegal picking and placing identification and dangerous action identification are respectively triggered. The processor records and stores various results and transmits the results to the mobile terminal of the manager, so that the management and control of the production line can be efficiently realized in multiple aspects.
Description
Technical Field
The application relates to the technical field of machine vision, in particular to a production line control method, a production line control device, a production line control system and electronic equipment based on vision.
Background
In an industrial production plant, some important process line processes require strict process control. Generally, products on certain key production lines are not expected to be taken away or put in midway, so that hidden dangers are brought to subsequent production processes, the quality of the products is possibly influenced, and then the products are required to be taken out and put into the production lines illegally. Meanwhile, some dangerous actions of related personnel around the production line also need to be identified in time, so that various hidden dangers are avoided, and the risks of injury of the personnel and damage of equipment are reduced.
Aiming at the identification of illegal taking and placing products, the traditional solution can generally select to carry out management and control on hardware, namely hardware facilities for limiting manual taking and placing are added on a production line, and corresponding alarm prompt is carried out. However, with the complexity of the production line and the different requirements of different production processes, it is no longer suitable to add corresponding hardware limiting devices at will. Furthermore, hardware-constrained devices are not easily traceable and identifiable by management personnel.
On the other hand, for the identification of dangerous actions of related personnel around the production line, a monitoring network camera is generally used for picture monitoring, and judgment is performed by means of an action identification algorithm, the traditional network camera is not easy to acquire images with high picture quality in workshops with large dynamic ranges, and motion blur is easy to occur to some special abnormal actions with rapid motion, which are not beneficial to the calculation processing of the action identification algorithm.
Disclosure of Invention
The embodiment of the application aims to provide a production line control method, a production line control device, a production line control system and electronic equipment based on vision, and aims to solve the technical problems that in the related technology, a production line is inconvenient to add hardware limiting equipment, illegal taking and placing conditions are inconvenient to trace and confirm, and errors are easy to occur in a traditional network camera action recognition algorithm in a workshop with a large dynamic range.
According to a first aspect of embodiments of the present application, there is provided a vision-based production line management and control method, including:
acquiring a first real-time video stream acquired by at least one network camera, wherein the visual field of each network camera only comprises open areas on a production line, and the open areas are not intersected with each other;
in the first real-time video stream, every T frames pass through a trained pick-and-place recognition neural network to identify a current key frame I0Performing primary classification prediction;
if the classified output of the neural network is a normal working class, continuing to perform next prediction;
if the classified output of the neural network is abnormal picking and placing class, continuing to collect the next key frame I1Through I0And I1Calculating a dense optical flow F, and counting the directional distribution of the F;
if F falls within the lower boundary θ of the take-out angle1And an upper bound on the extraction angle theta2Interval of composition [ theta ]1,θ2]If the proportion is larger than the taking-out identification threshold G, taking out the product if the violation exception is the violation exception;
if F falls within the lower boundary θ of the entry angle3And put into the upper bound of angle theta4Interval of composition [ theta ]3,θ4]If the proportion of the illegal product is larger than the putting identification threshold value P, the illegal product is put into the illegal product;
acquiring a first real-time event stream acquired by an event camera, wherein the visual field of the event camera comprises a movable area near a production line, accumulating each E event in the first real-time event stream into an event frame image, and storing the event frame image into a time window queue;
and predicting the time window queue through the trained action recognition neural network, and recognizing whether dangerous actions occur or not.
Further, the training process of the trained pick-and-place recognition neural network comprises the following steps:
extracting frame image pictures from a multi-section video of a production line, and making a data set;
extracting frame image pictures during normal work on a production line to serve as normal work classes of the data set;
extracting frame image pictures at the moment of abnormal picking and placing on a production line to serve as abnormal picking and placing types of the data set;
and training the classification neural network by using the data set to reach the required corresponding accuracy, and obtaining the trained picking and placing recognition neural network.
Further, by I0And I1Calculating dense optical flow F, and counting the directional distribution of F, comprising the following steps:
through an optical flow calculation method, two frames of images I with similar pictures and motion changes of the pictures are obtained0And I1And calculating a dense optical flow F, calculating an optical flow direction parameter value of each pixel point of the dense optical flow F, and counting the occupation ratio of the optical flow direction parameter values of all the pixel points in different angle intervals.
Further, storing the event frame image into a time window queue, comprising the steps of:
setting a time window queue with the length of LE, and setting all initial values to be 0;
and in the first real-time event stream, accumulating E events to obtain an event frame image, storing the event frame image into the time window queue, and obtaining the time window queue full of the event frame image in real time.
Further, the training process of the trained motion recognition neural network comprises the following steps:
respectively storing event stream information of general normal actions and event stream information of special dangerous actions of a movable area near a production line acquired by the event camera;
accumulating E events in each segment of event stream into a frame image, generating a video containing action information, and making a data set, wherein the data set comprises a normal action type event frame video and various abnormal action type event frame videos;
and training the action recognition neural network by using the data set to reach the required corresponding accuracy rate, and obtaining the trained action recognition neural network.
Further, still include:
when the event camera continuously works, the number of events output by the event camera in a time interval delta t is larger than a trigger threshold value EtTriggering the trained picking and placing recognition neural network and the trained action recognition neural network to work;
within the delay time interval delta T, if the event number output by the camera in the time interval delta T is always kept smaller than a trigger threshold EtAnd the trained pick-and-place recognition neural network and the trained action recognition neural network suspend working and wait for the next trigger starting.
According to a second aspect of the embodiments of the present application, there is provided a vision-based production line management and control device, including:
the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a first real-time video stream acquired by at least one network camera, the view field of each network camera only comprises open areas on a production line, and the open areas are not intersected with each other;
a first prediction module for recognizing the current key frame I by the trained pick-and-place recognition neural network every T frames in the first real-time video stream0Performing primary classification prediction;
the first judgment module is used for continuing to predict the next time if the classified output of the neural network is a normal pick-and-place type;
a second judgment module for continuing to collect the next key frame I if the neural network classification output is the abnormal pick-and-place classification1Through I0And I1Calculating a dense optical flow F, and counting the directional distribution of the F;
a third judging module for judging if the direction distribution of F falls on the lower boundary theta of the taking-out angle1And an upper bound on the extraction angle theta2Interval of composition [ theta ]1,θ2]If the proportion is larger than the taking-out identification threshold G, taking out the product if the violation exception is the violation exception;
a fourth judging module for judging if the direction distribution of F falls in the lower boundary theta of the putting-in angle3And put into the upper bound of angleθ4Interval of composition [ theta ]3,θ4]If the proportion of the illegal product is larger than the putting identification threshold value P, the illegal product is put into the illegal product;
the system comprises an acquisition and storage module, a time window queue and a time window processing module, wherein the acquisition and storage module is used for acquiring a first real-time event stream acquired by an event camera, the visual field of the event camera comprises a movable area near a production line, each E event in the first real-time event stream is accumulated into an event frame image, and the event frame image is stored in the time window queue;
and the second prediction module is used for predicting the time window queue through the trained action recognition neural network and recognizing whether dangerous actions occur or not.
According to a third aspect of embodiments of the present application, there is provided a vision-based production line management and control system, including:
the system comprises at least one network camera, a video acquisition unit and a display unit, wherein the network camera is used for acquiring a first real-time video stream, the view field of each network camera only comprises open areas on a production line, and the open areas are not intersected with each other;
an event camera for capturing a first real-time stream of events, said event camera having a field of view encompassing a movable area near a production line;
a switch to pass the first real-time video stream to a processor;
a processor for acquiring a first real-time video stream acquired by a network camera, wherein in the first real-time video stream, every T frames pass through a trained pick-and-place recognition neural network to identify a current key frame I0Performing primary classification prediction; if the classified output of the neural network is a normal working class, continuing to perform next prediction; if the classified output of the neural network is abnormal picking and placing class, continuing to collect the next key frame I1Through I0And I1Calculating a dense optical flow F, and counting the directional distribution of the F; if F falls within the lower boundary θ of the take-out angle1And an upper bound on the extraction angle theta2Interval of composition [ theta ]1,θ2]If the proportion is larger than the taking-out identification threshold G, taking out the product if the violation exception is the violation exception; if F falls within the lower boundary θ of the entry angle3And put inUpper bound of angle theta4Interval of composition [ theta ]3,θ4]If the proportion of the illegal product is larger than the putting identification threshold value P, the illegal product is put into the illegal product; acquiring a first real-time event stream acquired by an event camera, accumulating E events in the first real-time event stream into an event frame image, and storing the event frame image into a time window queue; and predicting the time window queue through the trained action recognition neural network, and recognizing whether dangerous actions occur or not.
According to a fourth aspect of embodiments of the present application, there is provided an electronic apparatus, including:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement a method as described in the first aspect.
According to a fifth aspect of embodiments herein, there is provided a processor readable storage medium having stored thereon processor instructions which, when executed by a processor, implement the steps of the method according to the first aspect.
The technical scheme provided by the embodiment of the application can have the following beneficial effects:
according to the technical scheme, the network camera is adopted to acquire the picture of the open area on the production line to perform illegal taking and placing identification alarm, so that the addition of hardware limiting equipment on a complex production line is avoided;
the picking and placing recognition neural network is adopted to judge the picture, so that the recognition effect is stable, and the higher recognition accuracy rate can be achieved for illegal picking and placing of a plurality of open area products on a production line;
the event camera is adopted to identify dangerous actions in the workshop, so that the problems that the image quality obtained by a traditional network camera in the workshop with a large dynamic range is poor and an action identification algorithm is unstable are solved, the method can be better suitable for monitoring workshop scene pictures with various dynamic ranges, and the action identification can be effectively carried out;
the event camera is used as the trigger of illegal picking and placing identification and dangerous action identification, the problems that some monitoring cameras or other limiting devices can only work all day long when used and the efficiency is low under the condition that no person is beside a production line are solved, and then the processing identification is carried out only when potential abnormality occurs, so that the resource utilization efficiency of the management and control method is high;
the network camera and the event camera are used as core sensors for management and control, pictures when illegal taking and dangerous actions occur can be recorded, and then interactive management and control can be carried out on the production line state, so that the production line state can be traced and confirmed conveniently.
It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the application.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and together with the description, serve to explain the principles of the application.
FIG. 1 is a flow chart illustrating a vision-based process control method according to an exemplary embodiment.
FIG. 2 is a schematic diagram illustrating a vision-based production line management system, according to an exemplary embodiment.
FIG. 3 is a block diagram illustrating a vision-based line management apparatus in accordance with an exemplary embodiment.
Detailed Description
Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the application, as detailed in the appended claims.
The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in this application and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that the term "and/or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.
It is to be understood that although the terms first, second, third, etc. may be used herein to describe various information, such information should not be limited to these terms. These terms are only used to distinguish one type of information from another. For example, first information may also be referred to as second information, and similarly, second information may also be referred to as first information, without departing from the scope of the present application. The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination", depending on the context.
FIG. 1 is a flow chart illustrating a vision-based process control method according to an exemplary embodiment. Referring to fig. 1, the method may include:
step S11, acquiring a first real-time video stream collected by at least one network camera, wherein the visual field of each network camera only comprises open areas on a production line, and the open areas are not intersected with each other;
step S12, in the first real-time video stream, every T frames pass through the trained picking and placing recognition neural network to the current key frame I0Performing primary classification prediction;
step S13, if the neural network classification output is normal working class, continuing to predict next time;
step S14, if the neural network classification output is abnormal pick-and-place type, continue to collect the next key frame I1Through I0And I1Calculating a dense optical flow F, and counting the directional distribution of the F;
in step S15, if the direction distribution of F falls within the lower boundary theta of the extraction angle1And an upper bound on the extraction angle theta2Interval of composition [ theta ]1,θ2]Is greater than the fetch identification threshold G,the rule violation exception is to take out the product;
step S16, if the direction distribution of F falls on the lower bound theta of the putting-in angle3And put into the upper bound of angle theta4Interval of composition [ theta ]3,θ4]If the proportion of the illegal product is larger than the putting identification threshold value P, the illegal product is put into the illegal product;
step S17, acquiring a first real-time event stream acquired by an event camera, wherein the visual field of the event camera comprises a movable area near a production line, accumulating each E event in the first real-time event stream into an event frame image, and storing the event frame image into a time window queue;
and step S18, predicting the time window queue through the trained motion recognition neural network, and recognizing whether dangerous motion occurs.
According to the embodiment, the network camera is adopted to acquire the pictures of the open area on the production line to perform illegal taking and placing identification alarm, so that the addition of hardware limiting equipment on a complex production line is avoided; the picking and placing recognition neural network is adopted to judge the picture, so that the recognition effect is stable, and the higher recognition accuracy rate can be achieved for illegal picking and placing of a plurality of open area products on a production line; the event camera is adopted to identify dangerous actions in the workshop, so that the problems that the image quality obtained by a traditional network camera in the workshop with a large dynamic range is poor and an action identification algorithm is unstable are solved, the method can be better suitable for monitoring workshop scene pictures with various dynamic ranges, and the action identification can be effectively carried out; the network camera and the event camera are used as core sensors for management and control, pictures when illegal taking and dangerous actions occur can be recorded, and then interactive management and control can be carried out on the production line state, so that the production line state can be traced and confirmed conveniently.
In a specific implementation of step S11, a first real-time video stream collected by at least one webcam is obtained, a field of view of each webcam only includes open areas on a production line, and the open areas are not intersected with each other.
Specifically, with reference to fig. 2, in the open area of the production line 1 (two of which are illustrated in the figure): the system comprises a first open area 2 and a second open area 3, a first network camera 4 and a second network camera 5 which are arranged, wherein the position and the angle of the first network camera 4 are adjusted to ensure that the visual field picture of the first network camera is almost only the first open area 2, and the position and the angle of the second network camera 5 are adjusted to ensure that the visual field picture of the second network camera is almost only the second open area 3.
In the specific implementation of step S12, in the first real-time video stream, every T frames pass through the trained pick-and-place recognition neural network to identify the current key frame I0And carrying out classification prediction once.
Specifically, in a real-time video stream with FPS (frame rate of motion-based video), the current key frame I is set every T frames0Inputting a trained pick-and-place recognition neural network, wherein the output result of the trained pick-and-place recognition neural network is a normal working class or an abnormal pick-and-place class, and not every frame is subjected to processing prediction, so that redundant information of a high frame rate can be omitted, and the real-time property is met.
In this embodiment, the training process of the trained pick-and-place recognition neural network includes the following steps:
a1, extracting frame image pictures from multi-segment videos of a production line, and making a data set;
a2, extracting the frame image picture of the production line in normal working as the normal working class of the data set;
a3, extracting frame image pictures at the moment of abnormal picking and placing on a production line, and taking the frame image pictures as abnormal picking and placing classes of the data set;
and A4, training the classification neural network by using the data set to reach the required corresponding accuracy, and obtaining the trained pick-and-place recognition neural network.
The trained picking and placing recognition neural network is adopted for picture judgment, so that the recognition effect is stable, the universality of a plurality of open areas on a production line is better, and meanwhile, higher recognition accuracy can be achieved.
In the embodiment of step S13, if the neural network classification output is the normal operation class, the next prediction is continued.
Specifically, if the neural network is classified and output as a normal working class, the operation is continued in the current state, and the next key frame is continuously acquired and predicted, which indicates that no abnormality occurs currently.
In the embodiment of step S14, if the neural network classification output is the abnormal pick-and-place class, the next key frame I is collected continuously1Through I0And I1And (5) calculating a dense optical flow F, and counting the direction distribution of the F.
Specifically, if the neural network classification output is an abnormal pick-and-place classification, the key frame I acquired next time is used1And the current key frame I0Calculating optical flow, and distinguishing the taking exception and the putting exception; and only when the neural network classification output is an abnormal pick-and-place class, the optical flow calculation is carried out, and when the neural network classification output is a normal working class, the optical flow calculation is not carried out, so that the calculation consumption of the system can be reduced as much as possible, and the real-time detection and alarm are met.
Further, by I0And I1Calculating dense optical flow F, and counting the directional distribution of F, comprising the following steps:
by an optical flow calculation method, two frames of images I with similar pictures and motion changes of the pictures are obtained0And I1And calculating a dense optical flow F, calculating an optical flow direction parameter value of each pixel point of the dense optical flow F, and counting the occupation ratio of the optical flow direction parameter values of all the pixel points in different angle intervals.
The direction of the optical flow represents the direction of the motion of an object in the picture, the proportion of different angle intervals can reflect the motion trend of a product, the taking out and the putting in are two abnormal categories with obvious motion direction difference, and the taking out or the putting in of the product can be effectively distinguished through the calculation and the statistics of the dense optical flow.
In the implementation of step S15, if F falls within the lower extraction angle boundary θ in the directional distribution1And an upper bound on the extraction angle theta2Interval of composition [ theta ]1,θ2]If the ratio of (1) is larger than the taking identification threshold value G, the violation exception is that the product is taken out.
Specifically, a take-out angle lower bound θ is set1And an upper bound on the extraction angle theta2And setting a fetch identification threshold G, if the directional distribution of F falls in the interval [ theta ]1,θ2]If the ratio of (1) is greater than the extraction recognition threshold value G, this abnormality is satisfied with the optical flow result at the time of extracting the product, and the abnormality in violation can be regarded as the extracted product.
In the implementation of step S16, if F falls within the lower entry angle boundary θ in the directional distribution3And put into the upper bound of angle theta4Interval of composition [ theta ]3,θ4]If the ratio is larger than the putting identification threshold value P, the violation exception is that the product is put in.
Specifically, a lower entry angle bound θ is set3And put into the upper bound of angle theta4And setting a put-in recognition threshold P if the directional distribution of F falls within a section [ theta ]3,θ4]If the ratio of (1) is greater than the input recognition threshold value P, the anomaly of the time meets the optical flow result when the product is input, and the violation anomaly of the time can be considered as the input product.
In a specific implementation of step S17, a first real-time event stream captured by an event camera is obtained, the field of view of the event camera includes a movable area near the production line, each E event in the first real-time event stream is accumulated into an event frame image, and the event frame images are stored in a time window queue.
Specifically, referring to fig. 2, the event camera 7 may capture motion information of a movable area near the production line 1, output a certain number of events corresponding to the current occurrence of human activities in the movable area near the production line 1, where potential abnormalities may occur, and perform identification and determination.
Further, storing the event frame image into a time window queue, comprising the steps of:
setting a time window queue with the length of LE, and setting all initial values to be 0;
and in the first real-time event stream, accumulating E events to obtain an event frame image, storing the event frame image into the time window queue, and obtaining the time window queue full of the event frame image in real time.
The time window queue stores the time sequence information of the current action in real time, the event frame image stores the spatial information of the current action, and the complete time window queue can be effectively processed and predicted by the trained action recognition neural network, so that an action recognition result is obtained. The use of time window queues may allow for real-time identification of dangerous actions.
In the specific implementation of step S18, the time window queue is predicted by the trained motion recognition neural network to identify whether a dangerous motion occurs.
Specifically, the action recognition neural network can predict a time window queue meeting the condition to obtain an action recognition result, the time window queue full of the event frame image is input into the trained action recognition neural network, and the trained action recognition neural network outputs the action recognition result, so that whether the dangerous action to be recognized occurs or not is judged.
Further, the training process of the trained motion recognition neural network comprises the following steps:
respectively storing event stream information of general normal actions and event stream information of special dangerous actions of a movable area near a production line acquired by the event camera;
accumulating E events in each segment of event stream into a frame image, generating a video containing action information, and making a data set, wherein the data set comprises a normal action type event frame video and various abnormal action type event frame videos;
and training the action recognition neural network by using the data set to achieve the required corresponding accuracy rate, and obtaining the trained action recognition neural network.
The event camera has the characteristics of low delay and high dynamic range, and can well identify the action in some workshop scenes with large dynamic range and under the condition of quick action which is easy to generate motion blur.
Further, the production line management and control method based on vision provided by the embodiment of the invention can further comprise the following steps:
step S19, when the event camera is continuously working, the event camera outputs in the time interval delta tIs greater than a trigger threshold EtTriggering the trained picking and placing recognition neural network and the trained action recognition neural network to work;
specifically, referring to fig. 2, the event camera 7 may capture motion information of a movable area near the production line 1, and the number of output events is large and corresponds to the current activity of a person, and a potential abnormality may occur, and an algorithm needs to be run for identification and determination. Setting the time interval Δ t and the trigger threshold E with the number of events output by the event camera 7 as the trigger condition of the systemtWhen the number of events output by the event camera in the time interval delta t is larger than a trigger threshold value EtAnd when the operation is finished, the trained pick-and-place recognition neural network and the trained action recognition neural network start to work, and corresponding prediction judgment is carried out.
Step S20, in the delay time interval Δ T, if the event camera keeps outputting the event number smaller than the trigger threshold E in the time interval Δ TtAnd the trained pick-and-place recognition neural network and the trained action recognition neural network suspend working and wait for the next trigger starting.
Specifically, a delay time interval Δ T is set, and if the event camera is in the delay time interval Δ T, the number of events output in the time interval Δ T is always kept smaller than a trigger threshold EtAnd the fact that no personnel activity exists in the current region for a period of time is shown, the trained pick-and-place recognition neural network and the trained action recognition neural network suspend working, and the next trigger starting is waited, so that the system does not need to work all weather, and only carries out processing recognition when potential abnormity occurs, and further the resource utilization efficiency of the management and control method is high.
An embodiment of the present invention further provides a production line management and control system based on vision, and referring to fig. 2, the system may include:
the system comprises at least one network camera, a video acquisition unit and a display unit, wherein the network camera is used for acquiring a first real-time video stream, the view field of each network camera only comprises open areas on a production line, and the open areas are not intersected with each other;
an event camera for capturing a first real-time stream of events, said event camera having a field of view encompassing a movable area near a production line;
a switch to pass the first real-time video stream to a processor;
a processor for acquiring a first real-time video stream acquired by a network camera, wherein in the first real-time video stream, every T frames pass through a trained pick-and-place recognition neural network to identify a current key frame I0Performing primary classification prediction; if the classified output of the neural network is a normal working class, continuing to perform next prediction; if the classified output of the neural network is the abnormal pick-and-place class, continuously collecting the next key frame I1Through I0And I1Calculating a dense optical flow F, and counting the directional distribution of the F; if F falls within the lower boundary θ of the take-out angle1And an upper bound on the extraction angle theta2Interval of composition [ theta ]1,θ2]If the proportion is larger than the taking-out identification threshold G, taking out the product if the violation exception is the violation exception; if F falls within the lower boundary θ of the entry angle3And put into the upper bound of angle theta4Interval of composition [ theta ]3,θ4]If the proportion of the illegal product is larger than the putting identification threshold value P, the illegal product is put into the illegal product; acquiring a first real-time event stream acquired by an event camera, accumulating each E events in the first real-time event stream into an event frame image, and storing the event frame image into a time window queue; and predicting the time window queue through the trained action recognition neural network, and recognizing whether dangerous actions occur or not.
Specifically, referring to fig. 2, taking two webcams as an example, a first webcam 4 and a second webcam 5 are connected to a network switch 6, and the network switch 6 is connected to a processor 8;
an event camera 7 is disposed near the production line 1, and the position and angle of the event camera 7 are adjusted so that its view screen includes a movable area near the production line 1. Connecting the event camera 7 to the processor 8;
the processor 8 stores the records of the various processing and analysis results and may further transmit the records to the mobile terminal 9.
Taking the first open area 2 and the first network camera 4 as an example (the algorithms of the second open area 3 and the second network camera 5 are the same), the specific contents for identifying the illegal taking and placing are as follows:
the processor 8 obtains a real-time video stream of the first open area 2 acquired by the first network camera 4, and in the real-time video stream with the frame rate of FPS, every T frames pass through a trained pick-and-place recognition neural network to identify the current key frame I0Performing primary classification prediction;
if the classified output of the neural network is a normal working class, continuing to perform next prediction;
if the classified output of the neural network is the abnormal pick-and-place class, continuously collecting the next key frame I1Through I0And I1Calculating a dense optical flow F, and counting the directional distribution of the F;
if F falls within the lower boundary θ of the take-out angle1And an upper bound on the extraction angle theta2Interval of composition [ theta ]1,θ2]If the ratio is greater than the taking-out identification threshold value G, the violation exception is that a product is taken out, the processor 8 records and stores the processing analysis result, and the processing analysis result can be further sent to the mobile terminal 9;
if F falls within the lower boundary θ of the entry angle3And put into the upper bound of angle theta4Interval of composition [ theta ]3,θ4]If the ratio of the violation exception is larger than the input identification threshold value P, the violation exception is that a product is input, and the processor 8 records and stores the processing and analysis result, and can further send the result to the mobile terminal 9.
According to the embodiment, the first network camera 4 can acquire the pictures of the first open area 2 on the production line 1 to perform illegal picking and placing identification alarm, meanwhile, the processor 8 can record and store the abnormal pictures and can further send the abnormal pictures to the mobile terminal 9, and a manager can check related pictures and other information, so that the tracing and confirmation are facilitated. The picking and placing recognition neural network is adopted to judge the picture, when an abnormal result is judged, the next key frame is obtained, the dense optical flow F is calculated, the direction distribution of the F is counted, and two states of picking and placing are distinguished.
Specifically, in a real-time video stream with the frame rate of FPS, the neural network classification prediction is performed every T frames, redundant information with a high frame rate can be omitted, and meanwhile, optical flow calculation is performed only when the prediction is abnormal, so that the calculation consumption of the system can be reduced as much as possible, and real-time detection and alarm are met.
Taking the event camera 7 and the production line 1 as an example, the specific contents for identifying the dangerous actions are as follows:
the processor 8 acquires a real-time event stream of a movable area near the production line 1, which is acquired by the event camera 7, accumulates each E event in the real-time event stream into an event frame image, and stores the event frame image into a time window queue;
and predicting the time window queue through the trained action recognition neural network, and recognizing whether dangerous actions occur or not.
According to the embodiment, the event camera is used for recognizing dangerous actions in the workshop, the problems that the image quality of a traditional network camera acquired in the workshop with a large dynamic range is poor and an action recognition algorithm is unstable are solved, the workshop scene pictures in various dynamic ranges can be better adapted to be monitored, action recognition can be effectively performed, and real-time recognition of the dangerous actions can be met by using the time window queue.
Corresponding to the embodiment of the vision-based production line management and control method, the application also provides an embodiment of the vision-based production line management and control device.
FIG. 3 is a block diagram illustrating a vision-based production line management and control apparatus, according to an exemplary embodiment.
Referring to fig. 3, the apparatus includes:
the system comprises an acquisition module 21, a processing module and a display module, wherein the acquisition module 21 is used for acquiring a first real-time video stream acquired by at least one network camera, the view field of each network camera only comprises open areas on a production line, and the open areas are not intersected with each other;
a first prediction module 22, configured to identify, in the first real-time video stream, the current key frame I every T frames through a trained pick-and-place recognition neural network0Performing primary classification prediction;
the first judging module 23 is configured to continue to perform next prediction if the neural network classification output is the normal pick-and-place classification;
a second judging module 24, configured to continue to acquire the next key frame I if the neural network classification output is the abnormal pick-and-place classification1Through I0And I1Calculating a dense optical flow F, and counting the directional distribution of the F;
a third determining module 25, configured to determine whether the direction distribution of F falls within the lower boundary θ of the extraction angle1And an upper bound on the extraction angle theta2Interval of composition [ theta ]1,θ2]If the proportion is larger than the taking-out identification threshold G, taking out the product if the violation exception is the violation exception;
a fourth determining module 26, configured to determine if the F falls within the lower limit θ of the input angle3And put into the upper bound of angle theta4Interval of composition [ theta ]3,θ4]If the proportion of the illegal product is larger than the putting identification threshold value P, the illegal product is put into the illegal product;
the acquisition and storage module 27 is configured to acquire a first real-time event stream acquired by an event camera, wherein a visual field of the event camera includes a movable area near a production line, each E event in the first real-time event stream is accumulated into an event frame image, and the event frame image is stored in a time window queue;
and the second prediction module 28 is used for predicting the time window queue through the trained action recognition neural network to recognize whether dangerous actions occur.
With regard to the apparatus in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment related to the method, and will not be described in detail here.
For the device embodiments, since they substantially correspond to the method embodiments, reference may be made to the partial description of the method embodiments for relevant points. The above-described embodiments of the apparatus are merely illustrative, and the units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of the application. One of ordinary skill in the art can understand and implement it without inventive effort.
Correspondingly, the present application also provides an electronic device, comprising: one or more processors; a memory for storing one or more programs; when executed by the one or more processors, cause the one or more processors to implement a vision-based line management method as described above.
Accordingly, the present application also provides a computer readable storage medium having stored thereon computer instructions, which when executed by a processor, implement a vision-based production line management and control method as described above.
Other embodiments of the present application will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the invention following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the invention pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.
It will be understood that the present application is not limited to the precise arrangements described above and shown in the drawings and that various modifications and changes may be made without departing from the scope thereof. The scope of the application is limited only by the appended claims.
Claims (10)
1. A vision-based production line management and control method is characterized by comprising the following steps:
acquiring a first real-time video stream acquired by at least one network camera, wherein the visual field of each network camera only comprises open areas on a production line, and the open areas are not intersected with each other;
in the first real-time video stream, every T frames pass through a trained pick-and-place recognition neural network to identify a current key frame I0Performing primary classification prediction;
if the classified output of the neural network is a normal working class, continuing to perform next prediction;
if the classified output of the neural network is abnormal picking and placing class, continuing to collect the next key frame I1Through I0And I1Calculating a dense optical flow F, and counting the directional distribution of the F;
if F falls within the lower boundary θ of the take-out angle1And an upper bound on the extraction angle theta2Interval of composition [ theta ]1,θ2]If the proportion is larger than the taking-out identification threshold G, taking out the product if the violation exception is the violation exception;
if F falls within the lower boundary θ of the entry angle3And put into the upper bound of angle theta4Interval of composition [ theta ]3,θ4]If the proportion of the illegal product is larger than the putting identification threshold value P, the illegal product is put into the illegal product;
acquiring a first real-time event stream acquired by an event camera, wherein the visual field of the event camera comprises a movable area near a production line, accumulating each E event in the first real-time event stream into an event frame image, and storing the event frame image into a time window queue;
and predicting the time window queue through the trained action recognition neural network, and recognizing whether dangerous actions occur or not.
2. The vision-based production line management and control method as claimed in claim 1, wherein the training process of the trained pick-and-place recognition neural network comprises the following steps:
extracting frame image pictures from a multi-section video of a production line, and making a data set;
extracting frame image pictures during normal work on a production line to serve as normal work classes of the data set;
extracting frame image pictures at the moment of abnormal picking and placing on a production line to serve as abnormal picking and placing types of the data set;
and training the classification neural network by using the data set to reach the required corresponding accuracy, and obtaining the trained picking and placing recognition neural network.
3. The vision-based production line management and control method as claimed in claim 1, characterized by comprising the following steps of I0And I1Calculating dense optical flow F, and counting the directional distribution of F, comprising the following steps:
by an optical flow calculation method, two frames of images I with similar pictures and motion changes of the pictures are obtained0And I1And calculating a dense optical flow F, calculating an optical flow direction parameter value of each pixel point of the dense optical flow F, and counting the occupation ratio of the optical flow direction parameter values of all the pixel points in different angle intervals.
4. The vision-based production line management and control method of claim 1, wherein the step of storing the event frame image into the time window queue comprises the following steps:
setting a time window queue with the length of LE, and setting all initial values to be 0;
and in the first real-time event stream, accumulating E events to obtain an event frame image, storing the event frame image into the time window queue, and obtaining the time window queue full of the event frame image in real time.
5. The vision-based production line management and control method according to claim 1, wherein the training process of the trained action recognition neural network comprises the following steps:
respectively storing event stream information of general normal actions and event stream information of special dangerous actions of a movable area near a production line acquired by the event camera;
accumulating E events in each segment of event stream into a frame image, generating a video containing action information, and making a data set, wherein the data set comprises a normal action type event frame video and various abnormal action type event frame videos;
and training the action recognition neural network by using the data set to reach the required corresponding accuracy rate, and obtaining the trained action recognition neural network.
6. The vision-based production line management and control method according to claim 1, further comprising:
when the event camera continuously works, the number of events output by the event camera in a time interval delta t is larger than a trigger threshold value EtTriggering the trained picking and placing recognition neural network and the trained action recognition neural network to work;
within the delay time interval delta T, if the number of events output by the camera in the time interval delta T is always smaller than a trigger threshold value EtAnd the trained pick-and-place recognition neural network and the trained action recognition neural network suspend working and wait for the next trigger starting.
7. The utility model provides a production line management and control device based on vision which characterized in that includes:
the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring a first real-time video stream acquired by at least one network camera, the view field of each network camera only comprises open areas on a production line, and the open areas are not intersected with each other;
a first prediction module for recognizing the current key frame I by the trained pick-and-place recognition neural network every T frames in the first real-time video stream0Performing primary classification prediction;
the first judgment module is used for continuing to predict the next time if the neural network classification output is a normal pick-and-place classification;
a second judgment module for continuing to collect the next key frame I if the classified output of the neural network is abnormal picking and placing1Through I0And I1Calculating a dense optical flow F, and counting the directional distribution of the F;
a third judging module for judging if the direction distribution of F falls on the lower boundary theta of the taking-out angle1And an upper bound on the extraction angle theta2Interval of composition [ theta ]1,θ2]If the proportion is larger than the taking-out identification threshold G, taking out the product if the violation exception is the violation exception;
a fourth judging module for judging if the direction distribution of F falls in the lower boundary theta of the putting-in angle3And put into the upper bound of angle theta4Interval of composition [ theta ]3,θ4]If the proportion of the illegal product is larger than the putting identification threshold value P, the illegal product is put into the illegal product;
the system comprises an acquisition and storage module, a time window queue and a time window processing module, wherein the acquisition and storage module is used for acquiring a first real-time event stream acquired by an event camera, the visual field of the event camera comprises a movable area near a production line, each E event in the first real-time event stream is accumulated into an event frame image, and the event frame image is stored in the time window queue;
and the second prediction module is used for predicting the time window queue through the trained action recognition neural network and recognizing whether dangerous actions occur or not.
8. A vision-based production line management and control system is characterized by comprising:
the system comprises at least one network camera, a video acquisition unit, a video processing unit and a display unit, wherein the network camera is used for acquiring a first real-time video stream, the view field of each network camera only comprises an open area on a production line, and the open areas are not intersected with each other;
an event camera for capturing a first real-time stream of events, said event camera having a field of view encompassing a movable area near a production line;
a switch to pass the first real-time video stream to a processor;
a processor for acquiring a first real-time video stream acquired by a network camera, wherein in the first real-time video stream, every T frames pass through a trained pick-and-place recognition neural network to identify a current key frame I0Performing primary classification prediction; if the classified output of the neural network is a normal working class, continuing to perform next prediction; if the classified output of the neural network is abnormal picking and placing class, continuing to collect the next key frame I1Through I0And I1Computing dense optical flow F, statisticsThe directional distribution of F; if F falls within the lower boundary θ of the take-out angle1And an upper extraction angle bound θ2Interval of composition [ theta ]1,θ2]If the proportion is larger than the taking-out identification threshold G, taking out the product if the violation exception is the violation exception; if F falls within the lower boundary θ of the entry angle3And put into the upper bound of angle theta4Interval of composition [ theta ]3,θ4]If the proportion of the illegal product is larger than the input identification threshold value P, the illegal product is input; acquiring a first real-time event stream acquired by an event camera, accumulating each E events in the first real-time event stream into an event frame image, and storing the event frame image into a time window queue; and predicting the time window queue through the trained action recognition neural network, and recognizing whether dangerous actions occur or not.
9. An electronic device, comprising:
one or more processors;
a memory for storing one or more programs;
when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-6.
10. A processor-readable storage medium having stored thereon processor instructions, which when executed by a processor, carry out the steps of the method according to any one of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210161702.2A CN114565870A (en) | 2022-02-22 | 2022-02-22 | Production line control method, device and system based on vision, and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210161702.2A CN114565870A (en) | 2022-02-22 | 2022-02-22 | Production line control method, device and system based on vision, and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114565870A true CN114565870A (en) | 2022-05-31 |
Family
ID=81714570
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210161702.2A Pending CN114565870A (en) | 2022-02-22 | 2022-02-22 | Production line control method, device and system based on vision, and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114565870A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115239205A (en) * | 2022-09-19 | 2022-10-25 | 武汉纺友技术有限公司 | Intelligent production method and device based on big data |
-
2022
- 2022-02-22 CN CN202210161702.2A patent/CN114565870A/en active Pending
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115239205A (en) * | 2022-09-19 | 2022-10-25 | 武汉纺友技术有限公司 | Intelligent production method and device based on big data |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190197319A1 (en) | Video surveillance system, video processing apparatus, video processing method, and video processing program | |
| CN107679471B (en) | Indoor personnel air post detection method based on video monitoring platform | |
| US8159537B2 (en) | Video surveillance equipment and video surveillance system | |
| CN110705482A (en) | Personnel behavior alarm prompt system based on video AI intelligent analysis | |
| CN110738135A (en) | worker work step specification visual identification judgment and guidance method and system | |
| CN104966304A (en) | Kalman filtering and nonparametric background model-based multi-target detection tracking method | |
| CN111783744A (en) | Operation site safety protection detection method and device | |
| CN112487891B (en) | Visual intelligent dynamic identification model construction method applied to electric power operation site | |
| CN113660483A (en) | Real-time monitoring method and device for video playing quality and computer equipment | |
| CN110619308A (en) | Aisle sundry detection method, device, system and equipment | |
| CN112637568B (en) | Distributed security monitoring method and system based on multi-node edge computing equipment | |
| CN104751164A (en) | Method and system for capturing movement trajectory of object | |
| CN113411542A (en) | Intelligent working condition monitoring equipment | |
| CN111652185A (en) | Safety construction method, system, device and storage medium based on violation behavior recognition | |
| CN114565870A (en) | Production line control method, device and system based on vision, and electronic equipment | |
| KR102142315B1 (en) | ATM security system based on image analyses and the method thereof | |
| CN111416960A (en) | Video monitoring system based on cloud service | |
| CN110895663B (en) | Two-wheel vehicle identification method and device, electronic equipment and monitoring system | |
| CN114764895A (en) | Abnormal behavior detection device and method | |
| CN112419639A (en) | Video information acquisition method and device | |
| CN111291728A (en) | Detection system, detection equipment and detection method for illegal crossing of transmission belt behavior | |
| CN111126100B (en) | Alarm method, alarm device, electronic equipment and storage medium | |
| CN113111847A (en) | Automatic monitoring method, device and system for process circulation | |
| EP3819817A1 (en) | A method and system of evaluating the valid analysis region of a specific scene | |
| CN111531582B (en) | Industrial robot fault detection method and system based on vision |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |