CN111460988A - Illegal behavior identification method and device - Google Patents
Illegal behavior identification method and device Download PDFInfo
- Publication number
- CN111460988A CN111460988A CN202010241982.9A CN202010241982A CN111460988A CN 111460988 A CN111460988 A CN 111460988A CN 202010241982 A CN202010241982 A CN 202010241982A CN 111460988 A CN111460988 A CN 111460988A
- Authority
- CN
- China
- Prior art keywords
- feature
- image
- optical flow
- illegal
- neural network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/045—Combinations of networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y04—INFORMATION OR COMMUNICATION TECHNOLOGIES HAVING AN IMPACT ON OTHER TECHNOLOGY AREAS
- Y04S—SYSTEMS INTEGRATING TECHNOLOGIES RELATED TO POWER NETWORK OPERATION, COMMUNICATION OR INFORMATION TECHNOLOGIES FOR IMPROVING THE ELECTRICAL POWER GENERATION, TRANSMISSION, DISTRIBUTION, MANAGEMENT OR USAGE, i.e. SMART GRIDS
- Y04S10/00—Systems supporting electrical power generation, transmission or distribution
- Y04S10/50—Systems or methods supporting the power network operation or management, involving a certain degree of interaction with the load-side end user applications
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- General Engineering & Computer Science (AREA)
- Evolutionary Computation (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computational Linguistics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Evolutionary Biology (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Multimedia (AREA)
- Image Analysis (AREA)
Abstract
The application provides a method and a device for identifying an illegal behavior, which belong to the field of image identification, and comprise the following steps: acquiring an image sequence acquired in real time based on a camera; inputting each frame of image of the image sequence into a spatial channel depth convolution neural network respectively to extract a first characteristic of each frame of image in the image sequence; calculating optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm; recording optical flow information in an image format to obtain an optical flow image; inputting the optical flow image into a time channel depth convolution neural network to extract a second feature of the optical flow image; merging the first feature and the second feature into a third feature; identifying whether an illegal action exists in the image sequence based on the third characteristic; and if the illegal behavior exists, outputting alarm information. According to the method and the device, the violation behaviors are identified through the deep convolutional neural network, and the working efficiency of identifying the violation behaviors of the personnel is improved.
Description
Technical Field
The application belongs to the field of image recognition, and particularly relates to a violation identification method and device.
Background
The transformer substation is an important link of electric energy transmission in an electric power system and has an irreplaceable important position in a power grid. Because the number of the transformer substations is greatly increased year by year, the number of the transformer substations and the number and requirements of equipment in the transformer substations are continuously increased, the work in the transformer substations is greatly increased, the number of operators in the transformer substations is more, the safety control difficulty of an operation site is increased, the frequent violation behaviors of the personnel site can be caused, the violation behaviors not only can cause potential safety hazards of equipment and a power grid, but also can cause personal safety accidents in severe cases. Therefore, real-time monitoring of the behavior of field workers is particularly important.
At present, for the detection of the violation of the personnel in the transformer substation, the inspection personnel still monitors the violation of the personnel in the transformer substation in the working site or monitors the violation of the personnel in the transformer substation through videos shot by a probe in the transformer substation, but the monitoring is carried out manually, so that the monitoring of the whole process without dead angles cannot be realized, the judgment accuracy is low, the real-time performance is poor, and the violation of the personnel in the site cannot be found and prevented in time.
Disclosure of Invention
The application aims to provide a violation identification method and device, and working efficiency of personnel violation identification is improved.
In order to achieve the above object, a first aspect of the present application provides a method for identifying an illegal action, including:
acquiring an image sequence acquired in real time based on a camera;
inputting each frame of image of the image sequence into a spatial channel deep convolutional neural network respectively to extract a first feature of each frame of image in the image sequence, wherein the spatial channel deep convolutional neural network is a deep convolutional neural network obtained by training based on a first training set in advance, and the first training set comprises: recording a static image with illegal behaviors and an illegal behavior type corresponding to the static image;
calculating optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm;
recording the optical flow information in an image format to obtain an optical flow image;
inputting the optical flow image into a time channel deep convolutional neural network to extract a second feature of the optical flow image, wherein the time channel deep convolutional neural network is a deep convolutional neural network trained on the basis of a second training set in advance, and the second training set comprises: recording a dynamic image with illegal behaviors and an illegal behavior type corresponding to the dynamic image;
merging the first feature and the second feature into a third feature;
identifying whether an illegal action exists in the image sequence based on the third characteristic;
and if the illegal behavior exists, outputting alarm information.
In a first possible implementation manner according to the first aspect of the present application, the recording the optical flow information in an image format to obtain an optical flow image includes:
and respectively storing the components of the optical flow information in the horizontal direction and the vertical direction and the magnitude of the optical flow into three channels of an RGB image to obtain a colorful optical flow image.
In a second possible implementation manner based on the first aspect of the present application or the first possible implementation manner of the first aspect of the present application, the merging the first feature and the second feature into a third feature specifically is:
and fusing the first feature and the second feature into a third feature based on a weighted average fusion algorithm.
In a second possible implementation manner based on the first aspect of the present application, in a third possible implementation manner, before the merging the first feature and the second feature into a third feature, the method further includes:
normalizing the first feature and the second feature;
the fusing the first feature and the second feature into a third feature based on the weighted average fusion algorithm specifically includes:
and fusing the normalized first feature and the normalized second feature into a third feature based on a weighted average fusion algorithm.
Based on the first aspect of the present application or the first possible implementation manner of the first aspect of the present application, in a fourth possible implementation manner, the outputting the alarm information includes:
and outputting a warning signal and an illegal image, wherein the warning signal is used for indicating that an illegal action exists in the image sequence, and the illegal image is at least one image recorded with a corresponding illegal action in the image sequence.
A second aspect of the present application provides an illegal behavior recognition device, including:
the acquisition module is used for acquiring an image sequence acquired in real time based on a camera;
a first feature extraction module, configured to input each frame of image of the image sequence into a spatial channel deep convolutional neural network, respectively, so as to extract a first feature of each frame of image in the image sequence, where the spatial channel deep convolutional neural network is a deep convolutional neural network obtained by training based on a first training set in advance, and the first training set includes: recording a static image with illegal behaviors and an illegal behavior type corresponding to the static image;
the computing module is used for computing the optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm;
the conversion module is used for recording the optical flow information in an image format to obtain an optical flow image;
a second feature extraction module, configured to input the optical flow image into a time channel deep convolutional neural network to extract a second feature of the optical flow image, where the time channel deep convolutional neural network is a deep convolutional neural network trained in advance based on a second training set, and the second training set includes: recording a dynamic image with illegal behaviors and an illegal behavior type corresponding to the dynamic image;
a fusion module for fusing the first feature and the second feature into a third feature;
the identification module is used for identifying whether violation behaviors exist in the image sequence or not based on the third characteristics;
and the output module is used for outputting alarm information if the violation behavior exists.
Based on the second aspect of the present application, in a first possible implementation manner, the fusion module is specifically configured to:
and fusing the first feature and the second feature into a third feature based on a weighted average fusion algorithm.
Based on the first possible implementation manner of the second aspect of the present application, in a second possible implementation manner, the violation behavior recognition apparatus further includes:
a processing module, configured to perform normalization processing on the first feature and the second feature;
the fusion module is specifically configured to: and fusing the normalized first feature and the normalized second feature into a third feature based on a weighted average fusion algorithm.
A third aspect of the present application provides an illegal action recognition device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the first aspect or any possible implementation manner of the first aspect when executing the computer program.
A fourth aspect of the present application provides a computer-readable storage medium, which stores a computer program that, when executed by a processor, performs the steps of the first aspect or any of the possible implementations of the first aspect.
In the method, firstly, an image sequence acquired in real time based on a camera is acquired; extracting first characteristics of each frame of image in the image sequence through a spatial channel depth convolution neural network; calculating optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm; recording optical flow information in an image format to obtain an optical flow image; and a second feature of the optical flow image of the neural network is deeply convolved through a time channel; then fusing the first feature and the second feature into a third feature and identifying whether an illegal behavior exists in the image sequence based on the third feature; if the illegal behavior exists, alarm information is output, complex processes such as image preprocessing in the manual feature extraction process are omitted, and the working efficiency of identifying the illegal behavior of personnel is improved.
Drawings
In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed to be used in the embodiments or the prior art descriptions will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present application, and it is obvious for those skilled in the art to obtain other drawings without creative efforts.
Fig. 1 is a schematic flowchart of a method for identifying an illegal action according to an embodiment of the present disclosure;
fig. 2 is a schematic structural diagram of an illegal behavior recognition device according to an embodiment of the present application;
fig. 3 is a schematic structural diagram of an illegal behavior recognition device according to another embodiment of the present application.
Detailed Description
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular system structures, techniques, etc. in order to provide a thorough understanding of the embodiments of the invention. It will be apparent, however, to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known systems, devices, circuits, and methods are omitted so as not to obscure the description of the present application with unnecessary detail.
It will be understood that the terms "comprises" and/or "comprising," when used in this specification and the appended claims, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof.
It is also to be understood that the terminology used in the description of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. As used in the specification of the present application and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
The technical solutions in the embodiments of the present application are clearly and completely described below with reference to the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only a part of the embodiments of the present application, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present application.
In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application, but the present application may be practiced in other ways than those described herein, and it will be apparent to those of ordinary skill in the art that the present application is not limited by the specific embodiments disclosed below.
Example one
An embodiment of the present application provides a method for identifying an illegal action, as shown in fig. 1, the method includes:
step 11: acquiring an image sequence acquired in real time based on a camera;
step 12: inputting each frame of image of the image sequence into a spatial channel depth convolution neural network respectively to extract a first characteristic of each frame of image in the image sequence;
the spatial channel deep convolutional neural network is a deep convolutional neural network obtained by training based on a first training set in advance, and the first training set comprises: recording a static image with illegal behaviors and an illegal behavior type corresponding to the static image;
specifically, the spatial channel depth convolution neural network extracts a static feature (i.e., a first feature) from each frame of image that is still in the image sequence.
Optionally, the first training set may be a sample video image sequence recorded with an illegal activity, and specifically, the sample video image sequence may include a still image recorded with types of illegal activities such as a person not wearing a safety helmet, a person taking off a safety helmet, and a person crossing a fence.
Optionally, when the deep convolutional neural network is trained based on training of the first training set, the deep convolutional neural network may be trained by using a static image with violations recorded in a sample video image sequence and corresponding violation types as inputs, so as to obtain the spatial channel deep convolutional neural network.
Step 13: calculating optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm;
optionally, the optical flow estimation algorithm may be a high-precision optical flow estimation algorithm based on a variational theory, and may accurately estimate the optical flow field and have good robustness.
Further, the continuous assumption of the gray values is specifically: when the position of each pixel point between frames changes, the gray value of each pixel point keeps unchanged, and the characteristic can be expressed by the following formula:
I(x,y,t)＝I(x+u,y+v,t+1)
wherein, I (x, y, t) represents the gray value information of each frame of image in the time sequence, and is a change vector of a certain pixel point from t time to t +1 time;
the gradient continuity assumption is specifically: because the gray value continuous hypothesis is sensitive to slight changes of light, the change of the gray value is properly allowed to be helpful for determining a real change vector, and at this time, a variable which is still unchanged when the gray value of the matched pixel point is changed, namely the gradient of the gray value image, needs to be found, and the specific formula is as follows:
wherein the content of the first and second substances,
representing the spatial gradient.
The smoothness assumption is specifically: the pixel displacement estimation based on the two assumptions only considers the pixel point itself, but does not consider other adjacent pixels around the pixel point; this leads to estimation errors once the gradient diverges somewhere or aperture problems occur, and therefore it is necessary to reintroduce smoothness assumptions into the optical flow field. Since the optimal displacement field may create a discontinuous situation at the object boundary, the smoothness assumption may be relaxed to only the situation that requires guaranteeing the piecewise smoothness of the optical flow field.
Based on the above three assumptions, the corresponding energy equation can be derived, specifically, let x be (x, y, t)T，w＝(u,v,1)TThe following energy equation can be derived:
where γ is a weight for balancing the gray value continuous hypothesis and the gradient continuous hypothesis, a concave function Ψ (S) may be added to the energy equation to further enhance the robustness of the energy equation2) Thus, the following calculation formula is derived:
wherein the content of the first and second substances,
the psi(s) is a small positive number, and since the psi(s) is small enough, the psi(s) can also be guaranteed to be a convex function, and the minimization of the energy equation can be guaranteed to be smoothly carried out; furthermore, Ψ does not introduce additional parametric variables, and can be set as a constant, and 0.001 can be used in the calculation.
Step 14: recording the optical flow information in an image format to obtain an optical flow image;
optionally, after obtaining the optical flow information, storing components of the optical flow information in the horizontal direction and the vertical direction and the magnitude of the optical flow into three channels of an RGB image, respectively, to obtain a color optical flow image.
Optionally, because the value of the image pixel value is [0, 255], and the calculation result of the optical flow estimation algorithm falls in a real number range close to 0 and having a positive value and a negative value, before obtaining the optical flow image, the optical flow information may be transformed into the corresponding pixel value, and the specific transformation formula is as follows:
F＝min[max[(flow×scale+128),255],0]
wherein F is the transformed pixel value; flow is the raw optical flow estimation result (component of optical flow information in the horizontal direction, component of optical flow information in the vertical direction, or magnitude of optical flow), i.e., optical flow information; scale is the magnification adjustment multiple, and when the value of scale is determined to be 16 through preliminary experiments, the optical flow calculation result can be magnified to the range with the span of 255; to prevent values with individual points from going out of range after amplification, a saturated non-linear transformation may be added at the upper and lower bounds of the transformation result.
Optionally, the components of the optical flow information in the horizontal direction and the vertical direction may be recorded by different single-channel grayscale images, so as to obtain two single-channel grayscale images as optical flow images.
Optionally, the obtained optical flow image may be directly input to the time channel depth convolution neural network for feature extraction, or the obtained optical flow image may be stored and then input to the time channel depth convolution neural network for feature extraction, which is not limited herein.
Step 15: inputting the optical flow image into a time channel depth convolution neural network to extract a second feature of the optical flow image;
the time channel deep convolutional neural network is a deep convolutional neural network obtained by training based on a second training set in advance, and the second training set comprises: recording a dynamic image with illegal behaviors and an illegal behavior type corresponding to the dynamic image;
specifically, the time-channel depth convolution neural network extracts a dynamic feature (i.e., a second feature) between every two adjacent frames of images in the image sequence based on the optical flow image.
Optionally, the second training set may be a sample video image sequence recorded with an illegal activity, and specifically, a dynamic image (i.e., an optical flow image) recorded with types of illegal activities such as a person not wearing a safety helmet, a person taking off a safety helmet, and a person crossing a fence may be obtained based on the sample video image sequence.
Optionally, when the deep convolutional neural network is trained based on training of the second training set, the dynamic image obtained based on the sample video image sequence and the corresponding violation type may be used as input to train the deep convolutional neural network, so as to obtain the time channel deep convolutional neural network.
Step 16: merging the first feature and the second feature into a third feature;
optionally, the first feature and the second feature are fused into a third feature based on a weighted average fusion algorithm, so as to reduce interference of useless frames or invalid frames in the image sequence.
Optionally, before the first feature and the second feature are combined into the third feature, the method further includes: normalizing the first feature and the second feature;
the fusing the first feature and the second feature into a third feature based on the weighted average fusion algorithm specifically includes: and fusing the normalized first feature and the normalized second feature into a third feature based on a weighted average fusion algorithm.
Specifically, the calculation formula for fusing the normalized first feature and the normalized second feature based on the weighted average fusion algorithm is specifically as follows:
wherein x isspatialAnd xtemporalThe first and second characteristics, cspatialAnd ctemporalWeighting coefficients, optionally c, for the spatial channel deep convolutional neural network and the temporal channel deep convolutional neural network, respectivelyspatialAnd ctemporal1/2 and 1/2 can be taken respectively, and 1/3 and 2/3 can be taken respectively, and the method is not limited here. Due to behavior analysisIn the research, the identification accuracy of the time channel depth convolution neural network is generally higher than that of the space channel depth convolution neural network, so that the weight of the second feature obtained by properly increasing the time channel depth convolution neural network is beneficial to the improvement of the final identification accuracy.
And step 17: identifying whether an illegal action exists in the image sequence based on the third characteristic;
the first features and the second features extracted by the two-channel (namely the time channel and the space channel) deep convolutional neural network are fused to obtain the third features, and whether the violation behaviors exist in the image sequence is identified based on the third features, so that the accuracy of the violation behavior identification result is greatly improved.
Step 18: and if the illegal behavior exists, outputting alarm information.
Optionally, the outputting the alarm information includes: and outputting a warning signal and an illegal image, wherein the warning signal is used for indicating that an illegal action exists in the image sequence, and the illegal image is at least one image recorded with a corresponding illegal action in the image sequence.
As can be seen from the above, the method for identifying an illegal action provided in the embodiment of the present application includes: acquiring an image sequence acquired in real time based on a camera; inputting each frame of image of the image sequence into a spatial channel depth convolution neural network respectively to extract a first characteristic of each frame of image in the image sequence; calculating optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm; recording optical flow information in an image format to obtain an optical flow image; inputting the optical flow image into a time channel depth convolution neural network to extract a second feature of the optical flow image; merging the first feature and the second feature into a third feature; identifying whether an illegal action exists in the image sequence based on the third characteristic; and if the illegal behavior exists, outputting alarm information. According to the method and the device, the violation behaviors are respectively identified through the deep convolutional neural network, and the working efficiency of identifying the violation behaviors of the personnel is improved.
Example two
The embodiment of the application provides a violation identification device, and fig. 2 shows a schematic structural diagram of the violation identification device provided by the embodiment of the application.
Specifically, referring to fig. 2, the violation identification apparatus 20 includes an obtaining module 21, a first feature extraction module 22, a calculation module 23, a conversion module 24, a second feature extraction module 25, a fusion module 26, an identification module 27, and an output module 28.
The acquiring module 21 is configured to acquire an image sequence acquired in real time based on a camera;
the first feature extraction module 22 is configured to input each frame of image of the image sequence into a spatial channel deep convolutional neural network, respectively, so as to extract a first feature of each frame of image in the image sequence, where the spatial channel deep convolutional neural network is a deep convolutional neural network obtained by training based on a first training set in advance, and the first training set includes: recording a static image with illegal behaviors and an illegal behavior type corresponding to the static image;
the calculation module 23 is configured to calculate optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm;
the conversion module 24 is used for recording the optical flow information in an image format to obtain an optical flow image;
the second feature extraction module 25 is configured to input the optical flow image into a time channel deep convolutional neural network to extract a second feature of the optical flow image, where the time channel deep convolutional neural network is a deep convolutional neural network trained based on a second training set in advance, and the second training set includes: recording a dynamic image with illegal behaviors and an illegal behavior type corresponding to the dynamic image;
a fusion module 26 for fusing the first feature and the second feature into a third feature;
the identification module 27 is configured to identify whether an illegal action exists in the image sequence based on the third feature;
the output module 28 is configured to output alarm information if there is an illegal action.
Optionally, the calculating module 23 is specifically configured to: and calculating the optical flow information of every two adjacent frames of images in the image sequence by using a high-precision optical flow estimation algorithm based on a variational theory.
Optionally, the conversion module 24 is specifically configured to: after the optical flow information is obtained, the components of the optical flow information in the horizontal direction and the vertical direction and the magnitude of the optical flow are respectively stored into three channels of an RGB image, and a colorful optical flow image is obtained.
Optionally, the conversion module 24 may be further configured to: before obtaining the optical flow image, the optical flow information is converted into corresponding pixel values.
Optionally, the fusion module 26 is specifically configured to: and fusing the first feature and the second feature into a third feature based on a weighted average fusion algorithm.
Optionally, the violation identification apparatus 20 further includes: a processing module (not shown in the figure) for performing normalization processing on the first feature and the second feature; the fusion module 26 is specifically configured to: and fusing the normalized first feature and the normalized second feature into a third feature based on a weighted average fusion algorithm.
Optionally, the output module 28 is specifically configured to: and outputting a warning signal and an illegal image, wherein the warning signal is used for indicating that an illegal action exists in the image sequence, and the illegal image is at least one image recorded with a corresponding illegal action in the image sequence.
As can be seen from the above, the violation identification device 20 provided in the embodiment of the present application can obtain an image sequence acquired in real time based on a camera; inputting each frame of image of the image sequence into a spatial channel depth convolution neural network respectively to extract a first characteristic of each frame of image in the image sequence; calculating optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm; recording optical flow information in an image format to obtain an optical flow image; inputting the optical flow image into a time channel depth convolution neural network to extract a second feature of the optical flow image; merging the first feature and the second feature into a third feature; identifying whether an illegal action exists in the image sequence based on the third characteristic; and if the illegal behavior exists, outputting alarm information. According to the method and the device, the violation behaviors are respectively identified through the deep convolutional neural network, and the working efficiency of identifying the violation behaviors of the personnel is improved.
EXAMPLE III
Referring to fig. 3, the violation identification apparatus includes a memory 31, a processor 32, and a computer program stored in the memory 31 and executable on the processor 32, where the memory 31 is used to store a software program and a module, and the processor 32 executes various functional applications and data processing by executing the software program and the module stored in the memory 31. The memory 31 and the processor 32 are connected by a bus 33. In particular, the processor 32, by running the above-mentioned computer program stored in the memory 31, implements the following steps:
acquiring an image sequence acquired in real time based on a camera;
inputting each frame of image of the image sequence into a spatial channel deep convolutional neural network respectively to extract a first feature of each frame of image in the image sequence, wherein the spatial channel deep convolutional neural network is a deep convolutional neural network obtained by training based on a first training set in advance, and the first training set comprises: recording a static image with illegal behaviors and an illegal behavior type corresponding to the static image;
calculating optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm;
recording the optical flow information in an image format to obtain an optical flow image;
inputting the optical flow image into a time channel deep convolutional neural network to extract a second feature of the optical flow image, wherein the time channel deep convolutional neural network is a deep convolutional neural network trained on the basis of a second training set in advance, and the second training set comprises: recording a dynamic image with illegal behaviors and an illegal behavior type corresponding to the dynamic image;
merging the first feature and the second feature into a third feature;
identifying whether an illegal action exists in the image sequence based on the third characteristic;
and if the illegal behavior exists, outputting alarm information.
Assuming that the above is the first possible embodiment, in a second possible embodiment provided based on the first possible embodiment, the recording the optical flow information in an image format to obtain an optical flow image includes:
and respectively storing the components of the optical flow information in the horizontal direction and the vertical direction and the magnitude of the optical flow into three channels of an RGB image to obtain a colorful optical flow image.
In a third possible embodiment based on the first possible embodiment or the second possible embodiment, specifically, the combination of the first feature and the second feature as a third feature is:
and fusing the first feature and the second feature into a third feature based on a weighted average fusion algorithm.
In a fourth possible embodiment based on the third possible embodiment, before the merging the first feature and the second feature into the third feature, the method further includes:
normalizing the first feature and the second feature;
the fusing the first feature and the second feature into a third feature based on the weighted average fusion algorithm specifically includes:
and fusing the normalized first feature and the normalized second feature into a third feature based on a weighted average fusion algorithm.
In a fifth possible embodiment based on the first possible embodiment or the second possible embodiment, the output alarm information includes:
and outputting a warning signal and an illegal image, wherein the warning signal is used for indicating that an illegal action exists in the image sequence, and the illegal image is at least one image recorded with a corresponding illegal action in the image sequence.
It should be understood that, in the embodiment of the present Application, the Processor 32 may be a Central Processing Unit (CPU), and the Processor 32 may also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate arrays (FPGAs) or other Programmable logic devices, discrete Gate or transistor logic devices, discrete hardware components, and the like. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like.
The memory 31 may include read-only memory, flash memory, and random access memory, and provides instructions and data to the processor. Some or all of the memory 31 may also include non-volatile random access memory.
As can be seen from the above, the violation identification device provided in the embodiment of the present application can obtain an image sequence collected in real time based on a camera; inputting each frame of image of the image sequence into a spatial channel depth convolution neural network respectively to extract a first characteristic of each frame of image in the image sequence; calculating optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm; recording optical flow information in an image format to obtain an optical flow image; inputting the optical flow image into a time channel depth convolution neural network to extract a second feature of the optical flow image; merging the first feature and the second feature into a third feature; identifying whether an illegal action exists in the image sequence based on the third characteristic; and if the illegal behavior exists, outputting alarm information. According to the method and the device, the violation behaviors are respectively identified through the deep convolutional neural network, and the working efficiency of identifying the violation behaviors of the personnel is improved.
It should be understood that the above-described integrated modules/units, if implemented in the form of software functional units and sold or used as separate products, may be stored in a computer readable storage medium. Based on such understanding, all or part of the flow in the method of the embodiments described above may be implemented by a computer program, which may be stored in a computer-readable storage medium and can implement the steps of the embodiments of the methods described above when the computer program is executed by a processor. The computer program includes computer program code, and the computer program code may be in a source code form, an object code form, an executable file or some intermediate form. The computer readable medium may include: any entity or device capable of carrying the above-mentioned computer program code, recording medium, usb disk, removable hard disk, magnetic disk, optical disk, computer Memory, Read-Only Memory (ROM), Random Access Memory (RAM), electrical carrier wave signal, telecommunication signal, software distribution medium, etc. It should be noted that the contents contained in the computer-readable storage medium can be increased or decreased as required by legislation and patent practice in the jurisdiction.
The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the application. Thus, the present application is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned functions may be distributed as different functional units and modules according to needs, that is, the internal structure of the apparatus may be divided into different functional units or modules to implement all or part of the above-mentioned functions. Each functional unit and module in the embodiments may be integrated in one processing unit, or each unit may exist alone physically, or two or more units are integrated in one unit, and the integrated unit may be implemented in a form of hardware, or in a form of software functional unit. In addition, specific names of the functional units and modules are only for convenience of distinguishing from each other, and are not used for limiting the protection scope of the present application. The specific working processes of the units and modules in the system may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
It should be noted that, the methods and the details thereof provided by the foregoing embodiments may be combined with the apparatuses and devices provided by the embodiments, which are referred to each other and are not described again.
Those of ordinary skill in the art would appreciate that the elements and algorithm steps of the examples described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the embodiments provided in the present application, it should be understood that the disclosed apparatus/terminal device and method may be implemented in other ways. For example, the above-described apparatus/device embodiments are merely illustrative, and for example, the division of the above-described modules or units is only one logical functional division, and the actual implementation may be implemented by another division, for example, a plurality of units or components may be combined or integrated into another system, or some features may be omitted, or not executed.
The above embodiments are only used to illustrate the technical solutions of the present application, and not to limit the same; although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present application and are intended to be included within the scope of the present application.
Claims (10)
1. A method for identifying an illegal action, comprising:
acquiring an image sequence acquired in real time based on a camera;
inputting each frame of image of the image sequence into a spatial channel deep convolutional neural network respectively to extract a first feature of each frame of image in the image sequence, wherein the spatial channel deep convolutional neural network is a deep convolutional neural network obtained by training based on a first training set in advance, and the first training set comprises: recording a static image with illegal behaviors and an illegal behavior type corresponding to the static image;
calculating optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm;
recording the optical flow information in an image format to obtain an optical flow image;
inputting the optical flow image into a time channel deep convolution neural network to extract a second feature of the optical flow image, wherein the time channel deep convolution neural network is a deep convolution neural network trained on the basis of a second training set in advance, and the second training set comprises: recording a dynamic image with illegal behaviors and an illegal behavior type corresponding to the dynamic image;
merging the first feature and the second feature into a third feature;
identifying whether a violation exists in the sequence of images based on the third features;
and if the illegal behavior exists, outputting alarm information.
2. The method for identifying violations of claim 1, wherein said recording of said optical flow information in an image format, and wherein obtaining an optical flow image comprises:
and respectively storing the components of the optical flow information in the horizontal direction and the vertical direction and the magnitude of the optical flow into three channels of an RGB image to obtain a colorful optical flow image.
3. The method for identifying an illegal activity according to claim 1 or 2, wherein the fusing the first feature and the second feature into a third feature is specifically:
fusing the first feature and the second feature into a third feature based on a weighted average fusion algorithm.
4. The method for identifying violations of claim 3, wherein fusing the first feature and the second feature into a third feature further comprises:
normalizing the first feature and the second feature;
the fusing the first feature and the second feature into a third feature based on the weighted average fusion algorithm specifically includes:
and fusing the normalized first feature and the normalized second feature into a third feature based on a weighted average fusion algorithm.
5. The violation behavior recognition method according to claim 1 or 2, wherein the outputting alarm information comprises:
and outputting an alarm signal and an illegal image, wherein the alarm signal is used for indicating that an illegal action exists in the image sequence, and the illegal image is at least one image recorded with a corresponding illegal action in the image sequence.
6. An apparatus for identifying an illegal action, comprising:
the acquisition module is used for acquiring an image sequence acquired in real time based on a camera;
a first feature extraction module, configured to input each frame of image of the image sequence into a spatial channel deep convolutional neural network, respectively, so as to extract a first feature of each frame of image in the image sequence, where the spatial channel deep convolutional neural network is a deep convolutional neural network obtained by training based on a first training set in advance, and the first training set includes: recording a static image with illegal behaviors and an illegal behavior type corresponding to the static image;
the calculation module is used for calculating optical flow information of every two adjacent frames of images in the image sequence based on an optical flow estimation algorithm;
the conversion module is used for recording the optical flow information in an image format to obtain an optical flow image;
a second feature extraction module, configured to input the optical flow image into a time channel deep convolutional neural network to extract a second feature of the optical flow image, where the time channel deep convolutional neural network is a deep convolutional neural network trained in advance based on a second training set, and the second training set includes: recording a dynamic image with illegal behaviors and an illegal behavior type corresponding to the dynamic image;
a fusion module for fusing the first feature and the second feature into a third feature;
the identification module is used for identifying whether violation behaviors exist in the image sequence or not based on the third features;
and the output module is used for outputting alarm information if the violation behavior exists.
7. The violation behavior recognition device according to claim 6, wherein the fusion module is specifically configured to:
fusing the first feature and the second feature into a third feature based on a weighted average fusion algorithm.
8. The violation identification apparatus of claim 7, wherein said violation identification apparatus further comprises:
the processing module is used for carrying out normalization processing on the first characteristic and the second characteristic;
the fusion module is specifically configured to: and fusing the normalized first feature and the normalized second feature into a third feature based on a weighted average fusion algorithm.
9. An illegal behavior recognition device comprising: memory, processor and computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any of claims 1 to 5 when executing the computer program.
10. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the steps of the method according to any one of claims 1 to 5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010241982.9A CN111460988B (en) | 2020-03-31 | 2020-03-31 | Illegal behavior recognition method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010241982.9A CN111460988B (en) | 2020-03-31 | 2020-03-31 | Illegal behavior recognition method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111460988A true CN111460988A (en) | 2020-07-28 |
| CN111460988B CN111460988B (en) | 2023-08-22 |
Family
ID=71685087
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010241982.9A Active CN111460988B (en) | 2020-03-31 | 2020-03-31 | Illegal behavior recognition method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111460988B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112347916A (en) * | 2020-11-05 | 2021-02-09 | 安徽继远软件有限公司 | Power field operation safety monitoring method and device based on video image analysis |
| CN112417989A (en) * | 2020-10-30 | 2021-02-26 | 四川天翼网络服务有限公司 | Invigilator violation identification method and system |
| CN112818868A (en) * | 2021-02-03 | 2021-05-18 | 招联消费金融有限公司 | Behavior sequence characteristic data-based violation user identification method and device |
| CN113284350A (en) * | 2021-05-21 | 2021-08-20 | 深圳市大道至简信息技术有限公司 | Laser-based illegal parking detection method and system |
| CN113361444A (en) * | 2021-06-22 | 2021-09-07 | 北京容联易通信息技术有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN114283485A (en) * | 2022-03-04 | 2022-04-05 | 杭州格物智安科技有限公司 | Safety helmet wearing detection method and device, storage medium and safety helmet |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102201165A (en) * | 2010-03-25 | 2011-09-28 | 北京汉王智通科技有限公司 | Monitoring system of vehicle traffic violation at crossing and method thereof |
| CN103116756A (en) * | 2013-01-23 | 2013-05-22 | 北京工商大学 | Face detecting and tracking method and device |
| DE102012109481A1 (en) * | 2012-10-05 | 2014-04-10 | Faro Technologies, Inc. | Device for optically scanning and measuring an environment |
| CN107666594A (en) * | 2017-09-18 | 2018-02-06 | 广东电网有限责任公司东莞供电局 | A kind of video monitoring monitors the method operated against regulations in real time |
| CN107730904A (en) * | 2017-06-13 | 2018-02-23 | 银江股份有限公司 | Multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks |
| CN108986473A (en) * | 2017-05-31 | 2018-12-11 | 蔚来汽车有限公司 | Vehicle mounted traffic unlawful practice identification and processing system and method |
| CN110135345A (en) * | 2019-05-15 | 2019-08-16 | 武汉纵横智慧城市股份有限公司 | Activity recognition method, apparatus, equipment and storage medium based on deep learning |
| CN110166741A (en) * | 2019-04-15 | 2019-08-23 | 深圳壹账通智能科技有限公司 | Environment control method, device, equipment and storage medium based on artificial intelligence |
-
2020
- 2020-03-31 CN CN202010241982.9A patent/CN111460988B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102201165A (en) * | 2010-03-25 | 2011-09-28 | 北京汉王智通科技有限公司 | Monitoring system of vehicle traffic violation at crossing and method thereof |
| DE102012109481A1 (en) * | 2012-10-05 | 2014-04-10 | Faro Technologies, Inc. | Device for optically scanning and measuring an environment |
| CN103116756A (en) * | 2013-01-23 | 2013-05-22 | 北京工商大学 | Face detecting and tracking method and device |
| CN108986473A (en) * | 2017-05-31 | 2018-12-11 | 蔚来汽车有限公司 | Vehicle mounted traffic unlawful practice identification and processing system and method |
| CN107730904A (en) * | 2017-06-13 | 2018-02-23 | 银江股份有限公司 | Multitask vehicle driving in reverse vision detection system based on depth convolutional neural networks |
| CN107666594A (en) * | 2017-09-18 | 2018-02-06 | 广东电网有限责任公司东莞供电局 | A kind of video monitoring monitors the method operated against regulations in real time |
| CN110166741A (en) * | 2019-04-15 | 2019-08-23 | 深圳壹账通智能科技有限公司 | Environment control method, device, equipment and storage medium based on artificial intelligence |
| CN110135345A (en) * | 2019-05-15 | 2019-08-16 | 武汉纵横智慧城市股份有限公司 | Activity recognition method, apparatus, equipment and storage medium based on deep learning |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112417989A (en) * | 2020-10-30 | 2021-02-26 | 四川天翼网络服务有限公司 | Invigilator violation identification method and system |
| CN112347916A (en) * | 2020-11-05 | 2021-02-09 | 安徽继远软件有限公司 | Power field operation safety monitoring method and device based on video image analysis |
| CN112347916B (en) * | 2020-11-05 | 2023-11-17 | 安徽继远软件有限公司 | Video image analysis-based power field operation safety monitoring method and device |
| CN112818868A (en) * | 2021-02-03 | 2021-05-18 | 招联消费金融有限公司 | Behavior sequence characteristic data-based violation user identification method and device |
| CN113284350A (en) * | 2021-05-21 | 2021-08-20 | 深圳市大道至简信息技术有限公司 | Laser-based illegal parking detection method and system |
| CN113284350B (en) * | 2021-05-21 | 2022-08-16 | 深圳市大道至简信息技术有限公司 | Laser-based illegal parking detection method and system |
| CN113361444A (en) * | 2021-06-22 | 2021-09-07 | 北京容联易通信息技术有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN113361444B (en) * | 2021-06-22 | 2021-12-14 | 北京容联易通信息技术有限公司 | Image processing method and device, electronic equipment and storage medium |
| CN114283485A (en) * | 2022-03-04 | 2022-04-05 | 杭州格物智安科技有限公司 | Safety helmet wearing detection method and device, storage medium and safety helmet |
| CN114283485B (en) * | 2022-03-04 | 2022-10-14 | 杭州格物智安科技有限公司 | Safety helmet wearing detection method and device, storage medium and safety helmet |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111460988B (en) | 2023-08-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111460988A (en) | Illegal behavior identification method and device | |
| WO2019014813A1 (en) | Method and apparatus for quantitatively detecting skin type parameter of human face, and intelligent terminal | |
| CN110287787B (en) | Image recognition method, image recognition device and computer-readable storage medium | |
| EP3282387A1 (en) | Fire detection method, fire detection apparatus and electronic equipment | |
| CN114998830A (en) | Wearing detection method and system for safety helmet of transformer substation personnel | |
| CN116386120B (en) | A noninductive control management system for wisdom campus dormitory | |
| CN109117855A (en) | Abnormal power equipment image identification system | |
| KR20110129158A (en) | Method and system for detecting a candidate area of an object in an image processing system | |
| CN111047598B (en) | Deep learning-based ultraviolet discharge light spot segmentation method and device for power transmission and transformation equipment | |
| CN107909542A (en) | Image processing method, device, computer-readable recording medium and electronic equipment | |
| CN108647605B (en) | Human eye gaze point extraction method combining global color and local structural features | |
| CN112749735B (en) | Converter tapping steel flow identification method, system, medium and terminal based on deep learning | |
| CN113222941A (en) | Method, system, equipment and medium for determining cutting state of continuous casting blank | |
| CN112001336A (en) | Pedestrian boundary crossing alarm method, device, equipment and system | |
| CN104615985B (en) | A kind of recognition methods of human face similarity degree | |
| LU501796B1 (en) | Intelligent calculation method of multi-camera earthwork coverage based on blockchain technology | |
| CN115830641A (en) | Employee identification method and device, electronic equipment and storage medium | |
| CN115601712A (en) | Image data processing method and system suitable for field safety measures | |
| CN112686851B (en) | Image detection method, device and storage medium | |
| CN114037873A (en) | Transformer substation secondary circuit protection pressing plate detection system and method based on artificial intelligence | |
| CN112651276A (en) | Power transmission channel early warning system based on double-light fusion and early warning method thereof | |
| CN112712077A (en) | Steel flow contour determination method, system, terminal and medium | |
| CN115205246B (en) | Method and device for extracting ultraviolet image characteristics of converter valve through corona discharge | |
| CN115909400A (en) | Identification method for using mobile phone behaviors in low-resolution monitoring scene | |
| CN113255595A (en) | Intelligent calculation method for multi-camera earthwork coverage rate based on block chain technology |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |