CN111144252A - Monitoring and early warning method for people stream analysis - Google Patents
Monitoring and early warning method for people stream analysis Download PDFInfo
- Publication number
- CN111144252A CN111144252A CN201911303501.6A CN201911303501A CN111144252A CN 111144252 A CN111144252 A CN 111144252A CN 201911303501 A CN201911303501 A CN 201911303501A CN 111144252 A CN111144252 A CN 111144252A
- Authority
- CN
- China
- Prior art keywords
- point cloud
- cloud data
- dimensional point
- dimensional
- monitoring processor
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/168—Feature extraction; Face representation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
- G06V20/53—Recognition of crowd images, e.g. recognition of crowd congestion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/172—Classification, e.g. identification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/174—Facial expression recognition
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02A—TECHNOLOGIES FOR ADAPTATION TO CLIMATE CHANGE
- Y02A90/00—Technologies having an indirect contribution to adaptation to climate change
- Y02A90/10—Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Theoretical Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Human Computer Interaction (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a monitoring and early warning method for people stream analysis, which comprises the following steps: the time of flight TOF camera carries out environment shooting on the monitored area according to the image acquisition instruction to obtain three-dimensional point cloud data, and sends the three-dimensional point cloud data and the camera ID to the monitoring processor; the monitoring processor carries out filtering processing on the three-dimensional point cloud data to obtain filtered three-dimensional point cloud data, carries out facial feature detection processing on the filtered three-dimensional point cloud data to obtain facial three-dimensional point cloud data, and stores the facial three-dimensional point cloud data in a facial three-dimensional point cloud data list; the monitoring processor carries out expression recognition processing based on the facial three-dimensional point cloud data to obtain an expression type of the facial three-dimensional point cloud data, when the expression type is determined to be a preset expression type, matching detection is carried out on the facial three-dimensional point cloud data and image data in a personnel information database to be recognized to obtain detection result data, and when the detection state is judged to be a preset state, an alarm message is generated according to personnel information and sent to the early warning terminal.
Description
Technical Field
The invention relates to the field of data processing, in particular to a monitoring and early warning method for people flow analysis.
Background
In recent years, with the rapid development of information technology, people apply advanced information technology, communication technology, computer processing technology and the like to various fields, and guarantee is provided for the living environment safety of people while bringing convenience to the life of people.
With the development of the camera shooting technology and the image processing technology, cameras are increasingly used in many places to monitor the living environment of people, and monitored data are stored, so that the safety of the living environment is improved. For example, cameras are installed in living places where people often move, such as shopping malls and banks, real-time images of the environment are acquired to be recorded and stored in a video mode, or the images are output through monitoring display equipment, so that the safety of the living environment is improved to a certain extent. However, since recording of the video and viewing of the monitored video require a large number of security personnel to monitor, a large amount of human resources are consumed.
Disclosure of Invention
Aiming at the defects Of the prior art, the embodiment Of the invention aims to provide a monitoring and early warning method for people flow analysis, which acquires images Of a life scene through a Time Of Flight (TOF) camera to generate three-dimensional point cloud data, identifies and judges the acquired three-dimensional point cloud data through a monitoring processor, generates alarm data when identifying people which possibly threaten the environmental safety, and sends the alarm data to terminal equipment Of safety management personnel.
In order to achieve the above object, an embodiment of the present invention provides a monitoring and early warning method for people flow analysis, including: the time of flight TOF camera carries out environment shooting on the monitored area according to the image acquisition instruction to obtain three-dimensional point cloud data; wherein the TOF camera has a camera ID;
the TOF camera sends the three-dimensional point cloud data and the camera ID to a monitoring processor;
the monitoring processor carries out filtering processing on the three-dimensional point cloud data to obtain filtered three-dimensional point cloud data; wherein the three-dimensional point cloud data comprises intensity data;
the monitoring processor carries out facial feature detection processing on the filtered three-dimensional point cloud data to obtain facial three-dimensional point cloud data, and the facial three-dimensional point cloud data are stored in a facial three-dimensional point cloud data list;
the monitoring processor carries out expression recognition processing on the basis of the facial three-dimensional point cloud data to obtain an expression type of the facial three-dimensional point cloud data;
when the monitoring processor judges that the expression type is a preset expression type, the monitoring processor performs matching detection on the facial three-dimensional point cloud data and image data in a personnel information database to be identified to obtain detection result data; wherein the detection result comprises detection state and personnel information data;
the monitoring processor judges whether the detection state is a preset state or not;
and when the detection state is a preset state, the monitoring processor generates an alarm message according to the personnel information and sends the alarm message to the early warning terminal.
Preferably, the monitoring processor performs facial feature detection processing on the filtered three-dimensional point cloud data to obtain facial three-dimensional point cloud data, and stores the facial three-dimensional point cloud data in a facial three-dimensional point cloud data list;
the monitoring processor carries out face detection processing on the intensity data of the filtered three-dimensional point cloud data based on an openCV (open content computer vision library) to obtain face intensity data, and the face intensity data is stored in a face intensity data list;
and the monitoring processor maps the face intensity data to the filtered three-dimensional point cloud data, extracts face three-dimensional point cloud data corresponding to the face intensity data from the filtered three-dimensional point cloud data, and stores the face three-dimensional point cloud data in a face three-dimensional point cloud data list.
Preferably, the monitoring processor performs expression recognition processing based on the facial three-dimensional point cloud data, and the expression type of the facial three-dimensional point cloud data is obtained by:
the monitoring processor carries out expression recognition processing on the facial three-dimensional point cloud data based on a pre-trained deep convolutional neural network model to obtain the expression type; wherein the expression types include anger, disgust, fear, happiness, depression, surprise, or neutrality.
Preferably, the information database of the person to be identified includes a plurality of first three-dimensional point cloud data and/or first two-dimensional image data, and the monitoring processor performs matching detection on the facial three-dimensional point cloud data and the image data in the information database of the person to be identified, and obtaining detection result data specifically includes:
the monitoring processor matches the facial three-dimensional point cloud data with the first three-dimensional point cloud data to obtain a maximum three-dimensional matching rate and optimal three-dimensional point cloud data;
the monitoring processor matches the intensity data of the facial three-dimensional point cloud data with the first two-dimensional image data to obtain a maximum two-dimensional matching rate and optimal two-dimensional image data;
the monitoring processor generates detection result data according to the maximum three-dimensional matching rate, the optimal three-dimensional point cloud data, the maximum two-dimensional matching rate and the optimal two-dimensional image data; wherein the detection result data includes the detection state.
Further preferably, the monitoring processor matches the facial three-dimensional point cloud data with the first three-dimensional point cloud data to obtain a maximum three-dimensional matching rate and optimal three-dimensional point cloud data specifically as follows:
the monitoring processor carries out normalization preprocessing on the facial three-dimensional point cloud data;
the monitoring processor obtains a three-dimensional average face image of all first three-dimensional point cloud data in the personnel information database to be identified,
the monitoring processor selects a plurality of feature points and a reference point on the three-dimensional average facial image, calculates the geodesic distance from each feature point to the reference point, establishes a feature point model according to the geodesic distance, and positions the feature points on the facial three-dimensional point cloud data by using the feature point model;
the monitoring processor extracts neighborhood characteristic relations of the characteristic points on the first three-dimensional point cloud data and the facial three-dimensional point cloud data by using a Gabor filter;
the monitoring processor respectively establishes a probability map model for each first three-dimensional point cloud data and the facial three-dimensional point cloud data in the personnel information database to be identified according to the neighborhood characteristic relationship;
and the monitoring processor calculates the similarity between the facial three-dimensional point cloud data and each first three-dimensional point cloud data in the personnel information database to be identified according to the probability map model, determines the maximum value of similarity as the maximum matching degree, and determines the highest first three-dimensional point cloud data of similarity as the optimal three-dimensional point cloud data.
Further preferably, the step of generating, by the monitoring processor, detection result data according to the maximum three-dimensional matching degree, the optimal three-dimensional point cloud data, the maximum two-dimensional matching degree, and the optimal two-dimensional image data specifically includes:
the monitoring processor determines the maximum three-dimensional matching degree, the maximum two-dimensional matching degree and the maximum value of the preset matching threshold;
when the maximum value is the maximum three-dimensional matching degree, the monitoring processor sets the detection state to be a success state;
the monitoring processor determines first person information according to the optimal three-dimensional point cloud data;
and the monitoring processor generates detection result data according to the detection state and the first person information.
Further preferably, the step of generating, by the monitoring processor, detection result data according to the maximum three-dimensional matching degree, the optimal three-dimensional point cloud data, the maximum two-dimensional matching degree, and the optimal two-dimensional image data specifically includes:
the monitoring processor determines the maximum three-dimensional matching degree, the maximum two-dimensional matching degree and the maximum value of the preset matching threshold;
when the maximum value is the maximum two-dimensional matching degree, the monitoring processor sets the detection state to be a success state;
the monitoring processor determines first person information according to the optimal two-dimensional image data;
and the monitoring processor generates detection result data according to the detection state and the first person information.
Further preferably, the step of generating, by the monitoring processor, detection result data according to the maximum three-dimensional matching degree, the optimal three-dimensional point cloud data, the maximum two-dimensional matching degree, and the optimal two-dimensional image data specifically includes:
the monitoring processor determines the maximum three-dimensional matching degree, the maximum two-dimensional matching degree and the maximum value of the preset matching threshold;
when the maximum value is the preset matching threshold value, the monitoring processor sets the detection state to be a failure state;
and the monitoring processor generates detection result data according to the detection state.
Preferably, before the time of flight TOF camera performs environmental shooting on the monitored area according to the image acquisition instruction, the method further comprises the following steps:
the monitoring processor receives a monitoring starting instruction and generates an image acquisition instruction according to an image acquisition time interval;
the monitoring processor sends the image acquisition instructions to the TOF camera.
The embodiment of the invention provides a monitoring and early warning method for people stream analysis, which is characterized in that a time of flight (TOF) camera is used for collecting an environment image of a monitored area to generate three-dimensional point cloud data, the collected three-dimensional point cloud data is subjected to face recognition, the recognized face three-dimensional point cloud data is subjected to expression type judgment, when the possibility that the three-dimensional point cloud data threatens the safety is obtained, the face three-dimensional point cloud data is further matched with image data in a preset personnel information database to be recognized, and when personnel information which is possibly threatened to the safety of the environment is obtained through analysis, alarm message data is generated and sent to an early warning terminal.
Drawings
Fig. 1 is a flowchart of a monitoring and early warning method for people flow analysis according to an embodiment of the present invention.
Detailed Description
The present application will be described in further detail with reference to the following drawings and examples. It is to be understood that the specific embodiments described herein are merely illustrative of the relevant invention and not restrictive of the invention. It should be further noted that, for the convenience of description, only the portions related to the related invention are shown in the drawings.
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
The invention relates to a monitoring and early warning method for people flow analysis, which is suitable for people flow intensive places such as shopping malls, museums, banks, airports, railway stations and the like. Fig. 1 is a flowchart of a monitoring and early warning method for people flow analysis according to an embodiment of the present invention. As shown in fig. 1, the method comprises the following steps:
and step 110, the TOF camera carries out environment shooting on the monitored area according to the image acquisition instruction to obtain three-dimensional point cloud data.
Specifically, TOF cameras are installed in different areas of a living site where a monitoring and early warning method needs to be used, and the TOF cameras have camera IDs. When the TOF is installed, the installation position and the shooting angle of the TOF camera can be selected or adjusted according to the range of the monitoring area required, so that the TOF camera can acquire clear face images as much as possible.
In a preferred scheme of the invention, before the time of flight TOF camera carries out environment shooting on a monitored area according to an image acquisition instruction, the monitoring processor receives a monitoring starting instruction and generates the image acquisition instruction according to an image acquisition time interval. That is, when the monitoring and early warning method provided by the embodiment of the invention is to be started, a manager inputs a monitoring starting instruction through an interactive screen of the monitoring processor; or the manager operates the hardware control equipment connected with the monitoring processor to generate a monitoring starting instruction and sends the monitoring starting instruction to the monitoring processor. And the monitoring processor reads a preset image acquisition time interval after receiving the monitoring starting instruction, generates an image acquisition instruction according to the image acquisition time interval and sends the image acquisition instruction to the TOF camera.
And when the TOF camera receives an image acquisition instruction sent by the monitoring processor, shooting an image of a frame monitoring area to generate three-dimensional point cloud data.
The TOF camera adopted in the embodiment of the invention transmits the optical signal through the built-in laser emission module and acquires the distance field depth data of the three-dimensional scene through the built-in Complementary Metal Oxide Semiconductor (CMOS) pixel array, the imaging rate can reach hundreds of frames per second, and meanwhile, the TOF camera has a compact structure and low power consumption. The three-dimensional data acquisition mode for the target scene is as follows: TOF cameras use an amplitude modulated light source that actively illuminates the target scene and is coupled to an associated sensor that is locked onto each pixel of the same frequency. The emission light of the built-in laser emission and the reflected light emitted after the emission light irradiates on the scene object have phase shift, and multiple measurements are obtained by detecting different phase shift amounts between the emission light and the reflected light. The amplitude modulation of the built-in laser transmitter is in the modulation frequency interval of 10-100MH, while the frequency controls the TOF camera sensor depth range and depth resolution. Meanwhile, a processing unit of the TOF camera independently executes phase difference calculation on each pixel to obtain depth data of a target scene, the processing unit of the TOF camera analyzes and calculates the reflection intensity of the reflected light to obtain intensity data of the target scene, and the intensity data of the target scene is analyzed and processed by combining the acquired two-dimensional data to obtain three-dimensional point cloud data of the target scene.
In a specific example of the embodiment of the present invention, the TOF camera uses a solid-state laser or an LED array as a built-in laser transmitter that transmits light waves with a wavelength around 850 nm. The emitting light source is continuous square wave or sine wave obtained by continuous modulation. The TOF camera processing unit obtains intensity data by calculating phase angles of emitted light and reflected light in a plurality of sampling samples and distances of target objects, analyzing and calculating current intensity converted by reflected light intensity, and then performing fusion processing by combining two-dimensional image data obtained by the optical camera to obtain three-dimensional point cloud data of a target scene.
In the process of collecting the environment image of the monitored area, the scene shooting is carried out through the invisible light actively emitted by the TOF camera, so that the clear three-dimensional point cloud data of the environment image of the monitored area can be obtained even under the dark condition. Therefore, the method provided by the embodiment of the invention is also suitable for use in night or dark environment with poor lighting state or even without lighting.
Specifically, each TOF camera stores a camera ID, and each camera ID corresponds to a monitoring area ID of a monitoring area to which the TOF camera belongs. The TOF camera sends the generated three-dimensional point cloud data to the monitoring processor along with the camera ID so that the monitoring processor can determine which TOF camera acquired the data when receiving the three-dimensional point cloud data.
And step 130, the monitoring processor performs filtering processing on the three-dimensional point cloud data to obtain filtered three-dimensional point cloud data.
Wherein the three-dimensional point cloud data comprises intensity data;
specifically, the monitoring processor selects a specific filtering mode to filter the received three-dimensional point cloud data, and removes noise in the three-dimensional point cloud data. The three-dimensional point cloud data is subjected to filtering processing using, for example, the following method:
in the embodiment of the present invention, the resolution of the TOF camera is M × N (M, N is all positive integers), for example, 320 × 240 or 640 × 480, and the like, so that a frame of three-dimensional point cloud data obtained by the TOF camera has M × N pixel points, and each pixel point further includes X, Y, Z three-dimensional coordinate values. Wherein, the TOF camera is used for converting original depth data into required 3-dimensional point cloud data: firstly, carrying out preliminary correction and temperature calibration on original depth data; secondly, distortion correction processing is carried out on the image; thirdly, the depth image coordinate system (x0, y0, z0) is converted into a camera coordinate system (x1, y1, z1), and the depth information on the image is converted into a three-dimensional coordinate system with the camera as an origin; finally, the camera coordinate system (x1, y1, z1) is converted to the required world coordinate system (x2, y2, z2) and the camera coordinate system is converted to the coordinate system required by the project, i.e. the coordinate system of the final point cloud. The data values of the X axis and the Y axis represent plane coordinate positions of scene points, and the data value of the Z axis represents an acquired actual depth value of the acquired scene.
The monitoring processor converts the three-dimensional point cloud data into an mxnx3 matrix, with each row representing a pixel arranged in the time-of-flight sensor. By resetting the M × N × 3 matrix to an M × N matrix and expressing the value of each element in the reset matrix with a depth value, the three-dimensional point cloud data is converted into two-dimensional planar image data.
The monitoring processor calculates the depth value of each pixel point of the two-dimensional plane image data by adopting a 3 multiplied by 3 space filtering operator based on the three-dimensional point cloud, and calculates the depth difference between the pixel of the central point and the pixel around the central point. And comparing the depth difference with a preset global threshold, judging that the depth value measured by the pixel point is a noise point when the depth difference is greater than the preset global threshold, and filtering the pixel point in the corresponding three-dimensional point cloud data. Otherwise, the corresponding pixel points in the three-dimensional point cloud data are reserved. And obtaining filtered three-dimensional point cloud data after processing. The filtered three-dimensional point cloud data also includes intensity data.
And 140, the monitoring processor performs facial feature detection processing on the filtered three-dimensional point cloud data to obtain facial three-dimensional point cloud data, and the facial three-dimensional point cloud data is stored in a facial three-dimensional point cloud data list.
Specifically, the monitoring processor performs facial feature detection on the filtered three-dimensional point cloud data by adopting a facial feature detection processing method, and extracts facial three-dimensional point cloud data from the facial feature detection processing method.
In a preferred version of the embodiment of the invention, an openCV computer vision library is installed in the monitoring processor. The method comprises the following steps of carrying out facial feature detection processing on filtered three-dimensional point cloud data based on an openCV (open view library), wherein the facial feature detection processing comprises the following steps:
firstly, the monitoring processor carries out face detection processing on the intensity data of the filtered three-dimensional point cloud data based on an openCV (open content computer vision library) to obtain face intensity data, and the face intensity data is stored in a face intensity data list. That is, the monitoring processor calls a corresponding function in the openCV library to perform face detection processing on the intensity data of the filtered three-dimensional point cloud data, and extracts face intensity data from the intensity data of the filtered three-dimensional point cloud data. And save the face intensity data in a face intensity data list.
Then, the monitoring processor maps the face intensity data to the filtered three-dimensional point cloud data, extracts face three-dimensional point cloud data corresponding to the face intensity data from the filtered three-dimensional point cloud data, and stores the face three-dimensional point cloud data in a face three-dimensional point cloud data list. More specifically, the face intensity data is a set of a plurality of pixel point data, the monitoring processor maps the face intensity data into the filtered three-dimensional point data, and the face three-dimensional point cloud data is extracted from the filtered three-dimensional point cloud data according to the corresponding relation between the face intensity data pixels and the filtered three-dimensional point cloud data, so that all face point cloud data are extracted from one frame of filtered three-dimensional point transportation and stored in a face three-dimensional point cloud data list.
Of course, there are many methods for detecting facial features, and the embodiment of the present invention is not limited to extracting facial three-dimensional point cloud data from filtered three-dimensional point cloud data by using the above method, and may also use its facial detection method for facial feature detection.
And 150, the monitoring processor performs expression recognition processing on the basis of the facial three-dimensional point cloud data to obtain the expression type of the facial three-dimensional point cloud data.
Specifically, the monitoring processor may perform expression recognition processing on the facial three-dimensional point cloud data by using a plurality of expression recognition methods.
In the preferred scheme of the embodiment of the invention, the monitoring processor adopts a deep convolutional neural network model training method in advance to train a deep convolutional neural network model capable of performing expression recognition. And the monitoring processor performs expression recognition processing on the facial three-dimensional point cloud data based on a pre-trained deep convolutional neural network model to obtain an expression type. Wherein the expression types include anger, disgust, fear, happiness, depression, surprise, or neutrality.
The deep convolutional neural network model adopted by the embodiment of the invention has high generalization capability, and can improve the recognition adaptability to the expression in the scenes of quotient excess and the like, thereby greatly improving the reliability and effectiveness of expression recognition.
And 160, when the monitoring processor judges that the expression type is the preset expression type, the monitoring processor performs matching detection on the facial three-dimensional point cloud data and the image data in the information database of the person to be identified to obtain detection result data.
Specifically, the preset expression type is stored in the monitoring processor in advance, and the preset expression type is determined by analyzing and determining the expression type by a person having professional criminal knowledge and experience, such as an expert having deep research on criminal psychology, a criminal investigation person, and the like. The preset expression type is used for judging the identified expression type, and when the monitoring processor judges that the identified expression type is the same as the preset expression type, the state of the person with the facial three-dimensional point cloud data is abnormal, and behaviors which have security threats to the public are possibly made. At this time, the monitoring processor needs to match the facial three-dimensional point cloud data with the image data in the information database of the person to be identified.
The information database of the person to be identified comprises a plurality of information data of the person, wherein each information data of the person comprises image data. The information database of the person to be identified can be obtained from data of national security agencies, such as information data of escaped persons and the like. Further, the image data may be three-dimensional point cloud data, may also be two-dimensional image data, and may also include both three-dimensional point cloud data and two-dimensional image data. The personnel information database to be identified comprises a plurality of personnel information data, so that the personnel information database to be identified also comprises a plurality of first three-dimensional point cloud data and a plurality of first two-dimensional image data.
The monitoring processor respectively performs matching detection on the facial three-dimensional point cloud data and a plurality of first two-dimensional image data in the information database of the personnel to be identified to obtain a detection result, and the method comprises the following steps:
firstly, matching facial three-dimensional point cloud data with first three-dimensional point cloud data by a monitoring processor to obtain the maximum three-dimensional matching rate and the optimal three-dimensional point cloud data;
specifically, the monitoring processor may match the three-dimensional point cloud data of the face with a plurality of first three-dimensional point cloud data in the information database of the person to be identified by using any identification method for identifying the face by using the three-dimensional point cloud, so as to obtain a maximum three-dimensional matching rate and optimal three-dimensional point cloud data. In an optional scheme of the embodiment of the invention, the monitoring processor identifies the facial three-dimensional point cloud data by adopting a three-dimensional face identification method based on a probability map model, and the matching process further specifically comprises the following steps:
step 1601, the monitoring processor performs normalization preprocessing on the facial three-dimensional point cloud data.
Step 1602, the monitoring processor finds the three-dimensional average face image of all the first three-dimensional point cloud data in the information database of the person to be identified.
Step 1603, the monitoring processor selects a plurality of feature points and a reference point on the three-dimensional average facial image, calculates the geodesic distance from each feature point to the reference point, establishes a feature point model according to the geodesic distance, and positions the feature points on the facial three-dimensional point cloud data by using the feature point model.
Step 1604, the monitoring processor extracts neighborhood feature relationships of the feature points on each of the first three-dimensional point cloud data and the facial three-dimensional point cloud data using a Gabor filter.
Step 1605, the monitoring processor establishes a probability map model for each first three-dimensional point cloud data and face three-dimensional point cloud data in the personnel information database to be identified according to the neighborhood characteristic relationship.
And 1606, calculating the similarity between the facial three-dimensional point cloud data and each first three-dimensional point cloud data in the personnel information database to be identified by the monitoring processor according to the probability map model, determining that the maximum value of the similarity is the maximum matching degree, and determining that the highest first three-dimensional point cloud data of the similarity is the optimal three-dimensional point cloud data.
And secondly, the monitoring processor matches the intensity data of the facial three-dimensional point cloud data with the first two-dimensional image data to obtain the maximum two-dimensional matching rate and the optimal two-dimensional image data.
Specifically, in the method for performing face recognition on two-dimensional image data, the monitoring processor matches the intensity data of the facial three-dimensional point cloud data with a plurality of first two-dimensional image data by using a face recognition method of elastic image matching, and confirms the matching rate with the highest matching rate and the first two-dimensional image data with the highest matching rate as the maximum two-dimensional matching rate and the optimal two-dimensional image data.
And finally, the monitoring processor generates detection result data according to the maximum three-dimensional matching rate, the optimal three-dimensional point cloud data, the maximum two-dimensional matching rate and the optimal two-dimensional image data. Wherein the detection result data includes a detection state.
Specifically, the monitoring processor determines the maximum three-dimensional matching degree, the maximum two-dimensional matching degree and the maximum value of a preset matching threshold:
and when the maximum value is the maximum three-dimensional matching degree, the monitoring processor sets the detection state to be a success state. And then the monitoring processor determines first person information according to the optimal three-dimensional point cloud data, and finally the monitoring processor generates detection result data according to the detection state and the first person information.
When the maximum value is the maximum two-dimensional matching degree, the monitoring processor sets the detection state to be a success state. And finally, the monitoring processor generates detection result data according to the detection state and the first person information.
And when the maximum value is a preset matching threshold value, the monitoring processor sets the detection state to be a matching failure state, and then generates detection result data according to the detection state. At this time, the person information data is empty.
In a specific example of the embodiment of the present invention, when the monitoring processor determines that the expression type obtained by the expression recognition processing is the preset expression type, the monitoring processor matches the facial three-dimensional point cloud data with all the first three-dimensional point cloud data in the information database of the person to be recognized, so as to obtain the similarity between the facial three-dimensional point cloud data and each first three-dimensional point cloud data, that is, the three-dimensional matching rate, and from these, it is determined that 76% of the maximum similarity is the maximum three-dimensional matching rate, and the first three-dimensional point cloud data having a similarity of 76% with the facial three-dimensional point cloud data is the optimal three-dimensional point cloud data. Meanwhile, the monitoring processor matches the intensity data of the facial three-dimensional point cloud data with all the first two-dimensional image data in the information database of the person to be identified to obtain the similarity between the intensity data of the facial three-dimensional point cloud data and each first two-dimensional image data, namely the two-dimensional matching rate, and determines that the maximum similarity of 93 percent is the maximum two-dimensional matching rate and the first two-dimensional image data with the similarity of 93 percent with the intensity data of the facial three-dimensional point cloud data is the optimal two-dimensional image data. Then, the monitoring processor compares the maximum three-dimensional matching rate, the maximum two-dimensional matching rate and a preset matching threshold value, and determines the matching state according to the maximum one of the maximum three-dimensional matching rate, the maximum two-dimensional matching rate and the preset matching threshold value, for example, the preset matching threshold value is 92%, the monitoring processor determines that the maximum two-dimensional matching rate is 93% as the maximum value, and at this time, the monitoring processor sets the detection state as a success state. Because the two-dimensional image data and the personnel information have an incidence relation, the monitoring processor can determine first personnel information according to the optimal two-dimensional image data, and the first personnel information can comprise names, ages, identification numbers, DNA data and the like. And finally, the monitoring processor generates detection result data according to the detection state and the first person information. The detection result data comprises a detection state and first person information data.
Through the matching process, the monitoring processor determines detection result data including detection state and personnel information data.
In step 170, the monitoring processor determines whether the detection status is a predetermined status.
Specifically, the monitoring processor determines whether the monitoring state is a preset state, and in an alternative embodiment of the present invention, the preset state is a success state. And when the monitoring processor judges that the detection state is a preset state, the monitoring processor shows that the information data of the person corresponding to the facial three-dimensional point cloud data, namely the person information data, is found in the person information database to be identified by the facial three-dimensional point cloud data. At this time, it is described that there is a security-threatening person in the monitoring area to which the TOF camera belongs, and step 180 is performed.
And step 180, the monitoring processor generates an alarm message according to the personnel information and sends the alarm message to the early warning terminal.
Specifically, the monitoring processor finds information of the monitoring processor according to the TOF camera ID, generates an alarm message according to the identified personnel information, and sends the alarm message to an early warning terminal used by a worker. The staff can execute the corresponding early warning scheme by viewing the content displayed on the terminal by the alarm message.
The embodiment of the invention provides a monitoring and early warning method for people stream analysis, which comprises the steps of collecting an environment image of a monitored area by using a time of flight (TOF) camera to generate three-dimensional point cloud data, carrying out face recognition on the collected three-dimensional point cloud data, carrying out expression type judgment on the recognized face three-dimensional point cloud data, further matching the face three-dimensional point cloud data with image data in a preset personnel information database to be recognized when the possibility of threatening the safety of the recognized face three-dimensional point cloud data is obtained through analysis, and generating alarm message data and sending the alarm message data to an early warning terminal when personnel information which possibly threatens the environmental safety is obtained through matching analysis and confirmation.
Those of skill would further appreciate that the various illustrative components and algorithm steps described in connection with the embodiments disclosed herein may be implemented as sub-hardware, computer software, or combinations of both, and that the various illustrative components and steps have been described above generally in terms of their functionality in order to clearly illustrate the interchangeability of hardware and software. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the implementation. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.
The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied in hardware, a software module executed by a processor, or a combination of the two. A software module may reside in Random Access Memory (RAM), memory, Read Only Memory (ROM), programmable ROM, erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.
The above embodiments are provided to further explain the objects, technical solutions and advantages of the present invention in detail, it should be understood that the above embodiments are merely exemplary embodiments of the present invention and are not intended to limit the scope of the present invention, and any modifications, equivalents, improvements and the like made within the spirit and principle of the present invention should be included in the scope of the present invention.
Claims (9)
1. A monitoring and early warning method for people stream analysis is characterized by comprising the following steps:
the time of flight TOF camera carries out environment shooting on the monitored area according to the image acquisition instruction to obtain three-dimensional point cloud data; wherein the TOF camera has a camera ID;
the TOF camera sends the three-dimensional point cloud data and the camera ID to a monitoring processor;
the monitoring processor carries out filtering processing on the three-dimensional point cloud data to obtain filtered three-dimensional point cloud data; wherein the three-dimensional point cloud data comprises intensity data;
the monitoring processor carries out facial feature detection processing on the filtered three-dimensional point cloud data to obtain facial three-dimensional point cloud data, and the facial three-dimensional point cloud data are stored in a facial three-dimensional point cloud data list;
the monitoring processor carries out expression recognition processing on the basis of the facial three-dimensional point cloud data to obtain an expression type of the facial three-dimensional point cloud data;
when the monitoring processor judges that the expression type is a preset expression type, the monitoring processor performs matching detection on the facial three-dimensional point cloud data and image data in a personnel information database to be identified to obtain detection result data; wherein the detection result comprises detection state and personnel information data;
the monitoring processor judges whether the detection state is a preset state or not;
and when the detection state is a preset state, the monitoring processor generates an alarm message according to the personnel information and sends the alarm message to the early warning terminal.
2. The monitoring and early warning method for people stream analysis according to claim 1, wherein the monitoring processor performs facial feature detection processing on the filtered three-dimensional point cloud data to obtain facial three-dimensional point cloud data, and stores the facial three-dimensional point cloud data in a facial three-dimensional point cloud data list;
the monitoring processor carries out face detection processing on the intensity data of the filtered three-dimensional point cloud data based on an openCV (open content computer vision library) to obtain face intensity data, and the face intensity data is stored in a face intensity data list;
and the monitoring processor maps the face intensity data to the filtered three-dimensional point cloud data, extracts face three-dimensional point cloud data corresponding to the face intensity data from the filtered three-dimensional point cloud data, and stores the face three-dimensional point cloud data in a face three-dimensional point cloud data list.
3. The monitoring and early warning method for people stream analysis according to claim 1, wherein the monitoring processor performs expression recognition processing based on the facial three-dimensional point cloud data, and the expression type of the facial three-dimensional point cloud data is specifically:
the monitoring processor carries out expression recognition processing on the facial three-dimensional point cloud data based on a pre-trained deep convolutional neural network model to obtain the expression type; wherein the expression types include anger, disgust, fear, happiness, depression, surprise, or neutrality.
4. The monitoring and early warning method for people stream analysis according to claim 1, wherein the database of information of people to be identified includes a plurality of first three-dimensional point cloud data and/or first two-dimensional image data, the monitoring processor performs matching detection on the facial three-dimensional point cloud data and the image data in the database of information of people to be identified, and obtaining detection result data specifically includes:
the monitoring processor matches the facial three-dimensional point cloud data with the first three-dimensional point cloud data to obtain a maximum three-dimensional matching rate and optimal three-dimensional point cloud data;
the monitoring processor matches the intensity data of the facial three-dimensional point cloud data with the first two-dimensional image data to obtain a maximum two-dimensional matching rate and optimal two-dimensional image data;
the monitoring processor generates detection result data according to the maximum three-dimensional matching rate, the optimal three-dimensional point cloud data, the maximum two-dimensional matching rate and the optimal two-dimensional image data; wherein the detection result data includes the detection state.
5. The monitoring and early warning method for people stream analysis according to claim 4, wherein the monitoring processor matches the facial three-dimensional point cloud data with the first three-dimensional point cloud data to obtain a maximum three-dimensional matching rate and optimal three-dimensional point cloud data, specifically:
the monitoring processor carries out normalization preprocessing on the facial three-dimensional point cloud data;
the monitoring processor obtains a three-dimensional average face image of all first three-dimensional point cloud data in the personnel information database to be identified,
the monitoring processor selects a plurality of feature points and a reference point on the three-dimensional average facial image, calculates the geodesic distance from each feature point to the reference point, establishes a feature point model according to the geodesic distance, and positions the feature points on the facial three-dimensional point cloud data by using the feature point model;
the monitoring processor extracts neighborhood characteristic relations of the characteristic points on the first three-dimensional point cloud data and the facial three-dimensional point cloud data by using a Gabor filter;
the monitoring processor respectively establishes a probability map model for each first three-dimensional point cloud data and the facial three-dimensional point cloud data in the personnel information database to be identified according to the neighborhood characteristic relationship;
and the monitoring processor calculates the similarity between the facial three-dimensional point cloud data and each first three-dimensional point cloud data in the personnel information database to be identified according to the probability map model, determines the maximum value of similarity as the maximum matching degree, and determines the highest first three-dimensional point cloud data of similarity as the optimal three-dimensional point cloud data.
6. The monitoring and early warning method for people stream analysis according to claim 4, wherein the step of generating detection result data by the monitoring processor according to the maximum three-dimensional matching degree, the optimal three-dimensional point cloud data, the maximum two-dimensional matching degree and the optimal two-dimensional image data specifically comprises the steps of:
the monitoring processor determines the maximum three-dimensional matching degree, the maximum two-dimensional matching degree and the maximum value of the preset matching threshold;
when the maximum value is the maximum three-dimensional matching degree, the monitoring processor sets the detection state to be a success state;
the monitoring processor determines first person information according to the optimal three-dimensional point cloud data;
and the monitoring processor generates detection result data according to the detection state and the first person information.
7. The monitoring and early warning method for people stream analysis according to claim 4, wherein the step of generating detection result data by the monitoring processor according to the maximum three-dimensional matching degree, the optimal three-dimensional point cloud data, the maximum two-dimensional matching degree and the optimal two-dimensional image data specifically comprises the steps of:
the monitoring processor determines the maximum three-dimensional matching degree, the maximum two-dimensional matching degree and the maximum value of the preset matching threshold;
when the maximum value is the maximum two-dimensional matching degree, the monitoring processor sets the detection state to be a success state;
the monitoring processor determines first person information according to the optimal two-dimensional image data;
and the monitoring processor generates detection result data according to the detection state and the first person information.
8. The monitoring and early warning method for people stream analysis according to claim 4, wherein the step of generating detection result data by the monitoring processor according to the maximum three-dimensional matching degree, the optimal three-dimensional point cloud data, the maximum two-dimensional matching degree and the optimal two-dimensional image data specifically comprises the steps of:
the monitoring processor determines the maximum three-dimensional matching degree, the maximum two-dimensional matching degree and the maximum value of the preset matching threshold;
when the maximum value is the preset matching threshold value, the monitoring processor sets the detection state to be a failure state;
and the monitoring processor generates detection result data according to the detection state.
9. The monitoring and early warning method for people stream analysis according to claim 1, wherein before the time of flight TOF camera performs environment shooting on the monitored area according to the image acquisition instruction, the method further comprises the following steps:
the monitoring processor receives a monitoring starting instruction and generates an image acquisition instruction according to an image acquisition time interval;
the monitoring processor sends the image acquisition instructions to the TOF camera.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911303501.6A CN111144252B (en) | 2019-12-17 | 2019-12-17 | Monitoring and early warning method for people stream analysis |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911303501.6A CN111144252B (en) | 2019-12-17 | 2019-12-17 | Monitoring and early warning method for people stream analysis |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111144252A true CN111144252A (en) | 2020-05-12 |
| CN111144252B CN111144252B (en) | 2023-09-05 |
Family
ID=70518640
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911303501.6A Active CN111144252B (en) | 2019-12-17 | 2019-12-17 | Monitoring and early warning method for people stream analysis |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111144252B (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112328995A (en) * | 2020-07-08 | 2021-02-05 | 德能森智能科技（成都）有限公司 | Social management system based on TOF image sensor verification |
| CN112530021A (en) * | 2020-12-24 | 2021-03-19 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for processing data |
| CN114067236A (en) * | 2021-10-28 | 2022-02-18 | 中国电子科技集团公司电子科学研究院 | Target person information detection device, detection method and storage medium |
| CN114913655A (en) * | 2022-06-16 | 2022-08-16 | 慧之安信息技术股份有限公司 | Museum intelligent security method based on edge calculation |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101221621A (en) * | 2007-01-12 | 2008-07-16 | 国际商业机器公司 | Method and system for warning a user about adverse behaviors |
| US20150042768A1 (en) * | 2010-04-15 | 2015-02-12 | Cedes Safety & Automation Ag | Time of Flight Camera Unit and Optical Surveillance System |
| CN104517104A (en) * | 2015-01-09 | 2015-04-15 | 苏州科达科技股份有限公司 | Face recognition method and face recognition system based on monitoring scene |
| JP2015162012A (en) * | 2014-02-26 | 2015-09-07 | 沖電気工業株式会社 | Face matching device, face matching method and program |
| CN105787469A (en) * | 2016-03-25 | 2016-07-20 | 广州市浩云安防科技股份有限公司 | Method and system for pedestrian monitoring and behavior recognition |
| US20170024881A1 (en) * | 2006-08-30 | 2017-01-26 | Anatomage Inc. | Patient-specific three-dimensional dentition model |
| US20170206423A1 (en) * | 2014-07-22 | 2017-07-20 | S-1 Corporation | Device and method surveilling abnormal behavior using 3d image information |
| CN108682050A (en) * | 2018-08-16 | 2018-10-19 | Oppo广东移动通信有限公司 | U.S. face method and apparatus based on threedimensional model |
| CN109670487A (en) * | 2019-01-30 | 2019-04-23 | 汉王科技股份有限公司 | A kind of face identification method, device and electronic equipment |
| KR20190101835A (en) * | 2018-02-23 | 2019-09-02 | 삼성전자주식회사 | Electronic device providing image including 3d avatar in which motion of face is reflected by using 3d avatar corresponding to face and method for operating thefeof |
| CN209784928U (en) * | 2019-05-14 | 2019-12-13 | 深圳市基鸿运科技有限公司 | Passenger flow statistics device based on TOF technology |
-
2019
- 2019-12-17 CN CN201911303501.6A patent/CN111144252B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20170024881A1 (en) * | 2006-08-30 | 2017-01-26 | Anatomage Inc. | Patient-specific three-dimensional dentition model |
| CN101221621A (en) * | 2007-01-12 | 2008-07-16 | 国际商业机器公司 | Method and system for warning a user about adverse behaviors |
| US20150042768A1 (en) * | 2010-04-15 | 2015-02-12 | Cedes Safety & Automation Ag | Time of Flight Camera Unit and Optical Surveillance System |
| JP2015162012A (en) * | 2014-02-26 | 2015-09-07 | 沖電気工業株式会社 | Face matching device, face matching method and program |
| US20170206423A1 (en) * | 2014-07-22 | 2017-07-20 | S-1 Corporation | Device and method surveilling abnormal behavior using 3d image information |
| CN104517104A (en) * | 2015-01-09 | 2015-04-15 | 苏州科达科技股份有限公司 | Face recognition method and face recognition system based on monitoring scene |
| CN105787469A (en) * | 2016-03-25 | 2016-07-20 | 广州市浩云安防科技股份有限公司 | Method and system for pedestrian monitoring and behavior recognition |
| KR20190101835A (en) * | 2018-02-23 | 2019-09-02 | 삼성전자주식회사 | Electronic device providing image including 3d avatar in which motion of face is reflected by using 3d avatar corresponding to face and method for operating thefeof |
| CN108682050A (en) * | 2018-08-16 | 2018-10-19 | Oppo广东移动通信有限公司 | U.S. face method and apparatus based on threedimensional model |
| CN109670487A (en) * | 2019-01-30 | 2019-04-23 | 汉王科技股份有限公司 | A kind of face identification method, device and electronic equipment |
| CN209784928U (en) * | 2019-05-14 | 2019-12-13 | 深圳市基鸿运科技有限公司 | Passenger flow statistics device based on TOF technology |
Non-Patent Citations (2)
| Title |
|---|
| B. MOGHADDAM,ET AL.: "Model-based 3D face capture with shape-from-silhouettes" * |
| 吴子扬: "表情稳健的三维人脸重建与识别" * |
Cited By (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112328995A (en) * | 2020-07-08 | 2021-02-05 | 德能森智能科技（成都）有限公司 | Social management system based on TOF image sensor verification |
| CN112530021A (en) * | 2020-12-24 | 2021-03-19 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for processing data |
| CN112530021B (en) * | 2020-12-24 | 2023-06-23 | 北京百度网讯科技有限公司 | Method, apparatus, device and storage medium for processing data |
| CN114067236A (en) * | 2021-10-28 | 2022-02-18 | 中国电子科技集团公司电子科学研究院 | Target person information detection device, detection method and storage medium |
| CN114913655A (en) * | 2022-06-16 | 2022-08-16 | 慧之安信息技术股份有限公司 | Museum intelligent security method based on edge calculation |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111144252B (en) | 2023-09-05 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111144252B (en) | Monitoring and early warning method for people stream analysis | |
| CN109461168B (en) | Target object identification method and device, storage medium and electronic device | |
| CN106600888B (en) | Automatic forest fire detection method and system | |
| CN112396658B (en) | Indoor personnel positioning method and system based on video | |
| CN111263114B (en) | Abnormal event alarm method and device | |
| CN112084963B (en) | Monitoring early warning method, system and storage medium | |
| CN110111515A (en) | A kind of border intrusion detection method, apparatus, server and system | |
| CN110708518B (en) | People flow analysis early warning dispersion method and system | |
| CN111899452A (en) | Forest fire prevention early warning system based on edge calculation | |
| CN109218667B (en) | Public place safety early warning system and method | |
| CN112257554B (en) | Forest fire recognition method, system, program and storage medium based on multiple spectra | |
| CN113096158A (en) | Moving object identification method and device, electronic equipment and readable storage medium | |
| CN110728810A (en) | Distributed target monitoring system and method | |
| CN112037252A (en) | Eagle eye vision-based target tracking method and system | |
| CN111723656B (en) | Smog detection method and device based on YOLO v3 and self-optimization | |
| KR20220066783A (en) | Estimation method of forest biomass | |
| CN116403377A (en) | Abnormal behavior and hidden danger detection device in public place | |
| CN116798176A (en) | Data management system based on big data and intelligent security | |
| CN113627321A (en) | Image identification method and device based on artificial intelligence and computer equipment | |
| Bosch et al. | Object discrimination by infrared image processing | |
| CN111612815A (en) | Infrared thermal imaging behavior intention analysis method and system | |
| CN111127436A (en) | Displacement detection early warning method for bridge | |
| CN106921846A (en) | Video mobile terminal legacy detection means | |
| CN111046765B (en) | Dangerous early warning method and system for high-speed rail | |
| CN111445507A (en) | Data processing method for non-visual field imaging |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |