CN112036267A - Target detection method, device, equipment and computer readable storage medium - Google Patents

Abstract

The invention discloses a target detection method, apparatus, device and computer-readable storage medium. The method includes: collecting image data of a detection environment and depth data corresponding to a target object every preset time period; and sequentially merging the image data and depth data collected at the same time in the order of the collection time from first to last to obtain the same time Corresponding image depth fusion data; according to the sequence of fusion time from first to last, sequentially input the image depth fusion data obtained by fusion into the pre-trained target detection model, and extract the image depth fusion data from the sequentially input image depth fusion data through the target detection model. The attribute data of the target object, and the three-dimensional motion trajectory data corresponding to the target object is generated according to the sequentially input image depth fusion data. The present invention does not require the user to stand around the target object, and only needs to check the attribute data and three-dimensional motion trajectory data of the target object output by the target detection model, and the monitoring process is more flexible and the monitoring efficiency is high.

Description

A target detection method, apparatus, device and computer-readable storage medium

technical field

The present invention relates to the technical field of image recognition, and in particular, to a target detection method, apparatus, device and computer-readable storage medium.

Background technique

In some application scenarios, it is necessary to pay attention to the dynamics of the target object in order to identify the needs of the target object. For example, in daily life, there are some people with special needs who cannot take care of themselves and have health problems, such as children, the elderly, and patients. These people with special needs need 24-hour supervision to avoid adverse consequences due to unsupervised supervision. For example, children are injured because they are left unattended, and patients are left unattended because they are unattended.

At present, the guardianship of people with special needs is mostly manual guardianship. The guardians generally need to accompany the wards around the ward for 24 hours, because after the guardians leave, the wards may be in unpredictable danger. However, the 24-hour manual monitoring method requires guardians to stay in the monitoring place. If multiple wards need to be monitored at the same time, one-to-one guardians are required, which makes the efficiency of this manual monitoring method relatively low.

SUMMARY OF THE INVENTION

The main purpose of the embodiments of the present invention is to provide a target detection method, apparatus, device and computer-readable storage medium, so as to solve the problem of low efficiency of the existing manual monitoring.

In view of the above-mentioned technical problems, the embodiments of the present invention are realized through the following technical solutions:

An embodiment of the present invention provides a target detection method, which includes: collecting image data of a detection environment and depth data corresponding to a target object every preset time period; and sequentially merging the collection at the same time in the order of the collection time from first to last The image data and the depth data are obtained, and the image depth fusion data corresponding to the same time is obtained; according to the sequence of fusion time from first to last, the image depth fusion data obtained by fusion is sequentially input into the pre-trained target A detection model, which extracts attribute data of the target object from the sequentially input image depth fusion data through the target detection model, and generates a three-dimensional image corresponding to the target object according to the sequentially input image depth fusion data Movement track data.

The sequential fusion of the image data and the depth data collected at the same time to obtain image depth fusion data corresponding to the same time includes: combining the image data and the depth data collected at the same time data to form a one-dimensional array of image depths corresponding to the same time, and use the one-dimensional array of image depths as the image depth fusion data corresponding to the same time.

Wherein, the target detection model includes: a YOLO model and a long short-term memory LSTM model that are connected to each other; the target detection model extracts the attribute data of the target object from the sequentially input image depth fusion data, And generating the three-dimensional motion trajectory data corresponding to the target object according to the sequentially inputted image depth fusion data, comprising: detecting the target object in the sequentially inputted image depth fusion data by using the YOLO model, Extract the three-dimensional coordinate data of the key points of the target object and the attribute data of the target object; in the sequentially inputted image depth fusion data through the LSTM model, the target detected by the YOLO model The object performs motion trajectory tracking to obtain the motion trajectory data of the target object, and according to the motion trajectory data of the target object and the YOLO model, the three-dimensional image of the key points of the target object extracted from the image depth fusion data is extracted. coordinate data to generate three-dimensional motion trajectory data of the target object.

Wherein, the YOLO model includes: a three-dimensional convolution layer; the extracting the three-dimensional coordinate data of the key points of the target object includes: extracting the image data part of the image depth fusion data through the three-dimensional convolution layer The two-dimensional feature data of the key points of the target object, extract the one-dimensional feature data of the key points of the target object in the depth data part of the image depth fusion data; according to the two-dimensional feature data and the one-dimensional feature data. dimensional feature data, and generate three-dimensional coordinate data of key points of the target object; wherein, the spatial dimension of the one-dimensional feature data is different from the spatial dimension of the two-dimensional feature data.

Wherein, the motion trajectory data is the regional position data of the target object in the image depth fusion data; the 3D motion trajectory data includes: multiple frames of 3D motion trajectory images; the motion trajectory according to the target object Data and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data, generating the three-dimensional motion trajectory data of the target object, including: constructing a three-dimensional coordinate space; In a first-come-last order, sequentially acquire the regional position data of the target object and the three-dimensional coordinate data of the key points in each of the image depth fusion data, and obtain multiple sets of the regional position data and the three-dimensional coordinates of the key points data; for each group of the regional position data and the three-dimensional coordinate data of the key points, according to the regional position data and the three-dimensional coordinate data of the key points, set the corresponding target object in the three-dimensional coordinate space. The 3D model generates a frame of 3D motion trajectory image.

Wherein, after obtaining the motion trajectory data of the target object, the method further includes: comparing the motion trajectory data of the target object with preset abnormal state data; When the similarity with the abnormal state data is greater than a preset similarity threshold, an abnormal alarm operation corresponding to the abnormal state data is performed.

Wherein, before sequentially inputting the image depth fusion data obtained by fusion into a pre-trained target detection model, the method further includes: simultaneously collecting image data of the detection environment and depth data corresponding to the target object; The image data and the depth data are obtained, and the sample image depth fusion data is obtained and the attribute data and the region position data are marked for the sample image depth fusion data; based on the sample image depth fusion, data enhancement processing is performed to obtain the sample image. For a plurality of enhanced image depth fusion data corresponding to the image depth fusion data, each of the enhanced image depth fusion data is regarded as a sample image depth fusion data, so as to train the target detection model by using all the obtained sample image depth fusion data.

The embodiment of the present invention also provides a target detection device, including: a collection module, used for collecting image data of the detection environment and depth data corresponding to the target object every preset time period; The image data and the depth data collected at the same time are fused in sequence to obtain the image depth fusion data corresponding to the same time. The image depth fusion data obtained by fusion is input into a pre-trained target detection model, and the attribute data of the target object is extracted from the sequentially input image depth fusion data by the target detection model. The inputted image depth fusion data generates three-dimensional motion trajectory data corresponding to the target object.

An embodiment of the present invention further provides a target detection device, where the target detection device includes a processor and a memory; the processor is configured to execute a target detection program stored in the memory, so as to achieve the target described in any one of the above detection method.

An embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned The target detection method of any one.

The beneficial effects of the embodiments of the present invention are as follows:

In the embodiment of the present invention, the image data of the detection environment where the target object is located and the depth data corresponding to the target object are collected, the image data and the depth data are fused into one data and input into a pre-trained target detection model, and the output of the target detection model is displayed. The attribute data and three-dimensional motion trajectory data of the target object, so that the user can intuitively see the attribute information of the target object and the motion trajectory of the target object. With the embodiment of the present invention, the user does not need to stick around the target object all the time, just check the attribute data and three-dimensional motion trajectory data of the target object output by the target detection model, the monitoring process is more flexible, and can monitor multiple monitoring objects at the same time. Low cost and high monitoring efficiency.

Description of drawings

The accompanying drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the attached image:

1 is a flowchart of a target detection method according to an embodiment of the present invention;

2 is a flowchart of processing steps of a target detection model according to an embodiment of the present invention;

3 is a flowchart of processing steps of an LSTM model according to an embodiment of the present invention;

4 is a structural diagram of a target detection device according to an embodiment of the present invention;

FIG. 5 is a structural diagram of a target detection device according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

According to an embodiment of the present invention, a target detection method is provided. As shown in FIG. 1 , it is a flowchart of a target detection method according to an embodiment of the present invention.

In step S110, image data of the detection environment and depth data corresponding to the target object are collected every preset time period.

The time length of the preset time period is an empirical value or a value obtained through experiments.

The detection environment refers to the environment in which the target object is located.

The target object refers to the image of the guardian object. For example, the guardianship objects are children, the elderly, and the sick. Of course, the guardianship objects can also be animals.

Specifically, the camera and the depth sensor are called at the same time, so that the camera collects image data of the detection environment, and the depth sensor collects the depth data corresponding to the target object. The shooting interval of the camera and the sampling interval of the depth sensor. The shooting interval and the sampling interval are the preset time period.

Image data refers to the data of one frame of image collected by the camera. Further, the viewing range of the camera is the detection environment.

The depth data refers to the depth value of the monitoring object in the detection environment with the position of the depth sensor as the starting point. Wherein, the depth value of the guardian object in the detection environment is taken as the depth value corresponding to the target object. Further, the depth sensor is facing the monitoring object or the monitoring position of the monitoring object. The monitoring location is, for example, a child's bed, a patient's bed. Further, if the depth sensor is facing the movable monitoring object, a positioning device can be set on the monitoring object, so that the positioning device transmits the position of the target object to the depth sensor in real time; or, an infrared receiver is set on the depth sensor. , Set up an infrared transmitter on the monitoring object, so that the infrared transmitter sends an infrared signal to the infrared receiver, so that the infrared receiver locates the position of the infrared transmitter according to the received infrared signal, and controls the depth sensor to align the infrared transmitter. s position.

In step S120, the image data and the depth data collected at the same time are sequentially fused according to the sequence of the collection time from first to last, to obtain the image depth fusion data corresponding to the same time.

The interval between two adjacent sampling moments is the above-mentioned preset time period. A set of image data and depth data can be collected at each collection moment. The earlier acquisition time is earlier than the later acquisition time.

According to the sequence of acquisition time from first to last, each group of image data and depth data are fused to obtain image depth fusion data corresponding to the group of image data and depth data.

Specifically, the fusion process includes: combining the image data and the depth data collected at the same time to form a one-dimensional array of image depths corresponding to the same time, and using the one-dimensional array of image depths as the same The image depth fusion data corresponding to the moment. The image data is taken as the image data part (image data element) in the image depth one-dimensional array, and the depth data is taken as the depth data part (depth data element) in the image depth one-dimensional array. Further, the order of the image data elements and the depth data elements in the one-dimensional array of image depths is not limited in this embodiment.

Step S130, according to the sequence of fusion time, input the image depth fusion data obtained by fusion into the pre-trained target detection model in sequence, extract the attribute data of the target object through the target detection model, and generate the target detection model. The three-dimensional motion trajectory data corresponding to the target object is described.

A target detection model is used to extract the attribute data of the target object from the sequentially input image depth fusion data, and generate three-dimensional motion trajectory data corresponding to the target object according to the sequentially input image depth fusion data .

The attribute data of the target object, including but not limited to: the category, name, age, height, skin color, dress and number of the target object. The categories of target objects are, for example, children, the elderly and the sick. The number is for example: the hospital number indicated on the wristband.

The three-dimensional motion track data refers to the video data of the target object moving in the three-dimensional coordinate space.

Three-dimensional motion track data, including: multiple frames of three-dimensional motion track images. Playing the multi-frame three-dimensional motion trajectory images is used to display the motion trajectory of the target object in the three-dimensional coordinate space.

Further, the three-dimensional motion trajectory data includes: the three-dimensional motion trajectory data of the target object (monitoring object) at the current collection moment and the three-dimensional motion trajectory data of the target object at the future moment. The future time refers to the time after the preset time length from the current collection time. That is to say, by training the target detection model, the target detection model predicts the three-dimensional motion trajectory data of the target object in the future according to the sequentially input image depth fusion data.

The attribute data of the target object output by the target detection model and the three-dimensional motion trajectory data corresponding to the target object are displayed on the preset display device.

In the embodiment of the present invention, the image data of the detection environment where the target object is located and the depth data corresponding to the target object are collected, the image data and the depth data are fused into one data and input into a pre-trained target detection model, and the output of the target detection model is displayed. The attribute data and three-dimensional motion trajectory data of the target object, so that the user can intuitively see the attribute information of the target object and the motion trajectory of the target object. With the embodiment of the present invention, the user does not need to stick around the target object all the time, just check the attribute data and three-dimensional motion trajectory data of the target object output by the target detection model, the monitoring process is more flexible, and can monitor multiple monitoring objects at the same time. Low cost and high monitoring efficiency.

Further, because the state of the monitoring object is flexible, the types of monitoring objects are diverse (children, elderly, various patients), and the moving path of the monitoring object in the detection environment is random, so the target detection model is used to detect the target object. (Monitoring object) to track the trajectory and generate three-dimensional motion trajectory data, which is of great significance to the automatic monitoring of the monitoring object.

Further, since the pre-trained target detection model can predict the three-dimensional motion trajectory data of the target object in the future, the user can make predictions according to the three-dimensional motion trajectory data of the target object in the future, and perform risk estimation on the guardian object. Solve the problems encountered by the guardian in a timely manner, ensure the safety of the guardian, and avoid the waste of time, labor and economic costs caused by problem discovery or untimely processing. For example, according to the image data and depth data collected over a period of time, it is predicted that the target object is about to pull out the infusion needle. At this time, the user can appear beside the target object in time to help or prevent the target object from pulling out the needle.

In order to make the target detection process of the embodiment of the present invention clearer, the structure and function of the target detection model are further described below.

The object detection model is a pre-trained model. The target detection model includes: a YOLO (YouOnly Look Once) model and an LSTM (Long Short-Term Memory, long short-term memory) model that are connected to each other.

The processing procedure of the target detection model is further described below.

As shown in FIG. 2 , it is a flowchart of processing steps of a target detection model according to an embodiment of the present invention.

Step S210, input the image depth fusion data obtained by fusion into the YOLO model in sequence.

The YOLO model is used to detect the target object in the image data of the image depth fusion data, extract the attribute data of the target object, detect the key points of the target object and extract the three-dimensional coordinate data of the key points. Wherein, the number of key points can be multiple.

The key point refers to the key position of the target object. For example: eyebrows, eye corners, mouth corners, earlobes, shoulders, elbows, knees, etc. Keypoints can outline the image area of the target object.

Step S220, the YOLO model is used to detect the target object in the sequentially input image depth fusion data, and extract the three-dimensional coordinate data of the key points of the target object and the attribute data of the target object.

The three-dimensional coordinate data of the key point refers to the coordinates of the key point in the three-dimensional space.

In this embodiment, in order to display the three-dimensional motion data of the target object more accurately, three-dimensional coordinate data of each key point among the multiple key points may be extracted.

The detected target object and the key points of the detected target object are output to the LSTM model through the YOLO model.

Step S230, in the sequentially inputted image depth fusion data through the LSTM model, the target object detected by the YOLO model is tracked by motion trajectory, to obtain the motion trajectory data of the target object, and according to the target object and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data to generate the three-dimensional motion trajectory data of the target object.

The LSTM model is used to track the motion trajectory of the target object according to the output result of the YOLO model, and generate the three-dimensional motion trajectory data of the target object.

Motion trajectory tracking is the tracking of key points of the target object.

The motion trajectory data includes: the region position data of the target object in the image depth fusion data. The area position data may be the area position of the graph formed by the key points of the target object. The graphic is similar to the target object.

The image depth fusion data includes an image data part and a depth data part, and the motion track data is the region position data of the target object in the image data of the image depth fusion data.

Since the key points of the target object can reflect the posture of the target object, the motion trajectory data can reflect the state of the guardian object (target object). The state of the monitoring object can be determined by using the motion track data of one frame of image data or the motion track data of multiple frames of image data. The state includes: health state, behavior state. For example, it is determined whether the monitoring object is within the preset range of the detection environment through the motion trajectory data of a frame of image data. Another example: through the motion trajectory data of a frame of image data, it is determined whether the monitoring object wears a mask and keeps a distance from others. For another example: through the motion track data of the multi-frame image data, it is determined whether the gait of the monitoring object is normal, and whether the monitoring object has convulsions.

In the target detection model, a basic network model may also be included, the output of the basic network model is connected to the input of the YOLO model, and the output of the YOLO model is connected to the input of the LSTM model. The basic network model is used to preprocess the image data in the image depth fusion data, convert the image data to grayscale data and reduce the noise in the image data.

The process of YOLO model detecting key points and extracting three-dimensional coordinate data of the key points is further described below.

After detecting the image of the target object, the YOLO model performs key point recognition in the image of the target object, and extracts the three-dimensional coordinate data of each key point for the identified key points of multiple preset types. Multiple preset types of keys include but are not limited to: eyebrow key, shoulder key, elbow key, waist key, knee key, footstep key.

Further, the YOLO model includes: a three-dimensional convolutional layer. Further, the YOLO model is the fourth version of the YOLO algorithm model (YOLOv4). Extend the 2D convolutional layer of the YOLO model to a 3D convolutional layer. The three-dimensional convolutional layer includes a three-channel convolutional layer. The structure of the convolutional layers of the three channels is the same, and the convolutional layer of each channel is used to extract feature data of one spatial dimension.

The two-dimensional feature data of the key points of the target object is extracted from the image data part of the image depth fusion data by the three-dimensional convolution layer, and the target object is extracted from the depth data part of the image depth fusion data The one-dimensional feature data of the key point of the target object; and the three-dimensional coordinate data of the key point of the target object is generated according to the two-dimensional feature data and the one-dimensional feature data.

Wherein, the spatial dimension of the one-dimensional feature data is different from the spatial dimension of the two-dimensional feature data. For example, the two-dimensional feature data includes: the coordinates of the X-axis and Y-axis corresponding to the key points; the one-dimensional feature data includes: the Z-axis coordinates corresponding to the key points; the X-axis, Y-axis and Z-axis coordinates corresponding to the key points are fused , to obtain the three-dimensional coordinate data corresponding to the key points.

The processing of the LSTM model is further described below. As shown in FIG. 3 , it is a flowchart of processing steps of the LSTM model according to an embodiment of the present invention.

Step S310, constructing a three-dimensional coordinate space.

The three-dimensional coordinate space may be a skybox with a preset viewing angle.

The preset angle of view is the same as the angle of view of the camera that collects image data.

Step S320, according to the order of fusion time from first to last, sequentially obtain the regional position data of the target object and the three-dimensional coordinate data of the key points in each of the image depth fusion data, and obtain multiple groups of regional position data and key points. 3D coordinate data.

The three-dimensional motion track data includes: multiple frames of three-dimensional motion track images. The three-dimensional motion trajectory image refers to the image of the three-dimensional model corresponding to the target object in the three-dimensional coordinate space.

The regional position data of the target object may reflect the regional position of the target object in the image data.

The LSTM model can extract the time series features of multiple frames of image data from the continuously input multiple image depth fusion data, and determine the regional position of the target object in the image data according to the time series features. The extracted time series features are the similarities and differences between the image data of the previous frame and the image data of the next frame. For example, people are coherent in the process of walking, and there are partial similarities and differences between two adjacent frames of image data from near to far (or from far to near) images of people, and temporal features are used to reflect this time sequence. The relationship can represent the similarities and differences between the two frames of image data before and after the time series.

Step S330, for each group of regional position data and the three-dimensional coordinate data of the key points, according to the regional position data and the three-dimensional coordinate data of the key points, set the three-dimensional model corresponding to the target object in the three-dimensional coordinate space, and generate a Frame 3D motion trajectory images.

Each group of regional position data and three-dimensional coordinate data of key points are extracted from the same image depth fusion data.

In this embodiment, Unity3D can be used to generate a three-dimensional motion trajectory image of the target object.

Specifically, the three-dimensional model of the monitoring object, that is, the three-dimensional model corresponding to the target object, may be collected in advance. The regional position data indicates the regional position of the target object in the XOY plane; the three-dimensional coordinate data of the key point indicates the position of the key point of the target object in the three-dimensional coordinate space; the three-dimensional model is set at the regional position of the target object in the XOY plane; Determine the mapping points to which each key point of the target object is mapped on the three-dimensional model, and adjust the three-dimensional coordinates of each mapping point to the three-dimensional coordinates of the corresponding key points; after the adjustment is completed, generate a three-dimensional model under a preset viewing angle , as a frame of three-dimensional motion trajectory image corresponding to the target object.

In this embodiment, since the motion trajectory data of the target object can be used to represent the state of the monitored object, in order to avoid missing the abnormal behavior of the monitored object, after the motion trajectory data of the target object is obtained, the The motion track data is compared with preset abnormal state data; when the similarity between the motion track data of the target object and the abnormal state data is greater than a preset similarity threshold, an abnormal alarm operation corresponding to the abnormal state data is performed.

The abnormal state data is the image data when the guardian object has abnormal behavior.

The similarity threshold is an empirical value or a value obtained through experiments.

The abnormal alarm operation includes: sending out a preset alarm tone, and/or sending information of the alarm content corresponding to the abnormal state data to the target user. The information is text information and/or voice information.

For example: the monitoring object is the patient, the movement trajectory data of the target object is that the patient is pulling out the infusion needle, and the abnormal state data is the image data of the monitoring object pulling out the infusion needle. In this way, the similarity between the movement trajectory data of the target object and the abnormal state data If it is greater than the similarity threshold, a preset alarm tone will be issued or the information that the patient is trying to withdraw the needle will be sent to the nurse station.

In this embodiment, the target detection model is obtained by pre-training. Before training an object detection model, a sample dataset needs to be set up for the object detection model. The sample dataset includes a number of annotated sample image depth fusion data.

Specifically, the image data of the detection environment and the depth data corresponding to the target object are collected at the same time; the image data and the depth data collected at the same time are fused to obtain sample image depth fusion data and annotate attributes for the sample image depth fusion data data and regional location data. Further, when collecting the image data of the detection environment, collect the image data from multiple angles such as the side, front, and back of the monitoring object as much as possible to ensure the diversity of the deep fusion data of the sample images, so as to better carry out the three-dimensional image of the target object. detection.

When the quantity of the sample image depth fusion data is small, the sample image depth fusion data can be enhanced, that is, on the basis of the existing sample image depth fusion data, the quantity of the sample image depth fusion data is expanded. Specifically, data enhancement processing is performed based on the deep fusion of the sample images to obtain a plurality of deep fusion data of the enhanced images corresponding to the deep fusion data of the sample images, and each of the deep fusion data of the enhanced images is regarded as a deep fusion data of the sample images , so as to train the target detection model using the obtained deep fusion data of all sample images.

Further, random cropping, scaling, rotating, flipping, mirroring, adding noise, and randomly adjusting contrast, brightness, chroma, etc. are performed on the image data in the sample image depth fusion data to obtain new image data, and the new image data and The depth data in the sample image depth fusion data is combined to obtain new image depth fusion data, and the new image depth fusion data is used as the sample image depth fusion data.

The obtained sample image depth fusion data is stored in a preset sample data set.

Before training the object detection model, set the training data for the object detection model. In this embodiment, the training data includes, but is not limited to, the number of iterations, the initial weight and network structure of the target detection model, the learning rate, and the convolution kernel.

When training the target detection model, the gradient descent method is used to train the target detection model. In the process of training the target detection model using the training data set, the parameters and weights of the target detection model are continuously adjusted according to the preset number of iterations. After the iteration, if the target detection model has not converged, adjust the network structure of the target detection model, and retrain the target detection model until the target detection model converges.

Further, in order to increase the accuracy and robustness of the YOLO model, multi-scale training is added to the YOLO model, the image data in the sample image deep fusion data is converted into multiple scales, and the YOLO model is trained using the image data of multiple scales. For example, set the image width and height in the configuration file of the YOLO model to 640*640 to improve the detection accuracy of the YOLO model for small targets.

In this embodiment, the loss function is used to determine the accuracy and stability of the target detection model. When the loss value of the target detection model is less than the preset loss threshold, it means that the target detection model has converged. Further, the loss function may be a confidence loss function, a classification loss function, or a loss function based on the target rectangle selection box. For example, you can use the Distance Intersection Over Union (DIOU) loss function or the Complete Intersection Over Union (CIOU) loss function to calculate the loss value of the target detection model, and if the loss value is greater than When the preset loss threshold is set, the parameters of the target detection model are adjusted.

The embodiment of the present invention also provides a target detection device. As shown in FIG. 4 , it is a structural diagram of a target detection apparatus according to an embodiment of the present invention.

The target detection device includes: a collection module 410 , a fusion module 420 and a detection module 430 .

The acquisition module 410 is configured to acquire image data of the detection environment and depth data corresponding to the target object every preset time period.

The fusion module 420 is configured to fuse the image data and the depth data collected at the same time in sequence according to the sequence of acquisition time, to obtain the image depth fusion data corresponding to the same time.

The detection module 430 is configured to sequentially input the image depth fusion data obtained by fusion into the pre-trained target detection model according to the sequence of fusion time, and use the target detection model in the sequentially inputted images. The attribute data of the target object is extracted from the depth fusion data, and three-dimensional motion trajectory data corresponding to the target object is generated according to the sequentially input image depth fusion data.

The functions of the apparatuses according to the embodiments of the present invention have been described in the foregoing method embodiments, so for details that are not described in this embodiment, reference may be made to the relevant descriptions in the foregoing embodiments, which will not be repeated here.

This embodiment provides a target detection device. As shown in FIG. 5 , it is a structural diagram of a target detection device according to an embodiment of the present invention.

In this embodiment, the target detection device includes but is not limited to: a processor 510 and a memory 520 .

The processor 510 is configured to execute the target detection program stored in the memory 520 to implement the above target detection method.

Specifically, the processor 510 is configured to execute the target detection program stored in the memory 520, so as to realize the following steps: collecting image data of the detection environment and depth data corresponding to the target object every preset time period; In a first-to-last order, the image data and the depth data collected at the same time are fused in sequence to obtain the image depth fusion data corresponding to the same time; The image depth fusion data is input into a pre-trained target detection model, and the attribute data of the target object is extracted from the sequentially input image depth fusion data by the target detection model, and according to the sequentially inputted The image depth fusion data generates three-dimensional motion trajectory data corresponding to the target object.

The sequential fusion of the image data and the depth data collected at the same time to obtain image depth fusion data corresponding to the same time includes: combining the image data and the depth data collected at the same time data to form a one-dimensional array of image depths corresponding to the same time, and use the one-dimensional array of image depths as the image depth fusion data corresponding to the same time.

Wherein, the target detection model includes: a YOLO model and a long short-term memory LSTM model that are connected to each other; the target detection model extracts the attribute data of the target object from the sequentially input image depth fusion data, And generating the three-dimensional motion trajectory data corresponding to the target object according to the sequentially inputted image depth fusion data, comprising: detecting the target object in the sequentially inputted image depth fusion data by using the YOLO model, Extract the three-dimensional coordinate data of the key points of the target object and the attribute data of the target object; in the sequentially inputted image depth fusion data through the LSTM model, the target detected by the YOLO model The object performs motion trajectory tracking to obtain the motion trajectory data of the target object, and according to the motion trajectory data of the target object and the YOLO model, the three-dimensional image of the key points of the target object extracted from the image depth fusion data is extracted. coordinate data to generate three-dimensional motion trajectory data of the target object.

Wherein, the YOLO model includes: a three-dimensional convolution layer; the extracting the three-dimensional coordinate data of the key points of the target object includes: extracting the image data part of the image depth fusion data through the three-dimensional convolution layer The two-dimensional feature data of the key points of the target object, extract the one-dimensional feature data of the key points of the target object in the depth data part of the image depth fusion data; according to the two-dimensional feature data and the one-dimensional feature data. dimensional feature data, and generate three-dimensional coordinate data of key points of the target object; wherein, the spatial dimension of the one-dimensional feature data is different from the spatial dimension of the two-dimensional feature data.

Wherein, the motion trajectory data is the regional position data of the target object in the image depth fusion data; the 3D motion trajectory data includes: multiple frames of 3D motion trajectory images; the motion trajectory according to the target object Data and the three-dimensional coordinate data of the key points of the target object extracted by the YOLO model in the image depth fusion data, generating the three-dimensional motion trajectory data of the target object, including: constructing a three-dimensional coordinate space; In a first-come-last order, sequentially acquire the regional position data of the target object and the three-dimensional coordinate data of the key points in each of the image depth fusion data, and obtain multiple sets of the regional position data and the three-dimensional coordinates of the key points data; for each group of the regional position data and the three-dimensional coordinate data of the key points, according to the regional position data and the three-dimensional coordinate data of the key points, set the corresponding target object in the three-dimensional coordinate space. The 3D model generates a frame of 3D motion trajectory image.

Wherein, after obtaining the motion trajectory data of the target object, the method further includes: comparing the motion trajectory data of the target object with preset abnormal state data; When the similarity with the abnormal state data is greater than a preset similarity threshold, an abnormal alarm operation corresponding to the abnormal state data is performed.

Wherein, before sequentially inputting the image depth fusion data obtained by fusion into a pre-trained target detection model, the method further includes: simultaneously collecting image data of the detection environment and depth data corresponding to the target object; The image data and the depth data are obtained, and the sample image depth fusion data is obtained and the attribute data and the region position data are marked for the sample image depth fusion data; based on the sample image depth fusion, data enhancement processing is performed to obtain the sample image. For a plurality of enhanced image depth fusion data corresponding to the image depth fusion data, each of the enhanced image depth fusion data is regarded as a sample image depth fusion data, so as to train the target detection model by using all the obtained sample image depth fusion data.

Embodiments of the present invention also provide a computer-readable storage medium. The computer-readable storage medium herein stores one or more programs. The computer-readable storage medium may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as read-only memory, flash memory, hard disk or solid-state hard disk; the memory may also include the above-mentioned types combination of memory.

When one or more programs in the computer-readable storage medium can be executed by one or more processors, the above-mentioned target detection method can be implemented. Since the object detection method has been described in detail above, it will not be repeated here.

The above description is only an embodiment of the present invention, and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included within the scope of the claims of the present invention.

Claims (10)

1. A method of object detection, comprising:

acquiring image data of a detection environment and depth data corresponding to a target object at intervals of a preset time period;

sequentially fusing the image data and the depth data acquired at the same time according to the sequence of the acquisition time from first to last to obtain image depth fusion data corresponding to the same time;

sequentially inputting the image depth fusion data obtained by fusion into a pre-trained target detection model according to the sequence of fusion time from first to last, extracting attribute data of the target object from the sequentially input image depth fusion data through the target detection model, and generating three-dimensional motion trajectory data corresponding to the target object according to the sequentially input image depth fusion data.

2. The method according to claim 1, wherein the sequentially fusing the image data and the depth data acquired at the same time to obtain image depth fusion data corresponding to the same time comprises:

and combining the image data and the depth data acquired at the same time to form an image depth one-dimensional array corresponding to the same time, and taking the image depth one-dimensional array as image depth fusion data corresponding to the same time.

3. The method of claim 1,

the target detection model includes: a connected YOLO model and a long-short term memory (LSTM) model;

the extracting, by the target detection model, attribute data of the target object from the sequentially input image depth fusion data, and generating three-dimensional motion trajectory data corresponding to the target object according to the sequentially input image depth fusion data, includes:

detecting the target object in the image depth fusion data sequentially input through the YOLO model, and extracting three-dimensional coordinate data of key points of the target object and attribute data of the target object;

and tracking the motion trail of the target object detected by the YOLO model in the sequentially input image depth fusion data through the LSTM model to obtain the motion trail data of the target object, and generating the three-dimensional motion trail data of the target object according to the motion trail data of the target object and the three-dimensional coordinate data of the key point of the target object extracted by the YOLO model in the image depth fusion data.

4. The method of claim 3,

the YOLO model includes: a three-dimensional convolutional layer;

the extracting three-dimensional coordinate data of the key points of the target object comprises:

extracting two-dimensional feature data of key points of the target object in an image data part of the image depth fusion data through the three-dimensional convolution layer, and extracting one-dimensional feature data of the key points of the target object in a depth data part of the image depth fusion data;

generating three-dimensional coordinate data of key points of the target object according to the two-dimensional characteristic data and the one-dimensional characteristic data; wherein the spatial dimension of the one-dimensional feature data is different from the spatial dimension of the two-dimensional feature data.

5. The method of claim 3,

the motion trail data is regional position data of the target object in the image depth fusion data;

the three-dimensional motion trajectory data includes: multi-frame three-dimensional motion trail images;

the generating three-dimensional motion trajectory data of the target object according to the motion trajectory data of the target object and the three-dimensional coordinate data of the key point of the target object extracted by the YOLO model in the image depth fusion data includes:

constructing a three-dimensional coordinate space;

sequentially acquiring the region position data of the target object and the three-dimensional coordinate data of the key points in each image depth fusion data according to the sequence of fusion time from first to last to obtain a plurality of groups of region position data and three-dimensional coordinate data of the key points;

and setting a three-dimensional model corresponding to the target object in the three-dimensional coordinate space according to the region position data and the three-dimensional coordinate data of the key points aiming at each group of the region position data and the three-dimensional coordinate data of the key points, and generating a frame of three-dimensional motion trail image.

6. The method of claim 3, wherein after said obtaining motion trajectory data of said target object, said method further comprises:

comparing the motion trail data of the target object with preset abnormal state data;

and when the similarity between the motion trail data of the target object and the abnormal state data is greater than a preset similarity threshold, executing abnormal alarm operation corresponding to the abnormal state data.

7. The method according to any one of claims 1-6, wherein before the sequentially inputting the image depth fusion data obtained by fusion into a pre-trained target detection model, the method further comprises:

simultaneously acquiring image data of a detection environment and depth data corresponding to a target object;

fusing the image data and the depth data which are acquired simultaneously to obtain sample image depth fusion data and marking attribute data and region position data for the sample image depth fusion data;

and performing data enhancement processing based on the sample image depth fusion to obtain a plurality of enhanced image depth fusion data corresponding to the sample image depth fusion data, and using each enhanced image depth fusion data as one sample image depth fusion data so as to train the target detection model by using all the obtained sample image depth fusion data.

8. An object detection device, comprising:

the acquisition module is used for acquiring image data of a detection environment and depth data corresponding to a target object at intervals of a preset time period;

the fusion module is used for sequentially fusing the image data and the depth data acquired at the same time according to the sequence of the acquisition time from first to last to obtain image depth fusion data corresponding to the same time;

the detection module is used for sequentially inputting the image depth fusion data obtained by fusion into a pre-trained target detection model according to the sequence of fusion time from first to last, extracting attribute data of the target object from the sequentially input image depth fusion data through the target detection model, and generating three-dimensional motion trajectory data corresponding to the target object according to the sequentially input image depth fusion data.

9. An object detection device, characterized in that the object detection device comprises a processor, a memory; the processor is used for executing the target detection program stored in the memory to realize the target detection method of any one of claims 1-7.

10. A computer-readable storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the object detection method of any one of claims 1-7.

Images