CN114882523A - Task target detection method and system based on fragmented video information - Google Patents

Abstract

The invention discloses a task target detection method and system based on fragmented video information. Based on an effective frame sequence extraction module, a depth convolution feature map extraction module, an optical flow information module, a deformation feature module, and a weight coefficient calculation module, a target is constructed. The person detection model completes the detection of the preset target person. The present invention improves the weight distribution of the aggregated frame through the fuzzy prior method, and calculates the weight of each frame of the image instead of assigning the same weight to each frame, which effectively improves the character. Accuracy and reliability of detection.

Description

A task target detection method and system based on fragmented video information

technical field

The invention relates to a task target detection method based on fragmented video information, and also relates to a system for realizing the task target detection method based on fragmented video information.

Background technique

Deep learning networks have made significant progress in object detection, and in recent years, excellent image-based object detection algorithms have been directly transferred to video object detection. Compared with still image object detection, video object detection is more challenging. Because the video detection scene is usually complex, such as the uncooperative of the detected person and the discontinuous video information obtained, the extracted image information is incomplete, such as motion blur, defocus, and rare gestures, which to a large extent Reduced detection accuracy.

Existing feature aggregation-based methods compensate for the misalignment between frames by aggregating features from multiple adjacent frames, and a key question is whether these frames should be treated equally. Now there are two ways to solve this problem, one solution is to treat each frame equally and give them the same weight, the other way is to use a light network to learn the weights during training, these two The solutions all lack special consideration for the effects of ambiguity.

In the present invention, we propose a human object detection method based on fragmented video information. Using fuzzy priors to improve the weight assignment of aggregated frames, in particular, a fuzzy mapping network is introduced to mark each pixel as fuzzy or non-blurred. The detection network focuses on the target and calibrates the saliency map to obtain a calibrated blur map focusing on the blur of the target, and calculates the weight of each frame of image; this method is more computationally intensive than the current most advanced video targets. The detection algorithm has better performance.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a task target detection method and system based on fragmented video information, improve the weight distribution of aggregated frames through the fuzzy prior method, and calculate the weight of each frame of image, instead of assigning the same weight to each frame , which effectively improves the accuracy and reliability of person detection.

In order to realize the above functions, a task target detection method based on fragmented video information, execute steps S1-step S7 according to a preset cycle, obtain a target person detection model, and then apply the target person detection model to complete the detection of the target person;

S1. Real-time acquisition of a video containing the walking of the target person, converting the video containing the walking of the target person into a sequence of video frames arranged in time sequence, and extracting continuous video frames of a preset number of frames at a preset position in the video frame sequence as a valid frame sequence, Taking the video frame sequence as the input and the valid frame sequence as the output, constructs the valid frame sequence extraction module;

S2. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the deep convolutional neural network, and using the depth convolution feature map corresponding to each video frame in the valid frame sequence as the output, construct the depth convolution feature map extraction module;

S3. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs formed by two video frames separated by a preset number of frames from each other in the valid frame sequence, calculate the The optical flow parameters of the group of video frame pairs are used as the motion information of the target person, and the optical flow parameters of each group of video frame pairs in the valid frame sequence are used as the output to construct an optical flow information module;

S4. Take the depth convolution feature maps output by the depth convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, based on the bilinear distortion function, take each The deformation features of the group video frame pairs are output, and the deformation feature module is constructed;

S5. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence. The weight coefficient of each video frame in the valid frame sequence is obtained based on the softmax classification network; the weight coefficient calculation module of each video frame in the valid frame sequence is used as the output to construct a weight coefficient calculation module;

S6. Deformation features of each group of video frame pairs output by the deformation feature module, each depth convolution feature map corresponding to each video frame output by the depth convolution feature map extraction module, and each video frame output by the weight coefficient calculation module. The weight coefficient is used as the input, and the aggregation features corresponding to the video frame group composed of video frame pairs are obtained, and the detection network module is constructed with the preset information of the target person as the output through the detection neural network;

S7. Take the real-time acquisition of the video frame sequence corresponding to the video of the target person walking as input, and take the preset information of the target person as output, based on the effective frame sequence extraction module, the depth convolution feature map extraction module, the optical flow information module, The deformation feature module and the weight coefficient calculation module are used to construct the target person detection model to be trained, and the target person detection model is obtained based on the participating training of the video samples containing the target person walking, and the target person detection is completed.

As a preferred technical solution of the present invention: step S3 takes the valid frame sequence output by the valid frame sequence extraction module as the input, based on the optical flow neural network, for two video frames in the valid frame sequence that are separated by a preset number of frames from each other. Each group of video frame pairs is calculated, the optical flow parameters of each group of video frame pairs are calculated as the motion information of the target person, and the optical flow parameters of each group of video frame pairs in the valid frame sequence are used as the output to construct the specific steps of the optical flow information module as follows:

S31: Define the t-th video frame It in the valid frame sequence as a reference frame, and the _t -τ-th video frame It _-τ and the t+τ-th video frame It ₊ τ in the valid frame sequence as support frames , input the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ into the optical flow neural network;

S32: The optical flow neural network includes a convolutional layer, an expansion layer, the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ pass through the contraction part composed of the convolutional layer of the optical flow neural network to obtain the reference frame It _. , the feature maps corresponding to the support frame It _-τ and the support frame It _+τ respectively;

S33: The feature maps corresponding to the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ respectively pass through the expansion layer of the optical flow neural network to obtain the reference frame It and the support frame whose size is expanded to the original image size _. The feature maps corresponding to It _-τ and the support frame It _+τ respectively;

S34: Perform optical flow prediction based on the feature maps corresponding to the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ obtained in step S33, respectively, using the reference frame It, the support frame It _{- τ} As a video frame pair, and the reference frame It and the support frame It _{+τ are} used as a video frame pair, the optical flow parameters M between the feature maps corresponding to the reference frame It and the support frame It _{-τ are} obtained respectively. _t-τ→t and the optical flow parameter M _{t+τ→t between} the feature maps corresponding to the reference frame It and the support frame It _+τ respectively are as follows:

M _t-τ→t =FlowNet(I _t-τ ,I _t )

M _t+τ→t =FlowNet(I _t+τ ,I _t )

In the formula, M _t-τ→t is the optical flow parameter between the feature maps corresponding to the reference frame It and the support frame It _-τ respectively, and _t -τ→ _t represents the reference frame It and the support frame _It- The correspondence between _τ , M _t+τ→t is the optical flow parameter between the feature maps corresponding to the reference frame It and the support frame It _+τ respectively, _t +τ→ _t is the reference frame It, the support frame I The corresponding relationship of _t+τ , FlowNet represents the optical flow neural network calculation.

As a preferred technical solution of the present invention: in step S4, the optical flow parameters of each group of video frame pairs in the effective frame sequence output by the depth convolution feature map extraction module and each depth convolution feature map output by the optical flow information module are input as input , based on the bilinear warping function, with the deformation features of each group of video frame pairs as the output, the specific method of constructing the deformation feature module is as follows:

f _t-τ→t =W(f _t-τ ,M _t-τ→t )

f _t+τ→t =W(f _t+τ ,M _t+τ→t )

In the formula, f _t-τ→t is the deformation feature between the reference frame It and the support frame It _-τ , and f _{t +τ→t is} the deformation feature between the reference frame It and the support frame It _+τ , W represents the calculation based on the bilinear distortion function, f _t-τ is the feature map corresponding to the support frame I _t-τ output by the depthwise convolution feature map extraction module, and f _t+τ is the output of the depthwise convolution feature map extraction module The feature map corresponding to the supporting frame It _-τ of .

As a preferred technical solution of the present invention: step S5 takes the valid frame sequence output by the valid frame sequence extraction module as input, and obtains the fuzzy features corresponding to each video frame in the valid frame sequence based on the fuzzy mapping network and the saliency detection network, respectively. , salient features, according to fuzzy features, salient features, and based on the softmax classification network, the weight coefficients of each video frame in the valid frame sequence are obtained; the weight coefficients of each video frame in the valid frame sequence are used as the output, and the weight coefficient calculation module is constructed The specific steps are as follows:

S51: input each video frame in the valid frame sequence into the fuzzy mapping network and the saliency detection network respectively, and obtain respectively the fuzzy feature and the saliency feature corresponding to each video frame;

S52: obtain the corresponding fuzzy map M _blur-sali of each video frame corresponding to each video frame by point multiplication step S51 corresponding fuzzy feature, salient feature;

S53: Binarize the corrected blur map M _blur-sali corresponding to each video frame based on a step function with a threshold of 0.5, wherein the step function is as follows:

In the formula, m is the value of the corrected blur map M _blur-sali corresponding to each video frame, and u(m) is the corrected blur map M _blur-sali corresponding to each video frame after binarization processing. value;

S54: for each video frame, add all u(m) obtained in step S53 to obtain the blur parameter Vcb of each video frame, and standardize the blur parameter Vcb of each video frame, the standardization method is as follows :

In the formula, Vcb _i represents the blurriness parameter of video frame i, VcbNorm _i represents the blurriness parameter of the normalized video frame i, and the value of i is {t-τ, t, t+τ};

S55: Input the ambiguity parameter _{VcbNorm i} of each standardized video frame obtained in step S54 into the softmax classification network, and obtain the weights corresponding to the support frame It _-τ , the reference frame It, and the support frame It _+τ respectively Coefficients ω _t-τ , ω _t , ω _t+τ .

As a preferred technical solution of the present invention: step S6 uses the deformation features of each group of video frame pairs output by the deformation feature module, the depth convolution feature maps corresponding to each video frame output by the depth convolution feature map extraction module, and The weight coefficient of each video frame output by the weight coefficient calculation module is used as input, and the aggregation features corresponding to the video frame group composed of video frame pairs are obtained, and through the detection neural network, the preset information of the target person is used as the output to construct a detection network. The specific steps of the module are as follows:

S61: Based on the deformed features f _t-τ→t and f _{t+τ→t output by the shape feature module, the feature map f t corresponding} to the reference frame It output by the depth convolution feature map extraction module, and the weight coefficient calculation module The weight coefficients ω _t-τ , ω _t , ω _t+τ corresponding to the output deformation features are obtained according to the following formula to obtain the video composed of the support frame It _-τ , the reference frame It , and the support frame It _{+ τ} The aggregated feature J of the frame group;

J=f _t-τ→t ω _t-τ +f _t ω _t +f _t+τ→t ω _t+τ

S62: Input the aggregated features into the detection neural network to obtain preset information of the target person.

The present invention also designs a task target detection system based on fragmented video information, including:

one or more processors;

a memory storing instructions operable that, when executed by the one or more processors, cause the one or more processors to perform operations to obtain a target person detection model by the following steps, and then apply the target person detection The model completes the detection of the preset target person:

S1. Real-time acquisition of a video containing the walking of the target person, converting the video containing the walking of the target person into a sequence of video frames arranged in time sequence, and extracting continuous video frames of a preset number of frames at a preset position in the video frame sequence as a valid frame sequence, Taking the video frame sequence as the input and the valid frame sequence as the output, constructs the valid frame sequence extraction module;

S2. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the deep convolutional neural network, and using the depth convolution feature map corresponding to each video frame in the valid frame sequence as the output, construct the depth convolution feature map extraction module;

S3. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs formed by two video frames separated by a preset number of frames from each other in the valid frame sequence, calculate the The optical flow parameters of the group of video frame pairs are used as the motion information of the target person, and the optical flow parameters of each group of video frame pairs in the valid frame sequence are used as the output to construct an optical flow information module;

S4. Take the depth convolution feature maps output by the depth convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, based on the bilinear distortion function, take each The deformation features of the group video frame pairs are output, and the deformation feature module is constructed;

S5. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence. The weight coefficient of each video frame in the valid frame sequence is obtained based on the softmax classification network; the weight coefficient calculation module of each video frame in the valid frame sequence is used as the output to construct a weight coefficient calculation module;

S6. Deformation features of each group of video frame pairs output by the deformation feature module, each depth convolution feature map corresponding to each video frame output by the depth convolution feature map extraction module, and each video frame output by the weight coefficient calculation module. The weight coefficient is used as the input, and the aggregation features corresponding to the video frame group composed of video frame pairs are obtained, and the detection network module is constructed with the preset information of the target person as the output through the detection neural network;

S7. Take the real-time acquisition of the video frame sequence corresponding to the video of the target person walking as input, and take the preset information of the target person as output, based on the effective frame sequence extraction module, the depth convolution feature map extraction module, the optical flow information module, The deformation feature module and the weight coefficient calculation module are used to construct the target person detection model to be trained, and the target person detection model is obtained based on the participating training of the video samples containing the target person walking, and the target person detection is completed.

The present invention also provides a computer-readable medium for storing software, wherein the readable medium includes instructions that can be executed by one or more computers, and the instructions are executed by the one or more computers. When , the operations of the task target detection method based on fragmented video information are performed.

Beneficial effects: Compared with the prior art, the advantages of the present invention include:

1. The present invention introduces the optical flow neural network to calculate the smoothness between any two frames, not only focusing on the features of a certain frame, but focusing on the relationship between the upper and lower frames.

2. The present invention proposes a new video target detection algorithm, which mainly studies the influence of blur on video target detection. Frames with clear object appearance contribute more to the results than frames with blurred object appearance.

3. The present invention, through the method for detecting human objects based on fragmented video information, helps to detect human characters when the video is discontinuous, and improves the detection accuracy of video objects.

Description of drawings

1 is a flowchart of a task target detection method based on fragmented video information provided according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of a network framework for task target detection based on fragmented video information provided according to an embodiment of the present invention.

Detailed ways

The present invention will be further described below in conjunction with the accompanying drawings. The following examples are only used to illustrate the technical solutions of the present invention more clearly, and cannot be used to limit the protection scope of the present invention.

Referring to FIG. 1 and FIG. 2, an embodiment of the present invention provides a task target detection method based on fragmented video information. Steps S1 to S7 are performed according to a preset cycle to obtain a target person detection model, and then the target person detection model is applied, Complete the detection of the preset target person;

S1. Real-time acquisition of a video containing the walking of the target person, converting the video containing the walking of the target person into a sequence of video frames arranged in time sequence, and extracting continuous video frames of a preset number of frames at a preset position in the video frame sequence as a valid frame sequence, Taking the video frame sequence as the input and the valid frame sequence as the output, constructs the valid frame sequence extraction module;

S2. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the deep convolutional neural network, and using the depth convolution feature map corresponding to each video frame in the valid frame sequence as the output, construct the depth convolution feature map extraction module;

In one embodiment, the deep convolutional neural network is Restnet-101.

S3. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs formed by two video frames separated by a preset number of frames from each other in the valid frame sequence, calculate the The optical flow parameters of the group of video frame pairs are used as the motion information of the target person, and the optical flow parameters of each group of video frame pairs in the valid frame sequence are used as the output to construct an optical flow information module;

Since the resolution of the optical flow parameters output by the optical flow neural network does not match the resolution of the depthwise convolutional feature maps, it is necessary to adjust the size of the optical flow parameters to match the feature maps.

In step S3, the valid frame sequence output by the valid frame sequence extraction module is used as input, and based on the optical flow neural network, for each group of video frame pairs formed by two video frames separated by a preset number of frames from each other in the valid frame sequence, calculate the The optical flow parameters of each group of video frame pairs are used as the motion information of the target person, and the optical flow parameters of each group of video frame pairs in the valid frame sequence are used as the output. The specific steps of constructing the optical flow information module are as follows:

S31: Define the t-th video frame It in the valid frame sequence as a reference frame, and the _t -τ-th video frame It _-τ and the t+τ-th video frame It ₊ τ in the valid frame sequence as support frames , input the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ into the optical flow neural network;

S32: The optical flow neural network includes a convolutional layer, an expansion layer, the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ pass through the contraction part composed of the convolutional layer of the optical flow neural network to obtain the reference frame It _. , the feature maps corresponding to the support frame It _-τ and the support frame It _+τ respectively;

S33: Since the size of each feature map is reduced in step S32, an expansion layer is required to expand the size of each feature map to the original size. The feature maps corresponding to the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ respectively pass through the expansion layer of the optical flow neural network to obtain the reference frame It and the support frame It whose size _is expanded to the original image size _{. -τ} and the feature maps corresponding to the support frame It _+τ respectively;

S34: The so-called optical flow parameter is to use the change of the pixels in each video frame in the valid frame sequence in the time domain and the correlation between the two video frames to find the corresponding relationship between the two video frames. Thereby, the motion information of the target person is calculated.

Optical flow prediction is performed based on the feature maps corresponding to the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ obtained in step S33, respectively, and the reference frame It and the support frame It _{-τ are} taken as one The video frame pair, the reference frame It and the support frame It _{+τ are} taken as a video frame pair to obtain the optical flow parameter M _t- between the feature maps corresponding to the reference frame It and the support frame It _{- τ} respectively _τ→t and the optical flow parameter M _{t+τ→t between} the feature maps corresponding to the reference frame It and the support frame It _+τ respectively are as follows:

M _t-τ→t =FlowNet(I _t-τ ,I _t )

M _t+τ→t =FlowNet(I _t+τ ,I _t )

In the formula, M _t-τ→t is the optical flow parameter between the feature maps corresponding to the reference frame It and the support frame It _-τ respectively, and _t -τ→ _t represents the reference frame It and the support frame _It- The correspondence between _τ , M _t+τ→t is the optical flow parameter between the feature maps corresponding to the reference frame It and the support frame It _+τ respectively, _t +τ→ _t is the reference frame It, the support frame I The corresponding relationship of _t+τ , FlowNet represents the optical flow neural network calculation.

S4. Referring to Figure 2, in the figure WARP represents the bilinear distortion function, aggregation represents the deformation feature, and each group in the effective frame sequence output by the depth convolution feature map extraction module and the effective frame sequence output by the optical flow information module is used. The optical flow parameters of the video frame pair are input, and the deformation feature module is constructed based on the bilinear distortion function with the deformation features of each group of video frame pairs as the output;

Step S4 takes each depthwise convolutional feature map output by the depthwise convolutional feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence outputted by the optical flow information module as input, based on the bilinear distortion function, with each The deformation features of the video frame pair are output, and the specific method of constructing the deformation feature module is as follows:

f _t-τ→t =W(f _t-τ ,M _t-τ→t )

f _t+τ→t =W(f _t+τ ,M _t+τ→t )

In the formula, f _t-τ→t is the deformation feature between the reference frame It and the support frame It _-τ , and f _{t +τ→t is} the deformation feature between the reference frame It and the support frame It _+τ , W represents the calculation based on the bilinear distortion function, f _t-τ is the feature map corresponding to the support frame I _t-τ output by the depthwise convolution feature map extraction module, and f _t+τ is the output of the depthwise convolution feature map extraction module The feature map corresponding to the supporting frame It _-τ of .

S5. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence. The weight coefficient of each video frame in the valid frame sequence is obtained based on the softmax classification network; the weight coefficient calculation module of each video frame in the valid frame sequence is used as the output to construct a weight coefficient calculation module;

The fuzzy mapping network is DBM, the saliency detection network is CSNet, the fuzzy mapping network is used to obtain the blur degree of the video frame, and the saliency detection network is used to eliminate the background interference in the image.

Step S5 takes the valid frame sequence output by the valid frame sequence extraction module as the input, and based on the fuzzy mapping network and the saliency detection network, respectively obtains the fuzzy features and salient features corresponding to each video frame. Based on the softmax classification network, the weight coefficient of each video frame is obtained; with the weight coefficient of each video frame in the valid frame sequence as the output, the specific steps of constructing the weight coefficient calculation module are as follows:

S51: input each video frame in the valid frame sequence into the fuzzy mapping network and the saliency detection network respectively, and obtain respectively the fuzzy feature and the saliency feature corresponding to each video frame;

S52: obtain the corresponding fuzzy map M _blur-sali of each video frame corresponding to each video frame by point multiplication step S51 corresponding fuzzy feature, salient feature;

S53: Binarize the corrected blur map M _blur-sali corresponding to each video frame based on a step function with a threshold of 0.5, wherein the step function is as follows:

In the formula, m is the value of the corrected blur map M _blur-sali corresponding to each video frame, and u(m) is the corrected blur map M _blur-sali corresponding to each video frame after binarization processing. value;

S54: for each video frame, add all u(m) obtained in step S53 to obtain the blur parameter Vcb of each video frame, and standardize the blur parameter Vcb of each video frame, the standardization method is as follows :

In the formula, Vcb _i represents the blurriness parameter of video frame i, VcbNorm _i represents the blurriness parameter of the normalized video frame i, and the value of i is {t-τ, t, t+τ};

S55: Input the ambiguity parameter _{VcbNorm i} of each standardized video frame obtained in step S54 into the softmax classification network, and obtain the weights corresponding to the support frame It _-τ , the reference frame It, and the support frame It _+τ respectively Coefficients ω _t-τ , ω _t , ω _t+τ .

S6. Deformation features of each group of video frame pairs output by the deformation feature module, each depth convolution feature map corresponding to each video frame output by the depth convolution feature map extraction module, and each video frame output by the weight coefficient calculation module. The weight coefficient is used as the input, and the aggregation features corresponding to the video frame group composed of video frame pairs are obtained, and the detection network module is constructed with the preset information of the target person as the output through the detection neural network;

In one embodiment, the detection neural network is Faster R-CNN.

Step S6 uses the deformation features of each group of video frame pairs output by the deformation feature module, each depth convolution feature map corresponding to each video frame output by the depth convolution feature map extraction module, and each video frame output by the weight coefficient calculation module. The weight coefficient is used as the input, and the aggregation features corresponding to the video frame group composed of video frame pairs are obtained, and the preset information of the target person is used as the output through the detection neural network. The specific steps of constructing the detection network module are as follows:

S61: Based on the deformed features f _t-τ→t and f _{t+τ→t output by the shape feature module, the feature map f t corresponding} to the reference frame It output by the depth convolution feature map extraction module, and the weight coefficient calculation module The weight coefficients ω _t-τ , ω _t , ω _t+τ corresponding to the output deformation features are obtained according to the following formula to obtain the video composed of the support frame It _-τ , the reference frame It , and the support frame It _{+ τ} The aggregated feature J of the frame group;

J=f _t-τ→t ω _t-τ +f _t ω _t +f _t+τ→t ω _t+τ

S62: Input the aggregated features into the detection neural network to obtain preset information of the target person.

S7. Take the real-time acquisition of the video frame sequence corresponding to the video of the target person walking as input, and take the preset information of the target person as output, based on the effective frame sequence extraction module, the depth convolution feature map extraction module, the optical flow information module, The deformation feature module and the weight coefficient calculation module are used to construct the target person detection model to be trained, and the target person detection model is obtained based on the participating training of the video samples containing the target person walking, and the target person detection is completed.

A task target detection system based on fragmented video information provided by an embodiment of the present invention includes:

one or more processors;

a memory storing instructions operable that, when executed by the one or more processors, cause the one or more processors to perform operations to obtain a target person detection model by the following steps, and then apply the target person detection The model completes the detection of the preset target person:

S1. Real-time acquisition of a video containing the walking of the target person, converting the video containing the walking of the target person into a sequence of video frames arranged in time sequence, and extracting continuous video frames of a preset number of frames at a preset position in the video frame sequence as a valid frame sequence, Taking the video frame sequence as the input and the valid frame sequence as the output, constructs the valid frame sequence extraction module;

S2. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the deep convolutional neural network, and using the depth convolution feature map corresponding to each video frame in the valid frame sequence as the output, construct the depth convolution feature map extraction module;

S3. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs formed by two video frames separated by a preset number of frames from each other in the valid frame sequence, calculate the The optical flow parameters of the group of video frame pairs are used as the motion information of the target person, and the optical flow parameters of each group of video frame pairs in the valid frame sequence are used as the output to construct an optical flow information module;

S4. Take the depth convolution feature maps output by the depth convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, based on the bilinear distortion function, take each The deformation features of the group video frame pairs are output, and the deformation feature module is constructed;

S5. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence. The weight coefficient of each video frame in the valid frame sequence is obtained based on the softmax classification network; the weight coefficient calculation module of each video frame in the valid frame sequence is used as the output to construct a weight coefficient calculation module;

S6. Deformation features of each group of video frame pairs output by the deformation feature module, each depth convolution feature map corresponding to each video frame output by the depth convolution feature map extraction module, and each video frame output by the weight coefficient calculation module. The weight coefficient is used as the input, and the aggregation features corresponding to the video frame group composed of video frame pairs are obtained, and the detection network module is constructed with the preset information of the target person as the output through the detection neural network;

S7. Take the real-time acquisition of the video frame sequence corresponding to the video of the target person walking as input, and take the preset information of the target person as output, based on the effective frame sequence extraction module, the depth convolution feature map extraction module, the optical flow information module, The deformation feature module and the weight coefficient calculation module are used to construct the target person detection model to be trained, and the target person detection model is obtained based on the participating training of the video samples containing the target person walking, and the target person detection is completed.

An embodiment of the present invention provides a computer-readable medium for storing software, where the readable medium includes instructions that can be executed by one or more computers, and when the instructions are executed by the one or more computers, The operations of the method for task target detection based on fragmented video information are performed.

The embodiments of the present invention have been described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned embodiments, and can also be made within the scope of knowledge possessed by those of ordinary skill in the art without departing from the purpose of the present invention. Various changes.

Claims (7)

1. a task target detection method based on fragmented video information, is characterized in that, execute step S1-step S7 by preset cycle, obtain target person detection model, then apply target person detection model, complete the detection to target person; S1. Real-time acquisition of a video containing the walking of the target person, converting the video containing the walking of the target person into a sequence of video frames arranged in time sequence, and extracting continuous video frames of a preset number of frames at a preset position in the video frame sequence as a valid frame sequence, Taking the video frame sequence as the input and the valid frame sequence as the output, constructs the valid frame sequence extraction module; S2. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the deep convolutional neural network, and using the depth convolution feature map corresponding to each video frame in the valid frame sequence as the output, construct the depth convolution feature map extraction module; S3. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs formed by two video frames separated by a preset number of frames from each other in the valid frame sequence, calculate the The optical flow parameters of the group of video frame pairs are used as the motion information of the target person, and the optical flow parameters of each group of video frame pairs in the valid frame sequence are used as the output to construct an optical flow information module; S4. Take the depth convolution feature maps output by the depth convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, based on the bilinear distortion function, take each The deformation features of the group video frame pairs are output, and the deformation feature module is constructed; S5. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the fuzzy mapping network and the saliency detection network, respectively obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence. The weight coefficient of each video frame in the valid frame sequence is obtained based on the softmax classification network; the weight coefficient calculation module of each video frame in the valid frame sequence is used as the output to construct a weight coefficient calculation module; S6. Deformation features of each group of video frame pairs output by the deformation feature module, each depth convolution feature map corresponding to each video frame output by the depth convolution feature map extraction module, and each video frame output by the weight coefficient calculation module. The weight coefficient is used as the input, and the aggregation features corresponding to the video frame group composed of video frame pairs are obtained, and the detection network module is constructed with the preset information of the target person as the output through the detection neural network; S7. Take the real-time acquisition of the video frame sequence corresponding to the video of the target person walking as input, and take the preset information of the target person as output, based on the effective frame sequence extraction module, the depth convolution feature map extraction module, the optical flow information module, The deformation feature module and the weight coefficient calculation module are used to construct the target person detection model to be trained, and the target person detection model is obtained based on the participating training of the video samples containing the target person walking, and the target person detection is completed. 2. a kind of task target detection method based on fragmented video information according to claim 1, is characterized in that, step S3 takes the effective frame sequence outputted by effective frame sequence extraction module as input, based on optical flow neural network, for by For each group of video frame pairs formed by two video frames separated by a preset number of frames in the valid frame sequence, the optical flow parameters of each group of video frame pairs are calculated as the motion information of the target person, and each group of video frames in the valid frame sequence is used. The correct optical flow parameter is the output, and the specific steps for constructing the optical flow information module are as follows: S31: Define the t-th video frame It in the valid frame sequence as a reference frame, and the _t -τ-th video frame It _-τ and the t+τ-th video frame It ₊ τ in the valid frame sequence as support frames , input the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ into the optical flow neural network; S32: The optical flow neural network includes a convolutional layer, an expansion layer, the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ pass through the contraction part composed of the convolutional layer of the optical flow neural network to obtain the reference frame It _. , the feature maps corresponding to the support frame It _-τ and the support frame It _+τ respectively; S33: The feature maps corresponding to the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ respectively pass through the expansion layer of the optical flow neural network to obtain the reference frame It and the support frame whose size is expanded to the original image size _. The feature maps corresponding to It _-τ and the support frame It _+τ respectively; S34: Perform optical flow prediction based on the feature maps corresponding to the reference frame It, the support frame It _{-τ ,} and the support frame It _+τ obtained in step S33, respectively, using the reference frame It, the support frame It _{- τ} As a video frame pair, and the reference frame It and the support frame It _{+τ are used} as a video frame _pair , the optical flow parameters M _{t- τ→t} and the optical flow parameter M _t+ τ _{→t between} the feature maps corresponding to the reference frame It and the support frame It _+τ respectively are as follows: M _t-τ→t =FlowNet(I _t-τ , I _t ) M _t+τ→t =FlowNet(I _t+τ , I _t ) In the formula, M _t-τ→t is the optical flow parameter between the feature maps corresponding to the reference frame It and the support frame It _-τ respectively, and _t -τ→ _t represents the reference frame It and the support frame _It- The correspondence between _τ , M _t+τ→t is the optical flow parameter between the feature maps corresponding to the reference frame It and the support frame It _+τ respectively, _t +τ→ _t is the reference frame It, the support frame I The corresponding relationship of _t+τ , FlowNet represents the optical flow neural network calculation. 3. a kind of task target detection method based on fragmented video information according to claim 2, is characterized in that, step S4 outputs each depth convolution feature map, optical flow information module output with depth convolution feature map extraction module output The optical flow parameters of each group of video frame pairs in the valid frame sequence of , are input, and based on the bilinear distortion function, the deformation features of each group of video frame pairs are used as the output, and the specific method of constructing the deformation feature module is as follows: f _t-τ→t =W(f _t-τ , M _t-τ→t ) f _t+τ→t =W(f _t+τ , M _t+τ→t ) In the formula, f _t-τ→t is the deformation feature between the reference frame It and the support frame It _-τ , and f _{t +τ→t is} the deformation feature between the reference frame It and the support frame It _+τ , W represents the calculation based on the bilinear distortion function, f _t-τ is the feature map corresponding to the support frame I _t-τ output by the depthwise convolution feature map extraction module, and f _t+τ is the output of the depthwise convolution feature map extraction module The feature map corresponding to the supporting frame It _-τ of . 4. a kind of task target detection method based on fragmented video information according to claim 3, is characterized in that, step S5 takes the effective frame sequence outputted by effective frame sequence extraction module as input, based on fuzzy mapping network, saliency detection network, respectively obtain the fuzzy features and salient features corresponding to each video frame in the valid frame sequence, and obtain the weight coefficients of each video frame in the valid frame sequence according to the fuzzy features and salient features, and based on the softmax classification network; The weight coefficient of each video frame in the sequence is the output, and the specific steps for constructing the weight coefficient calculation module are as follows: S51: Input each video frame in the valid frame sequence into a fuzzy mapping network and a saliency detection network, respectively, to obtain fuzzy features and saliency features corresponding to each video frame respectively; S52: obtain the corrected blur map M _blur-sali corresponding to each video frame by the fuzzy feature and the salient feature corresponding to each video frame obtained by the point multiplication step S51; S53: Binarize the corrected blur map M _blur-sali corresponding to each video frame based on a step function with a threshold of 0.5, where the step function is as follows:

In the formula, m is the value of the corrected blur map M _blur-sali corresponding to each video frame, and u(m) is the corrected blur map M _blur-sali corresponding to each video frame after binarization processing. value; S54: For each video frame, add all u(m) obtained in step S53 to obtain the blur degree parameter Vcb of each video frame, and perform normalization processing on the blur degree parameter Vcb of each video frame. The standardization processing method is as follows: :

In the formula, Vcb _i represents the ambiguity parameter of video frame i, VcbNorm _i represents the ambiguity parameter of the normalized video frame i, and the value of i is {t-τ, t, t+τ}; S55: Input the ambiguity parameter _{VcbNorm i} of each normalized video frame obtained in step S54 into the softmax classification network, and obtain the weights corresponding to the support frame It _-τ , the reference frame It, and the support frame It _+τ respectively Coefficients ω _t-τ , ω _t , ω _t+τ . 5. a kind of task target detection method based on fragmented video information according to claim 4, is characterized in that, step S6 with the deformation feature of each group of video frame pairs output by deformation feature module, depth convolution feature map extraction module Each depth convolution feature map corresponding to each output video frame and the weight coefficient of each video frame output by the weight coefficient calculation module are input, and the aggregated features corresponding to the video frame group composed of video frame pairs are obtained, and by detecting The neural network takes the preset information of the target person as the output, and the specific steps of constructing the detection network module are as follows: S61: Based on the deformed features f _t-τ→t and f _{t+τ→t output by the shape feature module, the feature map f t corresponding} to the reference frame It output by the depth convolution feature map extraction module, and the weight coefficient calculation module The weight coefficients ω _t-τ , ω _t , ω _t+τ corresponding to the output deformation features are obtained according to the following formula to obtain the video composed of the support frame It _-τ , the reference frame It , and the support frame It _{+ τ} The aggregated feature J of the frame group; J=f _t-τ→t ω _t-τ +f _t ω _t +f _t+τ→t ω _t+τ S62: Input the aggregated features into the detection neural network to obtain preset information of the target person. 6. A task target detection system based on fragmented video information is characterized in that, comprising: one or more processors; a memory storing instructions operable that, when executed by the one or more processors, cause the one or more processors to perform operations to obtain a target person detection model by the following steps, and then apply the target person detection The model completes the detection of the preset target person: S1. Real-time acquisition of a video containing the walking of the target person, converting the video containing the walking of the target person into a sequence of video frames arranged in time sequence, and extracting continuous video frames of a preset number of frames at a preset position in the video frame sequence as a valid frame sequence, Taking the video frame sequence as the input and the valid frame sequence as the output, constructs the valid frame sequence extraction module; S2. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the deep convolutional neural network, and using the depth convolution feature map corresponding to each video frame in the valid frame sequence as the output, construct the depth convolution feature map extraction module; S3. Taking the valid frame sequence output by the valid frame sequence extraction module as input, based on the optical flow neural network, for each group of video frame pairs formed by two video frames separated by a preset number of frames from each other in the valid frame sequence, calculate the The optical flow parameters of the group of video frame pairs are used as the motion information of the target person, and the optical flow parameters of each group of video frame pairs in the valid frame sequence are used as the output to construct an optical flow information module; S4. Take the depth convolution feature maps output by the depth convolution feature map extraction module and the optical flow parameters of each group of video frame pairs in the valid frame sequence output by the optical flow information module as input, based on the bilinear distortion function, take each The deformation features of the group video frame pairs are output, and the deformation feature module is constructed; S5. Taking the valid frame sequence output by the valid frame sequence extraction module as the input, based on the fuzzy mapping network and the saliency detection network, obtain the fuzzy features and saliency features corresponding to each video frame in the valid frame sequence, respectively. The weight coefficient of each video frame in the valid frame sequence is obtained based on the softmax classification network; the weight coefficient of each video frame in the valid frame sequence is used as the output, and the weight coefficient calculation module is constructed; S6. Deformation features of each group of video frame pairs output by the deformation feature module, each depth convolution feature map corresponding to each video frame output by the depth convolution feature map extraction module, and each video frame output by the weight coefficient calculation module. The weight coefficient is used as the input, and the aggregation features corresponding to the video frame group composed of video frame pairs are obtained, and the detection network module is constructed with the preset information of the target person as the output through the detection neural network; S7. Take the real-time acquisition of the video frame sequence corresponding to the video of the target person walking as input, and take the preset information of the target person as output, based on the effective frame sequence extraction module, the depth convolution feature map extraction module, the optical flow information module, The deformation feature module and the weight coefficient calculation module are used to construct the target person detection model to be trained, and the target person detection model is obtained based on the participation training of the video samples including the target person walking, and the target person detection is completed. 7. A computer-readable medium storing software, wherein the readable medium comprises instructions executable by one or more computers, the instructions, when executed by the one or more computers, The operation of the task target detection method based on fragmented video information according to any one of claims 1 to 5 is performed.

Images