TWI742690B - Method and apparatus for detecting a human body, computer device, and storage medium - Google Patents

Abstract

This disclosure provides a method and apparatus for detecting a human body, computer device and storage medium. According to the method, a to-be-detected image is obtained; location information of bone key points representing a skeleton structure of the human body, and location information of contour key points representing a human contour are determined based on the to-be-detected image; a detecting result of the human body is generated based on the location information of the bone key points and the contour key points.

Description

Human body detection method, device, computer equipment and storage medium

The present disclosure relates to the field of image processing technology, and in particular, to a human body detection method, device, computer equipment, and storage medium.

With the application of neural networks in image, video, voice, text and other fields, users have higher and higher requirements for the accuracy of various models based on neural networks. Human detection in images is an important application scenario of neural networks, which requires high precision in human detection and the amount of calculation data.

The purpose of the embodiments of the present disclosure is to provide a human body detection method, device, computer equipment, and storage medium.

In a first aspect, embodiments of the present disclosure provide a human body detection method, including: acquiring an image to be detected; determining the position information of key bone points used to characterize the bone structure of the human body based on the image to be detected; The position information of the contour key points of the human body contour; and the human body detection result is generated based on the position information of the bone key points and the position information of the contour key points.

The embodiment of the present disclosure can determine the position information of the key points of the skeleton used to characterize the human skeletal structure and the position information of the key points of the outline used to characterize the outline of the human body from the image to be detected, and based on the position information of the key points of the bone, As well as the position information of key points of the contour, the human body detection result is generated, which improves the precision of the representation while taking into account the amount of calculation data.

In addition, in the embodiments of the present disclosure, since the position information of the key points of the bones that characterize the bone structure of the human body and the position information of the key points of the outline that characterize the outline of the human body are used to obtain the human body detection results, the information that characterizes the human body is richer and has a broader scope. Application scenarios, such as image editing, body shape adjustment, etc.

In an optional embodiment, the contour key points include a main contour key point and an auxiliary contour key point; wherein there is at least one auxiliary contour key point between two adjacent main contour key points.

In this embodiment, the contour of the human body is represented by the position information of the main contour key points and the position information of the auxiliary contour key points, so that the identification of the human contour is more accurate and the amount of information is richer.

In an optional implementation manner, based on the image to be detected, determining the position information of key contour points used to characterize the contour of the human body includes: determining the position information of the key contour points of the main contour based on the image to be detected; Determine the human body contour information based on the position information of the main contour key points; and determine the position information of a plurality of the auxiliary contour key points based on the determined human contour information.

In this embodiment, the position information of the key points of the main contour and the position information of the key points of the auxiliary contour can be located more accurately.

In an optional implementation manner, the human body detection result includes one or more of the following: an image to be detected added with bone key point markers and contour key point markers; including position information of the bone key points and the contour The data group of the location information of the key point.

In this embodiment, the to-be-detected image including bone key point markers and contour key point markers can give people a more intuitive visual impression; a data group including the position information of the bone key points and the position information of the contour key points Easier to follow-up processing.

In an optional implementation manner, the method further includes: performing one or more of the following operations based on the human body detection result: human body motion recognition, human body posture detection, human body contour adjustment, human body image editing, and human body mapping.

In this embodiment, based on the human body detection result with higher characterization and less calculation data amount, more operations can be realized with higher accuracy and faster speed.

In an optional implementation manner, the determining, based on the image to be detected, the position information of the key points of the skeleton used to characterize the human skeletal structure and the position information of the key points of the outline used to characterize the outline of the human body includes: In the image to be detected, feature extraction is performed to obtain bone features and contour features, and the obtained bone features and contour features are feature-fused; based on the feature fusion result, the position information of the bone key points and the contour key are determined Point’s location information.

In this embodiment, feature extraction can be performed on the image to be detected to obtain bone features and contour features, and the obtained bone features and contour features can be feature-fused to obtain position information of bone key points used to characterize the bone structure of the human body. And the position information of the key points of the outline that can characterize the outline of the human body. The human body detection result obtained based on this method can not only represent the human body with a smaller amount of data, but also extract the bone features and contour features of the human body to represent the human body, taking into account the improvement of the fineness of the representation.

In an optional implementation manner, the feature extraction based on the image to be detected to obtain bone features and contour features, and feature fusion of the obtained bone features and contour features includes: based on the image to be detected , Perform feature extraction at least once, and perform feature fusion on the bone features and contour features obtained from each feature extraction. In the case of multiple feature extractions, perform the i+th feature fusion result based on the feature fusion result of the i-th feature fusion. 1 feature extraction, i is a positive integer; the determination based on the result of feature fusion to determine the position information of the key points of the skeleton used to characterize the human skeletal structure and the position information of the key points of the outline used to characterize the contour of the human body includes: based on the last The feature fusion result of a feature fusion determines the position information of the bone key points and the position information of the contour key points.

In this embodiment, at least one feature extraction is performed on the image to be detected, and the bone features and contour features obtained from each feature extraction are feature fused, so that the bone feature points and the contour feature points that have a positional relationship can be corrected mutually. , The final position information of the bone key points and the position information of the contour key points have higher accuracy.

In an optional implementation manner, the performing at least one feature extraction based on the image to be detected includes: in the first feature extraction, using a pre-trained first feature extraction network to extract from the image to be detected The first target skeletal feature matrix used to characterize the skeleton key points of the human skeleton feature; and the first target contour feature matrix of the contour key points used to characterize the contour feature of the human body is extracted; in the i+1th feature extraction, the pre- The trained second feature extraction network extracts the first target bone feature matrix of the bone key points used to characterize the human bone features from the feature fusion result of the i-th feature fusion; and extracts the contour key used to characterize the contour features of the human body The first target contour feature matrix of points; wherein the network parameters of the first feature extraction network and the second feature extraction network are different, and the network parameters of the second feature extraction network used for different times of feature extraction are different.

In this embodiment, the bone feature and the contour feature are extracted and merged at least once, and the position information of the bone key points and the position information of the contour key points are finally obtained with higher accuracy.

In an optional implementation manner, performing feature fusion of the extracted bone features and contour features includes: using a pre-trained feature fusion neural network to perform the feature fusion of the first target bone feature matrix and the first target contour feature matrix Feature fusion is performed to obtain a second target skeleton feature matrix and a second target contour feature matrix; wherein, the second target skeleton feature matrix is a three-dimensional bone feature matrix, and the three-dimensional bone feature matrix includes two corresponding bone key points. A three-dimensional skeleton feature matrix; the value of each element in the two-dimensional skeleton feature matrix represents the probability that the pixel corresponding to the element belongs to the corresponding bone key point; the second target contour feature matrix is a three-dimensional contour feature matrix, the The three-dimensional contour feature matrix includes a two-dimensional contour feature matrix corresponding to each contour key point; the value of each element in the two-dimensional contour feature matrix represents the probability that the pixel corresponding to the element belongs to the corresponding contour key point; The network parameters of the feature fusion neural network used in the secondary feature fusion are different.

In this embodiment, the bone features and contour features are fused based on the pre-trained feature fusion network, which can obtain better feature fusion results, so that the final position information of the bone key points and the position information of the contour key points have Higher accuracy.

In an optional embodiment, the determining the position information of the bone key points and the position information of the contour key points based on the feature fusion result of the last feature fusion includes: the second feature fusion obtained based on the last feature fusion The target skeleton feature matrix determines the position information of the skeleton key points; and based on the second target contour feature matrix obtained from the last feature fusion, the position information of the contour key points is determined.

In this embodiment, after at least one feature extraction and feature fusion, the position information of the bone key points and the position information of the contour key points finally obtained have higher accuracy.

In an alternative embodiment, the first feature extraction network includes: a common feature extraction network, a first bone feature extraction network, and a first contour feature extraction network; the first feature extraction network is used to extract the image from the image to be inspected. Extracting the first target skeleton feature matrix of the skeleton key points used to characterize the human skeleton features from the image; and extracting the first target contour feature matrix of the contour key points used to characterize the contour features of the human body includes: using the shared feature extraction network Road performs convolution processing on the image to be detected to obtain a basic feature matrix containing bone features and contour features; using the first bone feature extraction network to perform convolution processing on the basic feature matrix to obtain a first skeleton Feature matrix, and obtain a second skeleton feature matrix from the first target convolutional layer in the first skeleton feature extraction network; and obtain the first skeleton feature matrix based on the first skeleton feature matrix and the second skeleton feature matrix Target skeleton feature matrix; the first target convolutional layer is any convolutional layer except the last convolutional layer in the first bone feature extraction network; using the first contour feature extraction network, The basic feature matrix is subjected to convolution processing to obtain a first contour feature matrix, and a second contour feature matrix is obtained from the second target convolution layer in the first contour feature extraction network; based on the first contour feature matrix and The second contour feature matrix is used to obtain the first target contour feature matrix; the second target convolutional layer is any convolutional layer except the last convolutional layer in the first contour feature extraction network.

In this embodiment, the common feature extraction network is used to extract bone features and contour features, and other features other than bone features and contour features in the image to be detected are removed, and then the first bone feature extraction network is used to target the bone features. It uses the first contour feature extraction network to extract the contour features in a targeted manner, which requires less calculation.

In an optional implementation manner, the obtaining the first target bone feature matrix based on the first bone feature matrix and the second bone feature matrix includes: combining the first bone feature matrix and the second bone feature matrix The two bone feature matrices are spliced to obtain a first spliced bone feature matrix; the first spliced bone feature matrix is subjected to dimensional transformation processing to obtain the first target bone feature matrix; the first contour feature matrix is based on And the second contour feature matrix to obtain the first target contour feature matrix, including: concatenating the first contour feature matrix and the second contour feature matrix to obtain the first concatenated contour feature matrix; The first spliced contour feature matrix is subjected to dimensional transformation processing to obtain the first target contour feature matrix; wherein the dimension of the first target skeleton feature matrix is the same as the dimension of the first target contour feature matrix, and The first target skeleton feature matrix and the first target contour feature matrix have the same dimension in the same dimension.

In this embodiment, the first skeletal feature matrix and the second skeletal feature matrix are spliced, so that the first target skeletal feature matrix has richer skeletal feature information; at the same time, the first contour feature matrix and the second skeleton feature matrix are combined. The two contour feature matrices are spliced, so that the contour feature matrix of the first target has richer bone feature information. In the subsequent feature fusion process, the position information of the bone key points and the contour key points can be extracted with higher precision Location information.

In an alternative embodiment, the feature fusion neural network includes: a first convolutional neural network, a second convolutional neural network, a first transformation neural network, and a second transformation neural network; the use The feature fusion neural network performs feature fusion on the first target skeleton feature matrix and the first target contour feature matrix to obtain a second target skeleton feature matrix and a second target contour feature matrix, including: using the first The convolutional neural network performs convolution processing on the first target skeleton feature matrix to obtain a first intermediate skeleton feature matrix; and uses the second convolutional neural network to convolve the first target contour feature matrix Processing to obtain a first intermediate contour feature matrix; splicing the first intermediate contour feature matrix and the first target bone feature matrix to obtain a first splicing feature matrix; and using the first transformation neural network to Performing dimensional transformation on the first splicing feature matrix to obtain the second target bone feature matrix; performing splicing processing on the first intermediate bone feature matrix and the first target contour feature matrix to obtain a second splicing feature matrix, The second transformation neural network is used to perform dimensional transformation on the second splicing feature matrix to obtain the second target contour feature matrix.

In this embodiment, the first intermediate contour feature matrix and the first target bone feature matrix are spliced, and the second target bone feature matrix is obtained based on the splicing processing result. Fusion to achieve correction using the bone features obtained by contour feature extraction. In addition, by performing stitching processing on the first intermediate skeleton feature matrix and the first target contour feature matrix, and obtaining a second target contour feature matrix based on the result of the stitching processing, the skeletal features and contour features are merged, In order to achieve the use of bone features to correct the extracted contour features. Furthermore, the position information of the bone key points and the position information of the contour key points can be extracted with higher accuracy.

In an optional implementation manner, the feature fusion neural network includes: a first directional convolutional neural network, a second directional convolutional neural network, a third convolutional neural network, and a fourth convolutional neural network , The third transformation neural network, and the fourth transformation neural network; said using the feature fusion neural network to perform feature fusion on the first target skeleton feature matrix and the first target contour feature matrix to obtain the second The target skeleton feature matrix and the second target contour feature matrix include: performing directional convolution processing on the first target skeleton feature matrix using the first directional convolutional neural network to obtain the first directional skeleton feature matrix; And use a third convolutional neural network to perform convolution processing on the first directional bone feature matrix to obtain a second intermediate skeletal feature matrix; and use the second directional convolutional neural network to perform convolution processing on the first target Performing directional convolution processing on the contour feature matrix to obtain a first directional contour feature matrix; and using a fourth convolution neural network to perform convolution processing on the first directional contour feature matrix to obtain a second intermediate contour feature matrix; The second intermediate contour feature matrix and the first target bone feature matrix are spliced to obtain a third spliced feature matrix; and a third transformation neural network is used to perform dimensional transformation on the third spliced feature matrix to obtain The second target skeleton feature matrix; the second intermediate skeleton feature matrix and the first target contour feature matrix are spliced to obtain a fourth spliced feature matrix, and a fourth transform neural network is used to perform the splicing process on the first Four splicing feature matrices are subjected to dimensional transformation to obtain the second target contour feature matrix.

In this embodiment, the feature fusion processing is performed by means of directional convolution, and the position information of the bone key points and the position information of the contour key points can be extracted with higher accuracy.

In an optional implementation manner, the feature fusion neural network includes: a displacement estimation neural network, a fifth transformation neural network; the use of the feature fusion neural network to compare the feature matrix of the first target bone, and the Performing feature fusion on the first target contour feature matrix to obtain the second target skeleton feature matrix and the second target contour feature matrix includes: splicing the first target skeleton feature matrix and the first target contour feature matrix to obtain Fifth splicing feature matrix; input the fifth splicing feature matrix into the displacement estimation neural network, and perform displacement estimation on multiple sets of predetermined key point pairs to obtain a key point movement in each group of key point pairs Displacement information to another key point; each key point in each group of key point pairs is used as the current key point, and the three-dimensional feature matrix corresponding to the other key point paired with the current key point is obtained from the three-dimensional feature matrix corresponding to the other key point. The two-dimensional feature matrix corresponding to the other key point of the pair; according to the displacement information from the other key point of the pair to the current key point, the two-dimensional feature matrix corresponding to the other key point of the pair is The element undergoes position transformation to obtain the displacement feature matrix corresponding to the current key point; for each bone key point, the two-dimensional feature matrix corresponding to the bone key point is spliced with its corresponding displacement feature matrix to obtain the bone The spliced two-dimensional feature matrix of the key points; and input the spliced two-dimensional feature matrix of the bone key points to the fifth transformation neural network to obtain the target two-dimensional feature matrix corresponding to the bone key points; based on each bone key The target two-dimensional feature matrix corresponding to each point is generated to generate the second target bone feature matrix; for each contour key point, the two-dimensional feature matrix corresponding to the contour key point is spliced with its corresponding displacement feature matrix, Obtain the spliced two-dimensional feature matrix of the contour key point; input the spliced two-dimensional feature matrix of the contour key point to the fifth transformation neural network to obtain the target two-dimensional feature matrix corresponding to the contour key point; based on The target two-dimensional feature matrix corresponding to each contour key point respectively generates the second target contour feature matrix.

In this embodiment, the feature fusion is realized by performing displacement transformation on the bone key points and the contour key points, and the position information of the bone key points and the position information of the contour key points can be extracted with higher accuracy.

In an optional embodiment, the human body detection method is implemented by a human body detection model; the human body detection model includes: the first feature extraction network and/or the feature fusion neural network: the human body detection model is It is obtained by training using sample images in the training sample set, and the sample images are marked with actual position information of bone key points of the human skeletal structure and actual position information of the contour key points of the outline of the human body.

In this embodiment, the human body detection model obtained through the training method has higher detection accuracy, and the human body detection model can obtain the human body detection result that takes into account the fineness of the representation and the amount of calculation data.

In a second aspect, the embodiments of the present disclosure also provide a human body detection device, including: an acquisition module for acquiring an image to be detected; a detection module for determining a skeleton structure of the human body based on the image to be detected The position information of the bone key points and the position information of the contour key points used to represent the outline of the human body; a generating module for generating the human body based on the position information of the bone key points and the position information of the contour key points Test results.

In a third aspect, embodiments of the present disclosure also provide a computer device, including a processor, a non-transitory storage medium, and a bus, the non-transitory storage medium storing machine-readable instructions executable by the processor, when When the computer device is running, the processor and the storage medium communicate through a bus, and the machine-readable instruction executes the first aspect or any of the first aspects when the machine-readable instruction is executed by the processor. Steps in one possible implementation.

In a fourth aspect, the embodiments of the present disclosure also provide a computer-readable storage medium having a computer program stored on the computer-readable storage medium, and the computer program executes the first aspect or the first aspect when the computer program is run by a processor. The steps in any possible implementation of the aspect.

The embodiment of the present disclosure can determine the position information of the key points of the skeleton used to characterize the human skeletal structure and the position information of the key points of the outline used to characterize the outline of the human body from the image to be detected, and based on the position information of the key points of the bone, As well as the position information of key points of the contour, the human body detection result is generated, which improves the precision of the representation while taking into account the amount of calculation data.

In order to make the above objectives, features and advantages of the present disclosure more comprehensible, preferred embodiments accompanied with accompanying drawings are described in detail as follows.

In order to make the objectives, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be described clearly and completely in conjunction with the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only These are a part of the embodiments of the present disclosure, but not all of the embodiments. The components of the embodiments of the present disclosure generally described and illustrated in the accompanying drawings may be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in conjunction with the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but merely represents the embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work shall fall within the protection scope of the present disclosure.

Research has found that in human body detection, there are usually the following two methods: bone key point detection method and semantic segmentation method.

Skeleton key point detection method; in this method, the human bone key points are extracted from the image through the neural network model, and the corresponding human body detection results are obtained based on the bone key points; in this human body detection method, it uses The simple human body representation method has less data volume, so when other follow-up processing is performed based on the human body detection results obtained by this method, the amount of calculation required is less; more of it is used for the human body Gesture, action recognition and other fields; such as behavior detection, human-computer interaction based on human posture, etc. However, because this method cannot extract the contour information of the human body, the resulting human body detection results have low representation precision.

Semantic segmentation method; in this method, the probability that each pixel in the image belongs to the human body is recognized through the semantic segmentation model, and the human body detection result is obtained based on the probability that each pixel in the image belongs to the human body; in this kind of human body detection In this method, although the contour information of the human body can be obtained completely, the amount of calculation data contained in the human body recognition result is relatively large.

Therefore, a human body detection method that can take into account the fineness of representation and the amount of calculated data has become a problem that needs to be solved urgently.

Based on the above research, the present disclosure provides a human body detection method, device, computer equipment and storage medium, which can perform feature extraction on the image to be detected to extract the bone features and contour features of the human body, and extract the bone features and contour features obtained. The feature fusion is performed to obtain the position information of the key points of the bones used to characterize the skeleton structure of the human body, and the position information of the key points of the outline used to characterize the outline of the human body. The human body detection result obtained based on this method has less data volume, and reflects the skeletal characteristics and contour characteristics of the human body, and improves the fineness of representation.

In addition, in the embodiments of the present disclosure, since the position information of the key points of the bones that characterize the skeleton structure of the human body and the position information of the key points of the outline that characterize the outline of the human body are used to obtain the human body detection results, the information that characterizes the human body is richer and has a broader scope. Application scenarios.

The defects of the existing human detection methods need to be determined after repeated practice and careful study. Therefore, the discovery process of the existing problems and the solutions proposed in this disclosure should all fall within the scope of this disclosure.

The following describes in detail a human body detection method according to an embodiment of the present disclosure. The human body detection method can be applied to any device with data processing capability, such as a computer.

Refer to Fig. 1, which is a flowchart of a human body detection method provided by an embodiment of the present disclosure, in which:

Step S101: Obtain an image to be detected.

Step S102: Based on the image to be detected, determine the position information of the bone key points used to characterize the human skeletal structure and the position information of the contour key points used to characterize the outline of the human body.

Step S103: Generate a human body detection result based on the position information of the bone key points and the position information of the contour key points.

The above steps S101 to S103 are respectively described below.

I: In the above step S101, the image to be inspected may be, for example, the image to be inspected taken by a camera installed at the target location, the image to be inspected sent by other computer equipment, and it is pre-saved from the local database. The image to be detected, etc. The image to be detected may or may not include a human body image; if the image to be detected includes a human body image, the final human body detection result can be obtained based on the human body detection method provided by the embodiment of the present disclosure; If the image to be detected does not include a human body image, the human body detection result obtained is, for example, empty.

II: In the above step S102, as shown in FIG. 2a, the bone key points can be used to characterize the bone features of the human body, and the bone features include the characteristics of the joint parts of the human body. The joints are, for example, elbow joints, wrist joints, shoulder joints, neck joints, hip joints, knee joints, ankle joints, and the like. Exemplarily, bone key points can also be set on the human head.

Contour key points can be used to characterize the contour features of the human body, which can include: main contour key points, as shown in Figure 2a, or include: main contour key points and auxiliary contour key points, as shown in Figure 2b~Figure 2d; , Figures 2b~2d are partial diagrams of the parts within the line frame in Figure 2a.

Among them, the main contour key points are the contour key points that characterize the contours of the human body joints, as shown in Figure 2a, such as the contours of the elbow joint, the contour of the wrist joint, the contour of the shoulder joint, the contour of the neck joint, the contour of the hip joint, and the knee joint. The contours of the joints, the contours of the ankle joints, etc., generally appear corresponding to the key points of the bones that characterize the corresponding joints.

Auxiliary contour key points are contour key points that characterize the contour between the joints of the human body. There is at least one auxiliary contour key point between two adjacent main contour key points; in the example shown in Figure 2b, there are two main contour key points There is one auxiliary contour key point between the two; in the example shown in Figure 2c, there are two auxiliary contour key points between the two main contour key points; Figure 2d shows the example, one of the two main contour key points There are three key points in the auxiliary contour between.

The bone key points and outline key points involved in the above drawings and text descriptions are only examples to facilitate the understanding of the present disclosure. In actual applications, the number and positions of bone key points and contour key points can be appropriately adjusted according to the actual scene, which is not limited in the present disclosure.

For contour key points including: main contour key points and auxiliary contour key points, the following methods can be used to determine the position information of the contour key points used to characterize the contour of the human body based on the image to be detected:

Based on the image to be detected, determine the position information of the main contour key points; determine the human contour information based on the position information of the main contour key points; determine the position information of multiple auxiliary contour key points based on the determined human contour information.

For the case where the contour key points include the main contour key points, the position information of the main contour key points can be determined directly based on the image to be detected.

The embodiments of the present disclosure provide a specific method for determining the position information of the key points of bones used to characterize the human skeletal structure and the position information of the key points of the outline used to characterize the outline of the human body based on the image to be detected:

Based on the image to be detected, perform feature extraction to obtain bone features and contour features, and perform feature fusion on the obtained bone features and contour features; based on the feature fusion results, determine the position information of the bone key points and the position information of the contour key points .

Based on the image to be detected, the bone feature and contour feature extraction can be performed but not limited to any of the following A or B.

A: Perform a feature extraction on the image to be detected, and perform feature fusion on the bone features and contour features obtained by the feature extraction.

B: Perform multiple feature extraction on the image to be detected, and after each feature extraction, perform feature fusion on the bone features and contour features obtained from this feature extraction, and determine based on the feature fusion result of the last feature fusion The position information of the bone key points and the position information of the contour key points.

The following will firstly describe the situation A in detail.

In case A, based on the feature fusion result of this feature fusion, the position information of the bone key points used to characterize the human skeletal structure and the position information of the contour key points used to characterize the contour of the human body are determined.

The following describes the feature extraction process and feature fusion process in a1 and a2 respectively.

a1: Feature extraction process:

The pre-trained first feature extraction network can be used to extract the first target bone feature matrix of the bone key points used to characterize the human bone features from the image to be detected; A target contour feature matrix.

Specifically, referring to FIG. 3, an embodiment of the present disclosure provides a schematic structural diagram of a first feature extraction network. The first feature extraction network includes: a common feature extraction network, a first bone feature extraction network, and a first contour feature extraction network.

Referring to FIG. 4, an embodiment of the present disclosure also provides a specific process for extracting a first target skeleton feature matrix and a first target contour feature matrix from the image to be detected based on the first feature extraction network provided in FIG. 3, including the following step.

Step S401: Use the common feature extraction network to perform convolution processing on the image to be detected to obtain a basic feature matrix including bone features and contour features.

In specific implementation, the image to be detected can be represented as an image matrix; if the image to be detected is a single-color channel image, such as a grayscale image, it can be represented as a two-dimensional image matrix; Each element in the image matrix corresponds to the pixel of the image to be detected one-to-one; the value of each element in the two-dimensional image matrix is the pixel value of the pixel corresponding to each element. If the image to be detected is a multi-color channel image, such as an image in RGB format, it can be represented as a three-dimensional image matrix; the three-dimensional image matrix includes three different colors (for example, R, G , B) Two-dimensional image matrix with one-to-one channel correspondence; the value of each element in any two-dimensional image matrix is the pixel value corresponding to each element and the pixel value under the corresponding color channel.

The common feature extraction network includes at least one convolutional layer; after the image matrix of the image to be detected is input to the common feature extraction network, the common feature extraction network is used to perform convolution processing on the image matrix of the image to be detected. Extract the features in the image to be detected. In this case, the extracted features include both bone features and contour features.

Step S402: Use the first bone feature extraction network to perform convolution processing on the basic feature matrix to obtain a first bone feature matrix, and obtain a second bone feature matrix from the first target convolution layer in the first bone feature extraction network; Based on the first skeletal feature matrix and the second skeletal feature matrix, the first target skeletal feature matrix is obtained; the first target convolutional layer is any convolutional layer except the last convolutional layer in the first skeletal feature extraction network.

In specific implementation, the first bone feature extraction network includes multiple convolutional layers. The multiple convolutional layers are connected in sequence, and the input of the next convolutional layer is the output of the previous convolutional layer. The first skeleton feature extraction network with this structure can perform multiple convolution processing on the basic feature matrix, and obtain the first skeleton feature matrix from the last convolution layer. Here, the first skeleton feature matrix is a three-dimensional feature matrix; the three-dimensional feature matrix includes a plurality of two-dimensional feature matrices, and each two-dimensional feature matrix corresponds to a plurality of predetermined bone key points in a one-to-one correspondence. The value of an element in the two-dimensional feature matrix corresponding to a certain bone key point represents the probability that the pixel corresponding to the element belongs to the bone key point, and there are generally multiple pixels corresponding to one element.

In addition, through the multiple convolution processing of the basic feature matrix by the multi-layer convolutional layer, although the bone features of the human body can be extracted from the basic feature matrix, as the number of convolutions increases, some information in the image to be detected will be lost , These information may also include information related to the bone features of the human body; if the information in the image to be detected is lost too much, it may result in the first target bone feature matrix used to characterize the bones of the human body. Not precise enough. Therefore, in the embodiment of the present disclosure, the second skeletal feature matrix is also obtained from the first target convolutional layer of the first skeletal feature extraction network, and the first target is obtained based on the first skeletal feature matrix and the second skeletal feature matrix. Bone feature matrix.

Here, the first target convolutional layer is any convolutional layer except the last convolutional layer in the first bone feature extraction network. In the example of FIG. 3, the penultimate convolutional layer in the first bone feature extraction network is selected as the first target convolutional layer.

For example, the following method can be used to obtain the first target bone feature matrix based on the first bone feature matrix and the second bone feature matrix:

The first bone feature matrix and the second bone feature matrix are spliced to obtain the first spliced bone feature matrix; the first spliced bone feature matrix is subjected to dimensional transformation processing to obtain the first target bone feature matrix.

Here, in the case of performing dimensional transformation processing on the first spliced bone feature matrix, it can be input to the dimensional transformation neural network, and the dimensional transformation neural network is used to perform convolution processing on the first spliced bone feature matrix at least once, Obtain the first target bone feature matrix.

Here, the dimensional transformation neural network can merge the feature information carried in the first bone feature matrix and the second bone feature matrix, so that the obtained first target bone feature matrix contains richer information.

Step S403: Use the first contour feature extraction network to perform convolution processing on the basic feature matrix to obtain a first contour feature matrix, and obtain a second contour feature matrix from the second target convolution layer in the first contour feature extraction network ; Based on the first contour feature matrix and the second contour feature matrix, the first target contour feature matrix is obtained; the second target convolutional layer is any convolutional layer except the last convolutional layer in the first contour feature extraction network. In the example of FIG. 3, the penultimate convolutional layer in the first contour feature extraction network is selected as the second target convolutional layer.

In specific implementation, the first contour feature extraction network also includes multiple convolutional layers. The multiple convolutional layers are connected in sequence, and the input of the next convolutional layer is the output of the previous convolutional layer. The first contour feature extraction network with this structure can perform multiple convolution processing on the basic feature matrix, and obtain the first contour feature matrix from the last convolution layer. Here, the first contour feature matrix is a three-dimensional feature matrix; the three-dimensional feature matrix includes a plurality of two-dimensional feature matrices, and each two-dimensional feature matrix corresponds to a plurality of predetermined contour key points in a one-to-one correspondence. The value of an element in the two-dimensional feature matrix corresponding to a certain contour key point represents the probability that the pixel point corresponding to the element belongs to the contour key point, and there are generally multiple pixels corresponding to one element.

It should be noted here that the number of contour key points and the number of bone key points are generally different. Therefore, the number of two-dimensional feature matrices included in the obtained first contour feature matrix is different from that included in the first bone feature matrix. The number of two-dimensional feature matrices can be different.

For example, if the number of bone key points is 14 and the number of contour key points is 25, the number of two-dimensional feature matrices included in the first contour feature matrix is 25, and the two-dimensional feature matrix included in the first bone feature matrix is 25. The number of feature matrices is 14.

In addition, in order to make the first target contour feature matrix also contain richer information, the second contour feature can be obtained from the second target convolutional layer in the first contour feature extraction network in a manner similar to the above step S402. Matrix, and then based on the first contour feature matrix and the second contour feature matrix, the first target contour feature matrix is obtained.

Here, based on the first contour feature matrix and the second contour feature matrix, the manner of obtaining the first target contour feature matrix includes, for example:

The first contour feature matrix and the second contour feature matrix are spliced to obtain the first spliced contour feature matrix; the first spliced contour feature matrix is subjected to dimensional transformation processing to obtain the first target contour feature matrix.

It should be noted that in the above steps S402 and S403, the dimension of the first target skeleton feature matrix is the same as the dimension of the first target contour feature matrix, and the first target skeleton feature matrix and the first target contour feature matrix are in the same dimension. The dimensions of are the same, so that subsequent feature fusion processing is performed based on the first target skeleton feature matrix and the first target contour feature matrix.

For example, if the dimension of the first target skeleton feature matrix is 3, and the dimensions of each dimension are 64, 32, and 14, respectively, then the dimension of the first target skeleton feature matrix is expressed as 64*32*14; the first target The dimension of the contour feature matrix can also be expressed as 64*32*14.

In addition, in another embodiment, the first target skeleton feature matrix and the first target contour feature matrix can also be obtained in the following manner:

Use the common feature extraction network to perform convolution processing on the image to be detected to obtain a basic feature matrix containing bone features and contour features;

Use the first bone feature extraction network to perform convolution processing on the basic feature matrix to obtain the first bone feature matrix, and perform dimensional transformation processing on the first bone feature matrix to obtain the first target bone feature matrix;

Use the first contour feature extraction network to perform convolution processing on the basic feature matrix to obtain a first contour feature matrix, and perform dimensional transformation processing on the first contour feature matrix to obtain a first target contour feature matrix.

In this way, the bone features and contour features of the human body can also be extracted from the image to be detected with higher accuracy.

In addition, the first feature extraction network provided in the embodiments of the present disclosure is obtained by pre-training.

Here, the human body detection method provided by the embodiments of the present disclosure is implemented by a human body detection model; the human body detection model includes: a first feature extraction network and/or a feature fusion neural network;

The human body detection model is obtained by training using sample images in the training sample set. The sample images are marked with actual position information of bone key points of the human skeletal structure and actual position information of the contour key points of the outline of the human body.

Specifically, for the case where the human body detection model includes the first feature extraction network, the first feature extraction network can be trained separately, or it can be jointly trained with the feature fusion neural network, or it can be combined with separate training and joint training. .

The process of training to obtain the first feature extraction network includes but is not limited to the following (1) and (2).

(1) The individual training of the first feature extraction network includes, for example:

Step 1.1: Obtain multiple sample images and the annotation data of each sample image; the annotation data includes: the actual position information of the bone key points used to characterize the human skeletal structure, and the outline key points used to characterize the outline of the human body Actual location information;

Step 1.2: Input multiple sample images into the first basic feature extraction network to obtain the first sample target bone feature matrix and the first sample target contour feature matrix;

Step 1.3: Determine the first predicted position information of the skeleton key points based on the first sample target bone feature matrix; and determine the first predicted position information of the contour key points based on the first sample target contour feature matrix;

Step 1.4: Determine the first loss based on the actual position information of the bone key points and the first predicted position information of the bone key points; and determine the first loss based on the actual position information of the contour key points and the first predicted position information of the contour key points Second loss

Step 1.5: Perform this round of training on the first basic feature extraction network based on the first loss and the second loss;

After multiple rounds of training on the first basic feature extraction network, the first feature extraction network is obtained.

As shown in Figure 3, the first loss is LS1 in Figure 3; the second loss is LC1 in Figure 3. Based on the first loss and the second loss, supervise the training of the first basic feature extraction network to obtain a higher-precision first feature extraction network.

(2) Joint training of the first feature extraction network and the feature fusion neural network includes, for example:

Step 2.1: Obtain multiple sample images and the annotation data of each sample image; the annotation data includes: the actual position information of the bone key points used to characterize the human skeletal structure, and the outline key points used to characterize the outline of the human body Actual location information;

Step 2.2: Input multiple sample images into the first basic feature extraction network to obtain the first sample target bone feature matrix and the first sample target contour feature matrix;

Step 2.3: Use the basic feature fusion neural network to perform feature fusion on the first sample target skeleton feature matrix and the first sample target contour feature matrix to obtain the second sample target skeleton feature matrix and the second sample target contour feature matrix.

Step 2.4: Determine the second predicted position information of the skeleton key points based on the second sample target bone feature matrix; and determine the second predicted position information of the contour key points based on the second sample target contour feature matrix;

Step 2.5: Determine the third loss based on the actual position information of the bone key points and the second predicted position information of the bone key points; and determine the third loss based on the actual position information of the contour key points and the second predicted position information of the contour key points Fourth loss

Step 2.6: Perform this round of training on the first basic feature extraction network and the basic feature fusion neural network based on the third loss and the fourth loss;

After multiple rounds of training on the first basic convolutional neural network and the basic feature fusion neural network, the first feature extraction network and the feature fusion neural network are obtained.

(3) The process of combining individual training and joint training to obtain the first feature extraction network can be synchronized training using the processes in (1) and (2) above.

Alternatively, the process in (1) can be used to pre-train the first feature extraction network; the first feature extraction network obtained after the pre-training can be used with the feature fusion neural network to perform the above (2) Joint training.

It should be noted that for the separate training and joint training of the first feature extraction network, the sample images used can be the same or different.

Before the joint training of the first feature extraction network and the feature fusion neural network, the feature fusion neural network can also be pre-trained, and then the pre-trained feature fusion neural network can be used with the first feature extraction The network conducts joint training.

For the detailed process of individual training of the feature fusion neural network, please refer to the description of the embodiment shown in a2 below.

a2: Feature fusion process:

After obtaining the first target bone feature matrix of the bone key points used to characterize the human bone features, and the first target contour feature matrix of the contour key points used to characterize the human body contour features, it can be based on the first target bone feature moments and The first target contour feature matrix performs feature fusion processing.

Specifically, in the process of extracting bone features and contour features based on the image to be detected, although the basic feature matrix used is the same, the first bone feature extraction network extracts bone features from the basic feature matrix, and the first A contour feature extraction network extracts contour features from the basic feature matrix. The two processes exist independently of each other. But for the same human body, there is a correlation between contour features and bone features; the purpose of fusing contour features and bone features is to use the mutual influence relationship between bone features and contour features. For example, the position information of the final extracted key points of the skeleton can be corrected based on the contour features, and the position information of the final extracted key points of the contour can be corrected based on the bone features, so as to obtain a more accurate position of the bone key points The information and the position information of the key points of the contour can be used to obtain higher-precision human body detection results.

The embodiments of the present disclosure provide a specific method for feature fusion of extracted bone features and contour features, including: using a pre-trained feature fusion neural network to feature a first target bone feature matrix and a first target contour feature matrix Fusion, obtain the second target skeleton feature matrix and the second target contour feature matrix.

Wherein, the second target bone feature matrix is a three-dimensional bone feature matrix, and the three-dimensional bone feature matrix includes a two-dimensional bone feature matrix corresponding to each key point of the bone; The corresponding pixel point belongs to the probability of the corresponding bone key point (ie, the bone key point corresponding to the two-dimensional bone feature matrix); the second target contour feature matrix is the three-dimensional contour feature matrix, and the three-dimensional contour feature matrix includes the key points of each contour Corresponding two-dimensional contour feature matrix respectively; the value of each element in the two-dimensional contour feature matrix represents the probability that the pixel corresponding to the element belongs to the corresponding contour key point.

The feature fusion neural network provided in the embodiments of the present disclosure can be trained separately, can also be jointly trained with the first feature extraction network, or can be combined with separate training and joint training.

The process of jointly training the feature fusion neural network and the first feature extraction network can be referred to (2) above, and will not be repeated here.

For feature fusion neural networks of different structures, the training methods used will be different when they are trained separately. For the training methods of feature fusion neural networks of different structures, please refer to the following M1~M3 .

The process of feature fusion of bone features and contour features may include but is not limited to at least one of the following M1 to M3.

M1:

Referring to FIG. 5, an embodiment of the present disclosure provides a specific structure of a feature fusion neural network, including: a first convolutional neural network, a second convolutional neural network, a first transformation neural network, and a second transformation Neural network.

As shown in FIG. 6, an embodiment of the present disclosure also provides a feature fusion neural network based on the feature fusion neural network provided in FIG. 5 to perform feature fusion on the first target skeleton feature matrix and the first target contour feature matrix to obtain the second target skeleton feature matrix And the specific method of the second target contour feature matrix includes the following steps.

Step S601: Use the first convolutional neural network to perform convolution processing on the first target skeleton feature matrix to obtain the first intermediate skeleton feature matrix. Step S603 is executed.

Here, the first convolutional neural network includes at least one convolutional layer. If the first convolutional neural network has multiple layers, the multiple convolutional layers are connected in sequence; the input of the convolutional layer of this layer is the output of the previous convolutional layer. The first target skeleton feature matrix is input to the first convolutional neural network, and each convolution layer is used to perform convolution processing on the first target skeleton feature matrix to obtain the first intermediate skeleton feature matrix.

This process is to be able to further extract the bone features from the first target bone feature matrix.

Step S602: Use the second convolutional neural network to perform convolution processing on the first target contour feature matrix to obtain a first intermediate contour feature matrix. Step S604 is executed.

Here, the processing procedure is similar to the above step S601, and will not be repeated here.

It should be noted that there is no order of execution of step S601 and step S602. It can be executed synchronously or asynchronously.

Step S603: Perform splicing processing on the first intermediate contour feature matrix and the first target bone feature matrix to obtain the first spliced feature matrix; and use the first transformation neural network to perform dimensional transformation on the first spliced feature matrix to obtain the second target Bone feature matrix.

Here, the first intermediate contour feature matrix and the first target bone feature matrix are spliced to obtain the first spliced feature matrix, so that the obtained first spliced feature matrix includes both the contour feature and the bone feature.

Use the first transformation neural network to perform further dimensional transformation on the first splicing matrix. In fact, the first transformation neural network is used to extract the bone features from the first splicing feature matrix again; because the first splicing feature matrix is obtained In the process, other features except bone features and contour features in the image to be detected are removed, and only bone features and contour features are included. Therefore, the bones contained in the second target bone feature matrix obtained based on the first splicing feature matrix Features will be affected by contour features, can establish the relationship between bone features and contour features, and can realize the fusion of bone features and contour features.

Step S604: Perform splicing processing on the first intermediate skeleton feature matrix and the first target contour feature matrix to obtain a second spliced feature matrix, and use the second transform neural network to perform dimensional transformation on the second spliced feature matrix to obtain the second target Contour feature matrix.

Here, the process of splicing the first intermediate skeleton feature matrix and the first target contour feature matrix to obtain the second splicing feature matrix is similar to the process of obtaining the first splicing feature matrix in step S602, and will not be repeated here.

Similarly, the contour features contained in the contour feature matrix of the second target will be affected by the bone features, and the correlation between the bone features and the contour features is established, and the fusion of the bone features and the contour features is realized.

In another embodiment, the feature fusion neural network can be individually trained in the following manner.

Step 3.1: Obtain the first sample target bone feature matrix and the first sample target contour feature matrix of multiple sample images.

The method of obtaining is similar to the method of obtaining the first target skeleton feature matrix and the first target contour feature matrix in the foregoing embodiment, and will not be repeated here. It can be obtained in the case of joint training with the first feature extraction network, or can be obtained using a pre-trained first feature extraction network.

Step 3.2: Use the first basic convolutional neural network to perform convolution processing on the target bone feature matrix of the first sample to obtain the middle bone feature matrix of the first sample.

Step 3.3: Use the second basic convolutional neural network to perform convolution processing on the target contour feature matrix of the first sample to obtain the middle contour feature matrix of the first sample.

Step 3.4: Perform splicing processing on the middle contour feature matrix of the first sample and the target bone feature matrix of the first sample to obtain the spliced feature matrix of the first sample; and use the first basic transformation neural network to splice the features of the first sample The matrix undergoes dimensional transformation to obtain the target bone feature matrix of the second sample.

Step 3.5: Perform splicing processing on the middle skeleton feature matrix of the first sample and the target contour feature matrix of the first sample to obtain the second sample splicing feature matrix, and use the second basic transformation neural network to perform the splicing feature matrix of the second sample The dimension transformation is used to obtain the target contour feature matrix of the second sample.

Step 3.6: Determine the third predicted position information of the skeleton key point based on the second sample target skeleton feature matrix; and determine the third predicted position information of the contour key point based on the second sample target contour feature matrix.

Step 3.7: Determine the fifth loss based on the actual position information of the bone key points and the third predicted position information of the bone key points; and determine the fifth loss based on the actual position information of the contour key points and the third predicted position information of the contour key points Sixth loss.

Step 3.8: Based on the fifth loss and the sixth loss, perform the calculation of the first basic convolutional neural network, the second basic convolutional neural network, the first basic transform neural network, and the second basic transform neural network Round training

After multiple rounds of training on the first basic convolutional neural network, the second basic convolutional neural network, the first basic transformation neural network, and the second basic transformation neural network, a feature fusion neural network is obtained.

Here, the fifth loss is LS2 in FIG. 5; the sixth loss is LC2 in FIG. 5.

M2:

Referring to FIG. 7, the specific structure of another feature fusion neural network provided by the embodiment of the present disclosure includes: a first directional convolutional neural network, a second directional convolutional neural network, and a third convolutional neural network Way, the fourth convolutional neural network, the third transform neural network, and the fourth transform neural network.

Referring to FIG. 8, an embodiment of the present disclosure also provides a feature fusion neural network based on the feature fusion neural network provided in FIG. 7 to perform feature fusion on a first target skeleton feature matrix and a first target contour feature matrix to obtain a second target skeleton feature matrix And the specific method of the second target contour feature matrix includes the following steps.

Step S801: Use the first directional convolutional neural network to perform directional convolution processing on the first target bone feature matrix to obtain the first directional bone feature matrix. Use the third convolutional neural network to perform convolution processing on the first directional skeleton feature matrix to obtain the second intermediate skeleton feature matrix. Step S804 is executed.

Step S802: Use the second directional convolutional neural network to perform directional convolution processing on the first target contour feature matrix to obtain the first directional contour feature matrix; and use the fourth convolutional neural network to analyze the first directional contour feature The matrix is subjected to convolution processing to obtain the second intermediate contour feature matrix. Step S803 is executed.

Step S803: Perform splicing processing on the second intermediate contour feature matrix and the first target bone feature matrix to obtain a third spliced feature matrix; and use the third transformation neural network to perform dimensional transformation on the third spliced feature matrix to obtain the second target Bone feature matrix.

Step S804: Perform splicing processing on the second intermediate skeleton feature matrix and the first target contour feature matrix to obtain a fourth spliced feature matrix, and use the fourth transform neural network to perform dimensional transformation on the fourth spliced feature matrix to obtain the second target Contour feature matrix.

In a specific implementation, in the process of feature fusion of bone features and contour features, the bone key points are usually concentrated on the skeleton of the human body, while the contour key points are concentrated on the outline of the human body, that is, distributed around the skeleton. Therefore, it is necessary to perform local spatial transformations for bone features and contour features respectively. For example, transform the bone feature to the position of the contour feature in the contour feature matrix, and transform the contour feature to the position of the bone feature in the bone feature matrix, so as to better extract the bone features and contour features, and realize the bone features and contours. The fusion of features.

In order to achieve this objective, the embodiments of the present disclosure first use the first directional convolutional neural network to perform directional convolution processing on the first target bone feature matrix; the directional convolution can effectively realize the directional spatial transformation of bone features at the feature level . Then, a third convolutional neural network is used to perform convolution processing on the obtained first directional skeleton feature matrix to obtain a second intermediate skeleton feature matrix. In this case, since the skeletal feature has been oriented spatially transformed through the first directional convolution layer, the skeletal feature actually moves in the direction of the contour feature. Then, the second middle skeleton feature matrix and the first target contour feature matrix are spliced to obtain a fourth spliced feature matrix. The fourth splicing feature matrix includes not only contour features, but also bone features that have undergone directional spatial transformation. Then, the fourth transformation neural network is used to perform dimensional transformation on the fourth splicing feature matrix, that is, from the fourth splicing feature matrix, the contour feature is extracted again. The second target contour feature matrix obtained in this way will be affected by the bone features, and the fusion between the bone features and the contour features is realized.

In the same way, the embodiment of the present disclosure first uses the second directional convolutional neural network to perform directional convolution processing on the first target contour feature matrix, and the directional convolution can effectively realize the directional spatial transformation of the contour feature at the feature level. Then, a fourth convolutional neural network is used to perform convolution processing on the obtained first directional contour feature matrix to obtain a second intermediate contour feature matrix. In this case, since the contour feature has been oriented spatially transformed through the second directional convolution layer, the contour feature has actually moved in the direction of the bone feature. Then, the second intermediate contour feature matrix and the first target bone feature matrix are spliced to obtain a third spliced feature matrix. The third splicing feature matrix includes not only the bone features, but also the contour features that have undergone directional spatial transformation. Then, the third transformation neural network is used to perform dimensional transformation on the third splicing feature matrix, that is, the bone features are extracted again from the third splicing feature matrix. The second target bone feature matrix obtained in this way will be affected by the contour feature, and the fusion between the bone feature and the contour feature is realized.

Specifically, directional convolution consists of multiple iterative convolution steps, and effective directional convolution meets the following requirements:

(1) In each iteration of the convolution step, only update the element value of a group of elements in the feature matrix;

(2) After the last iteration of the convolution step, the element values of all elements should be updated only once.

，用於控制元素的更新順序。其中，函數F_k 的輸入是第一目標骨骼特徵矩陣中各元素的位置，而函數F_k 的輸出表示是否更新第k 次迭代中的元素。該輸出可以是1或0；1代表更新，0代表不更新。具體而言，在第k 次迭代過程中，只更新F_k = 1的區域中元素的元素值，而保持其他區域中元素的元素值不變。第i次迭代的更新可以表示為：Taking the directional convolution of the first target bone feature matrix as an example, in order to realize the directional convolution process, a feature function sequence can be defined

, Used to control the update sequence of elements. Among them, the input of the function F _k is the position of each element in the first target skeleton feature matrix, and the output of the function F _k indicates whether to update the element in the kth iteration. The output can be 1 or 0; 1 means update, 0 means no update. Specifically, in the k- th iteration process, only the element values of the elements in the region of F _k = 1 are updated, while the element values of the elements in other regions are kept unchanged. The update of the i-th iteration can be expressed as:

Among them, T ₀ (X) = X , X represents the input of directional convolution, that is, the first target bone feature matrix; W and b respectively represent the shared weight and bias during multiple iterations.

，分別為散射卷積算子

，和聚集卷積算子

。其中，散射卷積算子負責由內向外依次更新特徵矩陣中的元素；而聚集卷積算子由外向內依次更新特徵矩陣中的元素。In order to achieve the fusion of bone features and contour features, a pair of symmetrical directional convolution operators can be set, that is, the sequence of the above-mentioned feature functions

, Respectively, the scattering convolution operator

, And the aggregation convolution operator

. Among them, the scattering convolution operator is responsible for sequentially updating the elements in the feature matrix from the inside to the outside; while the gathering convolution operator sequentially updates the elements in the feature matrix from the outside to the inside.

；在使用第二定向卷積神經網路對第一目標輪廓特徵矩陣進行定向卷積處理的情況下，由於要將輪廓特徵元素定向空間變換至輪廓特徵矩陣中間的位置（與骨骼特徵更相關的位置），因此使用聚集卷積算子

。In the case of using the first directional convolutional neural network to perform directional convolution processing on the first target bone feature matrix, because the oriented space of the bone feature element is transformed to the position around the element (a position more related to the contour feature) ), so use the scattering convolution operator

; In the case of using the second directional convolutional neural network to perform directional convolution processing on the contour feature matrix of the first target, because the oriented space of the contour feature element is transformed to the position in the middle of the contour feature matrix (more relevant to the bone feature Position), so the aggregate convolution operator is used

.

Specifically, the first directional convolutional neural network performs directional convolution processing on the first target bone feature matrix as follows.

Divide the first target bone feature matrix into multiple sub-matrices, and each sub-matrix is called a grid; among them, if the first target bone feature matrix is a three-dimensional matrix, the dimensions of the three dimensions are: m, n, s , The dimension of the first target skeleton feature matrix is expressed as m*n*s; if the size of the grid is 5, that is, the dimension of each grid can be expressed as 5*5*s.

進行多次迭代卷積，得到目標子矩陣。如圖9a所示，提供了一種使用散射卷積算子

對網格大小為5的子矩陣中元素的元素值進行兩次迭代更新的過程。其中，圖9a中a表示原始子矩陣；b表示進行了一次迭代得到的子矩陣，c表示進行兩次迭代得到的子矩陣，也即目標子矩陣。Then for each grid, use the scattering convolution operator

Perform multiple iterative convolutions to obtain the target sub-matrix. As shown in Figure 9a, it provides a way to use the scattering convolution operator

The element value of the element in the sub-matrix with a grid size of 5 is updated twice iteratively. Among them, a in FIG. 9a represents the original sub-matrix; b represents the sub-matrix obtained by performing one iteration, and c represents the sub-matrix obtained by performing two iterations, that is, the target sub-matrix.

The target sub-matrices corresponding to each grid are spliced together to obtain the first directional bone feature matrix.

Similarly, the second directional convolutional neural network performs directional convolution processing on the first target contour feature matrix as follows.

Divide the first target contour feature matrix into multiple sub-matrices, and each sub-matrix is called a grid; among them, if the first target contour feature matrix is a three-dimensional matrix, the dimensions of the three dimensions are: m, n, s , The dimension of the first target contour feature matrix is expressed as m*n*s; if the size of the grid is 5, that is, the dimension of each grid can be expressed as 5*5*s.

進行多次迭代卷積，得到目標子矩陣。Then for each grid, use the aggregation convolution operator

Perform multiple iterative convolutions to obtain the target sub-matrix.

對網格大小為5的子矩陣中元素的元素值進行兩次迭代更新的過程。其中，圖9b中a表示原始子矩陣；b表示進行了一次迭代得到的子矩陣，c表示進行兩次迭代得到的子矩陣，也即目標子矩陣。As shown in Figure 9b, it provides a way to use the aggregated convolution operator

The element value of the element in the sub-matrix with a grid size of 5 is updated twice iteratively. Among them, a in FIG. 9b represents the original sub-matrix; b represents the sub-matrix obtained by performing one iteration, and c represents the sub-matrix obtained by performing two iterations, that is, the target sub-matrix.

The target sub-matrices corresponding to each grid are spliced together to obtain the first directional contour feature matrix.

It should be noted here that the iterative convolution process of each sub-matrix can be processed in parallel.

和聚集卷積算子

對子矩陣中元素的元素值進行迭代更新的示例。The examples in Figure 9a and Figure 9b only use the scattering convolution operator

And aggregate convolution operator

An example of iteratively updating the element values of the elements in the sub-matrix.

In another embodiment, the feature fusion neural network can be individually trained in the following manner.

Step 4.1: Obtain the first sample target bone feature matrix and the first sample target contour feature matrix of the multiple sample images.

The method of obtaining is similar to the method of obtaining the first target skeleton feature matrix and the first target contour feature matrix in the foregoing embodiment, and will not be repeated here. It can be obtained in the case of joint training with the first feature extraction network, or can be obtained using a pre-trained first feature extraction network.

Step 4.2: Use the first basic directional convolutional neural network to perform directional convolution processing on the first sample target bone feature matrix to obtain the first sample oriented bone feature matrix; use the first sample oriented bone feature matrix and contour The actual position information of the key point, the seventh loss is obtained. And based on the seventh loss, the first basic directional convolutional neural network is trained in this round.

Here, the seventh loss is LC3 in FIG. 7.

Here, the first basic directional convolutional neural network is used to perform directional convolution processing on the first sample target bone feature matrix, that is, to perform directional space transformation on the first sample target bone feature matrix. In this case, it is necessary to make the obtained position information of the key points represented by the oriented bone feature matrix of the first sample consistent with the position information of the key points of the contour as much as possible. Therefore, it is necessary to obtain the seventh loss based on the first sample orientation bone feature matrix and the actual position information of the outline key points, and use the seventh loss to supervise the training of the first basic orientation convolutional neural network.

Step 4.3: Use the second basic directional convolutional neural network to perform directional convolution processing on the first sample target contour feature matrix to obtain the first sample oriented contour feature matrix; use the first sample oriented contour feature matrix, and the skeleton The actual position information of the key point, get the eighth loss. And based on the eighth loss, the second basic directional convolutional neural network is trained in this round.

Here, the eighth loss is LS3 in FIG. 7.

Step 4.4: Use the fourth basic convolutional neural network to perform convolution processing on the oriented contour feature matrix of the first sample to obtain the middle contour feature matrix of the second sample; and make the obtained middle contour feature matrix of the second sample the same as the first The target bone feature matrix is spliced to obtain the third sample spliced feature matrix, and the third basic transformation neural network is used to perform dimensional transformation on the third sample spliced feature matrix to obtain the second sample target bone feature matrix.

Step 4.5: Determine the fourth predicted position information of the bone key point based on the second sample target bone feature matrix; determine the ninth loss based on the actual position information of the bone key point and the fourth predicted position information of the bone key point.

Here, the ninth loss is LS4 in FIG. 7.

Step 4.6: Use the third basic convolutional neural network to perform convolution processing on the oriented skeleton feature matrix of the first sample to obtain the middle skeleton feature matrix of the second sample; and make the obtained middle skeleton feature matrix of the second sample the same as the first sample The target contour feature matrix is spliced to obtain the fourth sample spliced feature matrix, and the fourth sample spliced feature matrix is dimensionally transformed using the fourth basic transformation neural network to obtain the second sample target contour feature matrix.

Step 4.7: Determine the fourth predicted position information of the contour key point based on the second sample target contour feature matrix; determine the tenth loss based on the actual position information of the contour key point and the fourth predicted position information of the contour key point.

Here, the tenth loss is LC4 in FIG. 7.

Step 4.8: Based on the ninth loss and the tenth loss, perform this round on the third basic convolutional neural network, the fourth basic convolutional neural network, the third basic transform neural network, and the fourth basic transform neural network train.

After the first basic directional convolutional neural network, the second basic directional convolutional neural network, the third basic convolutional neural network, the fourth basic convolutional neural network, the third basic transform neural network, and the first The four basic transformation neural network performs multiple rounds of training to obtain a well-trained feature fusion neural network.

M3:

Referring to FIG. 10, another specific structure of a feature fusion neural network provided by an embodiment of the present disclosure includes: a displacement estimation neural network and a fifth transformation neural network.

As shown in FIG. 11, an embodiment of the present disclosure also provides a feature fusion neural network based on the feature fusion neural network provided in FIG. 10 to perform feature fusion on a first target skeleton feature matrix and a first target contour feature matrix to obtain a second target skeleton feature matrix And the specific method of the second target contour feature matrix includes the following steps.

Step S1101: Perform splicing processing on the first target skeleton feature matrix and the first target contour feature matrix to obtain a fifth spliced feature matrix.

Step S1102: Input the fifth splicing feature matrix into the displacement estimation neural network, perform displacement estimation on multiple sets of predetermined key point pairs, and obtain the displacement of one key point in each key point pair to another key point Message; where two keypoints in each keypoint pair are adjacent, the two keypoints include a bone keypoint and a contour keypoint, or two bone keypoints, or two contour keypoints .

In specific implementation, multiple bone key points and multiple contour key points are determined in advance for the human body. As shown in FIG. 12, an example of multiple bone key points and contour key points determined in advance for the human body is provided. In this example, there are 14 bone key points, which are represented by the larger dots in Figure 12: the top of the head, the neck, the shoulders, the elbows, the wrists, the hips, the knees, and the ankles; the outline key There are 26 points, represented by the smaller dots in Figure 12. Except for the key points of the bones that characterize the top of the human head, there are two contour key points corresponding to each other bone key points. Among them, the bone key points of the double span correspond to the same contour key points.

Two key points adjacent to each other can form a key point pair. As shown in Figure 12, every two key points directly connected by a line segment can form a key point pair. That is, the composition of key point pairs may appear in the following three situations: (skeleton key points, bone key points), (contour key points, contour key points), (skeleton key points, contour key points).

The displacement estimation neural network includes a multi-layer convolutional layer, which is connected in turn, and is used to perform feature learning on the bone features and contour features in the fifth splicing feature matrix, so that one key point in each key point pair is moved to another Displacement information of key points. There are two sets of displacement information corresponding to each key point.

For example, if the key point pair is (P, Q), where P and Q respectively represent a key point. Then the displacement information of the key point pair includes: the displacement information from P to Q, and the displacement information from Q to P.

Each group of displacement information includes the moving direction and the moving distance.

Step S1103: Use each key point in each key point pair as the current key point, and obtain the corresponding key point corresponding to the other key point from the three-dimensional feature matrix corresponding to the other key point paired with the current key point. Two-dimensional feature matrix; among them, if the other key point of the pair is a bone key point, the three-dimensional feature matrix corresponding to the bone key point is the first bone feature matrix; if the other key point of the pair is a contour key point, the The three-dimensional feature matrix corresponding to the contour key points is the first contour feature matrix.

Step S1104: According to the displacement information from the other key point of the pair to the current key point, perform position transformation on the elements in the two-dimensional feature matrix corresponding to the other key point of the pair to obtain the displacement feature matrix corresponding to the current key point .

Here, still taking the key point pair (P, Q) as an example, first take P as the current key point, and obtain the two-dimensional feature matrix corresponding to Q from the three-dimensional feature matrix corresponding to Q.

Here, if Q is a bone key point, the three-dimensional feature matrix corresponding to Q is the first bone feature matrix (see step S402 above). If Q is the contour key point, the three-dimensional feature matrix corresponding to Q is the first contour feature matrix (see step S403 above).

Here, when Q is the key point of the skeleton, the first skeleton feature matrix is taken as the three-dimensional feature matrix of Q, and the two-dimensional feature matrix of Q is obtained from the first skeleton feature matrix. This is because the first bone feature matrix only includes bone features, which can make the bone features learned in the subsequent processing more targeted. Similarly, when Q is the key point of the contour, the first contour feature matrix is taken as the three-dimensional feature matrix of Q, and the two-dimensional feature matrix of Q is obtained from the first contour feature matrix. This is because only the contour features are included in the first contour feature matrix, which makes the contour features learned in the subsequent processing more pertinent.

After the two-dimensional feature matrix of Q is obtained, based on the displacement information from Q to P, the positions of the elements in the two-dimensional feature matrix of Q are transformed to obtain the displacement feature matrix corresponding to P.

For example, as shown in Figure 13, if the displacement information from Q to P is (2, 3), where 2 means that the distance moved in the first dimension is 2, and 3 means that the distance moved in the second dimension is 3, then The two-dimensional feature matrix of Q is shown in Figure 13 a; after the position transformation of the elements in the two-dimensional feature matrix of Q, the displacement feature matrix corresponding to P is obtained as shown in Figure 13 b. Here, only numbers are used to represent the relative displacement information. In actual implementation, the displacement information should be understood in conjunction with specific solutions. For example, the displacement information "2" can refer to 2 elements, 2 cells, and so on.

Then, taking Q as the current key point, and obtaining the two-dimensional feature matrix corresponding to P from the three-dimensional feature matrix corresponding to P. Then, based on the displacement information from P to Q, position transformation is performed on the elements in the two-dimensional characteristic matrix of P to obtain the displacement characteristic matrix Q corresponding to Q.

In this way, the displacement characteristic matrix corresponding to each bone key point and the displacement characteristic matrix corresponding to each contour key point can be generated.

It should be noted here that each bone key point may be paired with multiple key points. Therefore, there may be multiple displacement feature matrices for each bone key point; each contour key point may also be It may be paired with multiple key points, so there may be multiple displacement feature matrices for each contour key point. And for different contour key points, the number of corresponding displacement feature matrices may be different; for different bone key points, the number of corresponding displacement feature matrices may also be different.

Step S1105: For each bone key point, the two-dimensional feature matrix corresponding to the bone key point and each displacement feature matrix corresponding to the bone key point are spliced to obtain a spliced two-dimensional feature matrix of the bone key point; and The spliced two-dimensional feature matrix of the bone key points is input to the fifth transform neural network to obtain the target two-dimensional feature matrix corresponding to the bone key point; based on the target two-dimensional feature matrix corresponding to each bone key point, the first two-dimensional feature matrix is generated. 2. The target bone feature matrix.

Step S1106: For each contour key point, perform splicing processing on the two-dimensional feature matrix corresponding to the contour key point and each displacement feature matrix corresponding to the contour key point to obtain the spliced two-dimensional feature matrix of the contour key point; and The spliced two-dimensional feature matrix of the contour key points is input into the fifth transform neural network to obtain the target two-dimensional feature matrix corresponding to the contour key point; based on the target two-dimensional feature matrix corresponding to each contour key point, the first 2. Target contour feature matrix.

For example, if P is a bone key point, and the two-dimensional feature matrix corresponding to P is P', and P is located in three key point pairs, then based on the above process, three displacement feature matrices of P can be obtained, which are P1', P2' and P3', then P', P1', P2', and P3' are spliced to obtain the spliced two-dimensional feature matrix of P. In this case, the three displacement feature matrices of P may be obtained by transforming the positions of the elements in the two-dimensional feature matrix corresponding to the key points of the skeleton, as well as the elements in the two-dimensional feature matrix corresponding to the key points of the contour. Resulted from position transformation. Therefore, P', P1', P2', and P3' are spliced, so that the features of each key point adjacent to P in position are fused together. Then use the fifth transformation neural network to perform convolution processing on the spliced two-dimensional feature matrix of P, so that the obtained target two-dimensional feature matrix of P contains both bone features and contour features, realizing bone features and contour features Fusion.

In the same way, if P is the contour key point, the fusion of bone features and contour features can also be realized based on the above process.

In another embodiment, the feature fusion neural network can be individually trained in the following manner.

Step 5.1: Obtain the first sample target bone feature matrix and the first sample target contour feature matrix of multiple sample images.

The method of obtaining is similar to the method of obtaining the first target skeleton feature matrix and the first target contour feature matrix in the foregoing embodiment, and will not be repeated here. It can be obtained in the case of joint training with the first feature extraction network, or can be obtained using a pre-trained first feature extraction network.

Step 5.2: Perform splicing processing on the first sample target skeleton feature matrix and the first sample target contour feature matrix to obtain the fifth sample spliced feature matrix.

Step 5.3: Input the fifth sample splicing feature matrix into the basic displacement estimation neural network, perform displacement estimation on multiple sets of predetermined key point pairs, and obtain that one key point of each key point pair moves to another key point The predicted displacement information of each keypoint; where two keypoints in each keypoint pair are adjacent, the two keypoints include a bone keypoint and a contour keypoint, or include two bone keypoints, or two Outline key points.

Step 5.4: Take each key point in each key point pair as the current key point, and obtain the corresponding to the other key point of the pair from the sample three-dimensional feature matrix corresponding to the other key point paired with the current key point The two-dimensional feature matrix of the sample.

Step 5.5: According to the predicted displacement information from the other key point of the pair to the current key point, perform position transformation on the element in the two-dimensional feature matrix of the sample corresponding to the other key point of the pair to obtain the sample corresponding to the current key point Displacement feature matrix.

Step 5.6: Determine the displacement loss according to the sample displacement characteristic matrix corresponding to the current key point and the sample two-dimensional characteristic matrix corresponding to the current key point.

Step 5.7: Based on the displacement loss, perform this round of training on the displacement estimation neural network.

Step 5.8: For each bone key point, the sample two-dimensional feature matrix corresponding to the bone key point, and each sample displacement feature matrix corresponding to the bone key point are spliced to obtain the two-dimensional feature of the sample splicing of the bone key point Matrix; and input the sample splicing two-dimensional feature matrix of the bone key point into the fifth basic transformation neural network to obtain the two-dimensional feature matrix of the sample target corresponding to the bone key point; based on the sample target corresponding to each bone key point The two-dimensional feature matrix is used to generate the target bone feature matrix of the second sample.

Step 5.9: For each contour key point, the two-dimensional feature matrix of the sample corresponding to the contour key point and the displacement feature matrix of each sample corresponding to the contour key point are spliced to obtain the two-dimensional splicing feature of the sample of the contour key point Matrix; and input the two-dimensional feature matrix of the sample splicing of the contour key points into the fifth basic transformation neural network to obtain the two-dimensional feature matrix of the sample target corresponding to the contour key point; based on the sample target corresponding to each contour key point A two-dimensional feature matrix is used to generate a second sample target contour feature matrix.

Step 5.10: Determine the transformation loss based on the second sample target skeleton feature matrix, the second sample target contour feature matrix, the actual position information of the bone key points, and the actual position information of the contour key points. For example, the predicted position information of the skeleton key points may be determined based on the second sample target skeleton feature matrix, and the predicted position information of the contour key points may be determined based on the second sample target contour feature matrix. Based on the predicted position information and actual position information of the bone key points, and the predicted position information and actual position information of the contour key points, the transformation loss is determined.

Step 5.11: Based on the transformation loss, perform this round of training on the fifth basic transformation neural network.

Step 5.12: After multiple rounds of training on the basic displacement estimation neural network and the fifth basic transformation neural network, a feature fusion neural network is obtained.

B: Perform multiple feature extraction on the image to be detected, and after each feature extraction, perform feature fusion on the bone features and contour features obtained from this feature extraction, and determine based on the feature fusion result of the last feature fusion The position information of the bone key points and the position information of the contour key points.

In the case of multiple feature extractions, the i+1th feature extraction is performed based on the feature fusion result of the i-th feature fusion, and i is a positive integer.

In B, the process of performing the first feature extraction is consistent with the process of extracting bone features and contour features from the image to be detected in A, and will not be repeated here.

The specific process of each feature extraction except the first feature extraction in B includes:

Use the second feature extraction network to extract the first target skeleton feature matrix of the bone key points used to characterize the human skeleton from the feature fusion result of the last feature fusion; and extract the contour key points used to characterize the contour feature of the human body The contour feature matrix of the first target;

Wherein, the network parameters of the first feature extraction network and the second feature extraction network are different, and the network parameters of the second feature extraction network used for different times of feature extraction are different.

Here, both the first feature extraction network and the second feature extraction network include multiple convolutional layers. The network parameters of the first feature extraction network and the second feature extraction network include, but are not limited to: the number of convolutional layers, the size of the convolution kernel used by each convolutional layer, and the convolution kernel used by each convolutional layer. Quantity etc.

Referring to FIG. 14, an embodiment of the present disclosure provides a schematic structural diagram of a second feature extraction network. The second feature extraction network includes: a second bone feature extraction network and a second contour feature extraction network.

The feature fusion result of the last feature fusion using the second feature extraction network for this feature extraction includes: the second target skeleton feature matrix and the second target contour feature matrix; specifically, the second target skeleton feature matrix and the second target are obtained The process of the contour feature matrix is shown in A above, and will not be repeated here.

Use the second feature extraction network to extract the first target bone feature matrix of the bone key points used to characterize the human bone features from the feature fusion result of the last feature fusion; and extract the contour key points used to characterize the contour features of the human body The specific process of the first target contour feature matrix is, for example:

Use the second bone feature extraction network to perform convolution processing on the second target bone feature matrix obtained from the last feature fusion to obtain the third bone feature matrix, which is obtained from the third target convolution layer in the second bone feature extraction network The fourth skeletal feature matrix: Based on the third skeletal feature matrix and the fourth skeletal feature matrix, the fifth target skeletal feature matrix is obtained. Among them, the third target convolutional layer is any convolutional layer except the last convolutional layer in the second bone feature extraction network.

Use the second contour feature extraction network to perform convolution processing on the second target contour feature matrix obtained from the last feature fusion to obtain the third contour feature matrix, which is obtained from the fourth target convolution layer in the second contour feature extraction network The fourth contour feature matrix: Based on the third contour feature matrix and the fourth contour feature matrix, a sixth target contour feature matrix is obtained. The fourth target convolutional layer is any convolutional layer except the last convolutional layer in the second contour feature extraction network.

The specific processing method is similar to the specific process of using the first skeleton feature extraction network to extract the first target skeleton feature matrix and the first target contour feature matrix from the image to be detected in A, and will not be repeated here.

The above embodiments describe the method of determining the position information of the bone key points and the contour key points in the above II.

Ⅲ: After obtaining the position information of the bone key points and the position information of the contour key points based on the above Ⅱ, the position of each bone key point and the position of the contour key point can be determined from the image to be detected. The human body detection result can then be generated.

The human body detection result includes one or more of the following: a to-be-detected image including bone key point markers and contour key point markers; a data group including position information of the bone key points and position information of the contour key points.

Subsequently, based on the human body detection result, one or more of the following operations may be performed: human body motion recognition, human body posture detection, human body contour adjustment, human body image editing, and human body mapping.

Here, action recognition, for example, recognizes the current actions of the human body, such as fighting, running, etc.; human body posture recognition, for example, recognizes the current posture of the human body, such as lying down, whether to make a specified action, etc.; human contour adjustment, such as adjusting the body shape and height, etc. ; Human body image editing, such as zooming, rotating, and cutting the human body; human body mapping, for example, after detecting the human body in image A, paste the corresponding human body image into image B.

The embodiment of the present disclosure can determine the position information of the key points of the skeleton used to characterize the human skeletal structure and the position information of the key points of the outline used to characterize the outline of the human body from the image to be detected, and based on the position information of the key points of the bone, As well as the position information of key points of the contour, the human body detection result is generated, which improves the precision of the representation while taking into account the amount of calculation data.

In addition, in the embodiments of the present disclosure, since the position information of the key points of the bones that characterize the bone structure of the human body and the position information of the key points of the outline that characterize the outline of the human body are used to obtain the human body detection results, the information that characterizes the human body is richer and has a broader scope. Application scenarios, such as image editing, body shape adjustment, etc.

Based on the same technical concept, the embodiment of the present disclosure also provides a human body detection device corresponding to the human body detection method. Since the principle of the device in the embodiment of the present disclosure to solve the problem is similar to the above-mentioned human body detection method of the embodiment of the present disclosure, the implementation of the device You can refer to the implementation of the method, and the repetition will not be repeated.

15 is a schematic diagram of a human body detection device provided by an embodiment of the present disclosure. The device includes: an acquisition module 151, a detection module 152, and a generation module 153; wherein, the acquisition module 151 is used to acquire The image to be detected; the detection module 152, based on the image to be detected, is used to determine the position information of the bone key points used to characterize the human skeletal structure and the position information of the contour key points used to characterize the outline of the human body; The module 153 is configured to generate a human body detection result based on the position information of the bone key points and the position information of the contour key points.

In a possible implementation manner, the contour key points include a main contour key point and an auxiliary contour key point; wherein there is at least one auxiliary contour key point between two adjacent main contour key points.

In a possible implementation manner, the detection module 152 is configured to determine the position information of key contour points used to characterize the contour of the human body based on the image to be detected in the following manner: based on the image to be detected, Determine the position information of the main contour key points; determine the human body contour information based on the position information of the main contour key points; determine the position information results of a plurality of the auxiliary contour key points based on the determined human contour information.

In a possible implementation manner, the human body detection result includes one or more of the following: an image to be detected added with bone key point markers and contour key point markers; including position information of the bone key points and the contour The data group of the location information of the key point.

In a possible implementation manner, the human body detection device further includes: an execution module 154 for performing one or more of the following operations based on the human body detection result: human body motion recognition, human body posture detection, human body contour adjustment, and human body Image editing, and body stickers.

In a possible implementation manner, the detection module 152 is configured to determine the position information of key bone points used to characterize the skeleton structure of the human body and the information used to characterize the outline of the human body based on the image to be detected in the following manner: Location information of contour key points: based on the image to be detected, perform feature extraction to obtain bone features and contour features, and perform feature fusion on the obtained bone features and contour features; determine the bone key points based on the result of feature fusion The location information of and the location information of the key points of the outline.

In a possible implementation manner, the detection module 152 is configured to perform feature extraction based on the image to be detected in the following manner to obtain bone features and contour features, and perform feature fusion on the obtained bone features and contour features : Perform at least one feature extraction based on the image to be detected, and perform feature fusion on the bone features and contour features obtained from each feature extraction. In the case of performing multiple feature extractions, based on the i-th feature fusion The feature fusion result of is subjected to the i+1th feature extraction, and i is a positive integer; the detection module 152 is used to determine the position information of the bone key points used to characterize the human skeletal structure based on the feature fusion result in the following manner And the position information of the contour key points used to characterize the contour of the human body: based on the feature fusion result of the last feature fusion, the position information of the bone key points and the position information of the contour key points are determined.

In a possible implementation manner, the detection module 152 is configured to perform at least one feature extraction based on the image to be detected in the following manner: in the first feature extraction, a pre-trained first feature extraction is used The network extracts the first target skeleton feature matrix used to characterize the skeleton key points of the human skeleton feature and the first target contour feature matrix used to represent the contour key points of the human contour feature from the image to be detected; in the i+th In the first feature extraction, the pre-trained second feature extraction network is used to extract the first target bone feature matrix of the bone key points used to characterize the human bone feature from the feature fusion result of the i-th feature fusion; and the extraction is used The first target contour feature matrix for the contour key points that characterize the contour features of the human body; wherein the network parameters of the first feature extraction network and the second feature extraction network are different, and the second feature extraction used for different times of feature extraction The network parameters of the network are different.

In a possible implementation manner, the detection module 152 is configured to perform feature fusion of the extracted bone features and contour features in the following manner: use a pre-trained feature fusion neural network to perform feature fusion on the first target bone feature Matrix and the first target contour feature matrix are feature fused to obtain a second target skeleton feature matrix and a second target contour feature matrix; wherein, the second target skeleton feature matrix is a three-dimensional skeleton feature matrix, and the three-dimensional skeleton feature The matrix includes a two-dimensional bone feature matrix corresponding to each bone key point; the value of each element in the two-dimensional bone feature matrix represents the probability that the pixel corresponding to the element belongs to the corresponding bone key point; the second The target contour feature matrix is a three-dimensional contour feature matrix. The three-dimensional contour feature matrix includes a two-dimensional contour feature matrix corresponding to each contour key point; the value of each element in the two-dimensional contour feature matrix represents the corresponding element The probability that a pixel belongs to the key point of the corresponding contour; the network parameters of the feature fusion neural network used in different sub-feature fusion are different.

In a possible implementation manner, the detection module 152 is configured to determine the position information of the bone key points and the position information of the contour key points based on the feature fusion result of the last feature fusion in the following manner: Determine the position information of the skeleton key points based on the second target skeleton feature matrix obtained in the last feature fusion; and determine the position information of the contour key points based on the second target contour feature matrix obtained in the last feature fusion.

In a possible implementation manner, the first feature extraction network includes: a common feature extraction network, a first bone feature extraction network, and a first contour feature extraction network; the detection module 152 is configured to adopt the following methods Use the first feature extraction network to extract the first target bone feature matrix of the bone key points used to characterize the human bone features from the image to be detected; and extract the first target contour feature used to characterize the contour key points of the human body contour feature matrix:

Use the common feature extraction network to perform convolution processing on the to-be-detected image to obtain a basic feature matrix containing bone features and contour features; use the first bone feature extraction network to convolve the basic feature matrix Product processing to obtain a first bone feature matrix, and obtain a second bone feature matrix from the first target convolution layer in the first bone feature extraction network; based on the first bone feature matrix and the second bone feature Matrix to obtain the first target bone feature matrix; the first target convolutional layer is any convolutional layer except the last convolutional layer in the first bone feature extraction network; using the first contour The feature extraction network performs convolution processing on the basic feature matrix to obtain a first contour feature matrix, and obtains a second contour feature matrix from the second target convolution layer in the first contour feature extraction network; The first contour feature matrix and the second contour feature matrix are used to obtain the first target contour feature matrix; the second target convolutional layer is in the first contour feature extraction network, except for the last convolutional layer Any other convolutional layer.

In a possible implementation manner, the detection module 152 is configured to obtain the first target bone feature matrix based on the first bone feature matrix and the second bone feature matrix in the following manner: Performing splicing processing on the first bone feature matrix and the second bone feature matrix to obtain the first spliced bone feature matrix;

Performing dimensional transformation processing on the first spliced bone feature matrix to obtain the first target bone feature matrix;

The obtaining the first target contour characteristic matrix based on the first contour characteristic matrix and the second contour characteristic matrix includes: performing stitching processing on the first contour characteristic matrix and the second contour characteristic matrix , Obtain the first spliced contour feature matrix; perform dimensional transformation processing on the first spliced contour feature matrix to obtain the first target contour feature matrix; wherein the dimensions of the first target skeleton feature matrix are the same as those of the first The dimensions of the target contour feature matrix are the same, and the dimensions of the first target skeleton feature matrix and the first target contour feature matrix are the same in the same dimension.

In a possible implementation manner, the feature fusion neural network includes: a first convolutional neural network, a second convolutional neural network, a first transformation neural network, and a second transformation neural network;

The detection module 152 is configured to perform feature fusion on the first target skeleton feature matrix and the first target contour feature matrix using a feature fusion neural network in the following manner to obtain a second target skeleton feature matrix and Second target contour feature matrix: using the first convolutional neural network to perform convolution processing on the first target bone feature matrix to obtain a first intermediate bone feature matrix; and using the second convolutional neural network Perform convolution processing on the first target contour feature matrix to obtain a first intermediate contour feature matrix; perform splicing processing on the first intermediate contour feature matrix and the first target bone feature matrix to obtain a first spliced feature matrix And use the first transformation neural network to perform dimensional transformation on the first splicing feature matrix to obtain the second target skeleton feature matrix; combine the first intermediate skeleton feature matrix with the first target contour feature The matrix is spliced to obtain a second spliced feature matrix, and the second spliced feature matrix is dimensionally transformed using the second transformation neural network to obtain the second target contour feature matrix.

In a possible implementation manner, the feature fusion neural network includes: a first directional convolutional neural network, a second directional convolutional neural network, a third convolutional neural network, and a fourth convolutional neural network , The third transformation neural network, and the fourth transformation neural network;

The detection module 152 is configured to perform feature fusion on the first target skeleton feature matrix and the first target contour feature matrix using a feature fusion neural network in the following manner to obtain a second target skeleton feature matrix and Second target contour feature matrix: use the first directional convolutional neural network to perform directional convolution processing on the first target bone feature matrix to obtain the first directional bone feature matrix; and use the third convolutional neural network The network performs convolution processing on the first directional skeleton feature matrix to obtain a second intermediate skeleton feature matrix; and uses the second directional convolutional neural network to perform directional convolution on the first target contour feature matrix Processing to obtain a first directional contour feature matrix; and use a fourth convolutional neural network to perform convolution processing on the first directional contour feature matrix to obtain a second intermediate contour feature matrix; convert the second intermediate contour The feature matrix is spliced with the first target bone feature matrix to obtain a third spliced feature matrix; and a third transformation neural network is used to perform dimensional transformation on the third spliced feature matrix to obtain the second target bone feature Matrix; the second intermediate skeleton feature matrix and the first target contour feature matrix are spliced to obtain a fourth spliced feature matrix, and the fourth spliced feature matrix is dimensionally transformed using a fourth transformation neural network , To obtain the second target contour feature matrix.

In a possible implementation manner, the feature fusion neural network includes: a displacement estimation neural network and a fifth transformation neural network;

The detection module 152 is configured to perform feature fusion on the first target skeleton feature matrix and the first target contour feature matrix using a feature fusion neural network in the following manner to obtain a second target skeleton feature matrix and The second target contour feature matrix: the first target skeleton feature matrix and the first target contour feature matrix are spliced to obtain a fifth spliced feature matrix; the fifth spliced feature matrix is input to the displacement estimation nerve In the network, the displacement estimation is performed on a plurality of predetermined key point pairs, and the displacement information of one key point in each key point pair to another key point is obtained; each key point in each key point pair is obtained As the current key point, respectively, from the three-dimensional feature matrix corresponding to the other key point paired with the current key point, the two-dimensional feature matrix corresponding to the other key point of the pair is obtained; For the displacement information from the key point to the current key point, perform position transformation on the element in the two-dimensional feature matrix corresponding to the other key point of the pair to obtain the displacement feature matrix corresponding to the current key point; for each Bone key points, the two-dimensional feature matrix corresponding to the bone key point, and each displacement feature matrix corresponding to the bone key point are spliced to obtain the spliced two-dimensional feature matrix of the bone key point; Combine the two-dimensional feature matrix and input it into the fifth transformation neural network to obtain the target two-dimensional feature matrix corresponding to the bone key point; generate the second target based on the target two-dimensional feature matrix corresponding to each bone key point Skeleton feature matrix; for each contour key point, the two-dimensional feature matrix corresponding to the contour key point and each displacement feature matrix corresponding to the bone key point are spliced to obtain the spliced two-dimensional feature matrix of the contour key point; And input the spliced two-dimensional feature matrix of the contour key points into the fifth transform neural network to obtain the target two-dimensional feature matrix corresponding to the contour key point; based on the target two-dimensional feature matrix corresponding to each contour key point , Generating the second target contour feature matrix.

In a possible implementation manner, the human body detection method is implemented by a human body detection model; the human body detection model includes: the first feature extraction network and/or the feature fusion neural network; the human body detection model is It is obtained by training using sample images in the training sample set, and the sample images are marked with actual position information of bone key points of the human skeletal structure and actual position information of the contour key points of the outline of the human body.

For the description of the processing flow of each module in the device and the interaction flow between each module, reference may be made to the relevant description in the above method embodiment, which will not be described in detail here.

The embodiment of the present disclosure also provides a computer device, as shown in FIG. 16, which is a schematic structural diagram of the computer device provided by the embodiment of the present disclosure, including:

The processor 11, the storage medium 12, and the bus 13; the storage medium 12 is used to store execution instructions, including a random access memory 121 and an external memory 122; here, the random access memory 121 is also called random access memory storage The processor 11 is used to temporarily store the processed data in the processor 11 and the data exchanged with the external memory 122 such as a hard disk. The processor 11 exchanges data with the external memory 122 through the random access memory 121. When the computer device When 100 is running, the processor 11 communicates with the storage medium 12 through the bus 13, so that the processor 11 is executing the following instructions: acquiring an image to be detected; based on the image to be detected, Determine the position information of the bone key points used to characterize the skeleton structure of the human body and the position information of the contour key points used to characterize the outline of the human body; generate based on the position information of the bone key points and the position information of the contour key points Human body test results.

The embodiment of the present disclosure also provides a computer-readable storage medium, the computer-readable storage medium stores a computer program, and the computer program is executed by a processor to execute the human body detection method described in the above method embodiment step.

The computer program product of the human body detection method provided by the embodiment of the present disclosure includes a computer-readable storage medium storing program code. The program code includes instructions that can be used to execute the human body detection method described in the above method embodiment. For details of the steps, please refer to the above method embodiments, which will not be repeated here.

Those skilled in the art can clearly understand that, for the convenience and conciseness of the description, the specific working process of the system and device described above can refer to the corresponding process in the foregoing method embodiment, which will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed system, device, and method may be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined. Or it can be integrated into another system, or some features can be ignored or not implemented. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection may be indirect coupling or communication connection through some communication interfaces, devices or units, and may be in electrical, mechanical or other forms.

The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units . Some or all of the units may be selected according to actual needs to achieve the objectives of the solutions of the embodiments.

In addition, the functional units in the various embodiments of the present disclosure may be integrated in one processing unit, or each unit may exist separately, or two or more units may be integrated in one unit.

If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present disclosure essentially or the part that contributes to the existing technology or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several The instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present disclosure. The aforementioned storage media include: USB, removable hard disk, Read-Only Memory (Read-Only Memory, ROM), Random Access Memory (Random Access Memory, RAM), magnetic disks or optical disks, etc., which can store program codes media.

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than limit it. The protection scope of the present disclosure is not limited to this, although referring to the foregoing The embodiments describe the present disclosure in detail, and those of ordinary skill in the art should understand that any person skilled in the art can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present disclosure. Or it can be easily conceived of changes, or equivalent replacements of some of the technical features; and these modifications, changes or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered by the present disclosure. Within the scope of protection. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the patent application.

11: processor 12: storage media 121: Random Access Memory 122: external memory 13: Bus 151: Get Mods 152: Detection Module 153: Generate Module 154: Execution Module LC1: second loss LC2: sixth loss LC3: seventh loss LS1: first loss LS2: fifth loss LS3: Eighth loss S101～S103, S401～S403, S601～S604, S801～S804, S1101～S1106: steps

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the following will briefly introduce the drawings that need to be used in the embodiments. It should be understood that the following drawings only show some implementations of the present disclosure for illustrative purposes. The examples are not restrictive. For those of ordinary skill in the art, without creative work, other related drawings can be obtained from these drawings. The same or similar reference signs in the drawings represent the same or equivalent elements. Once a reference sign is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Fig. 1 shows a flowchart of a human body detection method provided by an embodiment of the present disclosure. Fig. 2a shows an example of the positions of contour key points and bone key points provided by an embodiment of the present disclosure. Fig. 2b shows an example of the positions of the main contour key points and the auxiliary contour key points provided by the embodiments of the present disclosure. Fig. 2c shows another example of the positions of the main contour key points and the auxiliary contour key points provided by the embodiments of the present disclosure. Fig. 2d shows another example of the positions of the main contour key points and the auxiliary contour key points provided by the embodiments of the present disclosure. Fig. 3 shows a schematic structural diagram of a first feature extraction network provided by an embodiment of the present disclosure. Fig. 4 shows a flowchart of a feature extraction method provided by an embodiment of the present disclosure. Fig. 5 shows a schematic structural diagram of a feature fusion network provided by an embodiment of the present disclosure. Fig. 6 shows a flowchart of a feature fusion method provided by an embodiment of the present disclosure. Fig. 7 shows a schematic structural diagram of another feature fusion network provided by an embodiment of the present disclosure. Fig. 8 shows a flowchart of another feature fusion method provided by an embodiment of the present disclosure. Fig. 9a shows a schematic diagram of an iterative update process using a scattering convolution operator provided by an embodiment of the present disclosure. Fig. 9b shows a schematic diagram of an iterative update process using a clustered convolution operator provided by an embodiment of the present disclosure. FIG. 10 shows a schematic structural diagram of another feature fusion network provided by an embodiment of the present disclosure. Fig. 11 shows a flowchart of another feature fusion method provided by an embodiment of the present disclosure. Fig. 12 shows examples of bone key points and contour key points provided by an embodiment of the present disclosure. FIG. 13 shows a specific example of performing displacement transformation on elements in a two-dimensional feature matrix provided by an embodiment of the present disclosure. FIG. 14 shows a schematic structural diagram of a second feature extraction network provided by an embodiment of the present disclosure. FIG. 15 shows a schematic diagram of a human body detection device provided by an embodiment of the present disclosure. FIG. 16 shows a schematic diagram of a computer device provided by an embodiment of the present disclosure.

S101~S103: steps

Claims (15)

A human body detection method includes: acquiring an image to be detected; based on the image to be detected, determining position information of key bone points used to characterize the skeleton structure of the human body and position information of key contour points used to characterize the outline of the human body; And based on the position information of the bone key points and the position information of the contour key points to generate a human body detection result, wherein the position of the bone key points used to characterize the bone structure of the human body is determined based on the image to be detected The steps of the information and the position information of the contour key points used to characterize the outline of the human body include: based on the image to be detected, feature extraction is performed to obtain bone features and contour features, and the obtained bone features and contour features are feature-fused And based on the feature fusion result, determining the position information of the bone key points and the position information of the contour key points. The human body detection method according to claim 1, wherein the contour key points include a main contour key point and an auxiliary contour key point, wherein there is at least one auxiliary contour key between two adjacent main contour key points Point, wherein based on the image to be detected, the step of determining the position information of the contour key points used to characterize the outline of the human body includes: determining the position information of the main contour key points based on the image to be detected; The position information of the key points of the main contour is used to determine the contour information of the human body; And based on the determined human body contour information, position information of a plurality of key points of the auxiliary contour is determined. The human body detection method according to claim 1, wherein the human body detection result includes one or more of the following: the to-be-detected image added with bone key point markers and contour key point markers; and including the bone key points The data set of the position information of the point and the position information of the outline key point, wherein the method further includes: performing one or more of the following operations based on the human body detection result: human body motion recognition, human body posture detection, and human body contour adjustment , Human body image editing, and body stickers. The human body detection method according to claim 1, wherein the step of performing feature extraction based on the image to be detected to obtain bone features and contour features, and performing feature fusion on the obtained bone features and contour features includes: For the image to be detected, at least one feature extraction is performed, and the bone features and contour features obtained from each feature extraction are feature fused, wherein, in the case of multiple feature extractions, the feature is based on the i-th feature fusion The fusion result performs the i+1th feature extraction, and i is a positive integer; wherein, based on the feature fusion result, the position information of the bone key points used to characterize the skeleton structure of the human body and the contour key points used to characterize the contour of the human body are determined The step of position information includes: determining the position information of the bone key points and the position information of the contour key points based on the feature fusion result of the last feature fusion. The human body detection method according to claim 4, wherein the step of performing at least one feature extraction based on the image to be detected includes: in the first feature extraction, using a pre-trained first feature extraction network from Extracting a first target skeleton feature matrix of bone key points for characterizing human bone features and a first target contour feature matrix of contour key points for characterizing human body contour features from the image to be detected; and In the first feature extraction, a pre-trained second feature extraction network is used to extract the first target skeleton feature matrix and the first target contour feature matrix from the feature fusion result of the i-th feature fusion, where the first target contour feature matrix is The network parameters of a feature extraction network and the second feature extraction network are different, and the network parameters of the second feature extraction network used for different times of feature extraction are different. The human body detection method according to claim 5, wherein the step of performing feature fusion on the extracted bone features and contour features further includes: using a pre-trained feature fusion neural network to perform the feature matrix of the first target bone and The first target contour feature matrix is feature fused to obtain a second target skeleton feature matrix and a second target contour feature matrix; wherein, the second target skeleton feature matrix is a three-dimensional skeleton feature matrix, and the three-dimensional skeleton feature matrix includes the The two-dimensional bone features corresponding to the key points of the bones The feature matrix; the value of each element in the two-dimensional bone feature matrix represents the probability that the pixel corresponding to the element belongs to the corresponding bone key point; the second target contour feature matrix is a three-dimensional contour feature matrix, the three-dimensional contour The feature matrix includes a two-dimensional contour feature matrix corresponding to each contour key point; the value of each element in the two-dimensional contour feature matrix represents the probability that the pixel corresponding to the element belongs to the corresponding contour key point; different sub-features The network parameters of the feature fusion neural network used for fusion are different. The human body detection method according to claim 5, wherein the first feature extraction network includes a common feature extraction network, a first bone feature extraction network, and a first contour feature extraction network, wherein the first feature is used The step of extracting the first target skeleton feature matrix and the first target contour feature matrix from the image to be detected by the extraction network includes: using the common feature extraction network to scroll the image to be detected Product processing to obtain a basic feature matrix containing bone features and contour features; use the first bone feature extraction network to perform convolution processing on the basic feature matrix to obtain a first bone feature matrix, and obtain a first bone feature matrix from the first bone The first target convolutional layer in the feature extraction network obtains a second bone feature matrix; based on the first bone feature matrix and the second bone feature matrix, the first target bone feature matrix is obtained; the first target The convolutional layer is any convolutional layer except the last convolutional layer in the first bone feature extraction network; and using the first contour feature extraction network to perform convolution processing on the basic feature matrix, Obtain the first contour feature matrix, and extract the network from the first contour feature The second target convolution layer in the road obtains a second contour feature matrix; based on the first contour feature matrix and the second contour feature matrix, the first target contour feature matrix is obtained; the second target convolution layer is In the first contour feature extraction network, any other convolutional layer except the last convolutional layer. The human body detection method according to claim 7, wherein the step of obtaining the first target bone feature matrix based on the first bone feature matrix and the second bone feature matrix includes: combining the first bone feature matrix The feature matrix and the second bone feature matrix are spliced to obtain a first spliced bone feature matrix; and the first spliced bone feature matrix is subjected to dimensional transformation processing to obtain the first target bone feature matrix, wherein Based on the first contour feature matrix and the second contour feature matrix, the step of obtaining the first target contour feature matrix further includes: performing stitching processing on the first contour feature matrix and the second contour feature matrix , Obtain a first spliced contour feature matrix; and perform dimensional transformation processing on the first spliced contour feature matrix to obtain the first target contour feature matrix; wherein the dimensions of the first target skeleton feature matrix are the same as the first A target contour feature matrix has the same dimensions, and the first target skeleton feature matrix and the first target contour feature matrix have the same dimension in the same dimension. The human body detection method according to claim 6, wherein the feature fusion neural network includes a first convolutional neural network, a second convolutional neural network, and a first transformation neural network. Through the network and the second transformation neural network, wherein the feature fusion neural network is used to perform feature fusion on the first target skeleton feature matrix and the first target contour feature matrix to obtain the second target skeleton feature The steps of the matrix and the second target contour feature matrix include: using the first convolutional neural network to perform convolution processing on the first target skeleton feature matrix to obtain a first intermediate skeleton feature matrix; using the second volume The product neural network performs convolution processing on the first target contour feature matrix to obtain a first intermediate contour feature matrix; splicing the first intermediate contour feature matrix and the first target skeleton feature matrix to obtain the first A splicing feature matrix; and using the first transformation neural network to perform dimensional transformation on the first splicing feature matrix to obtain the second target bone feature matrix; and combining the first intermediate bone feature matrix with the The first target contour feature matrix is spliced to obtain a second spliced feature matrix, and the second spliced feature matrix is dimensionally transformed using the second transformation neural network to obtain the second target contour feature matrix. The human body detection method according to claim 6, wherein the feature fusion neural network includes a first directional convolutional neural network, a second directional convolutional neural network, a third convolutional neural network, and a fourth volume Product neural network, third transformation neural network, and fourth transformation neural network, wherein the feature fusion neural network is used to specialize the first target skeleton feature matrix and the first target contour feature matrix The step of obtaining the second target skeleton feature matrix and the second target contour feature matrix includes: using the first directional convolutional neural network to perform directional convolution processing on the first target skeleton feature matrix to obtain the first target skeleton feature matrix. A directional skeleton feature matrix; and use a third convolutional neural network to perform convolution processing on the first directional skeleton feature matrix to obtain a second intermediate skeleton feature matrix; use the second directional convolutional neural network Perform directional convolution processing on the first target contour feature matrix to obtain a first directional contour feature matrix; and use a fourth convolution neural network to perform convolution processing on the first directional contour feature matrix to obtain the first directional contour feature matrix Two middle contour feature matrix; the second middle contour feature matrix and the first target bone feature matrix are spliced to obtain a third spliced feature matrix; and a third transform neural network is used to perform the splicing process on the third spliced feature The matrix is dimensionally transformed to obtain the second target skeleton feature matrix; and the second intermediate skeleton feature matrix and the first target contour feature matrix are spliced to obtain a fourth spliced feature matrix, and the fourth transformation is used The neural network performs dimensional transformation on the fourth splicing feature matrix to obtain the second target contour feature matrix. The human body detection method according to claim 6, wherein the feature fusion neural network includes a displacement estimation neural network and a fifth transformation neural network, and wherein the use of the feature fusion neural network to analyze the characteristics of the first target bone Matrix, to The step of performing feature fusion with the first target contour feature matrix to obtain a second target skeleton feature matrix and a second target contour feature matrix further includes: aligning the first target skeleton feature matrix and the first target contour feature matrix Perform splicing processing to obtain a fifth spliced feature matrix; input the fifth spliced feature matrix into the displacement estimation neural network, perform displacement estimation on a plurality of predetermined key point pairs, and obtain each group of key point pairs The displacement information of a key point of a key point to another key point; each key point in each key point pair is regarded as the current key point, from the three-dimensional feature matrix corresponding to the other key point paired with the current key point , Obtain a two-dimensional feature matrix corresponding to the other key point of the pair; according to the displacement information from the other key point of the pair to the current key point, compare the two key points corresponding to the other key point of the pair; The elements in the three-dimensional feature matrix undergo position transformation to obtain the displacement feature matrix corresponding to the current key point; for each bone key point, the two-dimensional feature matrix corresponding to the bone key point, and each bone key point corresponding to the two-dimensional feature matrix The displacement feature matrix is spliced to obtain the spliced two-dimensional feature matrix of the bone key point; and the spliced two-dimensional feature matrix of the bone key point is input to the fifth transformation neural network to obtain the corresponding bone key point A target two-dimensional feature matrix; generate the second target bone feature matrix based on the target two-dimensional feature matrix corresponding to each bone key point; and for each contour key point, a two-dimensional feature matrix corresponding to the contour key point, Each displacement feature matrix corresponding to the contour key point is spliced to obtain the spliced two-dimensional feature matrix of the contour key point; and the contour key point is spliced in two dimensions The feature matrix is input to the fifth transformation neural network to obtain a target two-dimensional feature matrix corresponding to the contour key point; based on the target two-dimensional feature matrix corresponding to each contour key point, the second target contour feature matrix is generated . The human body detection method according to claim 1, wherein the human body detection method is implemented by a human body detection model, and the human body detection model includes a first feature extraction network and/or a feature fusion neural network, wherein the human body detection model To be obtained by training using the sample images in the training sample set, the sample images are marked with actual position information of bone key points of the human skeletal structure and actual position information of the contour key points of the human body contour. A human body detection device includes: an acquisition module for acquiring an image to be detected; a detection module for determining, based on the image to be detected, position information of key bone points used to characterize human bone structure, and using The position information of the key points of the outline representing the outline of the human body; and a generating module for generating human detection results based on the position information of the bone key points and the position information of the outline key points, wherein the For the image to be detected, the step of determining the position information of the key points of the skeleton used to characterize the skeleton structure of the human body and the position information of the key points of the outline used to characterize the outline of the human body includes: performing feature extraction based on the image to be detected to obtain Bone features and contour features, and perform feature fusion on the obtained bone features and contour features; and based on the feature fusion result, determine the position information of the bone key points, and The location information of the key points of the outline. A computer device includes a processor, a non-transitory storage medium, and a bus. The non-transitory storage medium stores machine-readable instructions executable by the processor. When the computer device is running, the processor Communication with the non-transitory storage medium is performed through a bus, and the machine-readable instructions are executed by the processor to execute the steps of the method described in any one of claims 1-12. A computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program is executed by a processor to execute the steps of the method as described in any one of claims 1-12.

Images