WO2023155350A1

WO2023155350A1 - Crowd positioning method and apparatus, electronic device, and storage medium

Info

Publication number: WO2023155350A1
Application number: PCT/CN2022/100170
Authority: WO
Inventors: 杨昆霖; 刘诗男; 侯军; 伊帅
Original assignee: 上海商汤智能科技有限公司
Priority date: 2022-02-17
Filing date: 2022-06-21
Publication date: 2023-08-24
Also published as: CN114550086A

Abstract

The present invention relates to a crowd positioning method and apparatus, an electronic device, and a storage medium. The method comprises: performing positioning of human body key points of a crowd image to obtain an initial positioning map corresponding to the crowd image, wherein the initial positioning map is used for indicating positions of initial human body key points comprised in the crowd image; on the basis of the positions of the initial human body key points in the crowd image, determining a target neighborhood corresponding to the initial human body key points; and on the basis of the target neighborhood corresponding to the initial human body key points, filtering the initial positioning map to obtain a target positioning map, wherein the target positioning map is used for indicating positions of target human body key points comprised in the crowd image. According to embodiments of the present invention, a false detection probability that a same human body corresponds to a plurality of initial human body key points can be reduced, and the target positioning map having relatively high accuracy is obtained.

Description

A crowd positioning method and device, electronic equipment and storage medium

This application claims the priority of the Chinese patent application filed on February 17, 2022, with the application number 202210146593.7, and the title of the invention is "a crowd positioning method and device, electronic equipment and storage medium", the entire content of which is incorporated herein by reference Applying.

technical field

The present disclosure relates to the field of computer technology, and in particular to a crowd positioning method and device, electronic equipment, and a storage medium.

Background technique

With the growth of the population and the acceleration of the urbanization process, there are more and more behaviors where a large number of people gather, and the scale is getting bigger and bigger. Crowd analysis is of great significance to public safety and urban planning. Common crowd analysis tasks include crowd counting, crowd behavior analysis, crowd positioning, etc. Among them, crowd positioning is the basis of other crowd analysis tasks. Crowd positioning refers to estimating the position of the key points of the head included in the image or video through computer vision algorithms, and determining the coordinates of the key points of the head included in the image or video, so as to perform crowd analysis tasks such as subsequent crowd counting and group behavior analysis. Provide data basis. The accuracy of crowd positioning directly affects the accuracy of crowd counting and the results of crowd behavior analysis. Therefore, there is an urgent need for a crowd location method with high accuracy.

Contents of the invention

The disclosure proposes a crowd positioning method and device, electronic equipment and a technical solution of a storage medium.

According to an aspect of the present disclosure, a crowd positioning method is provided, including: performing key point positioning on a crowd image to obtain an initial positioning map corresponding to the crowd image, wherein the initial positioning map is used to indicate the crowd The position of the initial human body key point included in the image; based on the position of the initial human body key point in the crowd image, determine the target neighborhood corresponding to the initial human body key point; the target neighborhood, and filter the initial positioning map to obtain a target positioning map, wherein the target positioning map is used to indicate the positions of the key points of the target human body included in the crowd image.

In a possible implementation manner, the determining the target neighborhood corresponding to the initial human body key point based on the position of the initial human body key point in the crowd image includes: for any one of the initial human body key points point, based on the position of the initial human body key point in the crowd image and the preset perspective mapping relationship, determine the target neighborhood corresponding to the initial human body key point, wherein the preset perspective mapping relationship uses at image scales indicating different locations in the crowd image.

In a possible implementation manner, the initial human body key points are initial human head key points.

In a possible implementation manner, the determining the target neighborhood corresponding to the initial human body key point based on the position of the initial human body key point in the crowd image and a preset perspective mapping relationship includes : Based on the preset perspective mapping relationship, determine the target image scale corresponding to the position of the initial human head key point in the crowd image; based on the target image scale, determine the head frame height corresponding to the initial human head key point ; Based on the height of the head frame corresponding to the initial key point of the human head, determine the target neighborhood corresponding to the initial key point of the human head.

In a possible implementation manner, the determining the target neighborhood corresponding to the initial head key point based on the head frame height corresponding to the initial head key point includes: when the head frame height is greater than a preset In the case of the head frame height threshold, the target neighborhood is determined based on the first neighborhood radius; or, when the head frame height is less than or equal to the preset head frame height threshold, based on the second neighborhood radius, to determine the target neighborhood, wherein the radius of the first neighborhood is greater than the radius of the second neighborhood.

In a possible implementation manner, the positioning of the key points of the human body on the crowd image to obtain the initial positioning map corresponding to the crowd image includes: positioning the key points of the human body on the crowd image, and determining the corresponding The predicted positioning map, wherein the predicted positioning map is used to indicate the prediction confidence that the pixels in the crowd image are key points of the human body; based on the preset confidence threshold, image processing is performed on the predicted positioning map to obtain The initial positioning map.

In a possible implementation manner, the filtering of the initial positioning map based on the target neighborhood corresponding to the initial human key point to obtain the target positioning map includes: for any initial human head key point i , determine whether there is at least one other initial head key point in the target neighborhood corresponding to the initial head key point i; if there is at least one other initial head point j in the target neighborhood, based on the predicted location Figure, determining the prediction confidence corresponding to the initial head point i, and the prediction confidence corresponding to the at least one other initial head point j; based on the initial head key point i and the at least one other initial head key point j In , predict the initial head key point with the highest confidence, and determine the target head key point in the target neighborhood.

In a possible implementation manner, the method further includes: based on the target positioning map and the preset perspective mapping relationship, determining positions of key points of the target person's feet included in the crowd image.

In a possible implementation manner, the determining the positions of the key points of the target's feet included in the crowd image based on the target positioning map and the preset perspective mapping relationship includes: for any one of the targets For the key points of the human head, according to the target positioning map, determine the first image coordinates of the key points of the target human head in the crowd image; based on the preset perspective mapping relationship, perform coordinate transformation on the first image coordinates, Obtaining the second image coordinates of the target human foot key points corresponding to the target human head key points in the crowd image.

In a possible implementation manner, based on the preset perspective mapping relationship, coordinate transformation is performed on the coordinates of the first image to obtain key points of the target human feet corresponding to the key points of the target human head in the crowd image. The second image coordinates in include: based on the preset perspective mapping relationship, determining the target image scale corresponding to the key point of the target person's head; based on the target image scale, determining the relationship between the target person's head key point and the target person An image distance between foot key points; according to the first image coordinates and the image distance, determine the second image coordinates of the target person's foot key points in the crowd image.

In a possible implementation manner, the method further includes: acquiring a plurality of annotated human body frames obtained by annotating human body frames of pedestrians in different positions in the crowd image; based on the plurality of annotated human body frames, determining the Default perspective mapping relationship.

In a possible implementation manner, the determining the preset perspective mapping relationship based on the plurality of labeled human body frames includes: for any one of the labeled human body frames, determining a reference human body in the labeled human body frame The reference image scale corresponding to the key point; according to the third image coordinates of the reference human key point in the labeled human body frame, and the reference image scale corresponding to the reference human key point in the labeled human body frame, Fitting obtains the preset perspective mapping relationship.

According to an aspect of the present disclosure, a crowd positioning device is provided, including: a human body key point positioning module, configured to perform human body key point positioning on a crowd image, and obtain an initial positioning map corresponding to the crowd image, wherein the initial The positioning map is used to indicate the position of the initial human body key points included in the crowd image; the target neighborhood determination module is used to determine the initial human body key points based on the positions of the initial human body key points in the crowd image Corresponding target neighborhood; a filtering module, configured to filter the initial positioning map based on the target neighborhood corresponding to the initial human key point to obtain a target positioning map, wherein the target positioning map is used to indicate The positions of key points of the target human body included in the crowd image.

According to an aspect of the present disclosure, there is provided an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to call the instructions stored in the memory to execute the above-mentioned method.

According to one aspect of the present disclosure, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented.

In the embodiment of the present disclosure, the key points of the human body are located on the crowd image, and the initial positioning map indicating the position of the initial key points of the human body included in the crowd image can be obtained end-to-end. Position, determine the target neighborhoods corresponding to different initial human key points, and then use target neighborhoods of different sizes that match different initial human key points to filter the initial positioning map to reduce the number of initial human body key points corresponding to the same human body. The false detection probability of the point is obtained, and the target positioning map with high accuracy is obtained.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of drawings

The accompanying drawings here are incorporated into the description and constitute a part of the present description. These drawings show embodiments consistent with the present disclosure, and are used together with the description to explain the technical solution of the present disclosure.

FIG. 1 shows a flow chart of a crowd positioning method according to an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of a crowd image and its corresponding preset perspective mapping relationship according to an embodiment of the present disclosure;

Fig. 3 shows a block diagram of a crowd locating device according to an embodiment of the present disclosure;

Fig. 4 shows a block diagram of an electronic device according to an embodiment of the present disclosure;

FIG. 5 shows a block diagram of another electronic device according to an embodiment of the present disclosure.

Detailed ways

Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the figures indicate functionally identical or similar elements. While various aspects of the embodiments are shown in drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

The word "exemplary" is used exclusively herein to mean "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or better than other embodiments.

The term "and/or" in this article is just an association relationship describing associated objects, which means that there can be three relationships, for example, A and/or B can mean: A exists alone, A and B exist simultaneously, and there exists alone B these three situations. In addition, the term "at least one" herein means any one of a variety or any combination of at least two of the more, for example, including at least one of A, B, and C, which may mean including from A, Any one or more elements selected from the set formed by B and C.

In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific implementation manners. It will be understood by those skilled in the art that the present disclosure may be practiced without some of the specific details. In some instances, methods, means, components and circuits that are well known to those skilled in the art have not been described in detail so as to obscure the gist of the present disclosure.

Rapid crowd analysis is of great significance for public safety and urban planning. Common crowd analysis tasks include crowd counting, crowd behavior analysis, crowd positioning, etc. Among them, crowd positioning is the basis of other crowd analysis tasks. The accuracy of crowd positioning directly affects the accuracy of crowd counting and the results of crowd behavior analysis. Specifically, crowd positioning is a technology for estimating the positions of key points of the human body in the picture through computer vision algorithms for the video or image in the detection scene, and finally the coordinates of the key points of the human body in the picture can be obtained, which can be used for subsequent crowds. Analysis tasks such as counting and group behavior analysis provide basic data basis.

Common crowd localization methods rely on object detection algorithms. Transform the crowd positioning task into head target detection, and finally use the center of the head detection frame as the positioning result of the key points of the head. Crowd positioning algorithm based on head target detection: On the one hand, the training of the head target detection model based on convolutional neural network relies on a large amount of labeled data. For the task of crowd positioning, a large number of head frame labels are required. For very dense crowd scenes, the The cost of labeling the head frame is high; on the other hand, due to the small number of distant heads in crowded scenes, the labeled head frame is not accurate, which will affect the training of the head target detection model; in addition, the high performance of related technologies Most human head target detection frameworks are two-stage detection, that is, firstly extract features from the image through the pre-trained feature extraction network, and then send the extracted features to the ROI (Region Of Interest) network to obtain candidate areas that may have human heads , and then perform ROI pooling/ROI alignment on the candidate area to map the two-dimensional features into a fixed-length feature vector, and finally send the feature vector to two neural networks for classification and position regression respectively to obtain the detection result. The above-mentioned detection process is relatively complicated, and it is obviously not practical for crowd analysis with high real-time requirements.

In addition to the crowd positioning method based on head target detection, there is also a crowd positioning algorithm that directly generates a target positioning map. This method reduces the limitations of the head target detection algorithm: input the original crowd image, and directly End-to-end outputs an object localization map of the same size as the original crowd image. In the target localization map, the key point of the target head is represented by 1, otherwise it is 0. The crowd count metric is obtained by summing the object localization maps.

Then, the convolutional neural network can only output the initial positioning map, and image post-processing needs to be performed on the initial positioning map to obtain the final target positioning map. Filtering (for example, non-maximum suppression processing NMS) operations are required during image post-processing. In related technologies, a neighborhood with a fixed neighborhood radius is used for filtering. In this way, it is easy to cause false detection of multiple key points of the same human head in places with relatively large heads, or missed detection in places with relatively small heads.

An embodiment of the present disclosure provides a method for crowd positioning, which can be applied to crowd positioning in a dense scene. The human body key point positioning is performed on the crowd images collected in dense scenes, and the initial positioning map used to indicate the position of the initial human body key points included in the crowd image can be obtained end-to-end, and then based on the position of the initial human body key points in the crowd image , determine different target neighborhoods corresponding to different initial human key points, and then filter multiple initial human key points in the initial positioning map using target neighborhoods of different sizes to reduce the number of initial human key points corresponding to the same human body. The probability of false detection is higher, and the target positioning map with higher accuracy is obtained.

The crowd positioning method provided by the embodiments of the present disclosure will be described in detail below.

Fig. 1 shows a flow chart of a crowd locating method according to an embodiment of the present disclosure. The crowd positioning method can be performed by electronic devices such as terminal equipment or servers, and the terminal equipment can be user equipment (User Equipment, UE), mobile equipment, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (Personal Digital Assistant, PDA), handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc., the crowd positioning method can be realized by calling the computer-readable instructions stored in the memory by the processor. Alternatively, the crowd locating method can be executed by a server. As shown in Figure 1, the crowd positioning method may include:

In step S11 , the human body key points are located on the crowd image to obtain an initial positioning map corresponding to the crowd image, wherein the initial positioning map is used to indicate the position of the initial human body key points included in the crowd image.

The crowd image here is an image containing dense crowds, which can be obtained after the image acquisition device collects images of dense crowds within a certain spatial range, or it can be a key image frame containing dense crowds obtained from a video, or It may be obtained by other means, which is not specifically limited in the present disclosure.

Human body key points are located on the crowd image, and an initial positioning map used to indicate the position of the initial human body key points included in the crowd image is obtained end-to-end. For example, the head key point location is performed on the crowd image, and an initial positioning map used to indicate the position of the initial head key point included in the crowd image is obtained end-to-end. The specific process of head key point positioning will be described in detail later in combination with possible implementations of the present disclosure, and will not be repeated here.

In step S12, based on the positions of the initial human body key points in the crowd image, the target neighborhood corresponding to the initial human body key points is determined.

For the initial human key points included in the crowd image indicated by the initial positioning map, based on the positions of the initial human key points in the crowd image, the size of the target neighborhood for subsequent image processing of the initial human key points can be determined. The process of determining the target neighborhood corresponding to the initial human key point based on the position of the initial human key point in the crowd image will be described in detail later in combination with possible implementations of the present disclosure, and will not be repeated here.

In step S13, based on the target neighborhood corresponding to the initial key points of the human body, the initial positioning map is filtered to obtain the target positioning map, wherein the target positioning map is used to indicate the position of the key points of the target human body included in the crowd image.

According to different target neighborhoods corresponding to different initial human key points, the initial positioning map is filtered to obtain a target positioning map with high accuracy. The process of filtering the initial positioning map based on the target neighborhood corresponding to the initial key points of the human body will be described in detail later in combination with possible implementations of the present disclosure, and will not be repeated here.

In a possible implementation, the target neighborhood corresponding to the initial human key point is determined based on the position of the initial human key point in the crowd image, including: for any initial human key point, based on the position of the initial human key point in the crowd image The position in and the preset perspective mapping relationship determine the target neighborhood corresponding to the initial human body key point, wherein the preset perspective mapping relationship is used to indicate the image scale of different positions in the crowd image.

The image scale can be the number of pixel rows required to represent a unit height in the real world at a location in the crowd image. The unit height can be flexibly set according to actual conditions, for example, the unit height can be 1 meter, which is not specifically limited in the present disclosure.

In the crowd image, the image scale corresponding to the nearby pedestrians is large, and the image scale corresponding to the distant pedestrians is small. For example, for two pedestrians A and B with the same height of 1.7 meters, the number of pixel rows required by pedestrian A near the crowd image is p1, and the number of pixel rows required by pedestrian B far away from the crowd image is p2. Among them, p1>P2.

"Far" here refers to the distance between the real pedestrian corresponding to the pedestrian in the crowd image and the image acquisition device that collects the crowd image. The distance between the image acquisition devices is short.

The preset perspective mapping relationship may indicate image scales corresponding to different positions in the crowd image. The process of determining the preset perspective mapping relationship will be described in detail later in combination with possible implementations of the present disclosure, and will not be repeated here.

After determining the preset perspective mapping relationship corresponding to the crowd image, for any initial human key point indicated by the initial positioning map, the initial human key point can be determined based on the position of the initial human key point in the crowd image and the preset perspective mapping relationship. The image scale corresponding to the point, and then based on the image scale corresponding to the initial human key point, the target neighborhood corresponding to the initial human key point is determined.

Based on the position of the initial human key points in the crowd image, in addition to determining the image scale corresponding to the initial human key points based on the preset perspective mapping relationship to determine the target neighborhood corresponding to the initial human key points, it can also be directly based on the initial human key points The position of the point in the crowd image and the preset neighborhood radius corresponding to different positions in the preset crowd image are used to determine the target neighborhood corresponding to the initial key point of the head. Other methods of determining the target area can also be used. The implementation of this disclosure The example does not specifically limit this.

In a possible implementation, the crowd positioning method further includes: acquiring a plurality of marked human body frames obtained by marking pedestrians in different positions in the crowd image; determining a preset perspective mapping relationship based on the multiple marked human body frames .

Select pedestrians in different positions in the crowd image for human body frame labeling, and multiple labeled human body frames in the crowd image can be obtained. Based on the proportional relationship between the height of the marked human body frame and the actual height of the pedestrian, it can be determined The image scale corresponding to the limited position in the crowd image (where the body frame is marked), and then, based on the image scale corresponding to the limited position, further fitting is performed to effectively obtain the image scale corresponding to each position in the crowd image, that is, the preset perspective map is obtained relation.

Fig. 2 shows a schematic diagram of a crowd image and its corresponding preset perspective mapping relationship according to an embodiment of the present disclosure. As shown in Figure 2, pedestrians in different positions in the crowd image are selected for human body frame labeling, and four marked human body frames A, B, C, and D in different positions in the crowd image are obtained. Then, based on the four Label the human body frames A, B, C, and D, and fit them to effectively obtain the preset perspective mapping relationship corresponding to the crowd image.

In a possible implementation, determining the preset perspective mapping relationship based on multiple labeled human body frames includes: for any labeled human body frame, determining the reference image scale corresponding to the key points of the reference human body in the labeled human body frame; The third image coordinates of the key points of the reference human body marked in the human body frame and the reference image scale corresponding to the key points of the reference human body marked in the human body frame are fitted to obtain a preset perspective mapping relationship.

Along the column direction of the crowd image, the image scales at different positions change linearly. Therefore, after determining the image scales corresponding to the key points of the reference human body at limited positions in the crowd image according to the labeled human frame, it can be fitted by a linear function to effectively obtain The image scale corresponding to each position in the crowd image is to obtain the preset perspective mapping relationship corresponding to the crowd image.

Since pedestrians stand upright, the key points of human feet are used as reference key points of the human body, and the height of the marked human frame can be regarded as the height of the pedestrian in the crowd image. The height of the labeled human body frame can be expressed by the number of pixel rows occupied by the labeled human body frame. For example, if the labeled human frame occupies 17 rows of pixels in the crowd image, the height of the labeled human frame is 17. Assuming that the real height of the pedestrian corresponding to the marked human body frame is 1.7 meters, the position of the key point of the reference human foot in the marked human body frame can be determined, indicating that 1.7 meters in the real world requires 17 rows of pixels. Assuming that the unit height is 1m, therefore, at the position of the key points of the reference human feet in the body frame, it means that 1 meter in the real world requires 10 lines of pixels, that is, the scale of the reference image corresponding to the key points of the reference human feet in the body frame is 10. The real height of the pedestrian corresponding to the marked body frame can be selected as an appropriate value according to the actual situation, which is not specifically limited in the present disclosure.

Marking the key points of the reference human feet in the human body frame may be marking the midpoint of the bottom edge of the human body frame, or marking other pixel points in the human body frame, which is not specifically limited in the present disclosure.

Still taking the above-mentioned Figure 2 as an example, after the four marked human body frames A, B, C, and D are marked in the crowd image, the reference image scale corresponding to the reference human foot key points in each marked human body frame is determined according to the above method. Furthermore, a linear function fitting is performed according to the third image coordinates of the key points of the reference human feet in the four labeled human body frames and the corresponding reference image scales to obtain a linear mapping function p=a*y+b.

Wherein, the image coordinates refer to the position coordinates in the pixel coordinate system of the crowd image. For example, take the upper left corner of the crowd image as the coordinate origin (0, 0), the row direction parallel to the image is the x-axis direction, and the column direction parallel to the image is the y-axis direction to construct the pixel coordinate system of the crowd image, the image The units of the abscissa and ordinate of the coordinates are pixels. For example, if the image coordinates of the key points of the reference human feet are (10, 15), it indicates that the key points of the reference human feet are the pixel points at the 10th row and the 15th column in the crowd image.

The linear mapping function p=a*y+b is a functional representation of the preset perspective mapping relationship corresponding to the crowd image. Among them, a and b are the parameters obtained by linear function fitting, y is the vertical coordinate of the image coordinates of different positions in the crowd image, and p is the image scale corresponding to the position. Using the linear mapping function p=a*y+b, the image scale corresponding to each position in the crowd image can be determined.

Before or after determining the preset perspective mapping relationship corresponding to the crowd image, the key points of the human body are located on the crowd image to obtain an initial positioning map corresponding to the crowd image.

Due to the severe occlusion between the bodies of different pedestrians in dense crowd images, determining the key points of the head as the key points of the human body can effectively distinguish different pedestrians and improve the accuracy of crowd positioning.

The key point of the human head may be the center point of the human head, or other preset key points of the human head, which is not specifically limited in the present disclosure.

In a possible implementation, the human body key point positioning is performed on the crowd image to obtain the initial positioning map corresponding to the crowd image, including: performing head key point positioning on the crowd image, and determining a predicted positioning map corresponding to the crowd image, wherein, the predicted The positioning map is used to indicate the prediction confidence that the pixels in the crowd image are the key points of the head; based on the preset reliability threshold, image processing is performed on the predicted positioning map to obtain the initial positioning map.

Carry out head key point positioning on the crowd image, determine the prediction confidence of each pixel in the crowd image as the key point of the head end-to-end, and then perform threshold segmentation on the predicted positioning map by setting the reliability threshold to determine the initial information included in the crowd image The location of key points of the human head.

In an example, the trained human head key point positioning neural network can be used to perform head key point positioning on crowd images. Specifically, the crowd image is input into the trained human head key point positioning neural network, and the predicted positioning map is directly output after the positioning of the human head key point positioning neural network. The specific network structure and training process of the trained human head key point positioning neural network can adopt the network structure and training process in related technologies, which is not specifically limited in this disclosure.

In an example, the pixel value of each pixel in the predicted positioning map represents the prediction confidence of the pixel, that is, the probability that the pixel is a key point of a human head. A sigmoid operation is performed on the predicted positioning map, so that the pixel value of each pixel in the predicted positioning map is between 0 and 1. For example, if the pixel value of a certain pixel in the predicted localization map is 0.7, it means that the probability that the pixel is a key point of a human head is 0.7.

Since the predicted localization map is only used to indicate the prediction confidence that each pixel in the crowd image is a key point of the head, by presetting the confidence threshold, the predicted localization map is thresholded, so that it can be effectively used to indicate the crowd image. An initial localization map including the positions of the initial head keypoints. The specific value of the preset reliability threshold can be flexibly set according to the actual situation, which is not specifically limited in the present disclosure.

Compare the pixel value of each pixel in the predicted positioning map with the preset reliability threshold, and if the pixel value of a certain pixel in the predicted positioning map is greater than or equal to the preset reliability threshold, the relative The pixel value of the pixel at the same position is determined to be 1; when the pixel value of a certain pixel in the predicted positioning map is less than the preset reliability threshold, the pixel value of the pixel at the same relative position in the initial positioning map is determined as 0.

The initial positioning map and the crowd image have the same size, and the position of a pixel with a pixel value of 1 in the initial positioning map is used to indicate the position of the initial head key point included in the crowd image. For example, in the case where the pixel value of the pixel point whose image coordinates are (x, y) in the initial positioning map is 1, it can be determined that the pixel point whose image coordinates are (x, y) in the crowd image is the initial key point of the head; If the pixel value of the pixel with image coordinates (x, y) in the initial positioning image is 0, it can be determined that the pixel with image coordinates (x, y) in the crowd image is a part other than the initial key point of the head.

After determining the positions of the initial head key points included in the crowd image, the target neighborhood corresponding to each initial head key point may be determined based on a preset perspective mapping relationship.

In a possible implementation, based on the position of the initial human body key point in the crowd image and the preset perspective mapping relationship, the target neighborhood corresponding to the initial human body key point is determined, including: based on the preset perspective mapping relationship, determining the initial The target image scale corresponding to the position of the head key point in the crowd image; based on the target image scale, determine the head frame height corresponding to the initial head key point; based on the head frame height corresponding to the initial head key point, determine the target corresponding to the initial head key point Area.

Based on the preset perspective mapping relationship, the scale of the target image corresponding to the initial key points of the head can be quickly determined, and then the height of the head frame corresponding to the initial key points of the head at different positions can be determined, so that the matching target can be further determined according to the height of the head frame Area.

For example, for a certain initial head key point i in the initial positioning map, the image coordinates of the initial head key point i in the crowd image are (h _x , h _y ), then according to the preset perspective mapping relationship corresponding to the crowd image ( The linear mapping function p=a*y+b), it can be determined that the target scale corresponding to the initial key point i of the head is p _i =a*h _y +b. Assuming that the real head frame height corresponding to the pedestrian in the crowd image is 0.4m*0.4m, then the head frame height corresponding to the initial key point i of the head in the crowd image is s _i =0.4*p _i . According to the height of the head frame corresponding to the initial head key point is s _i =0.4*p _i , determine the target neighborhood matching its size.

In a possible implementation, based on the head frame height corresponding to the initial head key point, determining the target neighborhood corresponding to the initial head key point includes: when the head frame height is greater than the preset head frame height threshold, based on the first A neighborhood radius to determine the target neighborhood; or, when the height of the head frame is less than or equal to the preset head frame height threshold, based on the second neighborhood radius, determine the target neighborhood, wherein the first neighborhood radius is greater than the first Second neighborhood radius.

When the height of the head frame is greater than the preset height threshold of the head frame, it can be determined that the size of the head frame is relatively large, so a larger first neighborhood radius is used for subsequent filtering processing; when the height of the head frame is less than or equal to the preset When the height threshold of the head frame is set, it can be determined that the size of the head frame is relatively small, so a smaller second neighborhood radius is used for subsequent filtering processing. By flexibly determining the neighborhood radius, the accuracy of the filtering operation can be improved. The specific values of the preset head frame height threshold, the first neighborhood radius, and the second neighborhood radius can be flexibly set according to actual conditions, and this disclosure does not specifically limit them.

In one example, the height threshold of the head frame is 32. For an initial key point i, if the corresponding head frame height s _i >32, the target neighborhood is determined based on the first neighborhood radius 2; In the case of the corresponding head frame height s _i <= 32, the target neighborhood is determined based on the second neighborhood radius 1.

In the case where the first preset neighborhood radius is 2, the target neighborhood corresponding to the initial key point i of the human head includes pixels whose pixel distance from the initial key point i of the human head does not exceed 2 pixels. In the case that the second preset neighborhood radius is 1, the target neighborhood corresponding to the initial human head key point i includes pixels whose pixel distance from the initial human head key point i does not exceed 1 pixel point.

After determining the target neighborhood corresponding to each initial head key point in the initial positioning map based on the above method, use the target neighborhood corresponding to each initial head key point to filter the initial positioning map to obtain a target with high accuracy Location map.

In a possible implementation, based on the target neighborhood corresponding to the initial key points of the human body, the initial positioning map is filtered to obtain the target positioning map, including: for any initial key point i of the human head, determine the initial key point i In the corresponding target neighborhood, whether there is at least one other initial head key point; in the case of at least one other initial head point j in the target neighborhood, determine the prediction confidence corresponding to the initial head point i according to the predicted positioning map, and at least The prediction confidence corresponding to one other initial head point j; among the initial head key point i and at least one other initial head key point j, the initial head key point with the highest prediction confidence is determined as the target head key point in the target neighborhood.

In order to reduce the false detection probability of the same human head corresponding to multiple initial head key points, the initial localization map is filtered by further using the target neighborhood of the initial human head key points to obtain a high-precision target localization map.

For any initial head key point i in the initial positioning map, after determining its corresponding target neighborhood according to the above method, detect whether there are other initial head key points in the target neighborhood, if there are other image coordinates (x _j , y _j ), the prediction confidence corresponding to the initial head key point i and the prediction confidence corresponding to the initial head point j are determined according to the predicted positioning map. When the prediction confidence of the initial head key point i is greater than the prediction confidence of the initial head key point j, the pixel value of the pixel whose image coordinates are (xi _, y _i ) is kept as 1, and the image coordinates are (x _j , y _j ) The pixel value of the pixel point is updated to 0, that is, the initial head key point j in the initial positioning map is filtered out. By analogy, each initial head key point in the initial positioning map is traversed to obtain the final target positioning map.

In a possible implementation manner, the crowd positioning method further includes: based on the target positioning map and the preset perspective mapping relationship, determining the positions of the key points of the target person's feet included in the crowd image.

When analyzing the position of pedestrians in the detection system, it is generally necessary to know the position of the person's feet. For example, in the case of passenger traffic statistics, the line is generally drawn in advance according to the actual set position, and then it is judged whether to cross the line according to the positional relationship of the human feet in the front and rear frames. Therefore, in the crowd positioning task, how to accurately locate the position of human feet is very important.

After the target positioning map is determined based on the aforementioned method, the position of the key points of the target person's head included in the crowd image can be determined, and then based on the target positioning map and the preset perspective mapping relationship, the key points of the target person's feet included in the crowd image can be further quickly determined s position.

In a possible implementation, based on the target positioning map and the preset perspective mapping relationship, determining the position of the key point of the target person's foot included in the crowd image includes: for any key point of the target person's head, according to the target positioning map, determine The first image coordinates of the key points of the target person's head in the crowd image; based on the preset perspective mapping relationship, coordinate conversion is performed on the first image coordinates to obtain the first position of the key points of the target's feet corresponding to the key points of the target person's head in the crowd image Two image coordinates.

Since the preset perspective mapping relationship can indicate the image scales of different positions in the crowd image, based on the preset perspective mapping relationship, the coordinate conversion of the first image coordinates of the key points of the target head can be performed to obtain the target corresponding to the key points of the target head The second image coordinates of the key points of the human feet in the crowd image.

For example, for a key point of a target's head in the target location map, the first image coordinates of the key point of the target's head in the crowd image are (h _x , h _y ), and the first image coordinates are known. According to the preset perspective mapping relationship corresponding to the crowd image, coordinate transformation can be performed on the first image coordinates (h _x , h _y ), and the second image coordinates of the target human foot key points corresponding to the target human head key points are (f _x , f _y ).

In a possible implementation manner, based on the preset perspective mapping relationship, coordinate conversion is performed on the coordinates of the first image to obtain the second image coordinates of the key points of the target human feet corresponding to the key points of the target human head in the crowd image, including: based on Preset the perspective mapping relationship to determine the target image scale corresponding to the key points of the target person's head; determine the image distance between the key points of the target person's head and the key points of the target person's feet based on the target image scale; determine the target image according to the first image coordinates and image distance The second image coordinates of the key points of the human feet in the crowd image.

Since pedestrians stand upright, it can be determined that f _x =h _x . According to the preset perspective mapping relationship (linear mapping function p=a*y+b), it can be determined that the target scale corresponding to the key point of the target human foot is a*f _x +b. Assuming that in the real world, the distance from the center of the head to the feet of the person is 1.5 meters, that is, the real distance between the key points of the target person’s head and the key points of the target person’s feet is 1.5 meters, then the image of the key points of the target person’s feet and the key points of the target person’s head in the crowd image The distance is 1.5*(a*f _x +b), that is, h _x =f _y -1.5*(a*f _x +b), and then it can be determined that f _y =(h _y +1.5*b)/(1-1.5 *a), therefore, the second image coordinates of key points of the target human feet in the crowd image are (h _x , (h _y +1.5*b)/(1-1.5*a)).

By traversing each key point of the target's head in the target location map, the position of the key point of the target's foot corresponding to each key point of the target's head in the crowd image can be determined. Furthermore, according to the positions of the key points of the target person's feet in the adjacent and preceding frames of crowd images, crowd behavior analysis such as passenger flow statistics and behavior trajectory analysis can be performed, which is not specifically limited in this disclosure.

It can be understood that the above-mentioned method embodiments mentioned in this disclosure can all be combined with each other to form a combined embodiment without violating the principle and logic. Due to space limitations, this disclosure will not repeat them. Those skilled in the art can understand that, in the above method in the specific implementation manner, the specific execution order of each step should be determined according to its function and possible internal logic.

In addition, the present disclosure also provides crowd locating devices, electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any of the crowd locating methods provided in the present disclosure. For the corresponding technical solutions and descriptions, refer to the corresponding records in the method section ,No longer.

Fig. 3 shows a block diagram of a crowd locating device according to an embodiment of the present disclosure. As shown in Figure 3, the device 30 includes:

The human body key point positioning module 31 is configured to perform human body key point positioning on the crowd image to obtain an initial positioning map corresponding to the crowd image, wherein the initial positioning map is used to indicate the position of the initial human key point included in the crowd image;

The target neighborhood determination module 32 is used to determine the target neighborhood corresponding to the initial human body key point based on the position of the initial human body key point in the crowd image;

The filtering module 33 is configured to filter the initial positioning map based on the target neighborhoods corresponding to the initial key points of the human body to obtain the target positioning map, wherein the target positioning map is used to indicate the position of the key points of the target human body included in the crowd image.

In a possible implementation manner, the target neighborhood determination module 32 is specifically used for:

For any initial human body key point, based on the position of the initial human body key point in the crowd image and the preset perspective mapping relationship, determine the target neighborhood corresponding to the initial human body key point, wherein the preset perspective mapping relationship is used to indicate the crowd image Image scales at different locations in .

In a possible implementation, the target neighborhood determination module 32 includes:

The first determination sub-module is used to determine the target image scale corresponding to the position of the initial key point of the head in the crowd image based on the preset perspective mapping relationship;

The second determination submodule is used to determine the height of the head frame corresponding to the initial key point of the head based on the scale of the target image;

The third determination sub-module is configured to determine the target neighborhood corresponding to the initial key point of the human head based on the height of the head frame corresponding to the initial key point of the human head.

In a possible implementation manner, the third determination submodule is specifically used for:

When the height of the head frame is greater than the preset height threshold of the head frame, based on the first neighborhood radius, determine the target neighborhood; or,

When the height of the head frame is less than or equal to the preset height threshold of the head frame, the target neighborhood is determined based on the second neighborhood radius, wherein the first neighborhood radius is greater than the second neighborhood radius.

In a possible implementation, the human body key point positioning module 31 includes:

The human body key point positioning sub-module is used to locate the key points of the human body on the crowd image, and determine the predicted positioning map corresponding to the crowd image, wherein the predicted positioning map is used to indicate the prediction confidence that the pixel in the crowd image is the key point of the human body;

The fourth determination sub-module is configured to perform image processing on the predicted positioning map based on a preset reliability threshold to obtain an initial positioning map.

In a possible implementation manner, the filtering module 33 is specifically used for:

For any initial head key point i, determine whether there is at least one other initial head key point in the target neighborhood corresponding to the initial head key point i;

In the case where there is at least one other initial head point j in the target neighborhood, based on the predicted positioning map, determine the prediction confidence corresponding to the initial head point i, and the prediction confidence corresponding to at least one other initial head point j;

Based on the initial head key point i and at least one other initial head key point j, the initial head key point with the highest prediction confidence is determined to determine the target head key point in the target neighborhood.

In a possible implementation manner, the device 30 further includes:

The human foot key point determination module is configured to determine the position of the target human foot key points included in the crowd image based on the target positioning map and the preset perspective mapping relationship.

In a possible implementation, the human foot key point determination module includes:

The fifth determination sub-module is used to determine the first image coordinates of the key points of the target head in the crowd image according to the target positioning map for any key point of the target head;

The sixth determination sub-module is used to perform coordinate transformation on the first image coordinates based on the preset perspective mapping relationship, and obtain the second image coordinates of the key points of the target human feet corresponding to the key points of the target human head in the crowd image.

In a possible implementation manner, the sixth determination submodule is specifically used for:

Based on the preset perspective mapping relationship, determine the target image scale corresponding to the key points of the target head;

Based on the target image scale, determine the image distance between the key points of the target person's head and the key points of the target person's feet;

According to the first image coordinates and the image distance, the second image coordinates of the key points of the target person's feet in the crowd image are determined.

In a possible implementation manner, the device 30 further includes:

An acquisition module, configured to acquire a plurality of annotated human body frames obtained by annotating human body frames of pedestrians in different positions in the crowd image;

The perspective mapping relationship determination module is configured to determine a preset perspective mapping relationship based on a plurality of labeled human body frames.

In a possible implementation manner, the perspective mapping relationship determination module is specifically used for:

For any one marked human body frame, determine the reference image scale corresponding to the key points of the reference human body in the marked human body frame;

According to the third image coordinates of the key points of the reference human body marked in the frame of the human body, and the scale of the reference image corresponding to the key points of the reference human body in the frame of the marked human body, a preset perspective mapping relationship is obtained by fitting.

This method has a specific technical relationship with the internal structure of the computer system, and it can solve the technical problems of how to improve the hardware computing efficiency or execution effect (including reducing the amount of data storage, reducing the amount of data transmission, increasing the processing speed of the hardware, etc.), so as to obtain a natural The technical effect of regular computer system internal performance improvements.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the method embodiments above, and its specific implementation can refer to the description of the method embodiments above. For brevity, here No longer.

Embodiments of the present disclosure also provide a computer-readable storage medium, on which computer program instructions are stored, and the above-mentioned method is implemented when the computer program instructions are executed by a processor. Computer readable storage media may be volatile or nonvolatile computer readable storage media.

An embodiment of the present disclosure also proposes an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

An embodiment of the present disclosure also provides a computer program product, including computer-readable codes, or a non-volatile computer-readable storage medium carrying computer-readable codes, when the computer-readable codes are stored in a processor of an electronic device When running in the electronic device, the processor in the electronic device executes the above method.

Electronic devices may be provided as terminals, servers, or other forms of devices.

Fig. 4 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Referring to FIG. 4, the electronic device 800 may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle Devices, wearable devices and other terminal equipment.

Referring to FIG. 4, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, and a communication component 816 .

The processing component 802 generally controls the overall operations of the electronic device 800, such as those associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802 .

The memory 804 is configured to store various types of data to support operations at the electronic device 800 . Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like. The memory 804 can be implemented by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable Programmable Read Only Memory (EPROM), Programmable Read Only Memory (PROM), Read Only Memory (ROM), Magnetic Memory, Flash Memory, Magnetic or Optical Disk.

The power supply component 806 provides power to various components of the electronic device 800 . Power components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for electronic device 800 .

The multimedia component 808 includes a screen providing an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense a boundary of a touch or swipe action, but also detect duration and pressure associated with the touch or swipe action. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the electronic device 800 is in operation modes, such as call mode, recording mode and voice recognition mode. Received audio signals may be further stored in memory 804 or sent via communication component 816 . In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: a home button, volume buttons, start button, and lock button.

Sensor assembly 814 includes one or more sensors for providing status assessments of various aspects of electronic device 800 . For example, the sensor component 814 can detect the open/closed state of the electronic device 800, the relative positioning of components, such as the display and the keypad of the electronic device 800, the sensor component 814 can also detect the electronic device 800 or a Changes in position of components, presence or absence of user contact with electronic device 800 , electronic device 800 orientation or acceleration/deceleration and temperature changes in electronic device 800 . Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access wireless networks based on communication standards, such as wireless networks (Wi-Fi), second-generation mobile communication technologies (2G), third-generation mobile communication technologies (3G), fourth-generation mobile communication technologies (4G ), the long-term evolution (LTE) of the universal mobile communication technology, the fifth generation mobile communication technology (5G) or their combination. In an exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, electronic device 800 may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable A programmable gate array (FPGA), controller, microcontroller, microprocessor or other electronic component implementation for performing the methods described above.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as the memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to implement the above method.

This disclosure relates to the field of augmented reality. By acquiring the image information of the target object in the real environment, and then using various visual correlation algorithms to detect or identify the relevant features, states and attributes of the target object, and thus obtain the image information that matches the specific application. AR effect combining virtual and reality. Exemplarily, the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms may involve visual positioning, SLAM, 3D reconstruction, image registration, background segmentation, object key point extraction and tracking, object pose or depth detection, etc. Specific applications can not only involve interactive scenes such as guided tours, navigation, explanations, reconstructions, virtual effect overlays and display related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effect display, virtual Interactive scenarios such as model display. The relevant features, states and attributes of the target object can be detected or identified through the convolutional neural network. The above-mentioned convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

FIG. 5 shows a block diagram of another electronic device according to an embodiment of the present disclosure. Referring to FIG. 5, an electronic device 1900 may be provided as a server or terminal device. Referring to FIG. 5 , electronic device 1900 includes processing component 1922 , which further includes one or more processors, and a memory resource represented by memory 1932 for storing instructions executable by processing component 1922 , such as application programs. The application programs stored in memory 1932 may include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input-output (I/O) interface 1958 . The electronic device 1900 can operate based on the operating system stored in the memory 1932, such as the Microsoft server operating system (Windows Server ^TM ), the graphical user interface-based operating system (Mac OS X ^TM ) introduced by Apple Inc., and the multi-user and multi-process computer operating system (Unix ^™ ), a free and open source Unix-like operating system (Linux ^™ ), an open source Unix-like operating system (FreeBSD ^™ ), or the like.

In an exemplary embodiment, there is also provided a non-transitory computer-readable storage medium, such as the memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to implement the above method.

The present disclosure can be a system, method and/or computer program product. A computer program product may include a computer readable storage medium having computer readable program instructions thereon for causing a processor to implement various aspects of the present disclosure.

A computer readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. A computer readable storage medium may be, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or flash memory), static random access memory (SRAM), compact disc read only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded device, such as a printer with instructions stored thereon A hole card or a raised structure in a groove, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., pulses of light through fiber optic cables), or transmitted electrical signals.

Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to a respective computing/processing device, or downloaded to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or a network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .

Computer program instructions for performing the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or Source or object code written in any combination, including object-oriented programming languages—such as Smalltalk, C++, etc., and conventional procedural programming languages—such as the “C” language or similar programming languages. Computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement. In cases involving a remote computer, the remote computer can be connected to the user computer through any kind of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (such as via the Internet using an Internet service provider). connect). In some embodiments, an electronic circuit, such as a programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA), can be customized by utilizing state information of computer-readable program instructions, which can Various aspects of the present disclosure are implemented by executing computer readable program instructions.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It should be understood that each block of the flowcharts and/or block diagrams, and combinations of blocks in the flowcharts and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine such that when executed by the processor of the computer or other programmable data processing apparatus , producing an apparatus for realizing the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause computers, programmable data processing devices and/or other devices to work in a specific way, so that the computer-readable medium storing instructions includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks in flowcharts and/or block diagrams.

It is also possible to load computer-readable program instructions into a computer, other programmable data processing device, or other equipment, so that a series of operational steps are performed on the computer, other programmable data processing device, or other equipment to produce a computer-implemented process , so that instructions executed on computers, other programmable data processing devices, or other devices implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, a portion of a program segment, or an instruction that includes one or more Executable instructions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or they may sometimes be executed in the reverse order, depending upon the functionality involved. It should also be noted that each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by a dedicated hardware-based system that performs the specified function or action , or may be implemented by a combination of dedicated hardware and computer instructions.

The computer program product can be specifically realized by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) etc. wait.

The above descriptions of the various embodiments tend to emphasize the differences between the various embodiments, the same or similar points can be referred to each other, and for the sake of brevity, details are not repeated herein.

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

If the technical solution of this application involves personal information, the product applying the technical solution of this application has clearly notified the personal information processing rules and obtained the individual's independent consent before processing personal information. If the technical solution of this application involves sensitive personal information, the products applying the technical solution of this application have obtained individual consent before processing sensitive personal information, and at the same time meet the requirements of "express consent". For example, at a personal information collection device such as a camera, a clear and prominent sign is set up to inform that it has entered the scope of personal information collection, and personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed to agree to the collection of his personal information; or On the personal information processing device, when the personal information processing rules are informed with obvious signs/information, personal authorization is obtained through pop-up information or by asking individuals to upload their personal information; among them, the personal information processing rules may include Information such as the information processor, the purpose of personal information processing, the method of processing, and the type of personal information processed.

Having described various embodiments of the present disclosure above, the foregoing description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and alterations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principle of each embodiment, practical application or improvement of technology in the market, or to enable other ordinary skilled in the art to understand each embodiment disclosed herein.

Claims

A crowd positioning method, characterized by comprising:

Perform human body key point positioning on the crowd image to obtain an initial positioning map corresponding to the crowd image, wherein the initial positioning map is used to indicate the position of the initial human key point included in the crowd image;

Determining a target neighborhood corresponding to the initial human body key point based on the position of the initial human body key point in the crowd image;

Based on the target neighborhood corresponding to the initial human body key points, filter the initial positioning map to obtain a target positioning map, wherein the target positioning map is used to indicate the target human body key points included in the crowd image s position.
The method according to claim 1, wherein the determining the target neighborhood corresponding to the initial human key point based on the position of the initial human key point in the crowd image comprises:

For any one of the initial human body key points, based on the position of the initial human body key points in the crowd image and the preset perspective mapping relationship, determine the target neighborhood corresponding to the initial human body key points, wherein, The preset perspective mapping relationship is used to indicate image scales of different positions in the crowd image.
The method according to claim 1 or 2, wherein the initial human body key points are initial human head key points.
The method according to claim 3, wherein the target corresponding to the initial human body key point is determined based on the position of the initial human body key point in the crowd image and a preset perspective mapping relationship neighborhood, including:

Based on the preset perspective mapping relationship, determine the target image scale corresponding to the position of the initial human head key point in the crowd image;

Based on the target image scale, determine the height of the head frame corresponding to the initial key point of the head;

Based on the height of the head frame corresponding to the initial key point of the human head, the target neighborhood corresponding to the initial key point of the human head is determined.
The method according to claim 4, wherein the determining the target neighborhood corresponding to the initial key point of the human head based on the height of the head frame corresponding to the initial key point of the human head comprises:

In the case where the head frame height is greater than a preset head frame height threshold, based on a first neighborhood radius, determine the target neighborhood; or,

In the case that the head frame height is less than or equal to the preset head frame height threshold, the target neighborhood is determined based on a second neighborhood radius, wherein the first neighborhood radius is greater than the second neighborhood radius domain radius.
According to the method described in any one of claims 3 to 5, it is characterized in that the positioning of the key points of the human body on the crowd image to obtain the initial positioning map corresponding to the crowd image includes:

Perform human body key point positioning on the crowd image, and determine a predicted positioning map corresponding to the crowd image, wherein the predicted positioning map is used to indicate the prediction confidence that the pixels in the crowd image are key points of the human body;

Based on a preset reliability threshold, image processing is performed on the predicted positioning map to obtain the initial positioning map.
The method according to claim 6, wherein the initial positioning map is filtered based on the target neighborhood corresponding to the initial key points of the human body to obtain a target positioning map, comprising:

For any initial head key point i, determine whether there is at least one other initial head key point in the target neighborhood corresponding to the initial head key point i;

In the case where there is at least one other initial head point j in the target neighborhood, based on the predicted positioning map, determine the prediction confidence corresponding to the initial head point i, and the prediction confidence corresponding to the at least one other initial head point j prediction confidence;

Based on the initial head key point i and the at least one other initial head key point j with the highest prediction confidence, the target head key point in the target neighborhood is determined.
The method according to any one of claims 3 to 7, wherein the method further comprises:

Based on the target positioning map and the preset perspective mapping relationship, the positions of the key points of the target person's feet included in the crowd image are determined.
The method according to claim 8, wherein, based on the target positioning map and the preset perspective mapping relationship, determining the position of the key points of the target person's feet included in the crowd image includes:

For any key point of the target human head, according to the target positioning map, determine the first image coordinates of the key point of the target human head in the crowd image;

Based on the preset perspective mapping relationship, coordinate transformation is performed on the first image coordinates to obtain second image coordinates of target human foot key points corresponding to the target human head key points in the crowd image.
The method according to claim 9, characterized in that, based on the preset perspective mapping relationship, coordinate conversion is performed on the first image coordinates to obtain the key points of the target human feet corresponding to the key points of the target human head. The second image coordinates in the crowd image include:

Based on the preset perspective mapping relationship, determine the target image scale corresponding to the key points of the target head;

Based on the target image scale, determine the image distance between the key points of the target human head and the key points of the target human feet;

According to the first image coordinates and the image distance, second image coordinates of key points of the target person's feet in the crowd image are determined.
The method according to any one of claims 2 to 10, further comprising:

Obtaining multiple annotated human body frames obtained by annotating human body frames of pedestrians in different positions in the crowd image;

The preset perspective mapping relationship is determined based on the plurality of marked human body frames.
The method according to claim 11, wherein the determining the preset perspective mapping relationship based on the plurality of marked human body frames comprises:

For any one of the labeled human body frames, determine the reference image scale corresponding to the key points of the reference human body in the labeled human body frame;

The preset perspective is obtained by fitting according to the third image coordinates of the key points of the reference human body in the marked human body frame and the scale of the reference image corresponding to the key points of the reference human body in the marked human body frame. Mapping relations.
A crowd positioning device, characterized in that it includes:

The human body key point positioning module is configured to perform human body key point positioning on the crowd image to obtain an initial positioning map corresponding to the crowd image, wherein the initial positioning map is used to indicate the initial human body key points included in the crowd image Location;

A target neighborhood determination module, configured to determine a target neighborhood corresponding to the initial human body key point based on the position of the initial human body key point in the crowd image;

A filtering module, configured to filter the initial positioning map based on the target neighborhood corresponding to the initial key points of the human body to obtain a target positioning map, wherein the target positioning map is used to indicate that the crowd image includes The location of key points of the target human body.
An electronic device, characterized in that it comprises:

processor;

memory for storing processor-executable instructions;

Wherein, the processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1-12.
A computer-readable storage medium on which computer program instructions are stored, wherein the computer program instructions implement the method according to any one of claims 1 to 12 when executed by a processor.
A computer program product, comprising computer readable codes, or a non-volatile computer readable storage medium bearing computer readable codes, when the computer readable codes are run in a processor of an electronic device, the electronic The processor in the device executes the method for realizing any one of claims 1-12.