WO2023168957A1 - Pose determination method and apparatus, electronic device, storage medium, and program - Google Patents

Abstract

Embodiments of the present disclosure relate to a pose determination method and apparatus, an electronic device, a storage medium, and a program. The method comprises: determining at least one first feature point in an image to be recognized on a target object; determining at least one target image from among pre-stored images according to the target object in the image, wherein the target image has a second feature point and a corresponding three-dimensional point cloud, and the second feature point has a corresponding point in the three-dimensional point cloud; determining a target point of the at least one first feature point in the three-dimensional point cloud according to the at least one first feature point, the second feature point, and the point corresponding to the second feature point in the three-dimensional point cloud; and according to the target point, determining a target pose corresponding to the image. In the embodiments of the present disclosure, a three-dimensional feature point corresponding to each two-dimensional feature point in an image to be recognized can be determined by means of local feature matching, so that by means of the pose of an image collection apparatus when the image is collected according to the three-dimensional feature point, the accuracy of a determined target pose can be improved.

Description

Attitude determination method, device, electronic equipment, storage medium and program

Cross-references to related applications

This disclosure requires the priority of the Chinese patent application number 202210239443.0 submitted on March 11, 2022. The applicant is Zhejiang Shangtang Technology Development Co., Ltd. and the application is titled "Attitude Determination Method and Device, Electronic Equipment and Storage Medium". This application The entire text of is incorporated into this disclosure by reference.

Technical field

The present disclosure relates to the field of computer technology, and in particular, to a posture determination method, device, electronic device, storage medium and program.

Background technique

One of the important uses of augmented reality technology is to interact with real objects in the real world and render virtual effects based on them. Accurately estimating and tracking the six-dimensional (6D) pose of an object is a prerequisite for interactive rendering and is also a very important research issue in the field of computer vision. Among them, the specific definition of the 6D posture of the object is three degrees of freedom of displacement plus three degrees of freedom of rotation. The posture estimation results of related technologies will have certain deviations when the background of the object changes. At the same time, it is difficult to distinguish the difference in posture of objects in the object images collected at two approximate position angles.

Contents of the invention

Embodiments of the present disclosure propose an attitude determination method, device, electronic device, storage medium and program, aiming to improve the accuracy of the determined target attitude.

Embodiments of the present disclosure provide a gesture determination method, including: determining at least one first feature point on a target item in an image to be recognized; determining at least one target image from a pre-stored image based on the target item in the image to be recognized. , wherein the target image has a second feature point and a corresponding three-dimensional point cloud, and the second feature point has a corresponding point in the three-dimensional point cloud; according to the at least one first feature point, the third Two feature points, and the corresponding point of the second feature point in the three-dimensional point cloud, determine the target point of the at least one first feature point in the three-dimensional point cloud; determine the target point based on the target point The target pose corresponding to the image to be recognized.

An embodiment of the present disclosure provides a gesture determination device, including: a first determination part configured to determine at least one first feature point on a target item in an image to be recognized; a second determination part configured to determine according to the The target item in the image to be recognized determines at least one target image from the pre-stored image, wherein the target image has a second feature point and a corresponding three-dimensional point cloud, and the second feature point has a corresponding corresponding value in the three-dimensional point cloud. points; the target point matching part is configured to determine the at least one first feature point, the second feature point, and the corresponding point of the second feature point in the three-dimensional point cloud. A first feature point is a target point in the three-dimensional point cloud; the posture determination part is configured to determine the target posture corresponding to the image to be recognized based on the target point.

An embodiment of the present disclosure provides an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call instructions stored in the memory to perform the above-mentioned gesture determination. method.

Embodiments of the present disclosure provide a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the above-mentioned posture determination method is implemented.

An embodiment of the present disclosure provides a computer program. The computer program includes a computer readable code. When the computer readable code is run in an electronic device, the processor of the electronic device executes the above gesture. Determine the method.

In the embodiment of the present disclosure, first, through local feature matching, the two-dimensional second feature point corresponding to each two-dimensional first feature point in the image to be recognized in the pre-stored image is determined, and then, based on the corresponding two-dimensional second feature point with each first feature point, The three-dimensional feature point corresponding to the second feature point matched by the feature point determines the posture of the image acquisition device when collecting the image to be recognized, which can improve the accuracy of the determined target posture.

It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not restrictive of the disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings.

Description of the drawings

The accompanying drawings herein are incorporated into and constitute a part of this specification. They illustrate embodiments consistent with the disclosure and, together with the description, serve to explain the technical solutions of the disclosure.

Figure 1A is a schematic diagram of determining the six-dimensional posture of an object using a geometry-based method in the related art;

Figure 1B is a schematic diagram of determining the six-dimensional posture of an object based on template matching direct regression method in related technologies;

Figure 2 shows a flow chart of a gesture determination method according to an embodiment of the present disclosure;

Figure 3 shows a schematic diagram of a three-dimensional point cloud according to an embodiment of the present disclosure;

Figure 4 shows a schematic diagram of a second feature point matching process according to an embodiment of the present disclosure;

Figure 5 shows a schematic diagram of determining a target posture according to an embodiment of the present disclosure;

Figure 6A shows a schematic diagram of determining a reference posture according to an embodiment of the present disclosure;

Figure 6B shows a schematic diagram of determining the 6D posture of an object based on the posture determination method provided by the embodiment of the present application during practical application;

Figure 7 shows a schematic diagram of a gesture determination device according to an embodiment of the present disclosure;

Figure 8 shows a schematic diagram of an electronic device according to an embodiment of the present disclosure;

FIG. 9 shows a schematic diagram of another electronic device according to an embodiment of the present disclosure.

Detailed ways

Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numbers in the drawings identify functionally identical or similar elements. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless otherwise indicated.

The word "exemplary" as used herein means "serving as an example, example, or illustrative." Any embodiment described herein as "exemplary" is not necessarily to be construed as superior or superior to other embodiments.

The term "and/or" in this article is just an association relationship that describes related objects, indicating that three relationships can exist. For example, A and/or B can mean: A exists alone, A and B exist simultaneously, and they exist alone. B these three situations. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C, which can mean including from A, Any one or more elements selected from the set composed of B and C.

In addition, in order to better explain the present disclosure, numerous specific details are given in the following detailed description. It will be understood by those skilled in the art that the present disclosure may be practiced without certain specific details. In some instances, methods, means, components and circuits that are well known to those skilled in the art are not described in detail in order to emphasize the subject matter of the disclosure.

In related technologies, object 6D posture estimation methods can be divided into methods based on template matching to directly return the 6D posture of the object and geometry-based methods according to their mechanism of action; among them, geometry-based methods: This type of method usually determines the coordinate system of the object itself. Define three-dimensional (3 Dimension, 3D) feature points in three-dimensional space, and mark the two-dimensional (2 Dimension, 2D) feature points corresponding to these 3D feature points on the images from each perspective of the object. These images and annotations will be used to train a 2D feature point detection neural network. In actual use, the neural network will detect the predefined 2D feature points of the object from the input image, and combine the known matching relationship between 2D feature points and 3D feature points through N-point perspective (Perspective-N-Points, PnP) The algorithm solves the 6D posture of the object; as shown in Figure 1A, it is a schematic diagram of the geometry-based method in related technologies to determine the six-dimensional posture of the object; among them, first the 2D feature points of the object are detected on the input image, that is, execution 101; secondly, In the input image, the image of the area near the object is cropped, and 102 is executed; again, the probability distribution of the object's 2D feature points is estimated for the image of the area near the object, that is, 103 is executed; then, based on the probability distribution of the object's 2D feature points, Estimate and determine the pixel position of the object's 2D feature points, which is executed in 104; finally, use the known relationship between 2D and 3D feature points to determine the 6D posture of the object through the PnP algorithm, which is executed in 105.

At the same time, in the method of directly returning the 6D posture of the object based on template matching: it is necessary to obtain the image of the object at each viewing angle and its corresponding 6D posture in advance; and generate a matching template by encoding the images from each viewing angle, or directly use the neural network The information of the 6D pose of the object is encoded into the network through a learning-based method. And in actual use, the 6D posture of the target object is directly output by matching the input image directly or indirectly with the template. Here, as shown in Figure 1B , it is a schematic diagram of determining the six-dimensional posture of an object based on the direct regression method of template matching in related technologies; wherein, as shown in Figure 1A above, first 2D feature point detection is performed on the input image, that is, step 106 is performed; Then, crop the image of the area near the object in the input image, that is, execute 107; finally, the cropped image of the area near the object is directly outputted with a learning-based method based on the neural network, which directly outputs the 6D pose of the object, that is, execute 108.

Here, the above two types of methods have obvious problems: 1. They are sensitive to the object background. During template generation or neural network training, it will inevitably be interfered by background information in the image. Therefore, new background effects may be affected during actual use; 2. Insensitive to subtle changes in object posture, it is impossible to estimate a sufficiently accurate object posture, and whether it is template matching or neural network method, it is essentially They all encode and memorize image information. Therefore, if the posture of the object in the input image during inference has not appeared in the previous training data generated, the algorithm will estimate it as the posture closest to the image in the training data, and thus it will not be able to estimate the posture accurately enough.

In order to solve the problems that arise in related technologies, the usual solution of existing methods is to collect a large amount of data in different backgrounds and cover the viewing angles as densely as possible. This will increase the cost of data collection and acquisition; and due to the limited memory capacity of the neural network itself, it cannot essentially solve the problem, thereby limiting the application scenarios of the object 6D pose estimation algorithm in augmented reality. In addition, the representation of existing object 6D pose estimation methods cannot be directly extended to 6D pose tracking algorithms. They usually can only smooth between detection frames by using filtering methods, or additionally use other 6D pose tracking methods (such as Contour tracking) to complete 6D posture tracking, that is, it is impossible to use a unified representation to complete 6D posture detection and tracking of objects.

The gesture determination method in the embodiment of the present disclosure can be executed by an electronic device such as a terminal device or a server. Among them, the terminal device can be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (Personal Digital Assistant, PDA), a handheld device, a computing device, a vehicle-mounted device, or a portable device. Fixed or mobile devices such as wearables. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the gesture determination method of the embodiment of the present disclosure by calling the computer readable instructions stored in the memory through the processor.

In some embodiments of the present disclosure, the embodiments of the present disclosure are used to accurately determine the time to collect the image to be identified based on a three-dimensional point cloud constructed from multiple pre-stored images including the target item after acquiring an image to be identified. The posture of the image acquisition device.

FIG. 2 shows a flow chart of a gesture determination method according to an embodiment of the present disclosure. As shown in Figure 2, the object posture method according to the embodiment of the present disclosure may include the following steps S10 to S40:

Step S10: Determine at least one first feature point on the target item in the image to be recognized.

In some embodiments of the present disclosure, the posture determination method of the embodiment of the present disclosure is used to determine the target posture corresponding to the two-dimensional image to be recognized, that is, the posture of the image acquisition device when collecting the image to be recognized. The image to be identified is determined by the way in which the image acquisition device collects the target item, which can be a single image acquired through separate acquisition, or a frame of an image sequence acquired continuously by the moving image acquisition device during the movement. After determining the image to be recognized, the electronic device determines at least one first feature point on the target item in the image to be recognized. Among them, the target object is a static object that does not change its posture during the image collection process, and can be an inanimate still life or an animate object that is fixed during the image collection process.

In some embodiments of the present disclosure, the determination process of the first feature point on the image to be recognized may be to first obtain the target item in the image through object recognition, and then determine at least one local area on the target item that is used to characterize the target item. the first characteristic point. Each first feature point may also have a first descriptor used to describe local features of the first feature point, that is, the first descriptor is used to characterize the local area features of the target item in the image to be recognized.

In some embodiments of the present disclosure, the first descriptor may be determined by extracting preset features at the location of the first feature point, and determining a feature vector as the first descriptor based on the distribution of the preset features. The preset characteristics can be set according to actual needs, and can include at least one of color characteristics and texture characteristics. The first descriptor can be any feature descriptor, for example, it can be a Histogram of Oriented Gradients (HOG) feature descriptor. The determination process of the first descriptor may be to determine the gradient change of each pixel in the horizontal and vertical directions within the location of the first feature point, and determine the amplitude and direction of each pixel based on the gradient change. Generate a histogram of gradient magnitudes and directions. At the same time, the feature vector obtained through histogram normalization can also be used as the first descriptor.

Step S20: Determine at least one target image from pre-stored images according to the target item in the image to be recognized.

In some embodiments of the present disclosure, after determining the target item in the image to be recognized, a three-dimensional point cloud of the target item and at least one target image may also be determined. Each pre-stored image is an image obtained by pre-collecting the target object through an image collection device, and can be stored in a database locally or remotely connected to the electronic device. The at least one target image determined by the electronic device may be all pre-stored images, or part of the pre-stored images.

In some embodiments of the present disclosure, the electronic device can obtain at least one target image from the pre-stored images according to any preset rules. For example, the electronic device can first determine the posture information corresponding to each pre-stored image, and determine the initial posture corresponding to the image to be recognized, and then determine at least one of the pre-stored images based on the initial posture of the image to be recognized and the posture information corresponding to the pre-stored image. target image. In this way, the image whose relationship with the posture of the target item in the image to be recognized satisfies the preset relationship can be determined in the prestored image; for example, the similarity between the posture of the target item in the prestored image and the image to be identified can be determined to satisfy the preset relationship. Let the similarity image be used as the target image; this provides a basis for subsequently determining the second feature point of the target item in the target image.

In some embodiments of the present disclosure, the initial posture corresponding to the image to be recognized can be determined according to any existing method; for example, when the image to be recognized is a frame in a continuously collected image sequence, it can be determined that the position in the image sequence to be recognized is The previous posture corresponding to the multiple frames of images before the image is the posture of the image acquisition device when collecting the previous multiple frames of images. The extrapolation method is performed based on multiple previous postures to obtain the initial posture corresponding to the image to be recognized. Alternatively, the image to be recognized can also be directly input into a pre-trained attitude estimation model, and the corresponding initial attitude can be output after the attitude estimation. The initial posture can be represented by a vector, including three displacement parameters and three rotation parameters of the image acquisition device in the target three-dimensional coordinate system when collecting the image to be recognized. In this way, by interpolating the previous postures corresponding to the multiple frames of images before the image to be recognized, the resulting initial posture corresponding to the image to be recognized is more accurate, and the initial posture and the previous posture are more coherent.

In some embodiments of the present disclosure, the initial pose corresponding to the image to be recognized and the pose information corresponding to each pre-stored image can be represented by vectors. Therefore, the poses of the image to be recognized and the pre-stored images can be directly determined by calculating the vector distance. Match the situation, and filter out the pre-stored images whose vector distance is smaller than the preset distance threshold as the target image.

In some embodiments of the present disclosure, the target image has a second feature point and a corresponding three-dimensional point cloud, and the second feature point of the target image has a corresponding point in the three-dimensional point cloud. At the same time, the three-dimensional point cloud of the target object can be predetermined based on at least one pre-stored image. Alternatively, after determining the target item in the image to be recognized, multiple pre-stored images including the target item are obtained through screening in the database, and then the three-dimensional point cloud of the target item is determined based on the multiple pre-stored images. In some embodiments, the three-dimensional point cloud of the target object can be generated by an electronic device based on at least one pre-stored image, or by other devices such as a server based on at least one pre-stored image. In the case where the electronic device generates a three-dimensional point cloud, the electronic device may generate the three-dimensional point cloud before executing the attitude determination method of the embodiment of the present disclosure, or generate the three-dimensional point cloud of the target item during the execution of the attitude determination method of the embodiment of the present disclosure. cloud.

In some embodiments of the present disclosure, when the three-dimensional point cloud of the target object is generated by an electronic device, the process of determining the three-dimensional point cloud of the target object based on multiple pre-stored images may first determine at least one pre-stored image and obtain at least one of the pre-stored images. second feature points; secondly, match the second feature points of each pre-stored image to obtain multiple second feature point groups, wherein at least one second feature point in each second feature point group is used to characterize the target The same position on the item; finally, determine the three-dimensional point cloud corresponding to the pre-stored image based on the plurality of second feature point groups corresponding to the target item, wherein each point in the three-dimensional point cloud corresponds to at least one of the second feature point groups. Corresponds to the second feature point. That is to say, multiple pre-stored images can be screened through gesture screening to obtain an image similar to the posture of the object in the image to be recognized as the target image; in this way, the target image can be determined based on at least one pre-stored image through local feature matching. Identify the three-dimensional feature points corresponding to each two-dimensional feature point in the image; here, because each local feature point in the image only contains information about its nearby local area, compared with existing methods, it is more difficult to determine each two-dimensional feature point In the process of corresponding three-dimensional feature points, it is relatively less affected by the background and image perspective, which makes the three-dimensional feature points corresponding to the determined two-dimensional feature points more accurate.

Each pre-stored image may include a target item, and at least one second feature point is located on the target item in the corresponding pre-stored image, and the second feature point is used to characterize a local area of the target item. In some embodiments, each second feature point may also have a corresponding second descriptor for describing local features, that is, the second descriptor is used to characterize the local area features of the target item in the pre-stored image. Here, the second feature point matching process can be implemented based on the second descriptor, that is, the electronic device matches the second feature point of each prestored image according to the second descriptor of at least one second feature point of each prestored image. Multiple second feature point groups are obtained. At least one second feature point in each second feature point group is used to represent the same position on the target item. Then, a three-dimensional point cloud is determined based on a plurality of second feature point groups corresponding to the target item. Each point in the three-dimensional point cloud corresponds to at least one second feature point in a second feature point group.

In some embodiments of the present disclosure, the determination process of the plurality of second feature point groups may be to randomly select one of the plurality of pre-stored images as the target pre-stored image, and add each target in the target pre-stored image based on the second descriptor. The two feature points are respectively matched with other pre-stored images for local features, and a plurality of matched second feature points and the target feature point are combined to form a second feature point group. In some embodiments of the present disclosure, a pre-stored image in which an unmatched second feature point is located is re-determined as the target pre-stored image, and then the unmatched target second feature point is compared with other unmatched second feature points. Perform second descriptor matching on the second feature points to determine the second feature point group until all second feature points are matched, or the matching process of each second feature point has completed after all second feature points have been matched.

In some embodiments of the present disclosure, when every two second feature points are matched based on the second descriptor, the distance between the second descriptors of the two second feature points can be calculated to obtain the matching degree of the local features, The closer the distance, the higher the degree of matching. Among them, the second descriptor is a vector, and the distance between the two second descriptors can be obtained by directly performing an inner product on the vector. At the same time, the second feature point can also be matched through two parameters: local features and spatial location.

In some embodiments of the present disclosure, after obtaining multiple second feature point groups, the multiple second feature point groups can be solved to obtain a three-dimensional point cloud of the target item through a motion recovery structure algorithm. That is, the feature point tracking trajectory is determined based on the acquisition time of the image where the second feature point in each second feature point group is located, and a three-dimensional point of the target item is determined based on the multiple feature point tracking trajectories to form a three-dimensional point cloud. Therefore, each three-dimensional point in the three-dimensional point cloud corresponds to all the second feature points in a second feature point group. That is, during the construction process of the three-dimensional point cloud, it can be obtained that each three-dimensional point corresponds to at least one two-dimensional third feature point. The corresponding relationship between the two feature points; in this way, based on the motion recovery structure algorithm, the image acquisition device tracking and motion matching can be completed more flexibly, thereby making the obtained three-dimensional point cloud of the target object more accurate.

Figure 3 shows a schematic diagram of a three-dimensional point cloud according to an embodiment of the present disclosure. As shown in Figure 3, the three-dimensional point cloud 20 includes a plurality of three-dimensional points in the three-dimensional coordinate system, and each three-dimensional point has at least one corresponding two-dimensional second feature point.

FIG. 4 shows a schematic diagram of a second feature point matching process according to an embodiment of the present disclosure. As shown in FIG. 4 , the first pre-stored image 30 and the second pre-stored image 31 both include a target item 32 . The target item 32 in the first pre-stored image 30 and the second pre-stored image 31 each has a plurality of second feature points. When matching based on the corresponding second descriptor, the third point representing the same position of the target item 32 can be determined based on the local features. The two feature points are matched together.

In some embodiments of the present disclosure, each pre-stored image also has corresponding posture information, which is used to characterize the posture of the image acquisition device when collecting the pre-stored image. The posture information can be represented by a vector, including three displacement parameters and three rotation parameters of the image acquisition device in the target three-dimensional coordinate system when collecting the pre-stored image. The target three-dimensional coordinate system can be preset or selected.

In some embodiments of the present disclosure, the gesture information may be determined when the pre-stored image is acquired and stored together with the pre-stored image. Alternatively, the posture information can also be determined based on the corresponding three-dimensional points in the three-dimensional point cloud of the plurality of second feature points included in the three-dimensional point cloud after the three-dimensional point cloud is determined. That is to say, for each pre-stored image, an N-point perspective algorithm can be executed based on the correspondence between each second feature point included in the pre-stored image and the three-dimensional point in the three-dimensional point cloud, to obtain the posture information corresponding to the pre-stored image, that is, to collect each The posture of the image acquisition device when pre-storing images.

Step S30: Determine where the at least one first feature point is in the three-dimensional point cloud based on the at least one first feature point, the second feature point, and the corresponding point of the second feature point in the three-dimensional point cloud. Target points in a 3D point cloud.

In some embodiments of the present disclosure, after the electronic device determines at least one feature point of the target item in the image to be recognized and the second feature point of the image to be recognized corresponding to at least one target image, the electronic device can determine based on the plurality of first feature points, The second feature point and the corresponding point of the second feature point in the three-dimensional point cloud determine the target point of at least one first feature point in the three-dimensional point cloud. That is to say, the matching of the two-dimensional points and the three-dimensional points of the first feature point and the target point is indirectly achieved through the two-dimensional point matching of the first feature point and the second feature point.

In some embodiments of the present disclosure, the electronic device can perform feature point matching on the image to be recognized and the target image to obtain second feature points matching each first feature point. Then, the point corresponding to the second feature point matched by each first feature point in the three-dimensional point cloud is determined as the target point.

For example, in this embodiment of the present disclosure, the first feature point and the second feature point can be matched using descriptors. For example, the electronic device may perform feature point matching based on the first descriptor of each first feature point in the image to be recognized and the second descriptor of each second feature point in at least one target image to determine a plurality of first feature points. The second feature point matched by the feature point, and based on the corresponding relationship between the second feature point and the three-dimensional point in the target point cloud, the point corresponding to the first feature point in the three-dimensional point cloud is indirectly determined to be the target point. Optionally, according to the process of matching the first descriptor and the second descriptor to the second feature point corresponding to the first feature point, reference may be made to the process of matching the two second feature points in step S20.

In some embodiments of the present disclosure, for each first feature point in the image to be recognized, the closest third feature point can be determined by calculating the distance between the first descriptor and the second descriptor of the second feature point on each candidate image. Two feature points are matched, and the point corresponding to the second feature point matched by each first feature point in the three-dimensional point cloud is determined to be the target point.

Step S40: Determine the target posture corresponding to the image to be recognized according to the target point.

In some embodiments of the present disclosure, after determining the matching target point of each first feature point in the image to be recognized in the three-dimensional point cloud, it is determined that the position of the target item represented by each first feature point in the image to be recognized is at position in three-dimensional space.

In some embodiments of the present disclosure, the target posture corresponding to the image to be recognized can be determined based on the correspondence between at least one first feature point and each target point. For example, the target posture corresponding to the at least one first feature point in the three-dimensional point cloud can be determined. Perform N-point perspective algorithm to obtain the target posture corresponding to the image to be recognized. That is, based on the correspondence between multiple target points and the first feature point, a first-written equation can be constructed and solved to obtain the target posture.

Figure 5 shows a schematic diagram of determining a target posture according to an embodiment of the present disclosure. As shown in Figure 5, when it is necessary to determine the target posture corresponding to the image 41 to be recognized, posture matching is first performed based on the image 41 to be recognized and the pre-stored image 40, and at least one target image 42 is obtained from multiple pre-stored images 40, and then based on The first feature point in the image to be recognized 41 performs local feature matching with the second feature point in each target image 42 to obtain at least one second feature point that matches the first feature point as the matching feature point 43 . In some embodiments of the present disclosure, the point matched by each matching feature point 43 in the three-dimensional point cloud 44 is determined to be the target point 45, and then N is performed based on the target point 45 matched by each first feature point in the image 41 to be recognized. Point perspective algorithm is used to obtain the target pose 46 corresponding to the image to be recognized.

In some embodiments of the present disclosure, when the image to be recognized is a frame in a continuously collected image sequence, and it is necessary to determine the posture of the image acquisition device when each frame of image acquisition in the acquired image training, it can be based on the previous frame. The attitude of the image acquisition device during image acquisition determines the attitude of the image acquisition device during image acquisition of the current frame. For example, when acquiring a continuously collected image sequence, you can extract image frame 1 from the image sequence as the image to be recognized, and determine the 2nd to N (N is an integer greater than 2) frames in the image sequence as the reference image. After determining the corresponding relationship between each first feature point in the image to be recognized and the midpoint of the three-dimensional point cloud, determine a plurality of second features in the reference image based on the target point corresponding to each first feature point in the three-dimensional point cloud. point the corresponding reference point in the three-dimensional point cloud, and determine the reference pose of each reference image frame based on the correspondence between the third feature point and the reference point. In some embodiments of the present disclosure, since the reference poses determined multiple times are all based on accurate target poses, as the reference image becomes farther and farther away from the image to be recognized in the image sequence, the accuracy will become lower and lower. . In order to avoid the above problems, after determining that the Nth image frame is the reference image, the N+1th frame is extracted as the next image to be recognized. The target posture of the current image to be recognized is determined through the posture determination method of the embodiment of the present disclosure, so as to continue to determine the reference posture of the N+2 to N+N frames through a transfer method.

In some embodiments of the present disclosure, the process of determining the reference pose of the adjacent next frame reference image based on the target pose of the image to be recognized may include the following steps: first determine the next frame image of the image to be recognized in the image sequence as The reference image includes at least one third feature point on the target item. According to the target point corresponding to each first feature point on the image to be recognized, and according to the target point corresponding to each first feature point on the image to be recognized, the target point corresponding to each third feature point on the reference image is determined. Then, the reference posture corresponding to the reference image is determined based on the corresponding relationship between each third feature point and the target point. Among them, the corresponding relationship between each third feature point and the target point on the reference image can be determined based on the sparse optical flow algorithm, that is, according to the sparse optical flow algorithm, each first feature point on the image to be identified is tracked to obtain each first feature point. The third feature point matched on the reference image. The third feature point matching each first feature point in the reference image is determined to correspond to the target point corresponding to the first feature point in the three-dimensional point cloud. In some embodiments of the present disclosure, an N-view algorithm is executed based on each third feature point and the corresponding target point to obtain the reference pose corresponding to the reference image. At the same time, after determining the reference pose of the current reference image, the reference pose of the next frame image in the image sequence can be determined sequentially based on the correspondence between the third feature point and the target point in the current reference image.

FIG. 6A shows a schematic diagram of determining a reference posture according to an embodiment of the present disclosure. As shown in FIG. 6A , for the image 51 to be identified determined in the image sequence 50 , it can be determined based on local feature matching that each first feature point in the image 51 to be identified is in the target image 52 filtered from the pre-stored images. Matched second feature point 53. In some embodiments of the present disclosure, it is determined that the corresponding three-dimensional point in the three-dimensional point cloud 54 of the second feature point 53 matched by each first feature point is the target point 55, and the correspondence between the first feature point and the target point 55 is determined. The relationship determines the target pose 56 of the image to be recognized.

In some embodiments of the present disclosure, for the reference image 57 at the next frame position of the image 51 to be recognized in the image sequence 50 , the reference image 57 may be determined based on the image 51 to be recognized, the target point 55 and the reference image 57 based on a tracking algorithm. Reference posture. In some embodiments of the present disclosure, the corresponding relationship between each first feature point in the image to be recognized 51 and each third feature point in the reference image 57 can be determined according to a sparse optical flow algorithm, and based on each first feature point The corresponding relationship with the target point 55 determines the corresponding relationship between each third feature point and the target point 55 . The reference pose 58 of the reference image 57 is determined according to the corresponding relationship between each third feature point and the target point 55 .

Figure 6B shows a schematic diagram of determining the 6D posture of an object based on the posture determination method provided by the embodiment of the present application in the actual application process; as shown in Figure 6B, based on the posture determination method provided by the embodiment of the present application, that is, a method based on local feature point matching The prerequisite for estimating the object pose is to construct an object point cloud map and associate the local feature points on the data image with the objects in the three-dimensional world; as shown in Figure 6B, for a frame of input image 59, first, through the initial pose estimation, Find data images 510 with similar viewing angles among multiple pre-stored images; then, perform 2D feature point extraction and matching between the data image 510 and the input image 59 to obtain each 2D feature point in the input image 59, in the data image The matched 2D feature points in 510 determine the 2D-2D matching relationship between the input image 58 and the data image 510, as shown in 511 in Figure 6B; at the same time, by performing a motion recovery structure algorithm on the data image 510, the corresponding The object point cloud map 512; finally, based on the 2D-2D matching relationship between the input image 58 and the data image 510, and the object point cloud map 512, determine the relationship between the 2D feature points extracted in the input image 59 and the object point cloud map 512 The corresponding 2D-3D matching relationship is constructed, as shown in 513 in Figure 6B, and the 2D-3D matching relationship 513 is solved based on PnP to obtain the object pose 514 in the input image 59, that is, the object 6D attitude.

The embodiments of the present disclosure can determine the corresponding relationship between each two-dimensional feature point in the image to be identified and each two-dimensional feature point in the target image through local feature matching, and generate a three-dimensional point cloud based on multiple target images, thereby obtaining the target image. Identify the correspondence between each two-dimensional feature point in the image and the three-dimensional feature point in the three-dimensional point cloud to accurately determine the posture of the image acquisition device when collecting the image to be recognized, which can improve the accuracy of the determined target posture. At the same time, embodiments of the present disclosure can also perform preliminary screening of target images by estimating the preliminary posture of the image to be recognized, which can reduce the calculation amount of the two-dimensional feature point matching process, thereby improving the efficiency of the posture determination process.

The posture determination method provided by the embodiment of the present disclosure can determine the target posture corresponding to the image to be recognized based on at least one first feature point on the target item in the image to be recognized. Here, since the at least one first feature point usually only includes other Therefore, compared with existing methods, it is relatively less affected by the background and image perspective. Therefore, in actual use, only a small amount of training data is needed to generalize to different scenes. At the same time, for an input image, there are many local feature points that can be extracted, which results in higher fault tolerance and redundancy when solving the 6D posture corresponding to the image to be recognized. In addition, in the embodiment of the present disclosure, for object 6D posture detection and tracking, the same set of 2D-3D matching representations is used, which is more concise, elegant and efficient at the algorithm framework level. The two modules can support and help each other.

In some embodiments of the present disclosure, when determining the posture of multiple continuously collected images, the three-dimensional feature points corresponding to each two-dimensional feature point on the image to be recognized can be obtained through indirect matching through two-dimensional feature point matching. Determine the target attitude. Subsequently, the sparse optical flow algorithm is used to track and determine the correspondence between the two-dimensional feature points of the subsequent multi-frame images and the three-dimensional feature points on the image to be identified. In this way, the multi-frame images collected after the image to be identified can be quickly and accurately determined. reference attitude. At the same time, this method can also accurately perceive posture changes when the image collection position changes slightly.

It can be understood that the above-mentioned method embodiments mentioned in the embodiments of the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Those skilled in the art can understand that in the above-mentioned methods of specific embodiments, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, embodiments of the present disclosure also provide posture determination devices, electronic devices, computer-readable storage media, and programs, all of which can be used to implement any posture determination method provided by embodiments of the present disclosure. The corresponding technical solutions and descriptions and reference methods Part of the corresponding records.

FIG. 7 shows a schematic diagram of a gesture determination device according to an embodiment of the present disclosure. As shown in Figure 7, the attitude determination device according to the embodiment of the present disclosure includes:

The first determining part 60 is configured to determine at least one first feature point on the target item in the image to be recognized;

The second determination part 61 is configured to determine at least one target image from a pre-stored image according to the target item in the image to be recognized, wherein the target image has a second feature point and a corresponding three-dimensional point cloud, and the target The second feature point of the image has a corresponding point in the three-dimensional point cloud;

The target point matching part 62 is configured to determine the at least one first feature point according to the at least one first feature point, the second feature point, and the corresponding point of the second feature point in the three-dimensional point cloud. A target point of a feature point in the three-dimensional point cloud;

The posture determination part 63 is configured to determine the target posture corresponding to the image to be recognized according to the target point.

In some embodiments of the present disclosure, the device further includes: a pre-stored image determining part configured to determine at least one of the pre-stored images and obtain at least one second feature point of the pre-stored image; the first feature point matching The part is configured to match the second feature points of each of the pre-stored images to obtain a plurality of second feature point groups, wherein at least one second feature point in each of the second feature point groups Used to characterize the same position on the target item; the three-dimensional point matching part is configured to determine the three-dimensional point cloud corresponding to the pre-stored image according to the plurality of second feature point groups, wherein the three-dimensional point cloud Each point in the cloud corresponds to at least one second feature point in one of the second feature point groups.

In some embodiments of the present disclosure, the three-dimensional point matching part includes: a point cloud generation sub-part configured to obtain the target item by solving a plurality of the second feature point groups through a motion recovery structure algorithm. 3D point cloud.

In some embodiments of the present disclosure, the second determination part 61 includes: a first posture determination sub-part configured to determine posture information corresponding to each of the pre-stored images; a second posture determination sub-part configured to In order to determine the initial posture corresponding to the image to be recognized; the posture screening sub-section is configured to determine the at least one target image from the pre-stored image according to the initial posture and the posture information.

In some embodiments of the present disclosure, the first posture determination sub-part is further configured to, for each of the pre-stored images, determine at the three-dimensional point based on each of the second feature points included in the pre-stored image. The corresponding point in the cloud executes the N-point perspective algorithm to obtain the attitude information.

In some embodiments of the present disclosure, the image to be recognized is a frame in a continuously collected image sequence, and the second posture determination sub-part includes: a second posture determination part configured to determine the image sequence and a third posture determination part configured to perform an extrapolation method based on a plurality of the previous postures to obtain the initial posture.

In some embodiments of the present disclosure, the target point matching part 62 includes: a first feature point matching sub-part configured to perform feature point matching between the image to be recognized and the target image to obtain each a second feature point matched by the first feature point; a first target point matching sub-part configured to match the corresponding point of each second feature point matched by the first feature point in the three-dimensional point cloud , determined as the target point.

In some embodiments of the present disclosure, the posture determination part 63 includes: a third posture determination sub-part configured to perform an N-point perspective algorithm according to the target point to obtain the target posture corresponding to the image to be recognized.

In some embodiments of the present disclosure, the image to be recognized is a frame in a continuously collected image sequence, and the device further includes: a reference image determining part configured to determine whether the image to be recognized is in the image sequence. The next frame image in the image is used as a reference image, wherein the reference image includes at least one third feature point on the target item; the second feature point matching part is configured to match each image based on the image to be identified. The target point corresponding to the first feature point determines the target point corresponding to each third feature point in the reference image; the reference posture determination part is configured to determine the target point corresponding to each third feature point and the The corresponding relationship between the target points determines the reference posture corresponding to the reference image.

In some embodiments of the present disclosure, the second feature point matching part includes: a second feature point matching sub-part configured to track each of the first features in the image to be identified according to a sparse optical flow algorithm points to obtain third feature points matching each of the first feature points in the reference image; a second target point matching sub-part configured to determine that each of the first feature points matches in the reference image The matched third feature point corresponds to the target point corresponding to the first feature point in the three-dimensional point cloud.

In some embodiments, the functions or included parts of the device provided by the embodiments of the present disclosure can be used to perform the methods described in the above method embodiments. For specific implementation, reference can be made to the description of the above method embodiments.

Embodiments of the present disclosure also provide a computer-readable storage medium on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. Wherein, the computer-readable storage medium may be a volatile or non-volatile computer-readable storage medium.

An embodiment of the present disclosure also provides an electronic device, including: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to call instructions stored in the memory to execute the instructions provided by the above embodiments. attitude determination method.

Embodiments of the present disclosure also provide a computer program, the computer program including computer readable code, when the computer readable code is run in an electronic device, the processor of the electronic device executes the above implementation. The attitude determination method provided in the example.

Embodiments of the present disclosure also provide a computer program product, including computer readable code, or a non-volatile computer readable storage medium carrying the computer readable code. When the computer readable code is stored in a processor of an electronic device, When running, the processor in the electronic device executes the gesture determination method provided by the above embodiment.

The electronic device may be provided as a terminal, a server, or other forms of equipment.

FIG. 8 shows a schematic diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a PDA and other terminals.

Referring to Figure 8, the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (Input/Output, I/O) interface 812, Sensor component 814, and communication component 816.

Processing component 802 generally controls the overall operations of electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above method. Additionally, processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

Memory 804 is configured to store various types of data to support operations at electronic device 800 . Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (Electrically Erasable Programmable Read-Only Memory) Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (Electrical Programmable Read Only Memory, EPROM), Programmable Read-Only Memory (PROM), Read Only Memory (Read Only Memory, ROM), magnetic memory, flash memory, magnetic disk or optical disk.

Power supply component 806 provides power to various components of electronic device 800 . Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .

Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (Liquid Crystal Display, LCD) and a touch panel (TouchPanel, TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide action. In some embodiments, multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data. Each front-facing camera and rear-facing camera can be a fixed optical lens system or have a focal length and optical zoom capabilities.

Audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (Microphone, MIC). When the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. In some embodiments, the received audio signal may be stored in memory 804 or sent via communication component 816 . In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, etc. These buttons may include, but are not limited to: Home button, Volume buttons, Start button, and Lock button.

Sensor component 814 includes one or more sensors for providing various aspects of status assessment for electronic device 800 . For example, the sensor component 814 can detect the open/closed state of the electronic device 800, the relative positioning of components, such as the display and keypad of the electronic device 800, the sensor component 814 can also detect the electronic device 800 or an electronic device 800. The position of components changes, the presence or absence of user contact with the electronic device 800 , the orientation or acceleration/deceleration of the electronic device 800 and the temperature of the electronic device 800 change. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby items without any physical contact. Sensor assembly 814 may also include a light sensor, such as a Complementary Metal Oxide Semiconductor (CMOS) or Charge-coupled Device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2-Generation, 2G) or a third generation mobile communication technology (3-Generation, 3G), or they The combination. In one exemplary embodiment, the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be based on Radio Frequency Identification (RFID) technology, Infrared Data Association (IrDA) technology, Ultra Wideband (Ultra Wide Band, UWB) technology, Bluetooth (BitTorrent, BT) technology and other Technology to achieve.

In an exemplary embodiment, the electronic device 800 may be configured by one or more application specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processor (Digital Signal Processor, DSP), digital signal processing device (Digital Signal Processor Device). , DSPD), programmable logic device (Programmable Lofic Device, PLD), field programmable gate array (Field Programmable Gate Array, FPGA), controller, microcontroller, microprocessor or other electronic component implementation, used to perform the above method.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to complete the above method.

The present disclosure relates to the field of augmented reality. By acquiring image information of target items in the real environment, and then using various visual related algorithms to detect or identify the relevant characteristics, status and attributes of the target items, thereby obtaining information that matches specific applications. AR effect that combines virtuality and reality. For example, the target items may involve faces, limbs, gestures, actions, etc. related to the human body, or objects, markers, or sandboxes, display areas, or display items related to venues or places. Vision-related algorithms can involve visual positioning, SLAM, three-dimensional reconstruction, image registration, background segmentation, object key point extraction and tracking, object pose or depth detection, etc. Specific applications can not only involve interactive scenes such as tours, navigation, explanations, reconstructions, and virtual effects overlay displays related to real scenes or objects, but also involve special effects processing related to people, such as makeup beautification, body beautification, special effects display, virtual Model display and other interactive scenarios. Convolutional neural networks can be used to detect or identify the relevant features, status and attributes of target items. The above-mentioned convolutional neural network is a network model obtained through model training based on a deep learning framework.

FIG. 9 shows a schematic diagram of another electronic device 1900 according to an embodiment of the present disclosure. For example, electronic device 1900 may be provided as a server. Referring to FIG. 9 , electronic device 1900 includes a processing component 1922 , including one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, executable by processing component 1922 . The application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the above-described gesture determination method.

Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input-output (I/O) interface 1958 . The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as a Microsoft server operating system (Windows Server ^TM ), a graphical user interface operating system (Mac OS X ^TM ) launched by Apple, a multi-user multi-process computer operating system (Unix ^TM ), a free and open source Unix-like operating system (Linux ^TM ), an open source Unix-like operating system (FreeBSD ^TM ) or similar.

In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the above method.

The present disclosure may be a system, method, and/or computer program product. A computer program product may include a computer-readable storage medium having thereon computer-readable program instructions for causing a processor to implement aspects of the present disclosure.

Computer-readable storage media may be tangible devices that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: portable computer disk, hard disk, random access memory (RAM), ROM, EPROM or flash memory, SRAM, portable compact disk read-only Memory (Compact Disc-Read Only Memory, CD-ROM), Digital Versatile Disc (DVD), memory stick, floppy disk, mechanical encoding device, such as punched card or grooved embossed card with instructions stored on it structure, and any suitable combination of the above. As used herein, computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or through electrical wires. transmitted electrical signals.

Computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to various computing/processing devices, or to an external computer or external storage device over a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium in the respective computing/processing device .

Computer program instructions for performing operations of the present disclosure may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or in one or more Source code or object code written in any combination of programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server implement. In situations involving remote computers, the remote computer can be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or it can be connected to an external computer (e.g. Use an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, FPGA, or Programmable Logic Arrays (PLA), can be customized by utilizing state information of computer-readable program instructions. Program instructions are read to implement various aspects of the present disclosure.

Aspects of the present disclosure are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer-readable program instructions.

These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, thereby producing a machine that, when executed by the processor of the computer or other programmable data processing apparatus, , resulting in an apparatus that implements the functions/actions specified in one or more blocks in the flowchart and/or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium. These instructions cause the computer, programmable data processing device and/or other equipment to work in a specific manner. Therefore, the computer-readable medium storing the instructions includes An article of manufacture that includes instructions that implement aspects of the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.

Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other equipment, causing a series of operating steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executed on a computer, other programmable data processing apparatus, or other equipment to implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions that embody one or more elements for implementing the specified logical function(s). Executable instructions. In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two consecutive blocks may actually execute substantially in parallel, or they may sometimes execute in the reverse order, depending on the functionality involved. It will also be noted that each block of the block diagram and/or flowchart illustration, and combinations of blocks in the block diagram and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts. , or can be implemented using a combination of specialized hardware and computer instructions.

The computer program product can be implemented specifically through hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a Software Development Kit (SDK), etc. wait.

The above description of various embodiments is intended to emphasize the differences between the various embodiments, and the similarities or similarities may be referred to each other.

Those skilled in the art can understand that in the above-mentioned methods of specific embodiments, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The internal logic is determined.

If the disclosed technical solution involves personal information, the products applying the disclosed technical solution will clearly inform the personal information processing rules and obtain the individual's independent consent before processing personal information. If the disclosed technical solution involves sensitive personal information, the product applying the disclosed technical solution must obtain the individual's separate consent before processing the sensitive personal information, and at the same time meet the requirement of "express consent". For example, setting up clear and conspicuous signs on personal information collection devices such as cameras to inform them that they have entered the scope of personal information collection, and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed to have agreed to the collection of his or her personal information; or On personal information processing devices, when using obvious logos/information to inform personal information processing rules, obtain personal authorization through pop-up messages or asking individuals to upload their personal information; among them, personal information processing rules may include personal information processing rules. Information such as information processors, purposes of processing personal information, methods of processing, and types of personal information processed.

The embodiments of the present disclosure have been described above. The above description is illustrative, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Industrial applicability

Embodiments of the present disclosure relate to a gesture determination method, device, electronic equipment, storage medium and program, which determine at least one first feature point on a target object in an image to be recognized, and extract the pre-stored image from the pre-stored image according to the target object in the image to be recognized. Determine at least one target image, the target image has a second feature point and a corresponding three-dimensional point cloud, and the second feature point has a corresponding point in the three-dimensional point cloud; according to at least one first feature point, a second feature point, and a second The corresponding point of the feature point in the three-dimensional point cloud is determined, and the target point of at least one first feature point in the three-dimensional point cloud is determined, and the target posture corresponding to the image to be recognized is determined based on the target point.

Claims (14)

1. A posture determination method, the method includes:

   Determine at least one first feature point on the target item in the image to be recognized;

   At least one target image is determined from the pre-stored image according to the target item in the image to be recognized, wherein the target image has a second feature point and a corresponding three-dimensional point cloud, and the second feature point is in the three-dimensional point cloud. There are corresponding points in;

   According to the at least one first feature point, the second feature point, and the corresponding point of the second feature point in the three-dimensional point cloud, it is determined that the at least one first feature point is in the three-dimensional point cloud. target point in;

   The target posture corresponding to the image to be recognized is determined according to the target point.
2. The method of claim 1, further comprising:

   Determine at least one of the pre-stored images, and obtain at least one second feature point of the pre-stored image;

   Match the second feature points of each pre-stored image to obtain a plurality of second feature point groups, wherein at least one second feature point in each second feature point group is used to characterize the the same location on the target item;

   Determine a three-dimensional point cloud corresponding to the pre-stored image according to a plurality of second feature point groups, wherein each point in the three-dimensional point cloud corresponds to at least one second feature point group in one of the second feature point groups. Feature point correspondence.
3. The method according to claim 2, wherein determining the three-dimensional point cloud corresponding to the pre-stored image according to a plurality of the second feature point groups includes:

   Through a motion recovery structure algorithm, a plurality of second feature point groups are solved to obtain a three-dimensional point cloud of the target item.
4. The method according to any one of claims 1 to 3, wherein determining at least one target image from pre-stored images according to the target item in the image to be recognized includes:

   Determine the posture information corresponding to each of the pre-stored images;

   Determine the initial posture corresponding to the image to be recognized;

   The at least one target image is determined from the pre-stored image according to the initial posture and the posture information.
5. The method according to claim 4, wherein the determining the posture information corresponding to each of the pre-stored images includes:

   For each pre-stored image, an N-point perspective algorithm is performed based on the corresponding point of each second feature point included in the pre-stored image in the three-dimensional point cloud to obtain the posture information.
6. The method according to claim 4 or 5, wherein the image to be recognized is a frame in a continuously collected image sequence, and determining the initial posture corresponding to the image to be recognized includes:

   Determine the previous posture corresponding to the multiple frame images before the image to be recognized in the image sequence;

   An extrapolation method is performed based on a plurality of the previous postures to obtain the initial posture.
7. The method according to any one of claims 1 to 6, wherein the at least one first feature point, the second feature point, and the second feature point are in the three-dimensional point cloud. Corresponding points, determining the target point of the at least one first feature point in the three-dimensional point cloud includes:

   Match the feature points of the image to be recognized with the target image to obtain second feature points matching each of the first feature points;

   The point corresponding to the second feature point matched by each first feature point in the three-dimensional point cloud is determined as the target point.
8. The method according to any one of claims 1 to 7, wherein determining the target posture corresponding to the image to be recognized according to the target point includes:

   An N-point perspective algorithm is executed according to the target point to obtain the target posture corresponding to the image to be recognized.
9. The method according to any one of claims 1 to 8, wherein the image to be recognized is a frame in a continuously collected image sequence, and the method further includes:

   Determine the next frame image of the image to be recognized in the image sequence as a reference image, wherein the reference image includes at least one third feature point on the target item;

   Determine the target point corresponding to each third feature point in the reference image based on the target point corresponding to each first feature point in the image to be recognized;

   According to the corresponding relationship between each third feature point and the target point, the reference posture corresponding to the reference image is determined.
10. The method according to claim 9, wherein the target corresponding to each third feature point in the reference image is determined based on the target point corresponding to each first feature point in the image to be recognized. points, including:

    Track each first feature point in the image to be identified according to a sparse optical flow algorithm, and obtain a third feature point matched by each first feature point in the reference image;

    The third feature point matching each of the first feature points in the reference image is determined to correspond to the target point corresponding to the first feature point in the three-dimensional point cloud.
11. A posture determination device, the device includes:

    A first determining part configured to determine at least one first feature point on the target item in the image to be recognized;

    The second determination part is configured to determine at least one target image from a pre-stored image according to the target item in the image to be recognized, wherein the target image has a second feature point and a corresponding three-dimensional point cloud, and the second Feature points have corresponding points in the three-dimensional point cloud;

    The target point matching part is configured to determine the at least one first feature point according to the at least one first feature point, the second feature point, and the corresponding point of the second feature point in the three-dimensional point cloud. The target point of the feature point in the three-dimensional point cloud;

    The posture determination part is configured to determine the target posture corresponding to the image to be recognized according to the target point.
12. An electronic device including:

    processor;

    Memory used to store instructions executable by the processor;

    Wherein, the processor is configured to call instructions stored in the memory to execute the posture determination method according to any one of claims 1 to 10.
13. A computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the posture determination method according to any one of claims 1 to 10 is implemented.
14. A computer program, the computer program comprising computer readable code, when the computer readable code is run in an electronic device, the processor of the electronic device executes for implementing any one of claims 1 to 10 Described posture determination method.

Images