CN117724610A

CN117724610A - Data processing method and device for head display equipment, head-mounted equipment and medium

Info

Publication number: CN117724610A
Application number: CN202311720060.6A
Authority: CN
Inventors: 黄晓航; 贺江; 张双勇; 李启迪; 季春林
Original assignee: Guangdong Juhua New Display Research Institute
Current assignee: Guangdong Juhua New Display Research Institute
Priority date: 2023-12-13
Filing date: 2023-12-13
Publication date: 2024-03-19
Anticipated expiration: 2043-12-13
Also published as: CN117724610B

Abstract

The invention discloses a data processing method, a device, head-mounted equipment and a medium for head-mounted equipment, which are used for acquiring association relation information between plane point set information of the head-mounted equipment and space point set information corresponding to the plane point set information; candidate association relation information is determined according to the association relation information; and determining a matching result of the plane point set information and the space point set information according to the candidate association relation information. The invention can effectively improve the accuracy of positioning and matching.

Description

Data processing method and device for head display equipment, head-mounted equipment and medium

Technical Field

The invention relates to the technical field of positioning matching, in particular to a data processing method and device for head display equipment, head-mounted equipment and a medium.

Background

The infrared mark point positioning technology is widely applied to the fields of motion capturing, head display equipment and the like, and utilizes an external camera to capture a facula image of an infrared mark point on a target, combines the predefined relative spatial position information between the infrared mark points, and calculates the spatial position and the direction of the infrared mark point relative to the camera after matching the infrared mark point and the facula image with the predefined relative spatial position information, so that the target is positioned. However, the external camera may capture noise spots in addition to the spot of the infrared marker, which may affect the matching between the spot of the infrared marker and the predefined infrared marker, resulting in a mislocalization match.

Disclosure of Invention

The embodiment of the invention provides a data processing method, a device, equipment and a storage medium for head display equipment, which aim to effectively improve the accuracy of positioning matching.

In a first aspect, an embodiment of the present invention provides a data processing method for a head display device, where the method includes:

acquiring association relation information between plane point set information of head display equipment and space point set information corresponding to the plane point set information;

candidate association relation information is determined according to the association relation information;

and determining a matching result of the plane point set information and the space point set information according to the candidate association relation information.

Optionally, the determining, according to the candidate association relationship information, a matching result of the plane point set information and the space point set information includes:

obtaining target association relation information which is not repeatedly associated according to the candidate association information;

and determining a matching result of the plane point set information and the space point set information according to the target association relation information.

Optionally, the determining candidate association information according to the association information includes:

arranging the association relation information to obtain a plurality of arrangement information, and determining candidate matching results corresponding to the arrangement information;

Determining projection point set information of the space point set information according to each candidate matching result;

and determining the candidate association relation information according to the space points corresponding to the projection points and the plane points in the corresponding range of the projection points.

Optionally, the determining, according to the target association relationship information, a matching result of the plane point set information and the space point set information includes:

updating the matching score of the space point and the plane point in the target association relation information according to the target association relation information containing the space point aiming at each space point in the space point set information;

selecting a target plane point according to the matching score of the space point and the plane point;

updating the association relation information according to each space point and the target plane point corresponding to the space point;

and determining a matching result based on the updated association relation information.

Optionally, after updating the association information according to the target plane point corresponding to each spatial point and the spatial point, the method further includes:

if the update times of the association relation information is smaller than the preset update times, re-executing the step of determining candidate association relation information according to the association relation information based on the updated association relation information;

If the update times of the association relation information is greater than or equal to the preset update times, selecting a plurality of resolving association relation information from the updated association relation information;

and determining the matching result according to the plurality of the resolving association relation information.

Optionally, the determining the matching result according to the plurality of resolved association relation information includes:

arranging the solution candidate association relation information to obtain a plurality of solution arrangement information, and determining a solution candidate matching result corresponding to each solution arrangement information;

determining the association relation information of the solution candidates corresponding to the solution candidate matching results according to each solution candidate matching result, and obtaining the target association relation information of the solution candidate matching results based on the association relation information of the solution candidates;

and determining the matching result according to the number of each candidate matching result and the corresponding target resolving association relation information.

Optionally, the updating the matching score of the spatial point and the plane point in the target association relationship information according to the target association relationship information including the spatial point includes:

for each candidate matching result, acquiring the number of target association relation information corresponding to the candidate matching result, and determining an updated value of the target association relation information corresponding to the candidate matching result according to the number;

And updating the matching score of the spatial point and the plane point in the target association relation information according to the target association relation information containing the spatial point and the updated value of the target association relation information.

In a second aspect, an embodiment of the present invention provides a data processing apparatus for a head-mounted device, including:

the acquisition unit is used for acquiring association relation information between plane point set information of the head display equipment and space point set information corresponding to the plane point set information;

a candidate unit for determining candidate association information according to the association information;

the determining unit is used for determining a matching result of the plane point set information and the space point set information according to the candidate association relation information;

preferably, the determining unit determines a matching result of the plane point set information and the space point set information according to the candidate association relation information, including:

determining a matching result of the plane point set information and the space point set information according to the target association relation information;

Preferably, the candidate unit determines candidate association information according to the association information, including:

determining candidate association relation information according to the space points corresponding to the projection points and the plane points in the corresponding range of the projection points;

preferably, the determining unit determines a matching result of the plane point set information and the space point set information according to the target association relationship information, including:

determining a matching result based on the updated association relationship information;

Preferably, the determining unit updates the association information according to the target plane point corresponding to each spatial point and the spatial point, and further includes:

determining the matching result according to a plurality of the resolving association relation information;

preferably, the determining unit determines the matching result according to a plurality of the resolved association relation information, including:

determining the association relation information of the solution candidates corresponding to each solution candidate matching result, and processing the association relation information of the solution candidates based on an estimation module to obtain the number of the target association relation information of the solution candidates corresponding to the solution candidate matching result;

Determining the matching result according to the number of each candidate matching result and the corresponding target resolving association relation information;

preferably, the determining unit updates a matching score of the spatial point and a plane point in each target association relation information according to each target association relation information including the spatial point, including:

In a third aspect, an embodiment of the present invention further provides a head display device, including a memory storing a plurality of instructions; the processor loads instructions from the memory to execute any of the steps of the data processing method for the head display device provided by the embodiment of the invention.

In a fourth aspect, an embodiment of the present invention further provides a computer readable storage medium, where a plurality of instructions are stored, where the instructions are adapted to be loaded by a processor, to perform any of the steps of the data processing method for a head display device provided by the embodiment of the present invention.

Firstly, acquiring association relation information between plane point set information of head display equipment and space point set information corresponding to the plane point set information; candidate association relation information is determined according to the association relation information; and determining a matching result of the plane point set information and the space point set information according to the candidate association relation information. In this way, candidate association relation information is determined according to the association relation information of the plane point set information and the space point set information acquired by the head-mounted equipment, and based on the candidate association information after screening, a more accurate matching result of the space point set information and the plane point set information can be obtained, so that the accuracy of positioning matching is improved.

Drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings that are needed in the description of the embodiments will be briefly described below, it being obvious that the drawings in the following description are only some embodiments of the present invention, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a flow chart of an embodiment of a data processing method for a head-mounted device provided in an embodiment of the present invention;

FIG. 2 is a flow chart of another embodiment of a data processing method for a head-mounted device according to an embodiment of the present invention;

fig. 3 is a schematic diagram of an application scenario of a matching score provided in an embodiment of the present invention;

FIG. 4 is a schematic diagram of a data processing apparatus for a head-mounted display device according to an embodiment of the present invention;

fig. 5 is a schematic structural diagram of a head display device according to an embodiment of the present invention.

Detailed Description

The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to fall within the scope of the invention. Meanwhile, in the description of the embodiments of the present invention, the terms "first", "second", and the like are used only to distinguish the description, and are not to be construed as indicating or implying relative importance. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include one or more features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless explicitly defined otherwise.

The embodiment of the invention provides a data processing method and device for head display equipment, the head display equipment and a computer readable storage medium.

Specifically, the present embodiment will be described from the viewpoint of a data processing apparatus for a head-mounted device, which may be integrated in the head-mounted device in particular, that is, the data processing method for the head-mounted device of the embodiment of the present invention may be executed by the head-mounted device.

The following describes in detail the embodiments, taking the embodiment in which the execution subject is a head display device as an example, respectively, with reference to the accompanying drawings. The following description of the embodiments is not intended to limit the preferred embodiments. Although a logical order is depicted in the flowchart, in some cases the steps shown or described may be performed in an order different than depicted in the figures.

According to the background technology of the invention, in the related technology, due to the existence of noise light spots, the positioning matching based on the infrared mark points is easy to have error matching, so that the accuracy of the positioning matching is lower.

In order to solve the above problems, the present invention discloses a data processing method for a head display device, please refer to fig. 1, and the specific flow of the data processing method for a head display device may be as follows steps S10 to S40, wherein:

Step S10, acquiring association relation information between plane point set information of head display equipment and space point set information corresponding to the plane point set information;

in this embodiment, the main body that performs the data processing method for the head display device may be the head display device or other terminal devices, and the head display device, that is, the head-mounted display device, needs to locate the pose of the location target such as the operation handle, the action object, or the like. In order to more accurately locate the locating object, a plurality of marking points, which may be infrared LEDs, are arranged on the locating object, and the marking points arranged on the locating object and the relative positions between the marking points are preserved in advance. A space coordinate system is established, corresponding coordinate points are found in the coordinate system based on the relative positions among the marking points, and the space points corresponding to the coordinate points form space point set information.

When the positioning target moves, the marking point moves, so that the overall pose of the marking point is changed, and the overall pose of the marking point needs to be found, namely the positioning of the positioning target can be realized. The head display device is provided with a shooting device, a positioning image of a positioning target is acquired through the shooting device, such as an infrared camera array, light spots corresponding to marking points arranged on the positioning target can be shot on the positioning image, and in addition, some noise light spots can be shot. And establishing a plane coordinate system, and finding corresponding coordinate points in the plane coordinate system based on the light spots and the positions of the light spots in the positioning image, wherein the coordinate points form plane point set information acquired by the head display equipment.

Optionally, after the positioning image is obtained, a noise spot on the positioning image may be further subjected to preliminary filtration, specifically, the image is smoothed by using a gaussian blur algorithm to obtain a smoother spot profile, and then the image is subjected to binarization processing to facilitate the extraction of the spot profile. After the light spot contours are extracted, contours corresponding to noise light spots need to be removed: because the light spot corresponding to the infrared mark point is generally in an elliptical shape with the ratio of the long axis to the short axis within a certain range, the morphological attribute corresponding to the outline of the light spot can be judged, and the related morphological attribute comprises: spot area, spot roundness, spot convexity, ratio of the major and minor axes of the ellipse to which the spot profile is fitted. Noise spots can be screened out using predefined thresholds for various morphological properties.

After the plane point set of the head display device is obtained, spatial point set information pre-stored for mark points on a positioning target to be shot for obtaining a positioning image of the plane point set is obtained, the spatial point set information can represent an initial pose of the positioning target, after corresponding spatial points in the spatial point set information are matched with corresponding plane points in the plane point set information, differences between the plane point set information and the spatial point set information can be determined, differences between a real-time pose of the positioning target and the initial pose corresponding to the spatial point set information when the positioning image is shot can be determined based on the differences, and then the real-time pose of the positioning target can be obtained, so that the matching result of the spatial point set information and the plane point set information can be the real-time pose of the positioning target or other information capable of determining the real-time pose of the positioning target.

The association information is association information between the plane points corresponding to the plane point set information acquired by the head display device and the corresponding space points corresponding to the plane point set information, namely, a matching combination between the plane points and the space points, and when the head display device initially acquires the plane point set information which is not processed, if N plane points exist and M space points in the space point set information exist, m×n matching combinations may exist, namely, m×n association information exists. If the association relationship information between the plane point set information and the space point set information is set by the user or processed by the program, the number of the association relationship information is not necessarily M. It should be noted that, the obtained pipe association information may be all the matching combinations of the plane points and the space points, or may be the matching combinations of the plane points and the space points after the user setting or the program processing.

Step S20, candidate association information is determined according to the association information;

in this embodiment, the association relationship information may be processed based on some basic matching algorithm, and the matching combination between the spatial points corresponding to the spatial point set information and the planar points corresponding to the planar point set information may be calculated, so as to obtain the matching combination after screening as the candidate association relationship information.

And step S30, determining a matching result of the plane point set information and the space point set information according to the candidate association relation information.

In this embodiment, the association relationship information is subjected to preliminary processing to obtain candidate association relationship information, and a matching result between the plane points and the space points between the candidate plane point set information and the space point set information can be determined according to the candidate association relationship information.

In the technical scheme disclosed by the embodiment, the association relation information between the plane point set information of the head display device and the space point set information corresponding to the plane point set information is acquired; candidate association relation information is determined according to the association relation information; and determining a matching result of the plane point set information and the space point set information according to the candidate association relation information. In this way, candidate association relation information is determined according to the association relation information of the plane point set information and the space point set information acquired by the head-mounted equipment, and based on the candidate association information after screening, a more accurate matching result of the space point set information and the plane point set information can be obtained, so that the accuracy of positioning matching is improved.

Further, the step S30 includes:

In this embodiment, the association relationship information is initially processed, and candidate association relationship information is screened out, but the candidate association relationship information may have association relationship information of repeated association, and the association relationship information of repeated association is not necessarily that the spatial point and the plane point are not repeatedly associated, that is, the candidate association relationship information is inaccurate and needs further processing. The calculation module processes candidate association relation information based on the calculation module, and the calculation module can be a Hungary algorithm to obtain target association relation information which is not repeatedly associated and has the largest quantity, wherein the target association relation information is a matching combination between a target plane point and the target plane point, the target plane point only has a matching relation with one space point, namely the target space point, and the target space point also only has a matching relation with one plane point, namely the target plane point, so that the target plane point is not repeatedly associated.

After the target association relation information is determined, the matching relation between other plane points and space points can be further calculated according to the target association relation information, the matching score between the corresponding plane point and space point can be updated based on the target association relation information, after the matching score is updated in multiple rounds, the matching result of the plane point set information and the space point set information can be finally determined, the candidate association relation information of the upper layer of the obtained target association relation can be scored based on the number of the target association relation information, and whether the matching result between the space point set information and the plane point set information can be determined based on the candidate association relation information can be determined or not can be determined. Based on the final matching result, all or most of the space points can be determined to have corresponding target association relation information, the quantity of the target association information is maximum, and based on the matching result of the plane point set information and the space point set information, the pose of a positioning target corresponding to the plane point set information can be determined, so that high-precision pose determination or motion capture is realized.

In the technical scheme disclosed by the embodiment, based on candidate association relation information, target association relation information which is not repeatedly associated between a space point set and a plane point set is obtained; and determining a matching result of the plane point set information and the space point set information according to the target association relation information. In this way, candidate association relation information is determined according to the association relation information of the plane point set information and the space point set information acquired by the head-mounted equipment, and then target association relation information of the plane point and the space point is obtained by processing the candidate association relation information, so that a matching result is obtained, and therefore, the false matching of the plane point corresponding to the noise light spot and the space point can be filtered, and the accuracy of positioning matching is further improved.

Further, before step S20, the method includes:

In this embodiment, for the association information of the plane point set information and the space point set information corresponding to the plane point set information, the association information is arranged and combined according to a preset number to form a plurality of non-repeated arrangement information, for example, for the association information of a-a, a-B, a-C, B-a, B-B, B-C, C-a, C-B, C-C,6, the arrangement information can be arranged according to a preset number of 3 to obtainAnd the arrangement information. The preset number is determined according to the minimum association relation number required by the matching algorithm. And then solving each piece of arrangement information based on a matching algorithm to obtain candidate matching results corresponding to each piece of arrangement information, wherein the number of the solved candidate matching results is different according to the preset number or the difference of the matching algorithm. For example, 4 candidate matching results can be obtained through a P3P matching algorithm corresponding to the arrangement information with the preset number of 3. And corresponding to each candidate matching result of each arrangement information, projection can be performed to determine candidate association relation information. According to the candidate matching result, each space point in the space point set information is projected to the dimension of the plane point set information, namely the plane coordinate system of the plane point set information, so as to obtain the projection point of each space point set information on the plane coordinate system, the coordinates of the projection points form the projection point set information, and the projection points in the projection point set information are found And taking the projection point as a center, combining and matching the plane point in the preset range with the space point corresponding to the projection point in the radius of 3 pixels to obtain candidate association relation information, wherein the number of the candidate association relation information determined by each projection point is equal to the number of the plane point in the preset range, for example, if two plane points in the preset range exist, two candidate association relation information are correspondingly obtained, and the number of the candidate association relation information correspondingly obtained by each candidate matching result is equal to the sum of the candidate association relation information determined by each projection point.

Thus, a plurality of candidate matching results are calculated through the arrangement information composed of different association relation information. Different candidate association relation information is found by using different candidate matching results, and target association relation information which is not repeatedly associated is calculated and verified by using the candidate association relation information, so that the accuracy of the target association relation information can be improved.

Further, the determining, according to the target association relationship information, a matching result of the plane point set information and the space point set information includes:

And aiming at each candidate matching result, respectively taking candidate association relation information corresponding to the candidate matching result as input of a calculation module to obtain target association relation information corresponding to the candidate matching result.

In this embodiment, the estimation module may be a hungarian algorithm, and the hungarian algorithm may be used to estimate and mutually verify the association information of the plurality of association information, to estimate an estimation result with the maximum number of association information of non-repeated association, and to use the association information with the maximum number of non-repeated association corresponding to the estimation result as the target association information. For each candidate matching result, candidate association information which is determined corresponding to all candidate matching results and is determined based on the candidate matching result is taken as an input quantity, calculation of a hungarian algorithm is carried out, the hungarian algorithm finds the association information of the non-repeated association as far as possible, and finally the largest number of non-repeated association information is obtained, for example, if A can be matched with a and B, B can only be matched with B, if A is matched with B, B can not find a matched plane point, only the association information of the non-repeated association of A-B can be obtained, which is called as a calculation result (1), but if A is matched with a, the association information of the non-repeated association of two (A-a, B-B) can be obtained, which is called as a calculation result (2), obviously, the association information of the non-repeated association corresponding to the calculation result (2) can be found as the target association information.

The target association information is not obtained by selecting association information that does not repeatedly associate from the candidate association information as target association information, but by estimating and verifying each other, the maximum number of target association information is obtained. Therefore, more accurate matching modes of the space points and the plane points can be found by utilizing the target association relation information, matching results of the space point set information and the plane point set information can be obtained by directly utilizing the target association relation information, matching results of the space point set information and the plane point set information can be indirectly obtained by utilizing the target association relation information corresponding to all candidate matching results, the target association relation information is calculated by a plurality of candidate matching results and non-repeated matching, the accuracy of the target association relation information is higher, and the accuracy of positioning matching can be further improved.

Optionally, referring to fig. 2, based on any one of the foregoing embodiments, in another embodiment of the data processing method for a head-display device of the present invention, the determining, according to the target association relationship information, a matching result of the plane point set information and the spatial point set information includes:

Step S31, for each spatial point in the spatial point set information, updating the matching score of the spatial point and the plane point in the target association relationship information according to the target association relationship information containing the spatial point;

in this embodiment, the target association information may be used to update the matching score of each spatial point in the spatial point set information and each planar point in the planar point set information. Alternatively, the matching score of each spatial point in the spatial point set information and each planar point in the planar point set information may be represented by a matching histogram. If the space point set information has M space points and the plane point set information has N plane points, a matching histogram of m×n can be established. Illustratively, fig. 3 is a matching histogram constructed for 5 planar points and 4 spatial points, with the second row and second column being the matching score for spatial point P1 and spatial point P1.

Step S32, selecting a target plane point according to the matching score of the space point and the plane point;

for each spatial point in the spatial points, determining target association relation information comprising the spatial point, wherein the queried target association relation information comprises all target association relation information, namely target association relation information comprising all candidate matching results, updating the matching score between the spatial point and a plane point matched with the spatial point in the target association relation information according to the target association relation information, and increasing the matching score after updating, wherein the matching accuracy of the two points is higher. After the matching score of the space point is updated, the plane points are ordered from large to small according to the matching score between the space point and the plane points, the first n plane points are selected as target plane points of the space point, and n can be equal to the preset updating times of the association relation information-the current updating times of the association relation information.

Step S33, updating the association relation information according to each space point and the target plane point corresponding to the space point;

in this embodiment, matching is performed again according to each spatial point (number M) and the target plane point (number L) corresponding to the spatial point, so as to obtain association relationship information (number m×l) between each spatial point and each target plane point, and then update the association relationship information.

And step S34, determining a matching result based on the updated association relation information.

In this embodiment, the matching accuracy of the updated association relationship information is higher, and the matching result can be further determined by using a matching algorithm and an estimation module according to the updated association relationship information, so that the matching result is more accurately determined by using the updated association relationship information, and the matching result can also be directly determined based on the updated association relationship information.

In the technical scheme disclosed by the embodiment, the matching score between each space point and each plane point is updated through the target association relationship information, the association relationship information of the space point set information corresponding to the plane point set information updated by the target plane point is selected, so that a more accurate matching result is determined, and the accuracy of positioning matching is improved.

Further, the updating the matching score of the spatial point and the plane point in the target association relationship information according to the target association relationship information containing the spatial point includes:

In this embodiment, for each candidate matching result, a plurality of candidate association relationship information is calculated correspondingly, a plurality of target association relationship information can be determined correspondingly, and the target association relationship information indicates that under the candidate matching result, a plurality of association relationship information of spatial points and plane points which are not repeatedly associated can be calculated, and the more the number of the target association relationship information indicates that the candidate matching result is closer to an accurate matching result, the higher the accuracy of the corresponding target association relationship information is, and a higher matching score can be given to the target association relationship information. And setting an update value for the target association relation information corresponding to the candidate association relation information according to the number of the target association relation information corresponding to each candidate matching result, wherein the update value can be the number or can be in positive correlation with the number. And determining target association relation information comprising the space point for each space point, and updating a matching score between the space point and a plane point in the target association relation information according to an updating value corresponding to the target association relation information. For example, if the number of target association information corresponding to the candidate matching result corresponding to the target association information Pi-pj is 5, the matching score between the spatial point i and the plane point j in the target association information is increased by 5 points.

In this way, according to the number of the target association information corresponding to the candidate association information, the updating value of the target association information corresponding to the candidate matching result is set, and a higher matching score is given to the space point and the plane point in the target association information corresponding to the candidate matching result with higher accuracy, so that the updated association information used for determining the final matching result is more accurate to select, and the accuracy of positioning matching can be improved.

Further, after step S42, the method further includes:

In this embodiment, the updated association relationship information may be performed multiple times, the more the number of times of updating the association relationship information, the more accurate the updated association relationship information, and the more accurate the obtained matching result, but the matching cannot be performed for unlimited times, and the preset number of times of updating may be set as required. After each time of updating the association information, determining whether the update times of the current association information are greater than or equal to the preset update times, if the update times are less than the preset update times, determining new candidate association information based on the updated association information in turn, and calculating by the calculation module to obtain new target association information so as to update the association information again. If the update times of the association relation information is greater than or equal to the preset update times, the update is not needed, a plurality of pieces of resolving association relation information for resolving the matching result can be randomly selected from the updated association relation information, and the matching result of the plane point set information and the space point set information is obtained according to the resolving association relation information and a resolving algorithm corresponding to the matching result. In this way, based on the association relation information updated for a plurality of times, the resolving association relation information with higher matching accuracy is determined and used for resolving the matching result, so that the accuracy of positioning matching can be improved.

Further, the step of determining the matching result according to a plurality of the resolved association relation information includes:

In this example, the matching algorithm may calculate candidate matching results based on the association relationship information, and then the matching algorithm and the calculation module may also be used as a solution algorithm for the final matching result. According to the method, association relation information is calculated according to a preset number of arrangement combinations, a plurality of calculation arrangement information is obtained, calculation candidate matching results corresponding to the calculation arrangement information are determined based on a matching algorithm, for each calculation candidate matching result, projection point set information of a dimension where space point set information is projected to plane point set information is determined according to the calculation candidate matching results, calculation candidate association relation information is determined according to space points corresponding to projection points and plane points in a preset range corresponding to the projection points, calculation candidate association relation information corresponding to the candidate matching results is input into a calculation module, and target calculation association relation information corresponding to non-repeated matching of each candidate matching result is calculated. And sorting the calculated candidate matching results according to the number of the target calculated association relation information corresponding to each candidate matching result, and determining a final matching result based on the first R calculated candidate matching results in the sorting.

Specifically, the matching result may be pose information of the tracked target, and the solution candidate matching result may be candidate solution pose information of the tracked target. Therefore, the average value of the first R calculation candidate matching results can be calculated to obtain the initial value of the matching result, and then nonlinear optimization is carried out on the initial value of the matching result to obtain the final matching result.

Therefore, the accuracy of the matching result can be effectively improved and the accuracy of positioning matching can be improved through optimizing the matching for a plurality of times and processing the resolving candidate matching result corresponding to the resolving candidate association relation information.

For better understanding, the following association relationship information is exemplified by the update times of 2, and the matching result is exemplified by the pose of the positioning target corresponding to the plane point set information:

acquiring a positioning image captured by an infrared camera array, wherein the positioning image may have noise spots except for the spots of infrared mark points, and the image needs to be preprocessed firstly, including: (1) Smoothing the image by Gaussian blur to obtain a smoother light spot profile; (2) And carrying out binarization processing on the image so as to conveniently extract the light spot outline. After the candidate light spot contours are extracted, contours corresponding to noise light spots need to be removed: because the light spot corresponding to the infrared mark point is generally in an elliptical shape with the ratio of the long axis to the short axis within a certain range, the morphological attribute corresponding to the outline of the light spot can be judged, and the related morphological attribute comprises: spot area, spot roundness, spot convexity, ratio of the major and minor axes of the ellipse to which the spot profile is fitted. Part of the noise spots can be screened out using predefined thresholds for the various morphological properties.

After part of noise light spots are removed through preprocessing, 2D point set information P (the number is M) is determined according to the residual light spots on the positioning image, and in combination with prestored 3D point set information P (the number is N) determined based on infrared marking point positions on the tracking target, M x N2D-3D combinations are formed to serve as a set C of association relation information, 3 2D-3D combinations which are not repeated in the set C are selected to obtain all P3P candidate combinations serving as arrangement information, and an M x N matching histogram similar to that of FIG. 3 is established:

traversing all P3P candidate combinations, and obtaining 4 candidate matching results (candidate pose of positioning target) by a P3P algorithm (matching algorithm) for each group of P3P candidate matches. For each candidate pose: and projecting all 3d points in the point set information P onto the image by using the candidate pose to obtain 2d projection point set information P'. For each projection point p 'i in the range of 3 pixels in radius, a 2d point pj in the point set information p can be searched, the 3d point Pi corresponding to the p' i and the 2d point pj are a pair of 2d-3d candidate combinations, the 2d-3d candidate combination corresponding to each projection point is used as input of a Hungary algorithm, a set of 2d-3d combinations which are not repeatedly associated with the largest number can be obtained, the set is a set of target association relation information, and the matching score of the 2d-3d in the histogram is updated by using the target association relation information-2 d-3d combination in the set. The updating mode is to add the number of the target association relation information in the set to the original matching score. And finding each 3D point in P by using the matching histogram, and combining the 2D points corresponding to the 2D points before the matching score with the 2D points to form a new 2D-3D combination, so as to finally obtain an updated set C 'of the 2D-3D combination, and re-executing the process for the set C', and updating the histogram to obtain C ".

3 pairs of 2d-3d matches are randomly selected from C' to be used as the association relation information for resolving, 4 candidate poses are obtained by using a P3P algorithm, the projection process is carried out on each candidate pose, and a set of target association relation information corresponding to each candidate pose is obtained by using a Hungary algorithm. Comparing the number of the target association relation information in the corresponding set of each candidate pose, selecting n candidate poses with the number of the target association relation information being ranked n before average calculation is carried out, so as to obtain an initial value of the pose, and then carrying out nonlinear optimization on the initial value, so that the pose of the positioning target is used as a final matching result of the 3d point set P and the 2d point set P when the positioning image is finally shot, and the accuracy of positioning matching of the positioning target can be improved.

The embodiment also provides a data processing device for the head display device, which can be integrated in the head display device. For example, as shown in fig. 4, the data processing apparatus for a head display device may include:

an obtaining unit 1001, configured to obtain association relationship information between planar point set information of a head display device and spatial point set information corresponding to the planar point set information;

A candidate unit 1002, configured to determine candidate association information according to the association information;

a determining unit 1003, configured to determine a matching result of the plane point set information and the space point set information according to the candidate association relationship information;

preferably, the determining unit 1003 determines a matching result of the plane point set information and the space point set information according to the candidate association relationship information, including:

preferably, the candidate unit 1002 determines candidate association information according to the association information, including:

Preferably, the determining unit 1003 determines a matching result of the plane point set information and the space point set information according to the target association relationship information, including:

preferably, the determining unit 1003 further includes, after updating the association information according to the target plane point corresponding to each spatial point and the spatial point:

preferably, the determining unit 1003 determines the matching result according to a plurality of the resolved association relation information, including:

preferably, the determining unit 1003 updates, according to each of the target association relationship information including the spatial point, a matching score of the spatial point and a plane point in each of the target association relationship information, including:

In the embodiment, obtaining association relation information between plane point set information of head display equipment and space point set information corresponding to the plane point set information; candidate association relation information is determined according to the association relation information; and determining a matching result of the plane point set information and the space point set information according to the candidate association relation information. In this way, candidate association relation information is determined according to the association relation information of the plane point set information and the space point set information acquired by the head-mounted equipment, and based on the candidate association information after screening, a more accurate matching result of the space point set information and the plane point set information can be obtained, so that the accuracy of positioning matching is improved.

As shown in fig. 5, fig. 5 is a schematic structural diagram of a head display device according to an embodiment of the present invention. The head display device 1100 includes a processor 1101 having one or more processing cores, a memory 1102 having one or more computer readable storage media, and a computer program stored on the memory 1102 and executable on the processor. The processor 1101 is electrically connected to the memory 1102. It will be appreciated by those skilled in the art that the head display device structure shown in the figures does not constitute a limitation of the head display device, and may include more or less components than those illustrated, or may combine certain components, or may have a different arrangement of components.

The processor 1101 is a control center of the head display device 1100, connects respective portions of the entire head display device 1100 using various interfaces and lines, and performs various functions of the head display device 1100 and processes data by running or loading software programs and/or units stored in the memory 1102 and calling data stored in the memory 1102, thereby performing overall monitoring of the head display device 1100. The processor 1101 may be a processor CPU, a graphics processor GPU, a network processor (Network Processor, NP), etc., that may implement or perform the methods, steps and logic blocks disclosed in embodiments of the present invention.

In the embodiment of the present invention, the processor 1101 in the head display device 1100 loads the instructions corresponding to the processes of one or more application programs into the memory 1102 according to the following steps, and the processor 1101 executes the application programs stored in the memory 1102, so as to implement various functions, for example:

The specific implementation of each operation above may be referred to the previous embodiments, and will not be described herein.

Optionally, as shown in fig. 5, the head display device 1100 further includes: a touch display 1103, a radio frequency circuit 1104, an audio circuit 1105, an input unit 1106, and a power supply 1107. The processor 1101 is electrically connected to the touch display 1103, the radio frequency circuit 1104, the audio circuit 1105, the input unit 1106, and the power supply 1107, respectively. It will be appreciated by those skilled in the art that the head display device structure shown in fig. 5 is not limiting of the head display device and may include more or fewer components than shown, or may combine certain components, or a different arrangement of components.

The touch display 1103 may be used to display a graphical user interface and receive an operation instruction generated by a user acting on the graphical user interface. The touch display 1103 may include a display panel and a touch panel. Wherein the display panel may be used to display information entered by a user or provided to a user as well as various graphical user interfaces of the head-mounted device, which may be composed of graphics, text, icons, video, and any combination thereof. Alternatively, the display panel may be configured in the form of a liquid crystal display (LCD, liquid Crystal Display), an Organic Light-Emitting Diode (OLED), or the like. The touch panel may be used to collect touch operations on or near the user (such as operations on or near the touch panel by the user using any suitable object or accessory such as a finger, stylus, etc.), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Alternatively, the touch panel may include two parts, a touch detection device and a touch controller. The touch detection device detects the touch azimuth of a user, detects a signal brought by touch operation and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends the touch point coordinates to the processor 1101, and can receive and execute commands sent from the processor 1101. The touch panel may overlay the display panel, and upon detection of a touch operation thereon or thereabout, the touch panel is passed to the processor 1101 to determine the type of touch event, and the processor 1101 then provides a corresponding visual output on the display panel based on the type of touch event. In the embodiment of the present invention, the touch panel and the display panel may be integrated into the touch display 1103 to realize the input and output functions. In some embodiments, however, the touch panel and the touch panel may be implemented as two separate components to perform the input and output functions. I.e. the touch screen 1103 may also implement an input function as part of the input unit 1106.

The rf circuit 1104 may be configured to transmit and receive rf signals to and from a network device or other head-mounted device via wireless communication to and from the network device or other head-mounted device.

The audio circuit 1105 may be used to provide an audio interface between the user and the head-mounted device through a speaker, microphone. The audio circuit 1105 may transmit the received electrical signal after audio data conversion to a speaker, where the electrical signal is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signals into electrical signals, which are received by the audio circuit 1105 and converted into audio data, which are processed by the audio data output processor 1101, and transmitted to, for example, another head-mounted device via the radio frequency circuit 1104, or the audio data are output to the memory 1102 for further processing. The audio circuit 1105 may also include an ear bud jack to provide communication between the peripheral headphones and the head display device.

The input unit 1106 may be used to receive input numbers, character information, or user characteristic information (e.g., fingerprint, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

A power supply 1107 is used to power the various components of the head display device 1100. Alternatively, the power supply 1107 may be logically connected to the processor 1101 through a power management system, so as to perform functions of managing charging, discharging, and power consumption management through the power management system. The power supply 1107 may also include one or more of any of a direct current or alternating current power supply, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and the like.

Although not shown in fig. 5, the head display device 1100 may further include a camera, a sensor, a wireless fidelity module, a bluetooth module, etc., which will not be described herein.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.

Those of ordinary skill in the art will appreciate that all or a portion of the steps of the various methods of the above embodiments may be performed by instructions, or by instructions controlling associated hardware, which may be stored in a computer-readable storage medium and loaded and executed by a processor.

To this end, an embodiment of the present invention provides a computer readable storage medium in which a plurality of computer programs are stored, the computer programs being capable of being loaded by a processor to perform any one of the data processing methods for a head display device provided by the embodiment of the present invention. The computer program may execute the steps of a data processing method for a head-mounted device as follows:

Wherein the computer-readable storage medium may comprise: read Only Memory (ROM), random access Memory (RAM, random Access Memory), magnetic or optical disk, and the like.

Because the computer program stored in the computer readable storage medium can execute any one of the data processing methods for the head display device provided by the embodiments of the present invention, the beneficial effects that any one of the data processing methods for the head display device provided by the embodiments of the present invention can achieve can be achieved, which are detailed in the previous embodiments and will not be described herein.

In the embodiments of the data processing apparatus, the computer readable storage medium, the head display device, and the computer program product for a head display device, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments. It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the above-described data processing apparatus for head display device, computer readable storage medium, computer program product, specific working process of the head display device and its corresponding units and beneficial effects may refer to the description of the data processing method for head display device in the above embodiments, which is not repeated herein.

The foregoing has outlined some of the more detailed description of the data processing method, apparatus, head-mounted device, computer readable storage medium, and computer program product for a head-mounted device according to embodiments of the present invention, wherein the detailed description is provided for the purpose of illustrating the principles and embodiments of the present invention, and wherein the above description is only for the purpose of facilitating the understanding of the method and core idea of the present invention; meanwhile, as those skilled in the art will have variations in the specific embodiments and application scope in light of the ideas of the present invention, the present description should not be construed as limiting the present invention.

Claims

1. A data processing method for a head-mounted device, the method comprising:

2. The data processing algorithm according to claim 1, wherein the determining a matching result of the plane point set information and the space point set information according to the candidate association relation information comprises:

3. The data processing method of claim 2, wherein the determining candidate association information according to the association information comprises:

4. The data processing method according to claim 3, wherein the determining a matching result of the plane point set information and the space point set information according to the target association relation information includes:

5. The data processing method according to claim 4, wherein after updating the association information according to the target plane point for which each spatial point corresponds to the spatial point, further comprising:

6. The data processing method of claim 5, wherein the determining the matching result from the plurality of resolved association information comprises:

7. The data processing method of claim 4, wherein updating the matching score of the spatial point and the plane point in each of the target association information according to each of the target association information including the spatial point comprises:

8. A data processing apparatus for a head-mounted device, the data processing apparatus for a head-mounted device comprising:

9. The head display device is characterized by comprising a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to perform the steps of the data processing method for a head-mounted device as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that it stores a plurality of instructions adapted to be loaded by a processor to perform the steps of the data processing method for a head-display device according to any one of claims 1 to 7.