CN114329032A

CN114329032A - Image matching pixel retrieval method, device and storage medium

Info

Publication number: CN114329032A
Application number: CN202111613958.4A
Authority: CN
Inventors: 周杰
Original assignee: Beike Technology Co Ltd
Current assignee: Beike Technology Co Ltd
Priority date: 2021-12-27
Filing date: 2021-12-27
Publication date: 2022-04-12

Abstract

The present disclosure provides an image matching pixel retrieval method, an image matching pixel retrieval device, an electronic device and a storage medium, and relates to the technical field of computers, wherein the method comprises the following steps: correspondingly processing first image pixels and nodes in the first panoramic image and storing a spherical quadtree; determining an intersection circle of the epipolar plane corresponding to the second image pixel of the second panoramic image and the spherical surface, based on the epipolar constraints of the second panoramic image and the first panoramic image; performing search processing in the spherical quadtree based on the intersected circles to obtain candidate nodes corresponding to the pixels of the second image; in response to the coordinate information of the candidate node meeting the first search precision, taking a corresponding first image pixel of the candidate node as a matched pixel; the method, the device, the electronic equipment and the storage medium solve the problem of large search computation amount of the antipodal constraint of the panoramic imaging model, improve the computation efficiency, enable the work of three-dimensional reconstruction to run in real time and effectively improve the customer experience.

Description

Image matching pixel retrieval method, device and storage medium

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for retrieving image matching pixels, and a storage medium.

Background

In the process of VR acquisition and presentation, it is a more central technology to generate a three-dimensional model from a photo. At present, the three-dimensional reconstruction technology widely used in the industry is based on a pinhole imaging model, and a three-dimensional model is generated from a photo by a multi-view geometric method, and the retrieval of two-view matching points is a core step. The two-view matching point retrieval is carried out on the two panoramic photos, the panoramic photos need to be cut into split pinhole model photos, and then a corresponding epipolar constraint matching query method is applied, so that the operation scale is large, and the method cannot be applied to a real-time scene.

Disclosure of Invention

The present disclosure is proposed to solve the above technical problems. The embodiment of the disclosure provides an image matching pixel retrieval method and device, an electronic device and a storage medium.

According to a first aspect of the embodiments of the present disclosure, there is provided an image matching pixel retrieval method, including: triangulation processing is carried out on the spherical surface to construct a spherical quadtree; numbering nodes of the spherical quadtree, correspondingly processing first image pixels in a first panoramic image and the nodes, and storing the spherical quadtree; in response to a request to retrieve a second panorama and matching pixel points of the first panorama, determining an intersection circle where an epipolar plane corresponding to a second image pixel of the second panorama intersects the spherical surface based on epipolar constraints of the second panorama and the first panorama; searching the spherical quadtree based on the intersected circle to obtain a candidate node corresponding to the second image pixel; and in response to the coordinate information of the candidate node meeting a first search precision, taking the corresponding first image pixel of the candidate node as a matched pixel.

Optionally, the triangulation processing on the spherical surface includes: dividing the unit spherical surface to generate a plurality of spherical triangles; according to the tree depth information, carrying out recursive subdivision on each spherical triangle so as to construct the spherical quadtree; wherein the recursive subdivision processing comprises: and connecting the midpoints of each edge of the spherical triangle in pairs by using geodesic lines so as to divide the spherical triangle into four secondary spherical triangles.

Optionally, the constructing the spherical quadtree includes: constructing the spherical quadtree based on all spherical triangles; each spherical triangle is a node of the spherical quadtree; determining a circumscribed circle corresponding to the spherical triangle of each node, and determining a central vector and a circumscribed fillet radius corresponding to the center of the circumscribed circle.

Optionally, the numbering the nodes of the spherical quadtree includes: setting a corresponding first serial number for the node positioned at the top layer of the spherical quadtree, and setting a second serial number corresponding to a father node of the node to the other nodes; numbering all nodes according to a numbering rule and based on the first sequence number and/or the second sequence number.

Optionally, the processing the image pixels in the first panorama corresponding to the nodes includes: converting pixel coordinates of the first image pixels into spherical coordinates; starting from a root node, judging whether the spherical coordinates are inside or outside a node in the spherical quadtree, responding to the fact that the spherical coordinates are inside the node, determining whether the spherical coordinates are inside or outside a child node of the node, and until the spherical coordinates are determined to be inside a child node of the lowest layer, enabling the child node to be a pixel node corresponding to the first image pixel.

Optionally, the determining that the spherical coordinate is inside or outside a node in the spherical quadtree includes: determining a spherical triangle corresponding to the node; determining that the spherical coordinate is inside the node if the spherical coordinate is located within this spherical triangle.

Optionally, a mask corresponding to the first image pixel is constructed; and establishing and storing the corresponding relation among the serial number of the pixel node, the first image pixel and the mask.

Optionally, the performing search processing in the spherical quadtree based on the intersected circle to obtain a candidate node corresponding to the second image pixel includes: determining a normal vector of the intersecting circles; acquiring the central vector and the radius of the circumscribed fillet corresponding to the spherical triangle of the node; calculating a complementary angle of an included angle between the central vector and the normal vector; acquiring a node set intersected with the intersected circle, and selecting a node positioned at the lowest layer in the node set as the candidate node based on a search rule; wherein the search rules include: the fillet is greater than or equal to the circumscribed fillet radius, and a difference between the fillet and the circumscribed fillet radius is less than a second search precision.

Optionally, the first search precision comprises: and the included angle between the spherical coordinate of the first image pixel corresponding to the candidate node and the intersected circular plane is smaller than the included angle threshold value.

According to a second aspect of the embodiments of the present disclosure, there is provided an image matching pixel retrieving apparatus including: the quadtree construction module is used for triangulating the spherical surface and constructing a spherical quadtree; the pixel corresponding module is used for numbering the nodes of the spherical quadtree, correspondingly processing the first image pixel in the first panorama and the nodes and storing the spherical quadtree; an epipolar constraint processing module, configured to determine, in response to a request for retrieving a second panorama and matching pixel points of the first panorama, an intersection circle where an epipolar plane corresponding to a second image pixel of the second panorama intersects the spherical surface based on epipolar constraints of the second panorama and the first panorama; a candidate node determining module, configured to perform search processing in the spherical quadtree based on the intersected circle, so as to obtain a candidate node corresponding to the second image pixel; and the pixel matching determination module is used for responding to the condition that the coordinate information of the candidate node meets the search precision and taking the corresponding first image pixel of the candidate node as a matched pixel.

Optionally, the quadtree construction module is configured to segment the unit spherical surface to generate a plurality of spherical triangles; according to the tree depth information, carrying out recursive subdivision on each spherical triangle so as to construct the spherical quadtree; wherein the recursive subdivision processing comprises: and connecting the midpoints of each edge of the spherical triangle in pairs by using geodesic lines so as to divide the spherical triangle into four secondary spherical triangles.

Optionally, the quadtree construction module is specifically configured to construct the spherical quadtree based on all spherical triangles; each spherical triangle is a node of the spherical quadtree; determining a circumscribed circle corresponding to the spherical triangle of each node, and determining a central vector and a circumscribed fillet radius corresponding to the center of the circumscribed circle.

Optionally, the pixel correspondence module is specifically configured to set a first sequence number corresponding to a node located at a top level of the spherical quadtree, and set second sequence numbers corresponding to a parent node of the pixel correspondence module for the remaining nodes; numbering all nodes according to a numbering rule and based on the first sequence number and/or the second sequence number.

Optionally, the pixel correspondence module is specifically configured to convert pixel coordinates of the first image pixel into spherical coordinates; starting from a root node, judging whether the spherical coordinates are inside or outside a node in the spherical quadtree, responding to the fact that the spherical coordinates are inside the node, determining whether the spherical coordinates are inside or outside a child node of the node, and until the spherical coordinates are determined to be inside a child node of the lowest layer, enabling the child node to be a pixel node corresponding to the first image pixel.

Optionally, the pixel correspondence module is configured to determine a spherical triangle corresponding to the node; determining that the spherical coordinate is inside the node if the spherical coordinate is located within this spherical triangle.

Optionally, the pixel correspondence module is further configured to construct a mask corresponding to the first image pixel; and establishing and storing the corresponding relation among the serial numbers of the child nodes, the pixels of the first image and the mask.

Optionally, the candidate node determining module is configured to determine a normal vector of the intersecting circle; acquiring the central vector and the radius of the circumscribed fillet corresponding to the spherical triangle of the node; calculating a complementary angle of an included angle between the central vector and the normal vector; acquiring a node set intersected with the intersected circle, and selecting a node positioned at the lowest layer in the node set as the candidate node based on a search rule; wherein the search rules include: the complementary angle is greater than or equal to the radius of the circumscribed circle, and the difference between the complementary angle and the radius of the circumscribed circle is smaller than a first search precision.

Optionally, the search precision includes: and the included angle between the spherical coordinate of the first image pixel corresponding to the candidate node and the intersected circular plane is smaller than the included angle threshold value.

According to a third aspect of embodiments of the present disclosure, there is provided a computer-readable storage medium storing a computer program for executing the above-mentioned method.

According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including: a processor; a memory for storing the processor-executable instructions; the processor is used for executing the method.

According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program comprising computer readable code, characterized in that when the computer readable code is run on a device, a processor in the device executes a program for implementing the above-mentioned method.

Based on the image matching pixel retrieval method and device, the electronic device and the storage medium provided by the embodiment of the disclosure, the problem of large search computation amount of the polar constraint of the panoramic imaging model is solved, the search of the polar constraint of the spherical imaging model is converted into the query based on the spherical quadtree, the query time of the matching point is greatly shortened, the computation efficiency is improved, the work of three-dimensional reconstruction can be operated in real time, and the customer experience is effectively improved.

The technical solution of the present disclosure is further described in detail by the accompanying drawings and examples.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent by describing in more detail embodiments of the present disclosure with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the disclosure, and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the principles of the disclosure and not to limit the disclosure. In the drawings, like reference numbers generally represent like parts or steps.

FIG. 1 is a flow chart of one embodiment of an image matching pixel retrieval method of the present disclosure;

FIG. 2A is a schematic diagram of triangulation of a unit sphere; FIG. 2B is a schematic diagram of triangulation of a unit sphere;

FIG. 3 is a flow diagram of a candidate node in one embodiment of an image matched pixel retrieval method of the present disclosure;

FIG. 4 is a schematic diagram of node number filling in one embodiment of an image matching pixel retrieval method of the present disclosure;

FIG. 5 is a schematic structural diagram illustrating an embodiment of an image matching pixel retrieving apparatus according to the present disclosure;

FIG. 6 is a block diagram of one embodiment of an electronic device of the present disclosure.

Detailed Description

Example embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. It is to be understood that the described embodiments are merely a subset of the embodiments of the present disclosure and not all embodiments of the present disclosure, with the understanding that the present disclosure is not limited to the example embodiments described herein.

It should be noted that: the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

It will be understood by those of skill in the art that the terms "first," "second," and the like in the embodiments of the present disclosure are used merely to distinguish one element from another, and are not intended to imply any particular technical meaning, nor is the necessary logical order between them.

It is also understood that in embodiments of the present disclosure, "a plurality" may refer to two or more than two and "at least one" may refer to one, two or more than two.

It is also to be understood that any reference to any component, data, or structure in the embodiments of the disclosure, may be generally understood as one or more, unless explicitly defined otherwise or stated otherwise.

In addition, the term "and/or" in the present disclosure is only one kind of association relationship describing the associated object, and means that there may be three kinds of relationships, such as a and/or B, and may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" in the present disclosure generally indicates that the former and latter associated objects are in an "or" relationship.

It should also be understood that the description of the various embodiments of the present disclosure emphasizes the differences between the various embodiments, and the same or similar parts may be referred to each other, so that the descriptions thereof are omitted for brevity.

Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

Embodiments of the present disclosure may be implemented in electronic devices such as terminal devices, computer systems, servers, etc., which are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known terminal devices, computing systems, environments, and/or configurations that may be suitable for use with an electronic device, such as a terminal device, computer system, or server, include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set top boxes, programmable consumer electronics, network pcs, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above, and the like.

Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks may be performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

Summary of the application

In the process of realizing the present disclosure, the inventor finds that, when two-view matching point retrieval is performed in two panoramic photos, the existing retrieval method has a large operation scale and cannot be applied to a real-time scene. Therefore, a new repayment image matching pixel retrieval scheme is needed.

The image matching pixel retrieval method provided by the disclosure is used for triangulating a spherical surface to construct a spherical quadtree, correspondingly processing first image pixels and nodes in a first panoramic image and storing the spherical quadtree; determining an intersection circle of the epipolar plane corresponding to the second image pixel of the second panoramic image and the spherical surface, based on the epipolar constraints of the second panoramic image and the first panoramic image; performing search processing in the spherical quadtree based on the intersected circles to obtain candidate nodes corresponding to the pixels of the second image; in response to the coordinate information of the candidate node meeting the first search precision, taking a corresponding first image pixel of the candidate node as a matched pixel; the problem of the search operand of the panorama imaging model antipodal constraint is large is solved, the search of the spherical imaging model antipodal constraint is converted into the query based on the spherical quadtree, the operation efficiency is improved, the three-dimensional reconstruction work can run in real time, and the customer experience is effectively improved.

Exemplary method

Step numbers in the present disclosure, such as "step one", "step two", "S101", "S102", and the like, are only used to distinguish different steps, and do not represent the sequence of the steps, and the steps with different numbers may be performed in an adjustable sequence.

Panoramic cameras, such as binocular cameras, have their internal parameters calibrated and their own characteristic parameters, such as focal length, pixel size, etc. And the external parameters to be checked are parameters of the calibrated binocular camera in a world coordinate system, such as position, rotation direction and the like. Two cameras are used to take the same object at different positions, and if the scenes in the two pictures have overlapped parts, a certain corresponding relation exists between the two pictures. The most direct method is point-by-point matching, and if a certain constraint condition, namely epipolar constraint, is added, the search range can be greatly reduced and reduced to a line.

For the problem of searching matching points in two views (panoramic images) by epipolar constraint, at present, the searching process adopting a pinhole model is as follows: a certain imaging coordinate on the first frame image corresponds to a space point, the possible projection positions of the space point on the second frame image are constrained on a straight line by the geometrical relationship, and the retrieval process is to perform uniform iterative coordinate enumeration according to the parameters of the straight line. Different from the pinhole model, the epipolar constraint of the spherical imaging model is not a straight line any more, and if the epipolar constraint is mapped to the panorama, the epipolar constraint is mapped to a curve as a geodesic line is unfolded in the panorama, and the epipolar constraint cannot be obtained through coordinate iterative operation like the straight line epipolar constraint of the pinhole model.

Fig. 1 is a flowchart of an embodiment of an image matching pixel retrieval method of the present disclosure, and the method shown in fig. 1 includes the steps of: S101-S105. The following describes each step.

S101, triangulation processing is carried out on the spherical surface to construct a spherical quadtree.

And S102, numbering nodes of the spherical quadtree, correspondingly processing first image pixels and the nodes in the first panoramic image, and storing the spherical quadtree.

S103, in response to a request for searching matched pixel points of the second panoramic image and the first panoramic image, determining an intersected circle of an epipolar plane corresponding to a second image pixel of the second panoramic image and the spherical surface based on epipolar constraints of the second panoramic image and the first panoramic image.

And S104, performing searching processing in the spherical quadtree based on the intersected circles to acquire candidate nodes corresponding to the pixels of the second image.

And S105, in response to the coordinate information of the candidate node meeting the first searching precision, taking the corresponding first image pixel of the candidate node as a matched pixel.

Triangulation of the sphere can be performed using a variety of methods. For example, the unit spherical surface is divided to generate a plurality of spherical triangles. And performing recursive subdivision on each spherical triangle according to the tree depth information to construct a spherical quadtree, wherein the recursive subdivision comprises connecting the midpoints of each edge of each spherical triangle with each other by using geodesic lines, so that the spherical triangles are divided into four secondary spherical triangles.

And constructing a spherical quadtree based on all spherical triangles, wherein each spherical triangle is a node of the spherical quadtree. And determining a circumscribed circle corresponding to the spherical triangle of each node, and determining a central vector and a radius of a circumscribed fillet corresponding to the center of the circumscribed circle. And setting a first serial number corresponding to the node positioned at the top layer of the spherical quadtree, setting second serial numbers corresponding to self father nodes for other nodes, and numbering all nodes according to a numbering rule and based on the first serial number and/or the second serial number.

In one embodiment, as shown in FIG. 2A, a unit sphere is divided into 20 congruent spherical triangles from a regular 20-sided body. Setting the edge of a certain face f of the polyhedron surface as a set { e }, wherein two endpoints of e are vertexes of the polyhedron, and the endpoints and the o at the center of the polyhedron form a projective plane p by 3 points; assuming that the projective plane normal vector n always points inside f, p can represent a half-space h (half-space); the intersection of f with all the half-spaces h forms a cone r, and the spherical triangle corresponding to f can be obtained by the intersection of r with the unit sphere S2.

Performing recursion thinning construction tree structure on the triangulation, giving each spherical triangle t, connecting the midpoints P, S, U of each edge of each spherical triangle t with each other by using geodesic lines, and dividing the triangular into four spherical triangles; performing d recursion operations, based on the specified tree depth (number of levels) d, results in a quadtree representation of the unit sphere. The four spherical triangles obtained by each division are not equal in area, the area of the triangle surrounded by the midpoint connecting lines is larger than the area of the other three triangles, the difference is gradually reduced along with the increase of the thinning level, and when the tree level is deep enough, the influence of the difference on the operation process can be ignored.

All the spherical triangles form nodes of a quad-tree QuadTree, a circumscribed circle is calculated for the spherical triangle of each node, the center and the angular Radius (Constain Radius) r of the circumscribed circle are determined, the central vector of the circumscribed circle is a vector b, and the b is normalized into a unit vector. All nodes of the QuadTree are numbered and the number is expressed as a 32-bit integer: the serial numbers (first serial numbers) of the top 20 spherical triangle nodes are 1-20 in sequence, and the serial numbers of other intermediate nodes and child nodes are the serial numbers (second serial numbers are 0-3) of the nodes under the father node.

As shown in fig. 2B, the memory corresponding to the 32-bit integer data is divided into a plurality of segments, the initial value of all bits is set to 0 by default, for the first-level node serial number (1-20), the node serial number is converted into a 5-bit binary code and is filled into the first 5 bits, for the nodes of the lower layer, the corresponding serial number (0-3) is converted into a 2-bit binary code according to the node generation, and the serial numbers are sequentially filled into the subsequent positions step by step until the filling of all the node serial numbers is completed.

Converting the pixel coordinates of the first image pixels into spherical coordinates, judging the inside or outside of the spherical coordinates in the nodes of the spherical quadtree from the root node, responding to the inside of the nodes of the spherical coordinates, determining whether the spherical coordinates are inside or outside of the child nodes of the nodes until the spherical coordinates are determined to be inside the child nodes of the lowest layer, and connecting the child nodes with the pixel nodes corresponding to the first image pixels.

The method for judging the inside or outside of the node of the spherical coordinate in the spherical quadtree may be: a spherical triangle corresponding to the node is determined, and if the spherical coordinate is located within this spherical triangle, the spherical coordinate is determined to be inside the node. And constructing a mask corresponding to the first image pixel, and establishing and storing the serial number of the pixel node, the first image pixel and the corresponding relation of the mask.

In one embodiment, the quadtree structure corresponds to the first panorama, with a maximum supported photo resolution of 8192x4096 and a maximum tree depth of 13. For a first panorama with a specified resolution, corresponding all pixels to nodes of a quadtree, specifically: given a pixel coordinate, it is converted into a spherical coordinate (bearing vector) p, which is a three-dimensional vector of length 1. And starting from the root node, judging whether the coordinate is inside or outside the node on the quad-tree QuadTree, if the vector is inside the node, continuing to carry out recursive search in the child nodes of the node until the vector establishes a corresponding relation with the child nodes of the quad-tree QuadTree.

Judging whether the vector is in the node specifically comprises the following steps: the spherical triangle corresponding to each node of the QuadTree can be described as a part where three half-spaces (half-spaces) passing through the center of the sphere intersect with the spherical surface, and only the vector needs to be judged whether to meet the condition that the vector is simultaneously inside the three half-spaces, namely whether the directed distances from the point p to the half-space set { h } are the same as positive.

Serializing a quad-tree QuadTree to a disk, specifically: representing an image by a pixel matrix, and assuming that the width of the image is w and the height of the image is h; constructing a mask (mask) with the same size according to the resolution of the image, wherein the data type is 32-bit integer; determining the corresponding child node according to the method, determining the number of the child node, and assigning to the corresponding pixel on the mask (mask), wherein at the moment, any pixel on the mask (mask) records all relationship information from the top layer to the child node; and finally, storing the QuadTree (QuadTree) and the mask (mask) to a disk in the form of a yaml file.

The disclosed image matching pixel retrieval method provides a serialization and deserialization method of a quadtree on a spherical surface, an off-line tree structure can be constructed in advance when the resolution of an image is fixed, the structure is read through deserialization when the image is on line, and the construction process of the tree is accelerated through coding mapping; and the search of epipolar constraint is converted into query based on spherical quadtree, so that the query time of the matching points is greatly shortened.

Fig. 3 is a flowchart of a candidate node in an embodiment of the image matching pixel retrieving method of the present disclosure, and the method shown in fig. 3 includes the steps of: S301-S304. The following describes each step.

S301, determining a normal vector of the intersected circle.

S302, obtaining a central vector and a circumscribed fillet radius corresponding to the spherical triangle of the node.

And S303, calculating a complementary angle of an included angle between the central vector and the normal vector.

S304, acquiring a node set intersected with the intersected circle, and selecting a node positioned at the lowest layer in the node set as a candidate node based on a search rule; the search rules include: the residual angle is larger than or equal to the radius of the circumscribed fillet, and the difference between the residual angle and the radius of the circumscribed fillet is smaller than the second search precision.

And judging whether the coordinate information of the candidate node meets a first search precision, if so, taking a first image pixel corresponding to the candidate node as a matched pixel, wherein the first search precision comprises the condition that the included angle between the spherical coordinate of the first image pixel corresponding to the candidate node and the intersected circular plane is smaller than an included angle threshold value and the like.

In one embodiment, a QuadTree file constructed in advance is read into the memory. Knowing the epipolar constraints of the second panorama and the first panorama, the imaging model for the sphere can be expressed as a coplanar condition, that is, the three-dimensional coordinates of the matching pixel points of the second image pixels of the second panorama under the first panorama should fall at the intersection of the epipolar plane and the sphere of the first panorama (the sphere where the spherical quadtree is), that is, a large circle (intersection circle) of the sphere, as shown in fig. 4.

The epipolar constraint query procedure is to find the pixel coordinates corresponding to all points falling on the great circle. The epipolar constraint gives a normal n of the great circle, and a unit vector v corresponding to any point on the great circle is perpendicular to the normal n, namely < n, v > is 0. According to the judgment rule, all pixels meeting the constraint can be found on the quad-tree step by step, and the specific search algorithm is as follows: calculating a Radius (constant Radius) r of a circumscribed fillet by a spherical triangle of the node, wherein the center of the circumscribed circle corresponds to a vector b (central vector); and calculating a complementary angle a of an included angle between the vector b and the normal vector n, judging whether a node has an intersection with a great circle, if a > is r and a-r are smaller than the specified search precision or a < r, the node meets the condition, and performing the judgment in sequence and recurrently on all child nodes of the node until the child nodes are searched, wherein the child nodes are taken as candidate nodes.

Acquiring pixel coordinates contained in all searched child nodes, converting the pixel coordinates into a polar spherical coordinate (bearing vector) p, judging whether an included angle between the pixel coordinates and a large circular plane meets a specified first search precision, removing pixels which do not meet the specified first search precision, and obtaining the pixel coordinates which finally meet epipolar constraint as coordinates of matched pixels.

Exemplary devices

In one embodiment, as shown in fig. 5, the present disclosure provides an image matching pixel retrieving apparatus, including: a quadtree construction module 51, a pixel correspondence module 52, an epipolar constraint processing module 53, a candidate node determination module 54, and a pixel matching determination module 55.

The quadtree construction module 51 triangulates the sphere to construct a spherical quadtree. The pixel correspondence module 52 numbers the nodes of the spherical quadtree, and performs correspondence processing on the first image pixel in the first panorama and the node, and stores the spherical quadtree. The epipolar constraint processing module 53 determines, in response to a request to retrieve matching pixel points of the second panorama and the first panorama, an intersection circle where an epipolar plane corresponding to a second image pixel of the second panorama intersects with the spherical surface, based on the epipolar constraints of the second panorama and the first panorama.

The candidate node determination module 54 performs search processing in the spherical quadtree based on the intersecting circles to obtain candidate nodes corresponding to the pixels of the second image. If the coordinate information of the candidate node satisfies the search accuracy, the pixel matching determination module 55 takes the corresponding first image pixel of the candidate node as a matching pixel.

In one embodiment, the quadtree construction module 51 performs segmentation processing on the unit sphere to generate a plurality of spherical triangles. The quadtree construction module 51 performs recursive subdivision on each spherical triangle according to the tree depth information to construct a spherical quadtree; the recursive subdivision processing comprises the step of connecting the middle points of each edge of the spherical triangle in pairs by using geodesic lines so as to divide the spherical triangle into four secondary spherical triangles and the like.

The quadtree construction module 51 constructs a spherical quadtree based on all spherical triangles, each of which is a node of the spherical quadtree. The quadtree construction module 51 determines a circumscribed circle corresponding to the spherical triangle of each node, and determines a center vector and a circumscribed fillet radius corresponding to the center of the circumscribed circle.

In one embodiment, the pixel correspondence module 52 sets a first sequence number corresponding to the node located at the top level of the spherical quadtree, and sets a second sequence number corresponding to its parent node to the remaining nodes. The pixel correspondence module 52 numbers all nodes according to the numbering rule and based on the first sequence number and/or the second sequence number.

The pixel correspondence module 52 converts the pixel coordinates of the first image pixel into spherical coordinates. The pixel correspondence module 52 determines, starting from the root node, whether the spherical coordinates are inside or outside a node in the spherical quadtree, and determines, in response to the spherical coordinates being inside the node, whether the spherical coordinates are inside or outside a child node of the node until the spherical coordinates are determined to be inside a child node of the lowest layer, and associates the child node with the pixel node corresponding to the first image pixel.

Pixel correspondence module 52 determines the spherical triangle corresponding to the node and determines that the spherical coordinates are inside the node if the spherical coordinates are within this spherical triangle. The pixel correspondence module 52 constructs a mask corresponding to the first image pixel, and establishes and stores a correspondence relationship between the number of the child node, the first image pixel, and the mask.

In one embodiment, candidate node determination module 54 determines a normal vector of the intersecting circles. The candidate node determining module 54 obtains a center vector and an external fillet radius corresponding to the spherical triangle of the node, and calculates a complementary angle of an included angle between the center vector and the normal vector. The candidate node determining module 54 obtains a node set intersecting the intersecting circle, and selects a node located in the lowest layer from the node set as a candidate node based on a search rule; the search rule comprises that the residual angle is larger than or equal to the radius of the circumscribed fillet, and the difference between the residual angle and the radius of the circumscribed fillet is smaller than first search precision.

Fig. 6 is a block diagram of one embodiment of an electronic device of the present disclosure, as shown in fig. 6, the electronic device 61 includes one or more processors 611 and a memory 612.

The processor 611 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 61 to perform desired functions.

The memory 612 may include one or more computer program products that may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. Volatile memory, for example, may include: random Access Memory (RAM) and/or cache memory (cache), etc. The nonvolatile memory, for example, may include: read Only Memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on a computer readable storage medium and executed by processor 611 to implement the image matched pixel retrieval methods of the various embodiments of the present disclosure above and/or other desired functions. Various contents such as an input signal, a signal component, a noise component, etc. may also be stored in the computer-readable storage medium.

In one example, the electronic device 61 may further include: an input device 613, an output device 614, etc., which are interconnected by a bus system and/or other form of connection mechanism (not shown). The input device 613 may also include, for example, a keyboard, a mouse, and the like. The output device 614 can output various information to the outside. The output devices 614 may include, for example, a display, speakers, printer, and communication network and remote output devices connected thereto, among others.

Of course, for simplicity, only some of the components of the electronic device 61 relevant to the present disclosure are shown in fig. 6, omitting components such as buses, input/output interfaces, and the like. In addition, the electronic device 61 may include any other suitable components, depending on the particular application.

In addition to the above-described methods and apparatus, embodiments of the present disclosure may also be a computer program product comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image matched pixel retrieval method according to various embodiments of the present disclosure described in the "exemplary methods" section above of this specification.

The computer program product may write program code for performing the operations of embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the steps in the image matched pixel retrieval method according to various embodiments of the present disclosure described in the "exemplary methods" section above in this specification.

The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium may include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The foregoing describes the general principles of the present disclosure in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present disclosure are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present disclosure. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the disclosure is not intended to be limited to the specific details so described.

The image matching pixel retrieval method and device, the electronic device and the storage medium in the embodiment solve the problem of large search computation amount of the antipodal constraint of the panoramic imaging model, convert the search of the antipodal constraint of the spherical imaging model into the query based on the spherical quadtree, enable the matching process not to need to project and cut the panoramic photo in a slicing mode, greatly shorten the query time of the matching point, improve the computation efficiency, enable the three-dimensional reconstruction work to run in real time, and effectively improve the customer experience.

In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts in the embodiments are referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The block diagrams of devices, apparatuses, systems referred to in this disclosure are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, and systems may be connected, arranged, configured in any manner, as will be appreciated by those skilled in the art. Words such as "including," comprising, "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".

The methods and apparatus of the present disclosure may be implemented in a number of ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Further, in some embodiments, the present disclosure may also be embodied as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

It is also noted that in the devices, apparatuses, and methods of the present disclosure, each component or step can be decomposed and/or recombined. These decompositions and/or recombinations are to be considered equivalents of the present disclosure.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects, and the like, will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the disclosure to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims

1. An image matching pixel retrieval method, comprising:

triangulation processing is carried out on the spherical surface to construct a spherical quadtree;

numbering nodes of the spherical quadtree, correspondingly processing first image pixels in a first panoramic image and the nodes, and storing the spherical quadtree;

in response to a request to retrieve a second panorama and matching pixel points of the first panorama, determining an intersection circle where an epipolar plane corresponding to a second image pixel of the second panorama intersects the spherical surface based on epipolar constraints of the second panorama and the first panorama;

searching the spherical quadtree based on the intersected circle to obtain a candidate node corresponding to the second image pixel;

and in response to the coordinate information of the candidate node meeting a first search precision, taking the corresponding first image pixel of the candidate node as a matched pixel.

2. The method of claim 1, the triangulating the sphere comprising:

dividing the unit spherical surface to generate a plurality of spherical triangles;

according to the tree depth information, carrying out recursive subdivision on each spherical triangle so as to construct the spherical quadtree;

wherein the recursive subdivision processing comprises: and connecting the midpoints of each edge of the spherical triangle in pairs by using geodesic lines so as to divide the spherical triangle into four secondary spherical triangles.

3. The method of claim 2, the constructing the spherical quadtree comprising:

constructing the spherical quadtree based on all spherical triangles; each spherical triangle is a node of the spherical quadtree;

determining a circumscribed circle corresponding to the spherical triangle of each node, and determining a central vector and a circumscribed fillet radius corresponding to the center of the circumscribed circle.

4. The method of claim 3, the numbering nodes of the spherical quadtree comprising:

setting a corresponding first serial number for the node positioned at the top layer of the spherical quadtree, and setting a second serial number corresponding to a father node of the node to the other nodes;

numbering all nodes according to a numbering rule and based on the first sequence number and/or the second sequence number.

5. The method of claim 3, wherein said corresponding processing of image pixels in the first panorama with the nodes comprises:

converting pixel coordinates of the first image pixels into spherical coordinates;

starting from a root node, judging whether the spherical coordinates are inside or outside a node in the spherical quadtree, responding to the fact that the spherical coordinates are inside the node, determining whether the spherical coordinates are inside or outside a child node of the node, and until the spherical coordinates are determined to be inside a child node of the lowest layer, enabling the child node to be a pixel node corresponding to the first image pixel.

6. The method of claim 5, the determining that the spherical coordinates are inside or outside of a node in the spherical quadtree comprising:

determining a spherical triangle corresponding to the node;

determining that the spherical coordinate is inside the node if the spherical coordinate is located within this spherical triangle.

7. The method of claim 5, further comprising:

constructing a mask corresponding to the first image pixel;

and establishing and storing the corresponding relation among the serial number of the pixel node, the first image pixel and the mask.

8. The method of claim 3, wherein said performing a search process in the spherical quadtree based on the intersected circle to obtain candidate nodes corresponding to the second image pixel comprises:

determining a normal vector of the intersecting circles;

acquiring the central vector and the radius of the circumscribed fillet corresponding to the spherical triangle of the node;

calculating a complementary angle of an included angle between the central vector and the normal vector;

acquiring a node set intersected with the intersected circle, and selecting a node positioned at the lowest layer in the node set as the candidate node based on a search rule;

wherein the search rules include: the fillet is greater than or equal to the circumscribed fillet radius, and a difference between the fillet and the circumscribed fillet radius is less than a second search precision.

9. An image matching pixel retrieval apparatus comprising:

the quadtree construction module is used for triangulating the spherical surface and constructing a spherical quadtree;

the pixel corresponding module is used for numbering the nodes of the spherical quadtree, correspondingly processing the first image pixel in the first panorama and the nodes and storing the spherical quadtree;

an epipolar constraint processing module, configured to determine, in response to a request for retrieving a second panorama and matching pixel points of the first panorama, an intersection circle where an epipolar plane corresponding to a second image pixel of the second panorama intersects the spherical surface based on epipolar constraints of the second panorama and the first panorama;

a candidate node determining module, configured to perform search processing in the spherical quadtree based on the intersected circle, so as to obtain a candidate node corresponding to the second image pixel;

and the pixel matching determination module is used for responding to the condition that the coordinate information of the candidate node meets the first searching precision and taking the corresponding first image pixel of the candidate node as a matched pixel.

10. A computer-readable storage medium, characterized in that the storage medium stores a computer program for performing the method of any of the preceding claims 1-8.