CN115018904A

CN115018904A - Mask generation method and device for panoramic image

Info

Publication number: CN115018904A
Application number: CN202210623517.0A
Authority: CN
Inventors: 周杰; 李伟
Original assignee: You Can See Beijing Technology Co ltd AS
Current assignee: You Can See Beijing Technology Co ltd AS
Priority date: 2022-06-02
Filing date: 2022-06-02
Publication date: 2022-09-06
Anticipated expiration: 2042-06-02
Also published as: CN115018904B

Abstract

The embodiment of the disclosure discloses a mask generation method and a mask generation device for a panoramic image, which relate to the technical field of computers, wherein the method comprises the following steps: determining an initial polygon corresponding to a target area in the panoramic image on the unit spherical surface, and setting a target polygon corresponding to the initial polygon; constructing a spherical quadtree, numbering nodes of the spherical quadtree, and correspondingly processing spherical coordinates and the nodes to determine the nodes where the spherical coordinates are located; judging whether the spherical coordinates are in the target polygon; generating a mask for extracting a target area in the panoramic image according to the judgment result; the embodiment of the disclosure can solve the problems of large computation amount and low computation efficiency of mask generation, can support various polygons and determine the position relationship between pixels and polygons by adopting spherical quadtrees, improves the computation efficiency of the mask, and enables the work of mask generation to run in real time.

Description

Mask generation method and device for panoramic image

Technical Field

The present disclosure relates to the field of computer technologies, and in particular, to a method and an apparatus for generating a mask of a panoramic image, a storage medium, an electronic device, and a program product.

Background

Panoramic pictures are a technology for synthesizing pictures of all directions of a scene into one picture, and a user can view the pictures of all directions in the scene through one panoramic picture. For panoramic images, it is often necessary to separate a user-interested region from the panoramic image, where the user-interested region may be obtained by manual labeling or other methods as a polygonal region on a sphere, and the polygonal region is generally represented by a set of ordered three-dimensional spherical coordinate points, and the spherical coordinate points are sequentially connected to form a spherical polygon. The conversion operation is needed from the spherical polygon to the corresponding area on the panoramic photo, at present, the result of realizing the conversion operation is to generate a mask, but the operation amount of generating the mask is large, the running speed is slow, and the method cannot be applied to a real-time scene.

Disclosure of Invention

The present disclosure is proposed to solve the above technical problems. The embodiment of the disclosure provides a mask generation method and device for a panoramic image, a storage medium, an electronic device and a program product.

According to a first aspect of the embodiments of the present disclosure, there is provided a mask generation method for a panoramic image, including: determining an initial polygon corresponding to a target area in the panoramic image on a unit spherical surface; setting a target polygon corresponding to the initial polygon based on the shape feature of the initial polygon; wherein the target polygon is composed of at least one convex polygon; triangulation processing is carried out on the unit spherical surface, a spherical surface quadtree is constructed, and nodes of the spherical surface quadtree are numbered; acquiring spherical coordinates corresponding to two-dimensional pixel coordinates in the panoramic image, and performing corresponding processing on the spherical coordinates and the nodes to determine the nodes where the spherical coordinates are located; judging whether the spherical coordinates are in the target polygon or not according to the position information between the node and the target polygon; and generating a mask for extracting the target area in the panoramic image according to the judgment result.

Optionally, the setting, based on the shape feature of the initial polygon, a target polygon corresponding to the initial polygon includes: judging whether the initial polygon is a convex polygon or not based on the vertex information of the initial polygon; if so, determining the initial polygon as the target polygon; if not, the corresponding conversion processing is carried out on the initial polygon, and the target polygon is generated.

Optionally, the generating a mask for extracting the target region in the panoramic image according to the determination result includes: when the initial polygon is a convex polygon, if the spherical coordinate is within the target polygon, setting the pixel coordinate corresponding to the spherical coordinate to 1 in the mask, and if the spherical coordinate is not within the target polygon, setting the pixel coordinate corresponding to the spherical coordinate to 0 in the mask.

Optionally, the performing corresponding conversion processing on the initial polygon, and generating the target polygon includes: if the initial polygon is a concave polygon, generating a complementary set polygon corresponding to the initial polygon; combining the initial polygon and the complementary set polygon to generate an envelope polygon corresponding to the initial polygon; wherein the complementary set polygon and the envelope polygon are both convex polygons; generating the target polygon based on the complement polygon and the envelope polygon; wherein the target polygon is composed of the complement polygon and the envelope polygon.

Optionally, obtaining a vertex set of the initial polygon; and traversing each vertex of the vertex set, and if at least one internal angle corresponding to the vertex is larger than pi, determining that the initial polygon is a concave polygon.

Optionally, the generating a complement polygon corresponding to the initial polygon comprises: obtaining vertexes of which the internal angles are larger than pi in the vertex set, generating a spherical triangle formed by the vertexes of which the internal angles are larger than pi and other vertexes in a surrounding manner, and calculating the area of the spherical triangle; and taking the spherical triangle with the minimum area as a complementary polygon corresponding to the vertex of which the internal angle is greater than pi.

Optionally, the generating a mask for extracting the target region in the panoramic image according to the determination result includes: when the initial polygon is a concave polygon, if the spherical coordinates are within the envelope polygon, setting pixel coordinates corresponding to the spherical coordinates to 1 in a first mask, and if the spherical coordinates are not within the envelope polygon, setting pixel coordinates corresponding to the spherical coordinates to 0 in the first mask; setting the pixel coordinate corresponding to the spherical coordinate to 1 in a second mask if the spherical coordinate is within the complement polygon, and setting the pixel coordinate corresponding to the spherical coordinate to 0 in the second mask if the spherical coordinate is not within the complement polygon; and determining the difference value of the first mask and the second mask, and generating a mask for extracting the target area in the panoramic image based on the difference value.

Optionally, the performing corresponding conversion processing on the initial polygon, and generating the target polygon includes: if the initial polygon is determined to be a self-intersection polygon, performing segmentation processing on the initial polygon to obtain a plurality of sub-polygons; wherein none of the plurality of sub-polygons is a self-intersecting polygon; generating the target polygon based on the plurality of sub-polygons.

Optionally, the generating the target polygon based on the plurality of sub-polygons comprises: regarding a sub polygon with a convex polygon shape, taking the sub polygon as a first sub target polygon; for the sub-polygon with the shape of a concave polygon, generating a second sub-target polygon of the sub-polygon; wherein the second sub-target polygon is composed of a complementary polygon and an envelope polygon corresponding to the sub-polygon; generating the target polygon based on the first sub-target polygon and the second sub-target polygon.

Optionally, generating a normal vector of an edge based on the spherical coordinates of the vertex and the spherical coordinates of the adjacent vertex; calculating the outer product of the normal vectors of the two edges to obtain two intersection points of two circumscribed circles corresponding to the normal vectors of the two adjacent edges; determining that the two edges intersect and the initial polygon is a self-intersecting polygon if neither of the two intersection points is a vertex of the initial polygon and is located inside the two edges.

Optionally, the splitting the initial polygon includes: determining two intersected edges in the initial polygon and an intersection point of the two edges, and dividing the initial polygon into two sub-polygons according to the intersection point; wherein the areas of the two sub-polygons are smaller than the semi-spherical area; and if the sub-polygon is a self-intersection polygon, determining two intersected edges in the sub-polygon and an intersection point of the two edges, and dividing the sub-polygon into two new sub-polygons according to the intersection point until the two new sub-polygons are not self-intersection polygons.

Optionally, the generating a mask for extracting the target region in the panoramic image according to the determination result includes: generating a third mask corresponding to the first sub-target polygon; generating a fourth mask corresponding to the second sub-target polygon; generating a mask for extracting the target region in the panoramic image based on a combination of the third mask and the fourth mask.

Optionally, the determining, according to the position information between the node and the target polygon, whether the spherical coordinate is within the target polygon includes: determining that the spherical coordinates are within the target polygon if the node is inside the spherical polygon; if the node is on the spherical polygon boundary or the outside of the polygonal boundary, acquiring an edge normal vector corresponding to the spherical coordinate, calculating an inner product of the edge normal vector and the spherical coordinate, and if the inner product is greater than 0, determining that the spherical coordinate is located inside the spherical polygon.

Optionally, the triangulation processing on the unit sphere and the constructing of the sphere quadtree include: dividing the unit spherical surface to generate a plurality of spherical triangles; according to the tree depth information, carrying out recursive subdivision on each spherical triangle so as to construct the spherical quadtree; constructing the spherical quadtree based on all spherical triangles; wherein each spherical triangle is a node of the spherical quadtree.

Optionally, the numbering the nodes of the spherical quadtree includes: setting a corresponding first serial number for the node positioned at the top layer of the spherical quadtree, and setting a second serial number corresponding to a father node of the node to the other nodes; numbering all nodes according to a numbering rule and based on the first sequence number and/or the second sequence number; the corresponding processing of the spherical coordinates and the nodes comprises: converting the pixel coordinates of the panoramic image into spherical coordinates; and starting from a root node, judging whether the spherical coordinates are inside or outside a node in the spherical quadtree, determining whether the spherical coordinates are inside or outside a child node of the node in response to the spherical coordinates being inside the node, until the spherical coordinates are determined to be inside a child node of the lowest layer, and associating the child node with a pixel node corresponding to the first image pixel.

According to a second aspect of the embodiments of the present disclosure, there is provided a mask generating apparatus for a panoramic image, including: an initial region determining module for determining an initial polygon corresponding to a target region in the panoramic image on a unit spherical surface; a target area setting module, configured to set a target polygon corresponding to the initial polygon based on the shape feature of the initial polygon; wherein the target polygon is composed of at least one convex polygon; the quadtree construction module is used for triangulating the unit spherical surface, constructing a spherical quadtree and numbering nodes of the spherical quadtree; the pixel corresponding module is used for acquiring a spherical coordinate corresponding to a two-dimensional pixel coordinate in the panoramic image, and performing corresponding processing on the spherical coordinate and the node to determine the node where the spherical coordinate is located; the position judging module is used for judging whether the spherical coordinates are in the target polygon or not according to the position information between the nodes and the target polygon; and the mask drawing module is used for generating a mask for extracting the target area from the panoramic image according to the judgment result.

Optionally, the target area setting module is configured to determine whether the initial polygon is a convex polygon based on vertex information of the initial polygon; if so, determining the initial polygon as the target polygon; if not, performing corresponding conversion processing on the initial polygon to generate the target polygon.

Optionally, the mask drawing module is configured to, when the initial polygon is a convex polygon, set a pixel coordinate corresponding to the spherical coordinate in the mask to 1 if the spherical coordinate is within the target polygon, and set a pixel coordinate corresponding to the spherical coordinate in the mask to 0 if the spherical coordinate is not within the target polygon.

Optionally, the target area setting module includes: a first conversion unit configured to generate a complementary polygon corresponding to the initial polygon if the initial polygon is a concave polygon; combining the initial polygon and the complementary set polygon to generate an envelope polygon corresponding to the initial polygon; wherein the complementary polygon and the envelope polygon are both convex polygons; generating the target polygon based on the complement polygon and the envelope polygon; wherein the target polygon is composed of the complement polygon and the envelope polygon.

Optionally, the target area setting module includes: the first graph distinguishing unit is used for acquiring a vertex set of the initial polygon; and traversing each vertex of the vertex set, and if at least one internal angle corresponding to the vertex is larger than pi, determining that the initial polygon is a concave polygon.

Optionally, the first conversion unit is configured to obtain vertices of each internal angle greater than pi in the vertex set, generate a spherical triangle surrounded by the vertex of which the internal angle is greater than pi and other vertices, and calculate an area of the spherical triangle; and taking the spherical triangle with the smallest area as a complementary polygon corresponding to the vertex with the interior angle larger than pi.

Optionally, the mask drawing module is configured to, when the initial polygon is a concave polygon, set a pixel coordinate corresponding to the spherical coordinate to 1 in a first mask if the spherical coordinate is within the envelope polygon, and set a pixel coordinate corresponding to the spherical coordinate to 0 in the first mask if the spherical coordinate is not within the envelope polygon; setting the pixel coordinate corresponding to the spherical coordinate to 1 in a second mask if the spherical coordinate is within the complement polygon, and setting the pixel coordinate corresponding to the spherical coordinate to 0 in the second mask if the spherical coordinate is not within the complement polygon; and determining the difference value of the first mask and the second mask, and generating a mask for extracting the target area in the panoramic image based on the difference value.

Optionally, the target area setting module includes: the second conversion unit is used for performing segmentation processing on the initial polygon to obtain a plurality of sub-polygons if the initial polygon is determined to be a self-intersection polygon; wherein none of the plurality of sub-polygons is a self-intersecting polygon; generating the target polygon based on the plurality of sub-polygons.

Optionally, the second conversion unit is further configured to, for a sub-polygon with a convex polygon shape, take the sub-polygon as a first sub-target polygon; for the sub-polygon with the shape of a concave polygon, generating a second sub-target polygon of the sub-polygon; wherein the second sub-target polygon is composed of a complementary polygon and an envelope polygon corresponding to the sub-polygon; generating the target polygon based on the first sub-target polygon and the second sub-target polygon.

Optionally, the target area setting module includes: the second graph distinguishing unit is used for generating a normal vector of the edge based on the spherical coordinates of the vertex and the spherical coordinates of the adjacent vertex; calculating the outer product of the normal vectors of the two edges to obtain two intersection points of two circumscribed circles corresponding to the normal vectors of the two adjacent edges; determining that the two edges intersect and the initial polygon is a self-intersecting polygon if neither of the two intersection points is a vertex of the initial polygon and is located inside the two edges.

Optionally, the second conversion unit is further configured to determine two intersecting edges in the initial polygon and an intersection point of the two edges, and divide the initial polygon into two sub-polygons according to the intersection point; wherein the areas of the two sub-polygons are smaller than the semi-spherical area; and if the sub-polygon is a self-intersection polygon, determining two intersected edges in the sub-polygon and an intersection point of the two edges, and dividing the sub-polygon into two new sub-polygons according to the intersection point until the two new sub-polygons are not self-intersection polygons.

Optionally, the mask drawing module is configured to generate a third mask corresponding to the first sub-target polygon; generating a fourth mask corresponding to the second sub-target polygon; generating a mask for extracting the target region in the panoramic image based on a combination of the third mask and the fourth mask.

Optionally, the position determining module is configured to determine that the spherical coordinate is within the target polygon if the node is within the spherical polygon; if the node is on the spherical polygon boundary or the outside of the polygonal boundary, acquiring an edge normal vector corresponding to the spherical coordinate, calculating an inner product of the edge normal vector and the spherical coordinate, and if the inner product is greater than 0, determining that the spherical coordinate is located inside the spherical polygon.

Optionally, the quadtree construction module is configured to segment the unit spherical surface to generate a plurality of spherical triangles; according to the tree depth information, carrying out recursive subdivision on each spherical triangle so as to construct the spherical quadtree; constructing the spherical quadtree based on all spherical triangles; wherein each spherical triangle is a node of the spherical quadtree.

Optionally, the quadtree construction module is configured to set a first sequence number corresponding to a node located at a top level of the spherical quadtree, and set second sequence numbers corresponding to a parent node of the quadtree construction module to the remaining nodes; numbering all nodes according to a numbering rule and based on the first sequence number and/or the second sequence number; the pixel corresponding module is used for converting the pixel coordinates of the panoramic image into spherical coordinates; starting from a root node, judging whether the spherical coordinates are inside or outside a node in the spherical quadtree, responding to the fact that the spherical coordinates are inside the node, determining whether the spherical coordinates are inside or outside a child node of the node, and until the spherical coordinates are determined to be inside a child node of the lowest layer, enabling the child node to be a pixel node corresponding to the first image pixel.

According to a third aspect of embodiments of the present disclosure, there is provided a computer program comprising computer readable code, characterized in that when the computer readable code is run on a device, a processor in the device executes a program for implementing the above-mentioned method.

According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including: a processor; a memory for storing the processor-executable instructions; the processor is used for executing the method.

According to a fifth aspect of embodiments of the present disclosure, there is provided a computer program product comprising computer program instructions, characterized in that the computer program instructions, when executed by a processor, implement the method described above.

Based on the method and the device for generating the mask of the panoramic image, the storage medium, the electronic device and the program product, the problems of large computation amount and low computation efficiency of mask generation are solved, various polygons can be supported, the position relation between the pixels and the polygons can be determined by adopting the spherical quadtree, the computation efficiency of the mask is improved, the process of generating the mask is accelerated, the work of mask generation can be operated in real time, the method and the device are suitable for various scenes, and the customer experience is effectively improved.

The technical solution of the present disclosure is further described in detail by the accompanying drawings and examples.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent by describing in more detail embodiments of the present disclosure with reference to the attached drawings. The accompanying drawings are included to provide a further understanding of the embodiments of the disclosure, and are incorporated in and constitute a part of this specification, illustrate embodiments of the disclosure and together with the description serve to explain the principles of the disclosure and not to limit the disclosure. In the drawings, like reference numbers generally represent like parts or steps.

FIG. 1 is a flow chart of one embodiment of a method of mask generation for panoramic images of the present disclosure;

FIG. 2 is a schematic diagram of triangulation of a unit sphere;

fig. 3 is a schematic diagram of a target area in a panoramic image in an embodiment of a mask generation method for a panoramic image according to the present disclosure;

fig. 4 is a schematic diagram of processing convex polygons in an embodiment of a mask generation method for a panoramic image according to the present disclosure;

fig. 5 is a flowchart of a conversion process for a concave polygon in an embodiment of a mask generation method for a panoramic image of the present disclosure;

fig. 6 is a schematic diagram of a conversion process for a concave polygon in an embodiment of a mask generation method of a panoramic image of the present disclosure;

FIG. 7 is a flow chart of a conversion process for self-intersecting polygons in an embodiment of a method for mask generation of panoramic images of the present disclosure;

fig. 8 is a schematic diagram of a conversion process for a self-intersecting polygon in an embodiment of a mask generation method for a panoramic image of the present disclosure;

fig. 9 is a schematic structural diagram of an embodiment of a mask generating apparatus for a panoramic image according to the present disclosure;

fig. 10 is a schematic structural diagram of a target area setting module in an embodiment of the apparatus for generating a mask of a panoramic image according to the present disclosure;

FIG. 11 is a block diagram of one embodiment of an electronic device of the present disclosure.

Detailed Description

Example embodiments according to the present disclosure will be described in detail below with reference to the accompanying drawings. It is to be understood that the described embodiments are merely a subset of the embodiments of the present disclosure and not all embodiments of the present disclosure, with the understanding that the present disclosure is not limited to the example embodiments described herein.

It should be noted that: the relative arrangement of the components and steps, the numerical expressions, and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.

It will be understood by those of skill in the art that the terms "first," "second," and the like in the embodiments of the present disclosure are used merely to distinguish one element from another, and are not intended to imply any particular technical meaning, nor is the necessary logical order between them.

It is also understood that in embodiments of the present disclosure, "a plurality" may refer to two or more than two and "at least one" may refer to one, two or more than two.

It is also to be understood that any reference to any component, data, or structure in the embodiments of the disclosure, may be generally understood as one or more, unless explicitly defined otherwise or stated otherwise.

In addition, the term "and/or" in the present disclosure is only one kind of association relationship describing the associated object, and means that there may be three kinds of relationships, such as a and/or B, and may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" in the present disclosure generally indicates that the former and latter associated objects are in an "or" relationship.

It should also be understood that the description of the embodiments in the present disclosure emphasizes the differences between the embodiments, and the same or similar parts may be referred to each other, and are not repeated for brevity.

Meanwhile, it should be understood that the sizes of the respective portions shown in the drawings are not drawn in an actual proportional relationship for the convenience of description.

The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the disclosure, its application, or uses.

Techniques, methods, and apparatus known to those of ordinary skill in the relevant art may not be discussed in detail but are intended to be part of the specification where appropriate.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, further discussion thereof is not required in subsequent figures.

Embodiments of the present disclosure may be implemented in electronic devices such as terminal devices, computer systems, servers, etc., which are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well known terminal devices, computing systems, environments, and/or configurations that may be suitable for use with an electronic device, such as a terminal device, computer system, or server, include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, hand-held or laptop devices, microprocessor-based systems, set top boxes, programmable consumer electronics, network pcs, minicomputer systems, mainframe computer systems, distributed cloud computing environments that include any of the above, and the like.

Electronic devices such as terminal devices, computer systems, servers, etc. may be described in the general context of computer system-executable instructions, such as program modules, being executed by a computer system. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc. that perform particular tasks or implement particular abstract data types. The computer system/server may be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks may be performed by remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules may be located in both local and remote computer system storage media including memory storage devices.

Summary of the application

In the process of implementing the present disclosure, the inventor finds that the mask generation operation amount is large, the operation speed is slow, and the mask generation operation method cannot be applied to a real-time scene. Therefore, a new mask generation scheme for panoramic images is required.

The method for generating the mask of the panoramic image comprises the steps of determining an initial polygon corresponding to a target area in the panoramic image on a unit spherical surface; setting a target polygon corresponding to the initial polygon based on the shape characteristics of the initial polygon; triangulation processing is carried out on the unit sphere, a sphere quadtree is constructed, and nodes of the sphere quadtree are numbered; acquiring spherical coordinates corresponding to two-dimensional pixel coordinates in the panoramic image, and performing corresponding processing on the spherical coordinates and nodes to determine the nodes where the spherical coordinates are located; judging whether the spherical coordinates are in the target polygon or not according to the position information between the nodes and the target polygon; generating a mask for extracting a target area in the panoramic image according to the judgment result; the method solves the problems of large computation amount and low computation efficiency of mask generation, can support various polygons, and adopts the spherical quadtree to determine the position relation between the pixels and the polygons, thereby improving the computation efficiency of the mask, accelerating the process of generating the mask and enabling the work of generating the mask to run in real time.

Exemplary method

Step numbers in the present disclosure, such as "step one", "step two", "S101", "S102", and the like, are only used to distinguish different steps, and do not represent the sequence of the steps, and the steps with different numbers may be performed in an adjustable sequence.

Fig. 1 is a flowchart of an embodiment of a mask generation method for a panoramic image according to the present disclosure, where the method shown in fig. 1 includes the steps of: S101-S106. The following describes each step.

S101, determining an initial polygon corresponding to a target area in the panoramic image on a unit spherical surface. The panoramic image may be a panoramic image of a house, a stadium, etc. and the target area may be furniture, an item, a playing area, etc.

S102, setting a target polygon corresponding to the initial polygon based on the shape characteristics of the initial polygon; the target polygon is composed of at least one convex polygon, and the internal angle of each vertex of each convex polygon is smaller than pi.

In one embodiment, the initial polygon may be a spherical polygon such as a convex polygon, a concave polygon, etc., the initial polygons may be the same as or different from the target polygon, and the target polygon may be composed of one or more spherical convex polygons, i.e., the target polygon may be composed of one or more spherical convex polygons.

S103, triangulation processing is carried out on the unit spherical surface, a spherical surface quadtree is constructed, and nodes of the spherical surface quadtree are numbered.

And S104, acquiring spherical coordinates corresponding to the two-dimensional pixel coordinates in the panoramic image, and performing corresponding processing on the spherical coordinates and the nodes to determine the nodes where the spherical coordinates are located.

And S105, judging whether the spherical coordinates are in the target polygon or not according to the position information between the nodes and the target polygon. The position information includes information that the node is inside, boundary, outside, etc. of the target polygon.

And S106, generating a mask for extracting the target area in the panoramic image according to the judgment result.

A mask (mask) of the same size may be constructed according to the resolution of the panoramic image and assigned to the corresponding pixels on the mask. If the spherical coordinate corresponding to the pixel of the panoramic image is within the target polygon, the pixel coordinate is set to 1 in the mask, and if the spherical coordinate is not within the target polygon, the pixel coordinate corresponding to the spherical coordinate is set to 0 in the mask. The target region can be extracted from the panoramic image through the mask.

In one embodiment, the unit sphere is divided to generate a plurality of spherical triangles. And carrying out recursive subdivision on each spherical triangle according to the tree depth information so as to construct a spherical quadtree. And constructing a spherical quadtree based on all spherical triangles, wherein each spherical triangle is a node of the spherical quadtree. Setting a corresponding first serial number for a node positioned at the top layer of the spherical quadtree, and setting a second serial number corresponding to a father node of the node for the rest nodes; all nodes are numbered according to the numbering rule and based on the first sequence number and/or the second sequence number.

Converting pixel coordinates of the panoramic image into spherical coordinates; and starting from the root node, judging whether the spherical coordinates are inside or outside the nodes in the spherical quadtree, responding to the spherical coordinates inside the nodes, determining whether the spherical coordinates are inside or outside child nodes of the nodes until the spherical coordinates are determined inside the child nodes of the lowest layer, and connecting the child nodes with pixel nodes corresponding to the first image pixels.

For example, the unit spherical surface is divided to generate a plurality of spherical triangles. And performing recursive subdivision on each spherical triangle according to the tree depth information to construct a spherical quadtree, wherein the recursive subdivision comprises connecting the midpoints of each edge of each spherical triangle with each other by using geodesic lines, so that the spherical triangles are divided into four secondary spherical triangles.

And constructing a spherical quadtree based on all spherical triangles, wherein each spherical triangle is a node of the spherical quadtree. And determining a circumscribed circle corresponding to the spherical triangle of each node, and determining a central vector and a radius of a circumscribed fillet corresponding to the center of the circumscribed circle. And setting a first serial number corresponding to the node positioned at the top layer of the spherical quadtree, setting second serial numbers corresponding to self father nodes for other nodes, and numbering all nodes according to a numbering rule and based on the first serial number and/or the second serial number.

As shown in fig. 2, the unit spherical surface is divided into 20 congruent spherical triangles from the regular 20-sided surface body. Setting the edge of a certain face f of the polyhedron surface as a set { e }, wherein two endpoints of e are vertexes of the polyhedron, and the endpoints and the o at the center of the polyhedron form a projective plane p by 3 points; assuming that the projective plane normal vector n always points inside f, p can represent a half-space h (half-space); the intersection of f with all the half-spaces h forms a cone r, and the spherical triangle corresponding to f can be obtained by the intersection of r with the unit sphere S2.

Performing recursion thinning construction tree structure on the triangulation, giving each spherical triangle t, connecting the midpoints P, S, U of each edge of each spherical triangle t with each other by using geodesic lines, and dividing the triangular into four spherical triangles; performing d recursion operations, based on the specified tree depth (number of levels) d, results in a quadtree representation of the unit sphere. The four spherical triangles obtained by each division are not equal in area, the area of the triangle surrounded by the midpoint connecting lines is larger than the area of the other three triangles, the difference is gradually reduced along with the increase of the thinning level, and when the tree level is deep enough, the influence of the difference on the operation process can be ignored.

All the spherical triangles form nodes of a quad-tree QuadTree, a circumscribed circle is calculated for the spherical triangle of each node, the center and the angular Radius (Constain Radius) r of the circumscribed circle are determined, the central vector of the circumscribed circle is a vector b, and the b is normalized into a unit vector. All nodes of the QuadTree are numbered and the number is expressed as a 32-bit integer: the serial numbers (first serial numbers) of the top 20 spherical triangle nodes are 1-20 in sequence, and the serial numbers of other intermediate nodes and child nodes are the serial numbers (second serial numbers are 0-3) of the nodes under the father node.

Dividing a memory corresponding to 32-bit integer data into a plurality of segments, setting initial values of all bits as 0 by default, converting a node serial number into 5-bit binary codes for a first-level node serial number (1-20) and filling the 5-bit binary codes to the first 5 bits, converting the serial number (0-3) corresponding to the node generation into 2-bit binary codes for nodes of a lower layer, and filling the 2-bit binary codes to the subsequent positions step by step in sequence until the serial numbers of all the nodes are filled.

In one embodiment, two-dimensional pixel coordinates in the panoramic image are converted into spherical coordinates, the spherical coordinates are judged to be inside or outside a node in the spherical quadtree from a root node, whether the spherical coordinates are inside or outside a child node of the node is determined in response to the spherical coordinates being inside the node, and the child node is connected with a pixel node corresponding to a pixel of the first image until the spherical coordinates are determined to be inside a child node of the lowest layer. A spherical triangle corresponding to the node is determined, and if the spherical coordinate is located within this spherical triangle, the spherical coordinate is determined to be inside the node.

For example, the quadtree structure corresponds to a panoramic image, with a maximum supported photo resolution of 8192x4096 and a maximum tree depth of 13. And for the panoramic image with the specified resolution, corresponding all pixels to nodes of the quadtree. Given a pixel coordinate, it is converted into a spherical coordinate (bearing vector) p, which is a three-dimensional vector of length 1. And starting from the root node, judging whether the coordinate is inside or outside the node on the quad-tree QuadTree, if the vector is inside the node, continuing to carry out recursive search in the child nodes of the node until the vector establishes a corresponding relation with the child nodes of the quad-tree QuadTree.

The spherical triangle corresponding to each node of the QuadTree can be described as a part where three half-spaces (half-spaces) passing through the center of the sphere intersect with the spherical surface, and only the vector needs to be judged whether to meet the condition that the vector is simultaneously inside the three half-spaces, namely whether the directed distances from the point p to the half-space set { h } are the same as positive.

In one embodiment, various methods may be employed to determine whether the spherical coordinates are within the target polygon. For example, if the node is inside a spherical polygon, then the spherical coordinates are determined to be within the target polygon; if the node is outside the boundary of the spherical polygon or the boundary of the polygon, an edge normal vector corresponding to the spherical coordinate is obtained, the inner product of the edge normal vector and the spherical coordinate is calculated, and if the inner product is larger than 0, the spherical coordinate is determined to be inside the spherical polygon.

Judging whether the initial polygon is a convex polygon or not based on the vertex information of the initial polygon; if so, determining the initial polygon as a target polygon; if not, the corresponding conversion processing is carried out on the initial polygon, and a target polygon is generated.

When the initial polygon is a convex polygon, if the spherical coordinates are within the target polygon, the pixel coordinates corresponding to the spherical coordinates are set to 1 in the mask, and if the spherical coordinates are not within the target polygon, the pixel coordinates corresponding to the spherical coordinates are set to 0 in the mask.

In one embodiment, as shown in fig. 3, an initial polygon P corresponding to a target area in a panoramic image is determined on a unit sphere, and a vertex unit sphere coordinate array [ V ] of the initial polygon P is obtained]P the number of vertices is denoted m, where each vertex v _i The initial point is obtained by representing the closed graph by spherical coordinates (x, y, z) and connecting all the vertexes of P in sequenceA spherical polygon.

The purpose of the mask rendering is to mark the inside and outside of the polygon with different colors on a given panorama equidistant projection expansion map. Generating normal vectors corresponding to the edges of the spherical polygon according to the vertexes of the P to form an edge normal vector array [ N [ ]]The method specifically comprises the following steps: for each vertex v _i With the next adjacent vertex being v _i +1, the normal vector n of the edge _i ＝v _i ×v _i +1, symbol "x" represents the outer product operation.

The mask is M, a binary image representing the shape. The initial state of M is a zero matrix of size [ w, h ], w being the mask width and h being the mask height. As shown in fig. 3, it is determined whether the spherical coordinate P corresponding to the two-dimensional coordinate [ u, v ] in one panoramic image is inside the convex polygon P. If P is inside P, the pixel coordinate corresponding to the spherical coordinate is set to 1 in the mask M, and if P is not inside P, the pixel coordinate corresponding to the spherical coordinate is set to 0 in the mask M.

When P is a convex polygon, it is determined whether the coordinate P is inside the polygon P, and m sub-determination processes need to be performed. The normal vector of the edge corresponding to the vertex of the mark i of the polygon is n _i And n is _i The vertical large circular plane represents a half space, the spherical convex polygon is the intersection of all the half spaces and the unit sphere, and n is determined _i And whether the inner product of the P and the spherical coordinate is greater than 0, and if the inner product of the P and the spherical coordinate is greater than 0, determining that the P is in the inner part of the P.

And constructing a spherical quadtree, carrying out corresponding processing on the spherical coordinates and the nodes, determining the (leaf) node of the spherical coordinates P, judging whether the node is in the polygon P or on the boundary, recording the node set in the polygon P as { Li }, and recording the node on the boundary of the polygon P as { Le }. If the node belongs to { Li }, all spherical pixels in the node are in the polygon P, and if the node belongs to { Le }, whether the spherical coordinate P is in the polygon P needs to be judged.

The specific method for judging whether the spherical coordinate P is in the polygon P is as follows: the center (center) c and the angular Radius (Constain Radius) r of the node where the spherical coordinate p is located are read, assuming thatThe normal vector array of the edge of the spherical polygon P is [ N ]]Go over [ N ]]Element n in (1) _i If for any of the indices i, c and n _i Is greater than sin (r), the node is completely inside the polygon; if for any of the indices i, c and n _i Is greater than-sin (r), then the node may be on the boundary of the polygon; otherwise the node does not intersect the polygon P.

For any convex polygon, assuming that the size of a mask M to be drawn is [ w, h ], generating a spherical quadtree file with the size of [ w, h ], calculating an internal node set { Li }, a boundary node set { Le }, and setting the corresponding pixel coordinates of all { Li } in the mask M as 1; for all { Le }, if the spherical coordinates contained therein are determined to be within the polygon P, the corresponding pixel coordinates in the mask M are set to 1, and if the spherical coordinates contained therein are not determined to be within the polygon P, the corresponding pixel coordinates in the mask M are set to 0.

Fig. 5 is a flowchart of a conversion process for a concave polygon in an embodiment of a mask generation method for a panoramic image according to the present disclosure, and the method shown in fig. 5 includes the steps of: S501-S503. The following describes each step.

S501, if the initial polygon is a concave polygon, generating a complementary set polygon corresponding to the initial polygon.

In one embodiment, a vertex set of an initial polygon is obtained, each vertex of the vertex set is traversed, and if at least one vertex has an internal angle greater than pi, the initial polygon is determined to be a concave polygon.

Generating the complement polygon corresponding to the initial polygon may employ a variety of methods. For example, obtaining each vertex with an internal angle greater than pi in the vertex set, generating a spherical triangle surrounded by the vertex with the internal angle greater than pi and other vertices, and calculating the area of the spherical triangle; and taking the spherical triangle with the smallest area as a complementary set polygon corresponding to the vertex with the internal angle larger than pi.

S502, combining the initial polygon and the complementary set polygon to generate an envelope polygon corresponding to the initial polygon; wherein, the complementary polygon and the envelope polygon are both convex polygons.

S503, generating a target polygon based on the complementary set polygon and the envelope polygon; wherein the target polygon is composed of a complementary polygon and an envelope polygon.

When the initial polygon is a concave polygon, if the spherical coordinates are within the envelope polygon, the pixel coordinates corresponding to the spherical coordinates are set to 1 in the first mask, and if the spherical coordinates are not within the envelope polygon, the pixel coordinates corresponding to the spherical coordinates are set to 0 in the first mask.

If the spherical coordinate is in the complementary set polygon, setting the pixel coordinate corresponding to the spherical coordinate to be 1 in the second mask, and if the spherical coordinate is not in the complementary set polygon, setting the pixel coordinate corresponding to the spherical coordinate to be 0 in the second mask; a difference value between the first mask and the second mask is determined, and a mask for extracting a target region in the panoramic image is generated based on the difference value.

In one embodiment, the mask drawing problem of the concave polygon P may be converted into a problem of drawing a mask of a plurality of convex polygons. As shown in fig. 6, a convex envelope polygon H of the concave polygon P is calculated, and a complementary set { T } is calculated and recorded such that P ═ H- { T }, where each element in the complementary set { T } is a spherical convex triangle, one convex polygon corresponding to the concave polygon P can be obtained by a combination of the complementary set { T } and the convex envelope polygon H.

For example, traversing the vertices of the concave polygon P, for the vertex with index i, determining the internal angle corresponding to the vertex i, and if the internal angle is greater than pi, determining that the polygon is concave. Traversing the vertex of P, and calculating the corresponding spherical triangle complement set T for the vertex with the internal angle larger than pi and the subscript i, wherein T is an ordered vertex array [ v _i-1 ，v _i+1 ，v _i ]Forming a spherical triangle, and calculating the area S of the triangle.

According to the formula for calculating the area of the spherical polygon, assuming that the number of vertices of the spherical polygon is m, and the area S ═ sum (Ai) - (m-2) × pi, where sum (Ai) is the sum of the internal angles of the polygon, and Ai is the internal angle corresponding to the vertex with index i, the spherical triangle will beThe area S ═ sum (ai) -pi of T. Selecting the vertex with the subscript i corresponding to the complementary set T with the minimum area, and connecting the vertex v _i Removing from the vertex array of P and adding T to the set { T }; the above steps are repeated until all interior angles of P are non-concave.

According to the mask drawing method for convex polygons, a region H is drawn on a mask M, a new mask Mt of the same size as M is constructed for each element T in the complement { T }, the region T is drawn on Mt, and M-Mt is calculated, whereby a final mask for concave polygons can be obtained.

Fig. 7 is a flowchart of a conversion process for a self-intersecting polygon in an embodiment of a mask generation method for a panoramic image according to the present disclosure, where the method shown in fig. 7 includes the steps of: S701-S702. The following describes each step.

S701, if the initial polygon is determined to be a self-intersection polygon, segmenting the initial polygon to obtain a plurality of sub-polygons; wherein none of the plurality of sub-polygons is a self-intersecting polygon.

In one embodiment, a normal vector of an edge is generated based on the spherical coordinates of a vertex and the spherical coordinates of an adjacent vertex; calculating the outer product of the normal vectors of the two edges to obtain two intersection points of two circumscribed circles corresponding to the normal vectors of the two adjacent edges; if neither of the intersection points is a vertex of the initial polygon and is located inside the two edges, then the two edges are determined to intersect and the initial polygon is a self-intersecting polygon.

Determining two intersected edges in the initial polygon and an intersection point of the two edges, and dividing the initial polygon into two sub-polygons according to the intersection point; wherein, the areas of the two sub-polygons are both smaller than the semi-spherical area. If the sub-polygon is a self-intersection polygon, determining two intersected edges in the sub-polygon and an intersection point of the two edges, and dividing the sub-polygon into two new sub-polygons according to the intersection point until the two new sub-polygons are not self-intersection polygons.

S502, generating a target polygon based on the plurality of sub-polygons.

In one embodiment, for a sub-polygon shaped as a convex polygon, the sub-polygon is taken as a first sub-target polygon; for the sub-polygon with the shape of a concave polygon, generating a second sub-target polygon of the sub-polygon; wherein the second sub-target polygon is composed of a complement polygon and an envelope polygon corresponding to the sub-polygon. A target polygon is generated based on the first and second sub-target polygons.

And generating a third mask corresponding to the first sub-target polygon, generating a fourth mask corresponding to the second sub-target polygon, and generating a mask for extracting the target area in the panoramic image according to the combination of the third mask and the fourth mask.

In one embodiment, the self-intersecting polygons are segmented into sub-polygons without self-intersecting, and after corresponding masks are respectively drawn, merging processing is performed to generate a final target mask. Let i, j be the subscript of the vertex array, satisfy j>i，C＝n _i ×n _j And the vectors C and C represent two intersection points of the normal vectors corresponding to the great circles, and if the intersection points are not the vertexes of the polygon and are in the corresponding sides of i and j, the polygon is self-intersected.

As shown in FIG. 8, given two vertices with indices i, j and an intersection C, the starting point of the ith edge is v _i End point is v _i+1 The starting point of the jth edge is v _j End point is v _j+1 V is to be _i Rotate to v _i+1 Is angle of rotation _i The axis of rotation being axis _i V is to be _j Rotate to v _j+1 Is angle of rotation _j The axis of rotation being axis _j Calculating the starting point v in the same way _i Angle of rotation to C _ic The axis of rotation being axis _ic And a starting point v _j Angle of rotation to C _jc The axis of rotation being axis _jc 。

If the axis of rotation axis _i And angle _ic Are in the same direction and all parallel to n _i And the axis of rotation axis _j And angle _jc Are in the same direction and all parallel to n _j And the ratio angle _ic /angle _i Belongs to an open interval (0, 1) and a ratio angle _jc /angle _j If the interval belongs to the open interval (0, 1), the intersection point C falls within the side corresponding to i and j, which indicates that the sides corresponding to i and j intersect. For another intersection point-C, the same method is executed in the same way, if-C falls inside the corresponding edge, the edges corresponding to i and j are intersected.

Assume the polygon set is { P }, initially an empty set. When iteration starts, adding an original polygon P into a set; traversing all polygons P, and recording the number of vertexes as m; for polygon P, traverse all subscript pairs i, j (j)>i) If the corresponding edges of i and j are intersected and the intersection point is C, the polygon P is separated into two sub-polygons P1 and P2 according to C, and the ordered vertex array of P1 is [ v ₁ ,v ₂ ,…,v _i ,C,v _j+1 ,v _j+2 ,…,v _m ]The ordered vertex array of P2 is [ v ] _i+1 ,v _i+2 ,…,v _j-1 ,v _j ,C]Removing P from the set { P } and adding P1, P2 to the set { P }; this step is repeated until no self-intersecting polygons are present in the set { P }.

For the sub-polygons obtained by splitting, the vertex order may be reversed, and validity check needs to be performed. The area of the legal spherical polygon should not exceed the area of the semi-sphere, if the area is not the legal polygon, the vertex arrays of the legal spherical polygon need to be in reverse order, and the checking method comprises the following steps: calculating the area S according to the area formula of the spherical polygon, if the area S is less than 2x pi; the polygon is legal otherwise the polygon is illegal.

Exemplary devices

In one embodiment, as shown in fig. 9, the present disclosure provides a mask generating apparatus of a panoramic image, including: a starting region determining module 91, a target region setting module 92, a quadtree constructing module 93, a pixel corresponding module 94, a position judging module 95 and a mask drawing module 96. The initial region determining module 91 determines an initial polygon corresponding to a target region in the panoramic image on the unit sphere. The target region setting module 92 sets a target polygon corresponding to the initial polygon based on the shape characteristics of the initial polygon; wherein the target polygon is composed of at least one convex polygon.

The quadtree construction module 93 triangulates the unit sphere to construct a spherical quadtree, and numbers the nodes of the spherical quadtree. The pixel correspondence module 94 obtains spherical coordinates corresponding to two-dimensional pixel coordinates in the panoramic image, and performs corresponding processing on the spherical coordinates and nodes to determine the nodes where the spherical coordinates are located. The position determining module 95 determines whether the spherical coordinates are within the target polygon according to the position information between the node and the target polygon. The mask drawing module 96 generates a mask for extracting the target region in the panoramic image according to the determination result.

In one embodiment, if the node is inside a spherical polygon, the position determination module 95 determines that the spherical coordinates are within the target polygon. If the node is outside the boundary of the spherical polygon or the boundary of the polygon, the position determining module 95 obtains an edge normal vector corresponding to the spherical coordinate, calculates an inner product of the edge normal vector and the spherical coordinate, and if the inner product is greater than 0, the position determining module 95 determines that the spherical coordinate is inside the spherical polygon.

The quadtree construction module 93 divides the unit spherical surface to generate a plurality of spherical triangles; the quadtree construction module 93 performs recursive subdivision on each spherical triangle according to the tree depth information to construct a spherical quadtree. The quadtree construction module 93 constructs a spherical quadtree based on all spherical triangles; wherein, each spherical triangle is a node of the spherical quadtree.

The quadtree construction module 93 sets a first sequence number corresponding to a node located at the top level of the spherical quadtree, and sets a second sequence number corresponding to its parent node to the remaining nodes. The quadtree construction module 93 numbers all nodes according to the numbering rule and based on the first sequence number and/or the second sequence number.

The pixel correspondence module 94 converts the pixel coordinates of the first image pixel to spherical coordinates. The pixel correspondence module 94 determines whether the spherical coordinate is inside or outside a node in the spherical quadtree, starting from the root node, and determines whether the spherical coordinate is inside or outside a child node of the node in response to the spherical coordinate being inside the node until determining that the spherical coordinate is inside a child node of the lowest layer, and associates the child node with the pixel node corresponding to the first image pixel.

In one embodiment, the target region setting module 92 determines whether the initial polygon is a convex polygon based on vertex information of the initial polygon; if so, the target region setting module 92 determines that the initial polygon is the target polygon; if not, the target area setting module 92 performs corresponding conversion processing on the initial polygon to generate a target polygon.

When the initial polygon is a convex polygon, the mask drawing module 96 sets the pixel coordinate corresponding to this spherical coordinate to 1 in the mask if the spherical coordinate is within the target polygon, and the mask drawing module 96 sets the pixel coordinate corresponding to this spherical coordinate to 0 in the mask if the spherical coordinate is not within the target polygon.

In one embodiment, as shown in fig. 10, the target region setting module 92 includes a first conversion unit 921, a first image discriminating unit 922, a second conversion unit 923, and a second image discriminating unit 924. The first graph discrimination unit 922 acquires a vertex set of the initial polygon. The first graph discriminating unit 922 obtains a vertex set of the initial polygon to traverse each vertex of the vertex set, and if at least one vertex has an internal angle greater than pi, the first graph discriminating unit 922 obtains the vertex set of the initial polygon to determine that the initial polygon is a concave polygon.

The first conversion unit 921 generates a complementary polygon corresponding to the initial polygon if the initial polygon is a concave polygon. The first conversion unit 921 performs a combination process on the initial polygon and the complementary polygon to generate an envelope polygon corresponding to the initial polygon; wherein, the complementary polygon and the envelope polygon are both convex polygons. The first conversion unit 921 generates a target polygon based on the complementary polygon and the envelope polygon; wherein the target polygon is composed of a complementary polygon and an envelope polygon.

The first conversion unit 921 obtains vertices of each interior angle greater than pi in the vertex set, generates a spherical triangle surrounded by the vertices of the interior angle greater than pi and other vertices, and calculates an area of the spherical triangle. The first conversion unit 921 uses the spherical triangle having the smallest area as the complementary polygon corresponding to the vertex having the interior angle greater than pi.

When the initial polygon is a concave polygon, if the spherical coordinate is within the envelope polygon, the mask drawing module 96 sets the pixel coordinate corresponding to the spherical coordinate to 1 in the first mask, and if the spherical coordinate is not within the envelope polygon, the mask drawing module 96 sets the pixel coordinate corresponding to the spherical coordinate to 0 in the first mask;

if the spherical coordinate is within the complement polygon, the mask drawing module 96 sets the pixel coordinate corresponding to the spherical coordinate to 1 in the second mask, and if the spherical coordinate is not within the complement polygon, the mask drawing module 96 sets the pixel coordinate corresponding to the spherical coordinate to 0 in the second mask. The mask drawing module 96 determines a difference value of the first mask and the second mask, and generates a mask for extracting a target region in the panoramic image based on the difference value.

Second pattern discriminating section 924 generates a normal vector of an edge based on the spherical coordinates of a vertex and the spherical coordinates of an adjacent vertex. The second graph discriminating unit 924 calculates the outer product of the normal vectors of the two edges to obtain two intersection points of two circumscribed circles corresponding to the normal vectors of the two adjacent edges. If both of the intersection points are not the vertices of the initial polygon and are located inside the two edges, the second graph discrimination unit 924 determines that the two edges intersect and that the initial polygon is a self-intersecting polygon.

If the initial polygon is determined to be a self-intersecting polygon, the second conversion unit 923 performs segmentation processing on the initial polygon to obtain a plurality of sub-polygons; wherein none of the plurality of sub-polygons is a self-intersecting polygon. The second conversion unit 923 generates a target polygon based on the plurality of sub-polygons.

For a sub-polygon shaped as a convex polygon, the second conversion unit 923 treats this sub-polygon as a first sub-target polygon. For a sub-polygon with a concave polygon shape, the second conversion unit 923 generates a second sub-target polygon of the sub-polygon; wherein the second sub-target polygon is composed of a complement polygon and an envelope polygon corresponding to the sub-polygon. The second conversion unit 923 generates a target polygon based on the first sub-target polygon and the second sub-target polygon.

The second conversion unit 923 determines two intersecting edges in the initial polygon and an intersection point of the two edges, and divides the initial polygon into two sub-polygons according to the intersection point; wherein, the areas of the two sub-polygons are both smaller than the semi-spherical area. If the sub-polygon is a self-intersecting polygon, the second conversion unit 923 determines two intersecting edges in the sub-polygon and an intersection point of the two edges, and divides the sub-polygon into two new sub-polygons according to the intersection point until neither of the two new sub-polygons is a self-intersecting polygon.

The mask drawing module 96 generates a third mask corresponding to the first sub-target polygon and generates a fourth mask corresponding to the second sub-target polygon. The mask drawing module 96 generates a mask for extracting a target region in the panoramic image based on a combination of the third mask and the fourth mask.

FIG. 11 is a block diagram of one embodiment of an electronic device of the present disclosure, as shown in FIG. 11, electronic device 111 includes one or more processors 1111 and memory 1112.

The processor 1111 may be a Central Processing Unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 111 to perform desired functions.

Memory 1112 may store one or more computer program products and may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, Random Access Memory (RAM), cache memory (cache), and/or the like. The non-volatile memory may include, for example, Read Only Memory (ROM), hard disk, flash memory, etc. One or more computer program products may be stored on the computer-readable storage medium and executed by a processor. To implement the mask generation method of the panoramic image of the various embodiments of the present disclosure described above and/or other desired functions.

In one example, the electronic device 111 may further include: an input device 1113, and an output device 1114, among other components, interconnected by a bus system and/or other form of connection mechanism (not shown). The input device 1113 may also include, for example, a keyboard, mouse, or the like. The output device 1114 can output various information to the outside. The output devices 1114 may include, for example, a display, speakers, a printer, and a communication network and remote output devices connected thereto, among others.

Of course, for simplicity, only some of the components of the electronic device 111 relevant to the present disclosure are shown in fig. 11, omitting components such as buses, input/output interfaces, and the like. In addition, electronic device 111 may include any other suitable components, depending on the particular application.

In addition to the above-described methods and apparatuses, embodiments of the present disclosure may also be a computer program product comprising computer program instructions which, when executed by a processor, cause the processor to perform the steps in the mask generation method of a panoramic image according to various embodiments of the present disclosure described in the above section of this specification.

The computer program product may write program code for performing the operations of embodiments of the present disclosure in any combination of one or more programming languages, including an object oriented programming language such as Java, C + + or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.

Furthermore, embodiments of the present disclosure may also be a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, cause the processor to perform the steps in the mask generation method for a panoramic image according to various embodiments of the present disclosure described in the "exemplary methods" section above in this specification.

The computer-readable storage medium may take any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or a combination of any of the foregoing. More specific examples (a non-exhaustive list) of the readable storage medium may include: an electrical connection having one or more wires, a portable disk, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

The foregoing describes the general principles of the present disclosure in conjunction with specific embodiments, however, it is noted that the advantages, effects, etc. mentioned in the present disclosure are merely examples and are not limiting, and they should not be considered essential to the various embodiments of the present disclosure. Furthermore, the foregoing disclosure of specific details is for the purpose of illustration and description and is not intended to be limiting, since the disclosure is not intended to be limited to the specific details so described.

The method and the device for generating the mask of the panoramic image, the storage medium, the electronic device and the program product solve the problems of large computation amount and low computation efficiency of mask generation, can support various polygons and determine the position relation between pixels and the polygons by adopting the spherical quadtree, improve the computation efficiency of the mask, accelerate the process of generating the mask, enable the work of mask generation to run in real time, are suitable for various scenes and effectively improve the customer experience.

In the present specification, the embodiments are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same or similar parts in the embodiments are referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and for the relevant points, reference may be made to the partial description of the method embodiment.

The block diagrams of devices, apparatuses, systems referred to in this disclosure are only given as illustrative examples and are not intended to require or imply that the connections, arrangements, configurations, etc. must be made in the manner shown in the block diagrams. These devices, apparatuses, devices, and systems may be connected, arranged, configured in any manner, as will be appreciated by one skilled in the art. Words such as "including," comprising, "having," and the like are open-ended words that mean "including, but not limited to," and are used interchangeably therewith. The words "or" and "as used herein mean, and are used interchangeably with, the word" and/or, "unless the context clearly dictates otherwise. The word "such as" is used herein to mean, and is used interchangeably with, the phrase "such as but not limited to".

The methods and apparatus of the present disclosure may be implemented in a number of ways. For example, the methods and apparatus of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order for the steps of the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above unless specifically stated otherwise. Further, in some embodiments, the present disclosure may also be embodied as programs recorded in a recording medium, the programs including machine-readable instructions for implementing the methods according to the present disclosure. Thus, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

It is also noted that in the devices, apparatuses, and methods of the present disclosure, each component or step can be decomposed and/or recombined. These decompositions and/or recombinations are to be considered equivalents of the present disclosure.

The previous description of the disclosed aspects is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these aspects, and the like, will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The foregoing description has been presented for purposes of illustration and description. Furthermore, the description is not intended to limit embodiments of the disclosure to the form disclosed herein. While a number of example aspects and embodiments have been discussed above, those of skill in the art will recognize certain variations, modifications, alterations, additions and sub-combinations thereof.

Claims

1. A mask generation method of a panoramic image comprises the following steps:

determining an initial polygon corresponding to a target area in the panoramic image on a unit spherical surface;

setting a target polygon corresponding to the initial polygon based on the shape feature of the initial polygon; wherein the target polygon is composed of at least one convex polygon;

triangulation processing is carried out on the unit spherical surface, a spherical surface quadtree is constructed, and nodes of the spherical surface quadtree are numbered;

acquiring spherical coordinates corresponding to two-dimensional pixel coordinates in the panoramic image, and performing corresponding processing on the spherical coordinates and the nodes to determine the nodes where the spherical coordinates are located;

judging whether the spherical coordinates are in the target polygon or not according to the position information between the node and the target polygon;

and generating a mask for extracting the target area in the panoramic image according to the judgment result.

2. The method of claim 1, wherein setting a target polygon corresponding to the initial polygon based on the shape feature of the initial polygon comprises:

judging whether the initial polygon is a convex polygon or not based on the vertex information of the initial polygon;

if so, determining the initial polygon as the target polygon;

if not, the corresponding conversion processing is carried out on the initial polygon, and the target polygon is generated.

3. The method of claim 2, wherein the generating a mask for extracting the target region in the panoramic image according to the determination result comprises:

when the initial polygon is a convex polygon, if the spherical coordinate is within the target polygon, setting the pixel coordinate corresponding to the spherical coordinate to 1 in the mask, and if the spherical coordinate is not within the target polygon, setting the pixel coordinate corresponding to the spherical coordinate to 0 in the mask.

4. The method of claim 3, wherein the performing the corresponding transformation process on the initial polygon and the generating the target polygon comprises:

if the initial polygon is a concave polygon, generating a complementary set polygon corresponding to the initial polygon;

combining the initial polygon and the complementary set polygon to generate an envelope polygon corresponding to the initial polygon; wherein the complementary polygon and the envelope polygon are both convex polygons;

generating the target polygon based on the complement polygon and the envelope polygon; wherein the target polygon is composed of the complement polygon and the envelope polygon.

5. The method of claim 4, further comprising:

acquiring a vertex set of the initial polygon;

and traversing each vertex of the vertex set, and if at least one internal angle corresponding to the vertex is larger than pi, determining that the initial polygon is a concave polygon.

6. The method of claim 4, the generating a complementary polygon corresponding to the initial polygon comprising:

obtaining vertexes with the interior angle larger than pi in the vertex set, generating a spherical triangle formed by the vertexes with the interior angle larger than pi and other vertexes, and calculating the area of the spherical triangle;

and taking the spherical triangle with the smallest area as a complementary polygon corresponding to the vertex with the interior angle larger than pi.

7. The method of claim 4, wherein the generating a mask for extracting the target region in the panoramic image according to the determination result comprises:

when the initial polygon is a concave polygon, if the spherical coordinates are within the envelope polygon, setting pixel coordinates corresponding to the spherical coordinates to 1 in a first mask, and if the spherical coordinates are not within the envelope polygon, setting pixel coordinates corresponding to the spherical coordinates to 0 in the first mask;

setting the pixel coordinate corresponding to the spherical coordinate to 1 in a second mask if the spherical coordinate is within the complement polygon, and setting the pixel coordinate corresponding to the spherical coordinate to 0 in the second mask if the spherical coordinate is not within the complement polygon;

and determining a difference value of the first mask and the second mask, and generating a mask for extracting the target area in the panoramic image based on the difference value.

8. The method of claim 7, wherein the performing the corresponding transformation process on the initial polygon and the generating the target polygon comprises:

if the initial polygon is determined to be a self-intersection polygon, performing segmentation processing on the initial polygon to obtain a plurality of sub-polygons; wherein none of the plurality of sub-polygons is a self-intersecting polygon;

generating the target polygon based on the plurality of sub-polygons.

9. The method of claim 8, the generating the target polygon based on the plurality of sub-polygons comprising:

regarding a sub polygon with a convex polygon shape, taking the sub polygon as a first sub target polygon;

for the sub-polygon with the shape of a concave polygon, generating a second sub-target polygon of the sub-polygon; wherein the second sub-target polygon is composed of a complementary polygon and an envelope polygon corresponding to the sub-polygon;

generating the target polygon based on the first sub-target polygon and the second sub-target polygon.

10. A mask generating apparatus of a panoramic image, comprising:

an initial region determining module for determining an initial polygon corresponding to a target region in the panoramic image on a unit spherical surface;

a target area setting module, configured to set a target polygon corresponding to the initial polygon based on the shape feature of the initial polygon; wherein the target polygon is composed of at least one convex polygon;

the quadtree construction module is used for triangulating the unit spherical surface, constructing a spherical quadtree and numbering nodes of the spherical quadtree;

the pixel corresponding module is used for acquiring a spherical coordinate corresponding to a two-dimensional pixel coordinate in the panoramic image, and performing corresponding processing on the spherical coordinate and the node to determine the node where the spherical coordinate is located;

the position judging module is used for judging whether the spherical coordinates are in the target polygon or not according to the position information between the nodes and the target polygon;

and the mask drawing module is used for generating a mask for extracting the target area from the panoramic image according to the judgment result.