CN114896679B - Three-dimensional model optimization method of building, intelligent terminal and storage medium - Google Patents
Three-dimensional model optimization method of building, intelligent terminal and storage medium Download PDFInfo
- Publication number
- CN114896679B CN114896679B CN202210819365.1A CN202210819365A CN114896679B CN 114896679 B CN114896679 B CN 114896679B CN 202210819365 A CN202210819365 A CN 202210819365A CN 114896679 B CN114896679 B CN 114896679B
- Authority
- CN
- China
- Prior art keywords
- building
- point
- vertex
- image
- dimensional model
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/10—Geometric CAD
- G06F30/13—Architectural design, e.g. computer-aided architectural design [CAAD] related to design of buildings, bridges, landscapes, production plants or roads
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F30/00—Computer-aided design [CAD]
- G06F30/20—Design optimisation, verification or simulation
- G06F30/23—Design optimisation, verification or simulation using finite element methods [FEM] or finite difference methods [FDM]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
- G06T17/20—Finite element generation, e.g. wire-frame surface description, tesselation
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Geometry (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Hardware Design (AREA)
- Evolutionary Computation (AREA)
- General Engineering & Computer Science (AREA)
- Architecture (AREA)
- Civil Engineering (AREA)
- Structural Engineering (AREA)
- Computational Mathematics (AREA)
- Mathematical Analysis (AREA)
- Mathematical Optimization (AREA)
- Pure & Applied Mathematics (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- Processing Or Creating Images (AREA)
Abstract
The invention discloses a three-dimensional model optimization method of a building, an intelligent terminal and a storage medium, wherein the method comprises the following steps: acquiring a multi-view stereo satellite image, and reconstructing a three-dimensional model of a building based on the multi-view stereo satellite image; acquiring a building image of a building in a designated area, and extracting a structural line of the building in the building image based on the building image; recovering the spatial position of the structure line, filtering the structure line, and acquiring a first vertex and a second vertex which have the shortest endpoint distance between the three-dimensional model and the structure line; and obtaining a shortest path between the first vertex and the second vertex, and updating the coordinates of the vertex of the triangular surface on the shortest path into the coordinates of the projection point of the vertex of the triangular surface on the structural line. According to the invention, the three-dimensional model of the building produced by the satellite image is optimized through the structural line extracted from the unmanned aerial vehicle image, so that the three-dimensional model of the LOD 2-level building with higher precision and better visual effect is obtained.
Description
Technical Field
The invention relates to the technical field of photogrammetry and remote sensing, in particular to a three-dimensional model optimization method of a building, an intelligent terminal and a storage medium.
Background
The high-resolution stereoscopic image is easily affected by factors such as a camera carrying platform and electromagnetic interference in a data transmission process, and the high-resolution stereoscopic image is often not fine enough in the aspects of positioning accuracy, image resolution and the like, so that a three-dimensional model of a building produced by adopting the high-resolution stereoscopic image is not regular enough on the edge of a roof or a facade.
Most of building edge optimization methods in the prior art are used for regularizing house corners of two-dimensional buildings or performing plane fitting on building facades, so that the characteristics of structural characteristic lines of non-ground or roofs are degraded; the building three-dimensional model established based on the high-resolution stereo mapping satellite image has the problems of low edge regularity and poor visual effect.
Accordingly, the prior art is yet to be improved and developed.
Disclosure of Invention
The invention mainly aims to provide a building three-dimensional model optimization method, an intelligent terminal and a storage medium, and aims to solve the problems that in the prior art, a building three-dimensional model established based on a high-resolution stereo surveying and mapping satellite image is low in edge regularity and poor in visual effect.
In order to achieve the above object, the present invention provides a method for optimizing a three-dimensional model of a building, the method comprising the steps of:
acquiring a multi-view stereo satellite image, and reconstructing a three-dimensional model of a building based on the multi-view stereo satellite image;
acquiring a building image of a building in a designated area, and extracting a structure line of the building in the building image based on the building image;
recovering the spatial position of the structure line, filtering the structure line, and acquiring a first vertex and a second vertex which have the shortest endpoint distance between the three-dimensional model and the structure line;
and acquiring a shortest path between the first vertex and the second vertex, and updating the coordinates of the vertex of the triangular surface on the shortest path into the coordinates of the projection point of the vertex of the triangular surface on the structural line.
Optionally, the method for optimizing a three-dimensional model of a building, where the acquiring a multi-view stereo satellite image and reconstructing the three-dimensional model of the building based on the multi-view stereo satellite image specifically includes:
acquiring a high-resolution multispectral DOM image and a high-precision DSM image;
extracting a building based on the high-resolution multispectral DOM image, and resampling the high-precision DSM image in the building to recover a spatial point cloud;
and encrypting the space point cloud by using a quadratic linear interpolation method to obtain dense point cloud, and reconstructing a three-dimensional model of the building by a network construction algorithm based on the dense point cloud.
Optionally, the method for optimizing a three-dimensional model of a building, where the obtaining a building image of a building in a specified area and extracting a structural line of the building in the building image based on the building image specifically include:
acquiring a building image of a building in a designated area, and extracting a gray image in the building image;
after the gray level image is processed by an edge rendering algorithm, a set of edge pixel chains with adjacent pixels is generated;
and extracting a structural line from the edge pixel chain according to a least square straight line fitting method, and eliminating an error line segment by adopting a Helmholtz principle.
Optionally, the method for optimizing a three-dimensional model of a building, where the recovering the spatial position of the structure line and performing filtering processing on the structure line specifically include:
discretizing the structure line to obtain discrete points, projecting the discrete points to corresponding spatial positions through a collinearity equation to obtain projection points based on spatial information on an image where the discrete points are located, and fitting the projection points into straight line segments;
based on the straight line segment, retrieving a triangular surface patch of the straight line segment according to a K neighbor retrieval method, and obtaining a normal vector of the triangular surface patch;
calculating an included angle between the normal vector and the structural line based on the normal vector of each triangular patch;
if the included angle is larger than a preset threshold value, determining the straight line segment as a structural line of the building, and reserving the straight line segment;
and if the included angle is smaller than a preset threshold value, confirming that the straight line segment is a texture line, and removing.
Optionally, the method for optimizing a three-dimensional model of a building, wherein the collinearity equation is:
wherein, x is the abscissa of the image point, and y is the ordinate of the image point;x 0 is an element of the horizontal axis in the camera,y 0 is an element of the longitudinal axis within the camera,fis a vertical axis element in the camera;Xsis the abscissa of the camera station,Ysis the ordinate of the camera station,Zsas vertical coordinate, X, of camera-shooting station A Is the abscissa, Y, of the projected point in space A As ordinate, Z, of the projected point in space A Vertical coordinates of the spatial projection points;a i being imagesaAn orientation element for the orientation of the object,b i being imagesbThe orientation element is a position element of the display,c i being imagescAn orientation element.
Optionally, the method for optimizing a three-dimensional model of a building, where the obtaining a first vertex and a second vertex of the three-dimensional model having a shortest end-point distance from the structure line specifically includes:
taking the discrete point as a retrieval central point, and receiving triangular surface vertexes around the discrete point retrieved according to a space K nearest neighbor retrieval method;
and based on the vertex of the triangular surface, calculating the distances from the vertex of the triangular surface to the two endpoints of the structural line according to the triangular relation, and reserving a first vertex pt1 and a second vertex pt2 which are closest to the two endpoints.
Optionally, the method for optimizing a three-dimensional model of a building, where the obtaining a shortest path between the first vertex and the second vertex, and updating coordinates of a vertex of a triangular surface on the shortest path to coordinates of a projection point of the vertex of the triangular surface on the structural line, specifically includes:
setting the structural line as an axis and r as a radius, and constructing a buffer area, wherein the radius is set to be 5 times of the average ground resolution of the image;
based on the buffer area, obtaining a triangular surface vertex on the three-dimensional model, and constructing an undirected graph;
based on the undirected graph, performing space retrieval according to a shortest path algorithm to calculate a shortest path between the first vertex pt1 and the second vertex pt2, and obtaining a point set pt (i);
and when the vertex of the triangular surface in the point set pt (i) is projected to the structural line corresponding to the vertex of the triangular surface in the vertical direction to obtain a vertical point, calculating the coordinate of the vertical point, and updating the coordinate of the vertex of the triangular surface to the coordinate of the vertical point.
Optionally, the method for optimizing a three-dimensional model of a building, wherein the calculation process for calculating the coordinates of the vertical point specifically includes:
make the apex of the triangular surface
And a point S on said structure line 1 And a point S 2 Are respectively set as
、
、
Wherein the point S 1 And the point S 2 Are connected to form a straight line S 1 S 2 And the apex of the triangular face
On a straight line S 1 S 2 The projection coordinate in the vertical direction is point N
；
Based on the triangular surface vertex
Point S 1 And a point S 2 The three side vectors of the triangle are respectively
，
And
；
obtaining the following according to the vertical relation between the vectors:
；
from the vector collinearity we get:
；
obtaining the following result according to a vector vertical distance calculation formula:
；
solving the unknowns k as:
；
solving to obtain the coordinate of the projection coordinate N point;
wherein x is 0 Is a point
Abscissa of (a), y 0 Is a point
Ordinate of (a), z 0 Is a point
Vertical coordinate of (c), x 1 Is a point S 1 Abscissa of (a), y 1 Is a point S 1 Ordinate of (a), z 1 Is a point S 1 Vertical coordinate of (c), x 2 Is a point S 2 Abscissa of (a), y 2 Is a point S 2 Ordinate of (c), z 2 Is a point S 2 Vertical coordinate of (c), x n Is the abscissa of point N, y n Is the ordinate, z, of point N n Is the vertical coordinate of point N, and k is an unknown number.
In addition, in order to achieve the above object, the present invention further provides an intelligent terminal, wherein the intelligent terminal includes: a memory, a processor and a three-dimensional model optimization program of a building stored on the memory and executable on the processor, the three-dimensional model optimization program of a building when executed by the processor implementing the steps of the method of three-dimensional model optimization of a building as described above.
Further, to achieve the above object, the present invention also provides a computer readable storage medium, wherein the computer readable storage medium stores a three-dimensional model optimization program of a building, which when executed by a processor, implements the steps of the three-dimensional model optimization method of a building as described above.
In the invention, a multi-view stereo satellite image is collected, and a building three-dimensional model is reconstructed based on the multi-view stereo satellite image; acquiring a building image of a building in a designated area, and extracting a structural line of the building in the building image based on the building image; recovering the spatial position of the structure line, filtering the structure line, and acquiring a first vertex and a second vertex which have the shortest endpoint distance between the three-dimensional model and the structure line; and acquiring a shortest path between the first vertex and the second vertex, and updating the coordinates of the vertex of the triangular surface on the shortest path into the coordinates of the projection point of the vertex of the triangular surface on the structural line. The invention improves the traditional scheme of building three-dimensional reconstruction based on high-resolution three-dimensional surveying and mapping satellite images, and firstly, a multi-view three-dimensional satellite image is utilized to produce an LOD 2-level building three-dimensional model of a test area; then extracting a building structure line from the unmanned aerial vehicle image of the corresponding area, recovering the spatial position of the structure line, and filtering out texture lines; and finally, optimizing the building three-dimensional model by using the structural line for recovering the spatial position so as to obtain an LOD 2-level building three-dimensional model with higher precision and better visual effect.
Drawings
FIG. 1 is a flow chart of a preferred embodiment of a method of optimizing a three-dimensional model of a building of the present invention;
FIG. 2 is a flow chart of step S10 in the preferred embodiment of the method for optimizing the three-dimensional model of the building of the present invention;
FIG. 3 is a schematic view of a preferred embodiment of the building-based three-dimensional model optimization method of the present invention;
FIG. 4 is a flow chart of step S20 in the preferred embodiment of the method for optimizing the three-dimensional model of the building of the present invention;
FIG. 5 is a flow chart of the restoration and filtering of the structural lines in step S30 in the method for optimizing a three-dimensional model of a building according to the present invention;
FIG. 6 is a flowchart of the method for optimizing a three-dimensional model of a building according to the present invention, wherein the nearest triangle vertex is obtained in step S30;
FIG. 7 is a flowchart of step S40 in the method for optimizing a three-dimensional model of a building according to the present invention;
FIG. 8 is an overall flow chart of the method for optimizing the three-dimensional model of the building according to the present invention;
fig. 9 is a schematic operating environment diagram of an intelligent terminal according to a preferred embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention clearer and clearer, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
It should be noted that, if directional indications (such as up, down, left, right, front, back, 8230; etc.) are involved in the embodiment of the present invention, the directional indications are only used for explaining the relative positional relationship between the components, the motion situation, etc. in a specific posture (as shown in the figure), and if the specific posture is changed, the directional indications are correspondingly changed.
In addition, if there is a description relating to "first", "second", etc. in the embodiments of the present invention, the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying relative importance or to implicitly indicate the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, technical solutions between the embodiments may be combined with each other, but must be based on the realization of the technical solutions by a person skilled in the art, and when the technical solutions are contradictory to each other or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
As shown in fig. 1, the method for optimizing a three-dimensional model of a building according to a preferred embodiment of the present invention includes the following steps:
and S10, acquiring a multi-view stereo satellite image, and reconstructing a three-dimensional model of the building based on the multi-view stereo satellite image.
As shown in fig. 2, the step S10 includes:
s11, acquiring a high-resolution multispectral DOM image and a high-precision DSM image;
s12, extracting a building based on the high-resolution multispectral DOM image, and resampling the high-precision DSM image in the building to recover a spatial point cloud;
and S13, encrypting the space point cloud by using a quadratic linear interpolation method to obtain dense point cloud, and reconstructing a three-dimensional model of the building through a network construction algorithm based on the dense point cloud.
Specifically, as shown in fig. 3, a satellite is used for collecting a multi-view stereo satellite image Of a 2-Level building (Level Of Detail, which is a term describing the fineness Of BIM model information data) and an unmanned aerial vehicle (drone) is used for collecting an image Of the building in a test area, and the reconstruction Of the three-dimensional model Of the building through the multi-view stereo satellite image specifically includes that a multi-spectral image and a high-resolution panchromatic rear-view image with the same view angle are used for producing a high-resolution multi-spectral DOM image by using a Gram-Schmidt algorithm, and a multi-view panchromatic stereo image is used for producing a high-precision DSM image by using a semi-global stereo matching algorithm; then extracting a building from the high-resolution multispectral DOM image by adopting a machine learning or deep learning method; the method comprises the steps of resampling the high-precision DSM image in the building image spot to recover a space point cloud, encrypting the space point cloud by using a quadratic linear interpolation method to obtain a dense point cloud, wherein the encryption has the advantages that the density of the space point cloud obtained by directly carrying out dense matching is sparse due to low spatial resolution of the satellite image, more detailed information is lost if direct networking modeling is carried out, the point cloud needs to be encrypted to obtain the dense point cloud, and finally, the dense point cloud is used for reconstructing a three-dimensional model of the building through a networking algorithm.
And S20, acquiring a building image of a building in the designated area, and extracting a structural line of the building in the building image based on the building image.
As shown in fig. 4, the step S20 includes:
s21, obtaining a building image of a building in a designated area, and extracting a gray level image in the building image;
s22, after edge rendering algorithm processing is carried out on the gray level image, a set of edge pixel chains with adjacent pixels are generated;
and S23, extracting a structural line from the edge pixel chain according to a least square straight line fitting method, and eliminating an error line segment by adopting a Helmholtz principle.
Specifically, building images of LOD 2-level buildings in a test area are collected through an unmanned aerial vehicle, gray level images in the building images are extracted, an ED (edge drawing) edge detector is used for processing the gray level images through an edge drawing algorithm to generate a group of clean edge pixel chains with adjacent pixels, a structural line is extracted from the edge pixel chains according to a least square straight line fitting method, and a straight line segment with detection errors is eliminated by adopting a Helmholtz (Helmholtz) principle.
And S30, restoring the spatial position of the structure line, filtering the structure line, and acquiring a first vertex and a second vertex which have the shortest endpoint distance between the three-dimensional model and the structure line.
As shown in fig. 5, the restoring the spatial position of the structural line in step S30 includes:
step S311, discretizing the structure line to obtain discrete points, projecting the discrete points to corresponding spatial positions through a collinear equation to obtain projection points based on spatial information of the images where the discrete points are located, and fitting the projection points into straight line segments;
step S312, based on the straight line segment, retrieving a triangular surface patch of the straight line segment according to a K neighbor retrieval method, and obtaining a normal vector of the triangular surface patch;
step 313, calculating an included angle between the normal vector and the structural line based on the normal vector of each triangular patch;
step S314, if the included angle is larger than a preset threshold value, confirming that the straight line segment is a structural line of a building, and reserving the straight line segment;
and step S315, if the included angle is smaller than a preset threshold value, determining that the straight line segment is a texture line, and rejecting the texture line.
Specifically, the spatial position of the recovered structure line comprises two parts, namely characteristic line projection and spatial straight line fitting; the characteristic line projection part is used for discretizing the structural line to obtain a series of discrete points, then projecting the spatial information on the image where the series of discrete points are located to the corresponding spatial positions through a collinear equation to obtain projection points, and the collinear equation is used as follows:
；
wherein, x is the abscissa of the image point, and y is the ordinate of the image point;x 0 is an element of the horizontal axis in the camera,y 0 is an element of the longitudinal axis within the camera,fis a vertical axis element in the camera;Xsis the abscissa of the camera station,Ysis the ordinate of the camera's camera station,Zsas vertical coordinate, X, of camera-shooting station A Is the abscissa, Y, of the projected point in space A As ordinate, Z, of the projected point in space A Vertical coordinates of the spatial projection points;a i being imagesaAn orientation element for the orientation of the object,b i being imagesbThe orientation element is a position element of the display,c i being imagescAn orientation element; after the discrete points are projected, in order to recover the structural lines, the spatial straight line fitting is carried out, and the projected points are fitted into straight line segments by using RANSAC spatial straight line fitting or least square spatial straight line fitting algorithm; through the extraction of the line segments, a plurality of line segments in a scene can be obtained, wherein the line segments comprise texture lines generated by a large number of window or ground textures besides structural lines of buildings; in order to eliminate the texture lines which cannot be used, a K neighbor retrieval method is used for retrieving triangular patches near the straight line segment, after normal vectors of each triangular patch are calculated, the average value of the normal vectors is obtained, wherein the average value of the normal vectors is used for taking the average value of the normal vectors of local triangular patches as the normal vector of the local area, and then the structure line and the normal vector are calculatedIf the included angle is larger than a preset threshold value, determining that the straight line segment is a structural line of the building, and reserving the straight line segment; and if the included angle is smaller than a preset threshold value, confirming that the straight line segment is a texture line, and removing.
Further, as shown in fig. 6, the step S30 of obtaining the first vertex and the second vertex, where the distance between the three-dimensional model and the endpoint of the structure line is the shortest, includes:
s321, taking the discrete point as a retrieval central point, and receiving triangular surface vertexes around the discrete point retrieved according to a spatial K neighbor retrieval method;
and step S322, calculating the distances from the vertex of the triangular surface to the two end points of the structural line according to the triangular relationship based on the vertex of the triangular surface, and reserving a first vertex pt1 and a second vertex pt2 which are closest to the two end points.
Specifically, discrete points of the recovered space position structure line are used as retrieval central points, and triangular surface vertexes near the discrete points are retrieved by using a space K neighbor retrieval method so as to narrow a retrieval range; traversing all the triangular face vertexes where the search is located, then calculating the distances from the triangular face vertexes to the two end points of the structure line by using a triangular relation (sqrt (x + 2+ y + 2+ z + 2)) through two points with known coordinates in space, and keeping a first vertex pt1 and a second vertex pt2 which are closest to the two end points.
And S40, acquiring a shortest path between the first vertex and the second vertex, and updating the coordinates of the vertex of the triangular surface on the shortest path into the coordinates of the projection point of the vertex of the triangular surface on the structural line.
As shown in fig. 7, the step S40 includes:
s41, setting the structure line as an axis and r as a radius to construct a buffer area, wherein the radius is set to be 5 times of the average ground resolution of the image;
s42, acquiring a triangular surface vertex on the three-dimensional model based on the buffer area, and constructing an undirected graph;
step S43, based on the undirected graph, performing space retrieval according to a shortest path algorithm to calculate a shortest path between the first vertex pt1 and the second vertex pt2, and obtaining a point set pt (i);
and S44, when the vertex of the triangular surface in the point set pt (i) is projected to a structural line corresponding to the vertex of the triangular surface in the vertical direction to obtain a vertical point, calculating a coordinate of the vertical point, and updating the coordinate of the vertex of the triangular surface to the coordinate of the vertical point.
Specifically, the structure line is set as an axis, r is set as a radius, and a buffer area is constructed, wherein r is set as 5 times of the average ground resolution of the image; acquiring and searching all triangular surface vertexes V = (V) on the three-dimensional model in the buffer zone by adopting a k neighbor searching mode 1 ，V 2 ，…，V n ) (ii) a Constructing undirected graph G<V，E>If the vertices of the triangular surface are adjacent in the three-dimensional model, constructing an edge (E), wherein the weight of the edge is in direct proportion to the edge length in the Mesh model; solving a shortest path between the first vertex pt1 and the second vertex pt2 by utilizing a Dijkstra algorithm (shortest path algorithm), obtaining a point set pt (i), traversing all points in the point set pt (i), and calculating a vertical point coordinate of a projection from the vertex of the triangular surface to a vertical line direction of a structural line by utilizing the vertex of the traversed triangular surface and the structural line corresponding to the vertex of the traversed triangular surface; make the apex of the triangular surface
And a point S on said construction line 1 And point S 2 Are respectively set as
、
、
Wherein the point S 1 And the point S 2 Connected to form a straight line S 1 S 2 And the apex of the triangular face
In a straight line S 1 S 2 The coordinate of the vertical point projected in the vertical line direction is point N, and the coordinate of point N is
(ii) a Based on the triangular surface vertex
Point S 1 And a point S 2 The three side vectors of the triangle are respectively
，
And
from the vertical relationship between the vectors:
(1) (ii) a From the vector collinearity we get:
(2) (ii) a And then obtaining the following result according to a vector vertical distance calculation formula:
(3) (ii) a Substituting it into the solution unknowns k to obtain:
(4) (ii) a Substituting equation (4) into equation (3) to solve for x n 、y n And z n Namely the coordinate of the vertical point N, updating the coordinate of the traversed triangular surface vertex in the three-dimensional model to the coordinate corresponding to the coordinate of the vertical point N, namely finishing the updating of the point position.
Wherein x is 0 Is a point
Abscissa of (a), y 0 Is a point
Ordinate of (a), z 0 Is a point
Vertical coordinate of (a), x 1 Is a point S 1 Abscissa of (a), y 1 Is a point S 1 Ordinate of (c), z 1 Is a point S 1 Vertical coordinate of (c), x 2 Is a point S 2 Abscissa of (a), y 2 Is a point S 2 Ordinate of (a), z 2 Is a point S 2 Vertical coordinate of (c), x n Is the abscissa of point N, y n Is the ordinate, z, of point N n Is the vertical coordinate of point N, and k is an unknown number.
Further, as shown in fig. 8, the whole flow of the three-dimensional model optimization method of the building of the present invention is as follows:
s1, producing a three-dimensional model of a building by using a multi-view stereo satellite image;
s2, acquiring an unmanned aerial vehicle image;
s3, extracting a structural line of a building from the unmanned aerial vehicle image;
s4, restoring the position of the structural line in the space;
s5, filtering the structural lines;
s6, retrieving the vertex of the triangular surface closest to the two end points of the structural line on the three-dimensional model;
s7, searching the shortest path between the two triangular surface vertexes;
and step S8, updating the coordinates of the vertex of the triangular surface on the shortest path to the coordinates of the projection point of the triangular surface on the structural line.
Further, as shown in fig. 9, based on the method for optimizing the three-dimensional model of the building, the invention further provides an intelligent terminal, which includes a processor 10, a memory 20 and a display 30. Fig. 9 shows only some of the components of the intelligent terminal, but it should be understood that not all of the shown components are required to be implemented, and more or fewer components may be implemented instead.
The memory 20 may be an internal storage unit of the intelligent terminal in some embodiments, such as a hard disk or a memory of the intelligent terminal. The memory 20 may also be an external storage device of the Smart terminal in other embodiments, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) Card, a Flash memory Card (Flash Card), and the like, which are provided on the Smart terminal. Further, the memory 20 may also include both an internal storage unit and an external storage device of the smart terminal. The memory 20 is used for storing application software installed in the intelligent terminal and various data, such as program codes of the installed intelligent terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, the memory 20 stores a three-dimensional model optimization program 40 for a building, and the three-dimensional model optimization program 40 for a building can be executed by the processor 10, so as to implement the three-dimensional model optimization method for a building in the present application.
The processor 10 may be a Central Processing Unit (CPU), a microprocessor or other data Processing chip in some embodiments, and is used for executing program codes stored in the memory 20 or Processing data, such as performing a three-dimensional model optimization method for the building, and the like.
The display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch panel, or the like in some embodiments. The display 30 is used for displaying information at the intelligent terminal and for displaying a visual user interface. The components 10-30 of the intelligent terminal communicate with each other via a system bus.
In one embodiment, the following steps are implemented when the processor 10 executes the three-dimensional model optimization program 40 for a building in the memory 20:
acquiring a multi-view stereo satellite image, and reconstructing a three-dimensional model of a building based on the multi-view stereo satellite image;
acquiring a building image of a building in a designated area, and extracting a structural line of the building in the building image based on the building image;
recovering the spatial position of the structure line, filtering the structure line, and acquiring a first vertex and a second vertex which have the shortest endpoint distance between the three-dimensional model and the structure line;
and obtaining a shortest path between the first vertex and the second vertex, and updating the coordinates of the vertex of the triangular surface on the shortest path into the coordinates of the projection point of the vertex of the triangular surface on the structural line.
The acquiring of the multi-view stereo satellite image and the reconstruction of the three-dimensional building model based on the multi-view stereo satellite image specifically comprise:
acquiring a high-resolution multispectral DOM image and a high-precision DSM image;
extracting a building based on the high-resolution multispectral DOM image, and resampling the high-precision DSM image in the building to recover a spatial point cloud;
and encrypting the space point cloud by using a quadratic linear interpolation method to obtain dense point cloud, and reconstructing a three-dimensional model of the building by a network construction algorithm based on the dense point cloud.
The acquiring a building image of a building in a specified area and extracting a structural line of the building in the building image based on the building image specifically include:
acquiring a building image of a building in a designated area, and extracting a gray image in the building image;
after the gray level image is subjected to edge rendering algorithm processing, a set of edge pixel chains with adjacent pixels is generated;
and extracting a structural line from the edge pixel chain according to a least square straight line fitting method, and eliminating an error line segment by adopting a Helmholtz principle.
The recovering the spatial position of the structure line and filtering the structure line specifically include:
discretizing the structure line to obtain discrete points, projecting the discrete points to corresponding spatial positions through a collinear equation to obtain projection points based on spatial information on an image where the discrete points are located, and fitting the projection points into straight line segments;
based on the straight line segment, retrieving a triangular surface patch of the straight line segment according to a K neighbor retrieval method, and obtaining a normal vector of the triangular surface patch;
calculating an included angle between the normal vector and the structural line based on the normal vector of each triangular patch;
if the included angle is larger than a preset threshold value, confirming that the straight line segment is a structural line of the building, and reserving;
and if the included angle is smaller than a preset threshold value, confirming that the straight line segment is a texture line, and removing.
Wherein the collinearity equation is:
wherein, x is the abscissa of the image point, and y is the ordinate of the image point;x 0 is an element of the horizontal axis in the camera,y 0 is an element of the longitudinal axis within the camera,fis a vertical axis element in the camera;Xsis the abscissa of the camera station,Ysis the ordinate of the camera's camera station,Zsas vertical coordinate, X, of camera-shooting station A Is the abscissa, Y, of the projected point in space A As ordinate, Z, of spatial projection point A Vertical coordinates of the spatial projection points;a i being imagesaThe orientation element is a position element of the display,b i being imagesbAn orientation element for the orientation of the object,c i being imagescAn orientation element.
The obtaining of the first vertex and the second vertex, where the endpoint distance between the three-dimensional model and the structure line is the shortest, specifically includes:
taking the discrete point as a retrieval central point, and receiving triangular surface vertexes around the discrete point retrieved according to a space K nearest neighbor retrieval method;
and based on the vertex of the triangular surface, calculating the distances from the vertex of the triangular surface to the two endpoints of the structural line according to the triangular relation, and reserving a first vertex pt1 and a second vertex pt2 which are closest to the two endpoints.
The obtaining a shortest path between the first vertex and the second vertex and updating a coordinate of a vertex of the triangular surface on the shortest path to a projection point coordinate of the vertex of the triangular surface on the structural line specifically includes:
setting the structural line as an axis and r as a radius, and constructing a buffer area, wherein the radius is set to be 5 times of the average ground resolution of the image;
based on the buffer area, obtaining a triangular surface vertex on the three-dimensional model, and constructing an undirected graph;
based on the undirected graph, carrying out space retrieval according to a shortest path algorithm to calculate a shortest path between the first vertex pt1 and the second vertex pt2, and obtaining a point set pt (i);
and when the vertex of the triangular surface in the point set pt (i) is projected to the structural line corresponding to the vertex of the triangular surface in the vertical direction to obtain a vertical point, calculating the coordinate of the vertical point, and updating the coordinate of the vertex of the triangular surface into the coordinate of the vertical point.
The calculation process for calculating the coordinates of the vertical point specifically comprises the following steps:
make the apex of the triangular surface
And a point S on said construction line 1 And point S 2 Are respectively set as
、
、
Wherein the point S 1 And the point S 2 Are connected to form a straight line S 1 S 2 And the apex of the triangular face
In a straight line S 1 S 2 The projection coordinate in the vertical direction is point N
；
Based on the triangular surface vertex
Point S 1 And point S ` The three side vectors constructing the triangle are respectively
，
And
；
obtaining the following according to the vertical relation between the vectors:
；
from the vector collinearity we get:
；
obtaining the following result according to a vector vertical distance calculation formula:
；
solving the unknowns k as:
；
solving to obtain the coordinate of the projection coordinate N point;
wherein x is 0 Is a point
Abscissa of (a), y 0 Is a point
Ordinate of (a), z 0 Is a point
Vertical coordinate of (a), x 1 Is a point S 1 Abscissa of (a), y 1 Is a point S 1 Ordinate of (a), z 1 Is a point S 1 Vertical coordinate of (a), x 2 Is a point S 2 Abscissa of (a), y 2 Is a point S 2 Ordinate of (c), z 2 Is a point S 2 Vertical coordinate of (a), x n Is the abscissa of point N, y n Is the ordinate, z, of point N n Is the vertical coordinate of point N, and k is an unknown number.
The present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a three-dimensional model optimization program of a building, which when executed by a processor, implements the steps of the three-dimensional model optimization method of a building as described above.
In summary, the present invention provides a method for optimizing a three-dimensional model of a building, the method comprising: acquiring a multi-view stereo satellite image, and reconstructing a three-dimensional model of a building based on the multi-view stereo satellite image; acquiring a building image of a building in a designated area, and extracting a structure line of the building in the building image based on the building image; recovering the spatial position of the structure line, filtering the structure line, and acquiring a first vertex and a second vertex which have the shortest endpoint distance between the three-dimensional model and the structure line; and obtaining a shortest path between the first vertex and the second vertex, and updating the coordinates of the vertex of the triangular surface on the shortest path into the coordinates of the projection point of the vertex of the triangular surface on the structural line. The invention improves the traditional scheme of building three-dimensional reconstruction based on high-resolution three-dimensional surveying and mapping satellite images, and firstly, a multi-view three-dimensional satellite image is utilized to produce an LOD 2-level building three-dimensional model of a test area; then extracting a building structure line from the unmanned aerial vehicle image of the corresponding area, recovering the spatial position of the structure line, and filtering out texture lines; and finally, optimizing the building three-dimensional model by using the structural line for restoring the spatial position so as to obtain the LOD 2-level building three-dimensional model with higher precision and better visual effect.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one of 8230, and" comprising 8230does not exclude the presence of additional like elements in a process, method, article, or apparatus comprising the element.
Of course, it will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer readable storage medium, and when executed, the program can include the processes of the embodiments of the methods described above. The computer readable storage medium may be a memory, a magnetic disk, an optical disk, etc.
It is to be understood that the invention is not limited to the examples described above, but that modifications and variations may be effected thereto by those of ordinary skill in the art in light of the foregoing description, and that all such modifications and variations are intended to be within the scope of the invention as defined by the appended claims.
Claims (7)
1. A method for optimizing a three-dimensional model of a building, the method comprising:
acquiring a multi-view stereo satellite image, and reconstructing a three-dimensional model of a building based on the multi-view stereo satellite image;
acquiring a building image of a building in a designated area, and extracting a structure line of the building in the building image based on the building image;
recovering the spatial position of the structure line, filtering the structure line, and acquiring a first vertex and a second vertex which have the shortest endpoint distance between the three-dimensional model and the structure line;
acquiring a shortest path between the first vertex and the second vertex, and updating the coordinate of the vertex of the triangular surface on the shortest path into the projection point coordinate of the vertex of the triangular surface on the structural line;
the recovering of the spatial position of the structure line and the filtering of the structure line specifically include:
discretizing the structure line to obtain discrete points, projecting the discrete points to corresponding spatial positions through a collinear equation to obtain projection points based on spatial information on an image where the discrete points are located, and fitting the projection points into straight line segments;
based on the straight line segment, retrieving a triangular surface patch of the straight line segment according to a K neighbor retrieval method, and obtaining a normal vector of the triangular surface patch;
calculating an included angle between the normal vector and the structural line based on the normal vector of each triangular patch;
if the included angle is larger than a preset threshold value, confirming that the straight line segment is a structural line of the building, and reserving;
if the included angle is smaller than a preset threshold value, confirming that the straight line segment is a texture line, and removing;
the obtaining of the first vertex and the second vertex, where the endpoint distance between the three-dimensional model and the structure line is shortest, specifically includes:
taking the discrete point as a retrieval central point, and receiving triangular surface vertexes around the discrete point retrieved according to a space K nearest neighbor retrieval method;
based on the triangular surface vertex, calculating the distances from the triangular surface vertex to two endpoints of the structural line according to a triangular relation, and keeping a first vertex pt1 and a second vertex pt2 which are closest to the two endpoints;
the obtaining a shortest path between the first vertex and the second vertex, and updating a coordinate of a vertex of a triangular surface on the shortest path to a projection point coordinate of the vertex of the triangular surface on the structural line, specifically includes:
setting the structural line as an axis and r as a radius, and constructing a buffer area, wherein the radius is set to be 5 times of the average ground resolution of the image;
based on the buffer area, obtaining a triangular surface vertex on the three-dimensional model, and constructing an undirected graph;
based on the undirected graph, carrying out space retrieval according to a shortest path algorithm to calculate a shortest path between the first vertex pt1 and the second vertex pt2, and obtaining a point set pt (i);
and when the vertex of the triangular surface in the point set pt (i) is projected to the structural line corresponding to the vertex of the triangular surface in the vertical direction to obtain a vertical point, calculating the coordinate of the vertical point, and updating the coordinate of the vertex of the triangular surface into the coordinate of the vertical point.
2. The method for optimizing the three-dimensional model of the building according to claim 1, wherein the acquiring the multi-view stereoscopic satellite image and reconstructing the three-dimensional model of the building based on the multi-view stereoscopic satellite image specifically comprises:
acquiring a high-resolution multispectral DOM image and a high-precision DSM image;
extracting a building based on the high-resolution multispectral DOM image, and resampling the high-precision DSM image in the building to recover a spatial point cloud;
and encrypting the space point cloud by using a quadratic linear interpolation method to obtain dense point cloud, and reconstructing a three-dimensional model of the building by a network construction algorithm based on the dense point cloud.
3. The method for optimizing the three-dimensional model of the building according to claim 1, wherein the obtaining of the building image of the building in the designated area and the extracting of the structural line of the building in the building image based on the building image specifically comprise:
acquiring a building image of a building in a designated area, and extracting a gray image in the building image;
after the gray level image is processed by an edge rendering algorithm, a set of edge pixel chains with adjacent pixels is generated;
and extracting a structural line from the edge pixel chain according to a least square straight line fitting method, and eliminating an error line segment by adopting a Helmholtz principle.
4. The method of optimizing a three-dimensional model of a building of claim 1, wherein the collinearity equation is:
wherein, x is the abscissa of the image point, and y is the ordinate of the image point;
is an element of the horizontal axis in the camera,
is an internal camera longitudinal axis element, and f is an internal camera vertical axis element;
being the abscissa of the camera station,
is the ordinate of the camera's camera station,
for erecting camera-shooting stationsThe coordinates of the position of the object to be measured,
is the abscissa of the projected point in space,
is the ordinate of the projection point in space,
vertical coordinates of the spatial projection points;
is an a-direction element of the image,
is a b-directional element of the image,
is the c-orientation element of the image.
5. The method for optimizing the three-dimensional model of the building according to claim 1, wherein the calculation process for calculating the coordinates of the vertical point is specifically:
make the apex of the triangular surface
And a point S on said structure line 1 And point S 2 Are respectively set as
、
、
Wherein the point S1 and the pointS2 are connected to form a straight line S 1 S 2 And the apex of the triangular face
In a straight line S 1 S 2 The projection coordinate in the vertical direction is point N
；
Based on the triangular surface vertex
Three side vectors of a triangle are respectively constructed as a point S1 and a point S2
，
And
；
obtaining the following according to the vertical relation between the vectors:
；
from the vector collinearity we get:
；
obtaining the following result according to a vector vertical distance calculation formula:
；
solving the unknowns k as:
；
solving to obtain the coordinate of the projection coordinate N point;
wherein x is 0 Is a point
Abscissa of (a), y 0 Is a point
Ordinate of (a), z 0 Is a point
Vertical coordinate of (a), x 1 Is a point S 1 Abscissa of (a), y 1 Is a point S 1 Ordinate of (a), z 1 Is a point S 1 Vertical coordinate of (a), x 2 Is a point S 2 Abscissa of (a), y 2 Is a point S 2 Ordinate of (a), z 2 Is a point S 2 Vertical coordinate of (a), x n Is the abscissa of point N, y n Is the ordinate, z, of point N n Is the vertical coordinate of point N, and k is an unknown number.
6. An intelligent terminal, characterized in that the intelligent terminal comprises a memory, a processor and a three-dimensional model optimization program of a building stored on the memory and executable on the processor, the three-dimensional model optimization program of a building implementing the steps of the method for three-dimensional model optimization of a building according to any one of claims 1 to 5 when executed by the processor.
7. A computer-readable storage medium having stored thereon a computer program, the computer-readable storage medium having stored thereon a three-dimensional model optimization program of a building, which when executed by a processor, performs the steps of the method for three-dimensional model optimization of a building of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210819365.1A CN114896679B (en) | 2022-07-13 | 2022-07-13 | Three-dimensional model optimization method of building, intelligent terminal and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210819365.1A CN114896679B (en) | 2022-07-13 | 2022-07-13 | Three-dimensional model optimization method of building, intelligent terminal and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114896679A CN114896679A (en) | 2022-08-12 |
| CN114896679B true CN114896679B (en) | 2022-10-04 |
Family
ID=82729562
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210819365.1A Active CN114896679B (en) | 2022-07-13 | 2022-07-13 | Three-dimensional model optimization method of building, intelligent terminal and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114896679B (en) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN115409941B (en) * | 2022-08-31 | 2023-05-30 | 中南大学 | Three-dimensional ground object model fusion method and system in three-dimensional road scene |
| CN115982542B (en) * | 2023-03-21 | 2023-08-29 | 湖北泰跃卫星技术发展股份有限公司 | Method, device and storage medium for calculating adjacent relation between lands |
| CN116342848B (en) * | 2023-03-28 | 2024-02-02 | 云阳县优多科技有限公司 | Intelligent manufacturing method and system for toy |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109269472A (en) * | 2018-08-08 | 2019-01-25 | 广州市城市规划勘测设计研究院 | Oblique photograph measures building feature line drawing method, apparatus and storage medium |
| CN110838129A (en) * | 2019-11-18 | 2020-02-25 | 四川视慧智图空间信息技术有限公司 | Three-dimensional building model contour characteristic line extraction method based on oblique photogrammetry |
| WO2021197341A1 (en) * | 2020-04-03 | 2021-10-07 | 速度时空信息科技股份有限公司 | Monocular image-based method for updating road signs and markings |
| CN113516771A (en) * | 2021-06-21 | 2021-10-19 | 深圳市武测空间信息有限公司 | Building change feature extraction method based on live-action three-dimensional model |
| CN114255314A (en) * | 2022-02-28 | 2022-03-29 | 深圳大学 | Automatic texture mapping method, system and terminal for shielding avoidance three-dimensional model |
| CN114359437A (en) * | 2021-11-02 | 2022-04-15 | 杭州富阳富创大数据产业创新研究院有限公司 | Building structure two-dimensional plane map reconstruction method based on point cloud |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114119884A (en) * | 2021-11-09 | 2022-03-01 | 中国科学院空天信息创新研究院 | Building LOD1 model construction method based on high-score seven-satellite image |
-
2022
- 2022-07-13 CN CN202210819365.1A patent/CN114896679B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109269472A (en) * | 2018-08-08 | 2019-01-25 | 广州市城市规划勘测设计研究院 | Oblique photograph measures building feature line drawing method, apparatus and storage medium |
| CN110838129A (en) * | 2019-11-18 | 2020-02-25 | 四川视慧智图空间信息技术有限公司 | Three-dimensional building model contour characteristic line extraction method based on oblique photogrammetry |
| WO2021197341A1 (en) * | 2020-04-03 | 2021-10-07 | 速度时空信息科技股份有限公司 | Monocular image-based method for updating road signs and markings |
| CN113516771A (en) * | 2021-06-21 | 2021-10-19 | 深圳市武测空间信息有限公司 | Building change feature extraction method based on live-action three-dimensional model |
| CN114359437A (en) * | 2021-11-02 | 2022-04-15 | 杭州富阳富创大数据产业创新研究院有限公司 | Building structure two-dimensional plane map reconstruction method based on point cloud |
| CN114255314A (en) * | 2022-02-28 | 2022-03-29 | 深圳大学 | Automatic texture mapping method, system and terminal for shielding avoidance three-dimensional model |
Non-Patent Citations (5)
| Title |
|---|
| Combined Rule-Based and Hypothesis-Based Method for Building Model Reconstruction from Photogrammetric Point Clouds;Linfu Xie 等;《Remote Sensing》;20210314;第13卷(第06期);第1107页 * |
| Integration of aerial oblique imagery and terrestrial imagery for optimized 3D modeling in urban areas;Bo Wu 等;《ISPRS Journal of Photogrammetry and Remote Sensing》;20180313;第139卷;第119-132页 * |
| 三角网模型多目标加权最短路径的特征线提取;朱庆 等;《西南交通大学学报》;20210228;第56卷(第01期);第116-122页 * |
| 基于帧缓存的多角度影像精细纹理映射方法;朱庆 等;《西南交通大学学报》;20180413;第54卷(第02期);第269-277页 * |
| 建筑点云几何模型重建方法研究进展;杜建丽 等;《遥感学报》;20190525;第23卷(第03期);第374-391页 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114896679A (en) | 2022-08-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN114896679B (en) | Three-dimensional model optimization method of building, intelligent terminal and storage medium | |
| Wu et al. | Integration of aerial oblique imagery and terrestrial imagery for optimized 3D modeling in urban areas | |
| US9025861B2 (en) | System and method for floorplan reconstruction and three-dimensional modeling | |
| US9858640B1 (en) | Device and method for merging 3D point clouds from sparsely distributed viewpoints | |
| Eo et al. | Coregistration of terrestrial lidar points by adaptive scale-invariant feature transformation with constrained geometry | |
| Barazzetti et al. | True-orthophoto generation from UAV images: Implementation of a combined photogrammetric and computer vision approach | |
| Maurer et al. | Tapping into the Hexagon spy imagery database: A new automated pipeline for geomorphic change detection | |
| Alidoost et al. | An image-based technique for 3D building reconstruction using multi-view UAV images | |
| CN109416843A (en) | Real-time height mapping | |
| Kuschk | Large scale urban reconstruction from remote sensing imagery | |
| Cosido et al. | Hybridization of convergent photogrammetry, computer vision, and artificial intelligence for digital documentation of cultural heritage-a case study: the magdalena palace | |
| CN113689535B (en) | Building model generation method and device based on unmanned aerial vehicle image | |
| CN115272494B (en) | Calibration method and device for camera and inertial measurement unit and computer equipment | |
| JP4568845B2 (en) | Change area recognition device | |
| Kim et al. | Block world reconstruction from spherical stereo image pairs | |
| CN113077552A (en) | DSM (digital communication system) generation method and device based on unmanned aerial vehicle image | |
| Zhang et al. | Lidar-guided stereo matching with a spatial consistency constraint | |
| CN115564926A (en) | Three-dimensional patch model construction method based on image building structure learning | |
| Zhu et al. | Propagation strategies for stereo image matching based on the dynamic triangle constraint | |
| CN113566793A (en) | True orthoimage generation method and device based on unmanned aerial vehicle oblique image | |
| CN112002007A (en) | Model obtaining method and device based on air-ground image, equipment and storage medium | |
| Haggag et al. | Towards automated generation of true orthoimages for urban areas | |
| Sohn et al. | Sequential modelling of building rooftops by integrating airborne LiDAR data and optical imagery: preliminary results | |
| CN111583331B (en) | Method and device for simultaneous localization and mapping | |
| Nakagawa | Point cloud clustering using panoramic layered range image |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |