CN113674145B - Spherical surface splicing and real-time alignment method for PTZ (pan-tilt-zoom) moving image - Google Patents
Spherical surface splicing and real-time alignment method for PTZ (pan-tilt-zoom) moving image Download PDFInfo
- Publication number
- CN113674145B CN113674145B CN202010411353.6A CN202010411353A CN113674145B CN 113674145 B CN113674145 B CN 113674145B CN 202010411353 A CN202010411353 A CN 202010411353A CN 113674145 B CN113674145 B CN 113674145B
- Authority
- CN
- China
- Prior art keywords
- image
- background
- target
- focal length
- images
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 69
- 239000011159 matrix material Substances 0.000 claims description 51
- 230000008569 process Effects 0.000 claims description 20
- 230000006870 function Effects 0.000 description 15
- 238000012545 processing Methods 0.000 description 11
- 230000004927 fusion Effects 0.000 description 8
- 238000001514 detection method Methods 0.000 description 6
- 238000006243 chemical reaction Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 230000036961 partial effect Effects 0.000 description 3
- 230000003068 static effect Effects 0.000 description 3
- 238000003066 decision tree Methods 0.000 description 2
- 230000002452 interceptive effect Effects 0.000 description 2
- 239000003550 marker Substances 0.000 description 2
- 238000003672 processing method Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 230000009466 transformation Effects 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000004364 calculation method Methods 0.000 description 1
- 230000000670 limiting effect Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000005654 stationary process Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000036962 time dependent Effects 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T3/00—Geometric image transformations in the plane of the image
- G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
- G06T3/4038—Image mosaicing, e.g. composing plane images from plane sub-images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/33—Determination of transform parameters for the alignment of images, i.e. image registration using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10016—Video; Image sequence
- G06T2207/10021—Stereoscopic video; Stereoscopic image sequence
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Image Analysis (AREA)
- Studio Devices (AREA)
Abstract
The application provides a spherical surface splicing and real-time alignment method of PTZ moving images, which comprises the steps of obtaining a three-dimensional image of a panoramic background and a panoramic feature point set in a target scene; acquiring a target image in a target scene shot by a PTZ camera; determining a panoramic local area characteristic point set corresponding to the target image from the panoramic characteristic point set according to the target image; acquiring a target image feature point set of a target image; and adjusting the characteristic points in the characteristic point set of the target image according to the characteristic point set of the local area and the characteristic point set of the target image, so that the difference between the positions of the characteristic points in the characteristic point set of the target image and the positions of the characteristic points in the characteristic point set of the local area is smaller than the preset difference, obtaining an adjusted target image, and projecting the adjusted target image into the three-dimensional image to obtain a fused three-dimensional image. Therefore, the method can realize the accurate matching of the target image in the target scene and the three-dimensional image of the panoramic background, and solve the problem of real-time alignment.
Description
Technical Field
The embodiment of the application relates to the technical field of virtual fusion, in particular to a spherical surface splicing and real-time alignment method of PTZ moving images.
Background
The virtual-real fusion technology is to display images or video streams in a three-dimensional scene, namely, fusion display is carried out on the images and the three-dimensional model, and a user can watch the images or the video streams at any view angle. The technology has important application in the fields of urban roaming, traffic analysis, video monitoring and the like. Because the omnibearing moving and lens zooming and zooming control (PTZ) camera has the characteristics of wide field of view, adjustable gesture, zooming and the like, the PTZ camera greatly enhances the information capturing amount of virtual and real fusion scenes and improves the utilization rate of camera resources.
In the prior art, a registration method of a PTZ camera in a three-dimensional scene is a model positioning-based method, and the registration method further estimates the transformation relationship between a model and an image by finding the corresponding relationship between key points on the model and characteristic points of the image to be calibrated. For example, in a sports field scene (for example, a ball race field), a random decision tree method is used for training the spatial correspondence relationship between the characteristic points of sports field markers (such as goalkeeper limit and doorframe in a football field) in an image and the key points of a three-dimensional model of a sports field aiming at the sports field model, the characteristic points of the sports field markers in the image to be calibrated are matched with the key points of the three-dimensional model of the sports field, and the registration between the image to be calibrated and the three-dimensional model of the sports field is further realized.
However, the method is mainly suitable for a scene with an explicit marker, and for a scene with an ambiguous marker, such as an indoor scene, the characteristic points may be confused in matching, so that accurate registration between the image to be calibrated and the three-dimensional model cannot be realized, and further the problem of real-time alignment of the PTZ camera in the motion process is caused.
Disclosure of Invention
The embodiment of the application provides a method for splicing and aligning a spherical surface of a PTZ moving image in real time, which aims to solve the problem of real-time alignment of a PTZ camera in the prior art in the moving process.
In a first aspect, an embodiment of the present application provides a method for spherical stitching and real-time alignment of a PTZ moving image, where the method includes: acquiring a three-dimensional image of a panoramic background in a target scene and a panoramic feature point set; acquiring a target image in a target scene shot by a PTZ camera; determining a panorama local area characteristic point set corresponding to the target image from the panorama characteristic point set according to the target image; acquiring a target image feature point set of the target image; according to the panorama local area feature point set and the target image feature point set, twisting and adjusting feature points in the target image feature point set, so that the difference between the positions of the feature points in the target image feature point set and the positions of the feature points in the matched panorama local area feature point set is smaller than the preset difference, and an adjusted target image is obtained; and projecting the adjusted target image into the three-dimensional image of the panoramic background according to a real-time texture projection principle to obtain a fused three-dimensional image.
Optionally, the determining, according to the target image, a panorama local area feature point set corresponding to the image from the panorama feature point set includes: determining the position of the target image mapped into the panoramic background image according to the rotation angle of the target image shot by the PTZ camera relative to the initial image shot and the focal length of the target image shot by the PTZ camera; and determining the panorama local area characteristic point set corresponding to the position from the panorama characteristic point set of the three-dimensional image of the panorama background.
Optionally, the acquiring the three-dimensional image of the panoramic background in the target scene includes: acquiring an image of a panoramic background in the target scene; and carrying out three-dimensional modeling according to the image of the panoramic background to obtain a three-dimensional image of the panoramic background.
Optionally, the acquiring the image of the panoramic background in the target scene includes: acquiring images of a plurality of backgrounds in a target scene; and carrying out panoramic stitching on the images with the multiple backgrounds to obtain an image with a panoramic background.
Optionally, the images of the multiple backgrounds are obtained according to images shot by the PTZ camera at different focal length levels, and the panoramic stitching is performed on the images of the multiple backgrounds to obtain the image of the panoramic background, including: acquiring homography matrixes and rotation matrixes among the images of the multiple backgrounds under the same focal length level according to the images of the multiple backgrounds under the same focal length level; acquiring an internal reference matrix of the PTZ camera under the focal length grade according to the homography matrix and the rotation matrix, wherein the internal reference matrix comprises a focal length corresponding to the focal length grade and principal point coordinates of an image of a background under the focal length grade; correcting principal point coordinates of an image of a background in a back focal length grade in two adjacent focal length grades according to an internal reference matrix of the PTZ camera in the two adjacent focal length grades by the following formula; obtaining target principal point coordinates according to the corrected principal point coordinates of the background images under different focal length levels; and carrying out panoramic stitching on the images of the multiple backgrounds obtained under the different focal length grades according to the focal length respectively corresponding to the different focal length grades, the focal length corresponding to the adjacent two focal length grades and the target principal point coordinate, so as to obtain the image of the panoramic background.
In the method, in the process of the invention,indicating a focus level of Z i When the background image is used, the pixel coordinate value of any characteristic point of the background image is obtained; />Indicating a focus level of Z i-1 When the background image is used, the pixel coordinate value of any characteristic point of the background image is obtained; f (Z) i ) Indicating a focus level of Z i A focal length value of the PTZ camera at that time; f (Z) i-1 ) Indicating a focus level of Z i-1 A focal length value of the PTZ camera at that time; (p) x ,p y ) And the principal point coordinates of the background image at the last focal length level in the two corrected adjacent focal length levels are shown.
Optionally, the acquiring the images of the multiple backgrounds in the target scene includes: acquiring a plurality of images in a target scene shot by a PTZ camera at different positions; calculating a loss value according to a plurality of images photographed under a position and a loss function aiming at the position; if the loss value is larger than the preset threshold, the loss value is used as an iteration error, and the loss value is recalculated according to a plurality of images shot at the position and the loss function; and if the loss value is smaller than a preset threshold value, determining the image of the background obtained by the loss value as the image of the background of the position.
The loss function is:
wherein I is K Representing a kth image of a plurality of images in a target scene photographed at the position; i represents an image of the background at the location; infinity represents an element-by-element product; w (W) K A two-dimensional matrix of weights representing pixel differences of the K-th image at the location and the image of the background, each element of the matrix being Representing the pixel value of the kth image at the same position in the ith row and jth column, ε represents the iteration error, +.>A weight value representing a pixel difference value between a kth image at the position and an image of the background at an ith row and a jth column; pixels I of the ith row and jth column of image I of the background at said location ij The initial value is a weighted average of a plurality of image pixels in the target scene taken at that position, +.>
Optionally, the method further comprises: and displaying the fused three-dimensional image.
According to the spherical surface splicing and real-time alignment method for the PTZ moving image, a three-dimensional image of a panoramic background and a panoramic feature point set in a target scene are obtained; acquiring a target image in a target scene shot by a PTZ camera; determining a panorama local area characteristic point set corresponding to the target image from the panorama characteristic point set according to the target image; acquiring a target image feature point set of the target image; according to the panorama local area feature point set and the target image feature point set, twisting and adjusting feature points in the target image feature point set, so that the difference between the positions of the feature points in the target image feature point set and the positions of the feature points in the matched panorama local area feature point set is smaller than the preset difference, and an adjusted target image is obtained; and projecting the adjusted target image into the three-dimensional image of the panoramic background according to a real-time texture projection principle to obtain a fused three-dimensional image. Therefore, the method can realize the accurate matching of the target image in the target scene shot by the PTZ camera and the acquired three-dimensional image of the panoramic background in the target scene, and solve the problem of real-time alignment in the prior art.
Drawings
In order to more clearly illustrate the embodiments of the present application or the technical solutions of the prior art, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it will be obvious that the drawings in the following description are some embodiments of the present application, and that other drawings can be obtained according to these drawings without inventive effort to a person skilled in the art.
Fig. 1 is a flowchart of a method for spherical stitching and real-time alignment of a PTZ moving image according to an embodiment of the present application;
fig. 2 is a flowchart of a method for spherical stitching and real-time alignment of a PTZ moving image according to another embodiment of the present application;
fig. 3 is a three-dimensional image perspective view of a panoramic background according to an embodiment of the present application;
FIG. 4 is a schematic diagram illustrating a target image feature point distortion adjustment matching according to an embodiment of the present application;
fig. 5 is a schematic perspective view of a fused three-dimensional image according to an embodiment of the present application.
Detailed Description
Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.
The spherical surface splicing and real-time alignment method of PTZ moving images mainly relates to two aspects. One aspect is PTZ camera registration in a three-dimensional scene. The registration of the PTZ camera refers to determining the correct pose of the PTZ camera in the three-dimensional scene, such as the position, orientation, focal length, etc. of the PTZ camera in the three-dimensional scene. The registration method of the PTZ camera in the prior art under the three-dimensional scene is a model positioning-based method, and the registration method further estimates the transformation relation between the model and the image by finding the corresponding relation between key points on the model and characteristic points of the image to be calibrated. For example, in a sports field scene (for example, a ball race field), a random decision tree method is used for training the spatial correspondence between the characteristic points of sports field markers (for example, goalkeeper limit, penalty points and the like in a football field) in an image and the key points of a three-dimensional model of a sports field aiming at the sports field model, the characteristic points of the sports field markers in the image to be calibrated are matched with the key points of the three-dimensional model of the sports field, and the registration between the image to be calibrated and the three-dimensional model of the sports field is further realized. However, the method is mainly suitable for scenes with clear markers, and for scenes with undefined markers, such as indoor scenes, no obvious markers exist in the scenes, so that characteristic points can be matched to be confused, accurate registration between an image to be calibrated and a three-dimensional model can not be realized, and further the problem of real-time alignment of the PTZ camera in the motion process is caused.
Based on the technical problems, the application provides a spherical surface splicing and real-time alignment method of a PTZ moving image. According to the method, the characteristic points in the image shot in real time by the PTZ camera are mainly adjusted, so that the characteristic points in the image shot in real time by the PTZ camera are matched with the three-dimensional image characteristic points of the background in the corresponding target scene, the pose of the PTZ camera is solved through a motion camera model in the motion process (such as rotation) of the PTZ camera, then the adjusted image shot in real time by the PTZ camera is projected into the three-dimensional image of the background in the corresponding target scene, and the fusion of the image shot in real time by the PTZ camera and the three-dimensional image of the background in the motion process is realized. In the method, the characteristic points of the background in the image shot by the PTZ camera in real time are matched with the characteristic points of the background in the target scene, and the characteristic points of the target object in the image shot by the PTZ camera in real time are removed as error points, so that the accurate registration between the shot and acquired image and the three-dimensional image can be realized, and the problem of real-time alignment of the PTZ camera in the motion process is solved.
The technical solution of the present application will be described below with reference to several specific embodiments.
Fig. 1 is a flow chart of a method for spherical surface stitching and real-time alignment of a PTZ moving image according to an embodiment of the present application, as shown in fig. 1, the method according to the embodiment of the present application may include:
s201, acquiring a three-dimensional image of a panoramic background in a target scene.
A three-dimensional image of the panoramic background in the target scene is acquired, the three-dimensional image containing images of all the backgrounds in the target scene, e.g., a conference room. The image of the background may be an image formed by an object that is stationary in the conference room, i.e., an image of the background does not include a moving object such as a user.
S202, acquiring a panoramic feature point set of a three-dimensional image of a panoramic background in a target scene.
According to the feature point detection algorithm, a panoramic feature point set of the three-dimensional image of the panoramic background is extracted from the three-dimensional image of the panoramic background in the target scene acquired in S201, and the feature point set is called a panoramic feature point set. The feature point detection algorithm belongs to the prior art, and is not described herein.
S203, acquiring a target image in a target scene shot by the PTZ camera.
A PTZ camera is placed in a target scene, and the PTZ camera can capture and acquire a video image in the target scene in real time, and acquire a frame of image in the target scene from the captured video image as a target image.
S204, determining a panorama local area characteristic point set corresponding to the target image from the panorama characteristic point set according to the target image.
And according to the target image shot by the PTZ camera in S203, finding out the corresponding area of the target image in the three-dimensional image of the panoramic background. Then, a feature point set of a corresponding region of the target image in the three-dimensional image of the panoramic background is determined from the panoramic feature point set of the three-dimensional image of the panoramic background acquired in S202, and the feature point set is called a panoramic local region feature point set. The processing method of each frame of image in the video image shot by the PTZ camera in real time is similar to the processing method of the target image, and will not be described here again.
S205, acquiring a target image feature point set of the target image.
According to the feature point detection algorithm, a feature point set of an image, which is called a target image feature point set, is extracted from a target image in a target scene photographed by the PTZ camera in S203. Wherein, the feature point detection algorithm belongs to the prior art, and is not described herein.
Alternatively, the execution sequence of S204 and S205 is not sequential, and S204 may be executed first and S205 may be executed second, or S205 may be executed first and S204 may be executed second.
S206, according to the panorama local area feature point set and the target image feature point set, twisting and adjusting feature points in the target image feature point set, so that the difference between the positions of the feature points in the target image feature point set and the positions of the feature points in the matched panorama local area feature point set is smaller than the preset difference, and an adjusted target image is obtained.
According to the panorama local area feature point set obtained in the step S204 and the target image feature point set obtained in the step S205, a random sampling coincidence (Random Sample Consensus, RANSAC) algorithm is adopted for the feature points in the panorama local area feature point set and the target image feature point set, and the feature points in the target image feature point set of the target image and the feature points in the panorama local area feature point set corresponding to the feature points in the three-dimensional image of the panorama background are matched, so that initial matching of the target image and the three-dimensional image of the panorama background is realized. And then obtaining the best matching point pair (for example, the shortest distance between the matching points) according to the matching relation between the target image and the characteristic points of the three-dimensional image of the panoramic background. If the difference between the positions of the feature points in the feature point set of the target image in the matching point pair and the positions of the feature points in the feature point set of the partial region of the matched panoramic image is smaller than the preset difference, the feature points in the feature point set of the target image are not required to be adjusted, and at the moment, the target image is any frame of image in the video shot by the PTZ camera. If the difference between the positions of the characteristic points in the characteristic point set of the target image in the matching point pair and the positions of the characteristic points in the characteristic point set of the partial region of the matched panoramic image is greater than or equal to the preset difference, the characteristic points in the characteristic point set of the target image are required to be subjected to distortion adjustment, so that the difference between the positions of the characteristic points in the characteristic point set of the target image and the positions of the characteristic points in the characteristic point set of the partial region of the matched panoramic image is smaller than the preset difference, and the adjusted target image is obtained.
S207, projecting the adjusted target image into the three-dimensional image of the panoramic background according to the real-time texture projection principle, and obtaining a fused three-dimensional image.
According to the real-time texture projection principle, the adjusted target image obtained in the step S206 is projected into the three-dimensional image of the panoramic background in the target scene obtained in the step S201, and a fused three-dimensional image is obtained, so that the accurate registration of the image shot by the PTZ camera in real time and the three-dimensional image is realized, the image display under the virtual-real fusion scene is realized, and the visual experience of a user is improved.
According to the spherical surface splicing and real-time alignment method for the PTZ moving image, a three-dimensional image of a panoramic background and a panoramic feature point set in a target scene are obtained; acquiring a target image in a target scene shot by a PTZ camera; determining a panoramic local area characteristic point set corresponding to the target image from the panoramic characteristic point set according to the target image; acquiring a target image feature point set of a target image; according to the panorama local area feature point set and the target image feature point set, twisting and adjusting feature points in the target image feature point set, so that the difference between the positions of the feature points in the target image feature point set and the positions of the feature points in the matched panorama local area feature point set is smaller than the preset difference, and an adjusted target image is obtained; and projecting the adjusted target image into the three-dimensional image of the panoramic background according to the real-time texture projection principle to obtain a fused three-dimensional image. Therefore, the method can realize the accurate matching of the target image in the target scene shot by the PTZ camera and the acquired three-dimensional image of the panoramic background in the target scene, and solve the problem of real-time alignment in the prior art.
In some embodiments, fig. 2 is a schematic flow chart of a method for spherical stitching and real-time alignment of a PTZ moving image according to another embodiment of the present application, as shown in fig. 2, on the basis of the embodiment shown in fig. 1, the method according to the embodiment of the present application may include:
s301, acquiring an image of a panoramic background in a target scene.
The PTZ camera shoots a plurality of images in the target scene, and then carries out corresponding processing on the plurality of images in the target scene to obtain a plurality of images of the background in the target scene. And then performing spherical panorama conversion, seam estimation and multi-band fusion on the images of the multiple backgrounds in the target scene, so as to realize panorama stitching of the images of the multiple backgrounds in the target scene, and repairing a panorama stitching result in an interactive mode to obtain the images of the panorama backgrounds in the target scene. The technology of performing spherical panorama conversion, joint estimation and multi-band fusion on images of multiple backgrounds in a target scene, implementing panorama stitching of the images of the multiple backgrounds in the target scene, and repairing panorama stitching results in an interactive manner belongs to the prior art, and is not described herein.
Optionally, one possible implementation of S301 is:
s301a, acquiring images of a plurality of backgrounds in a target scene.
The PTZ camera shoots a plurality of images in the target scene, and then carries out corresponding processing on the plurality of images in the target scene to obtain images of the backgrounds of the plurality of images in the target scene. Corresponding processing of multiple images in a target scene is described in the following embodiments
Optionally, one possible implementation of S301a is:
s301a1, acquiring a plurality of images in a target scene photographed by a PTZ camera at different positions.
The plurality of images in the target scene are a plurality of images acquired by the PTZ camera taken at different focal length levels and different positions.
The PTZ camera rotates 360 degrees in the horizontal direction and 90 degrees in the vertical direction under the same focal length level in a plurality of focal length levels, and performs full-sphere traversal shooting to acquire a plurality of images in a target scene. In this embodiment, two different focal length levels of the PTZ camera are taken as an example, and the PTZ camera is located at three different positions at each focal length level. The PTZ camera shoots and acquires a plurality of images in a target field at a first position under a first focal length level; shooting and acquiring a plurality of images in a target field at a second position; a plurality of images in the target field are captured at a third location. The PTZ camera shoots and acquires a plurality of images in a target field at a first position under a second focal length level; shooting and acquiring a plurality of images in a target field at a second position; a plurality of images in the target field are captured at a third location. The target scene, for example, a plurality of images in a conference room, includes both moving objects, for example, a person, and stationary objects, for example, a table chair in the conference room.
According to the method, multiple images in the target scene shot by the PTZ camera at each position corresponding to different positions under different focal length levels can be acquired.
S301a2, for one position, a loss value is calculated from a plurality of images captured in the position and a loss function.
For example, for a plurality of images in the target field captured at the first position at the first focal length level in S301a1 described above, a loss function is used to calculate the loss value of the plurality of images. Similarly, loss values for multiple images at different locations at different focus levels are calculated using a loss function.
S301a3, judging whether the loss value is smaller than a preset threshold value or not; if the loss value is greater than or equal to the preset threshold, taking the loss value as an iteration error, and returning to execute S301a2; if the loss value is smaller than the preset threshold, S301a4 is executed.
S301a4, the image of the background obtained by the loss value is determined as the image of the background of the position.
Specifically, for a position at the same focal length level, a loss function is used to calculate a loss value for a plurality of images captured at the position.
The loss function is:
in formula I, I K Representing a kth image of a plurality of images in a target scene photographed at the position; i represents an image of the background at the location; infinity represents an element-by-element product; w (W) K A two-dimensional matrix of weights representing pixel differences of the K-th image at the location and the image of the background, each element of the matrix being Representing the pixel value of the kth image at the same position in the ith row and jth column, ε represents the iteration error, +.>A weight value representing a pixel difference value between a kth image at the position and an image of the background at an ith row and a jth column; pixel initial value I of ith row and jth column of image I of background at said position ij For a weighted average of a plurality of image pixels in the target scene taken at that position, +.>
The specific calculation process of the loss value is as follows:
1) Calculating the initial value I of the pixel of the ith row and jth column of the image I of the background at the position ij ，I ij For a weighted average of a plurality of image pixels in the target scene taken at that location, formula two is calculated.
In the second formula, the first formula is a formula,representing pixel values of a kth image in an ith row and a jth column in a plurality of images acquired at the same position; /> A weight value representing a pixel difference value between a kth image of the plurality of images at the position and a jth column of the image of the background at an ith row, and a pixel initial value I of the jth column of the ith row of the image I of the background at the position is obtained ij When the iteration error epsilon=0 is taken; p=0.5.
2) Will be initial value I ij Substituting the value into the formula I to obtain the loss value of the loss function.
3) If the calculated loss value epsilon is larger than the preset threshold value, taking the loss value as an iteration error, and substituting the iteration error epsilon intoIn (2) re-finding the new +.>Then new +.>Substituting the pixel value I into the formula II to obtain the pixel value I of the ith row and the jth column of the image I of the background at the position ij . And finally, executing the step 2) to obtain the loss value of the loss function until the loss value of the loss function is smaller than a preset threshold value. And when the loss value of the loss function is smaller than a preset threshold value, determining the image of the background obtained by the loss value as the image of the background at the position.
4) Repeating the steps 1) -3), and acquiring images of the background at different positions under the same focal length level. The acquired images of the background at different positions in the same focal length level are, for example, the image of the background at the first position in the first focal length level, the image of the background at the second position in the first focal length level, and the image of the background at the third position in the first focal length level, and at this time, the images of the three backgrounds at different positions are corresponding to the image of the background at the first focal length level.
Similarly, three images of the background at the second focus level are acquired according to the above steps.
S301b, carrying out panoramic stitching on the images of the multiple backgrounds to obtain an image of the panoramic background.
And (3) correspondingly processing the images of the multiple backgrounds at the multiple positions under different focal lengths acquired in the step (S301 a) to acquire the image of the panoramic background. The corresponding processing of the plurality of images in the target scene is described in the following embodiments.
Optionally, one possible implementation of S301b is:
s301b1, obtaining a homography matrix and a rotation matrix between the images of the multiple backgrounds under the same focal length level according to the images of the multiple backgrounds under the same focal length level.
And obtaining a homography matrix and a rotation matrix between every two adjacent background images under the same focal length level according to every two adjacent background images in the multiple background images under the same focal length level. And then, carrying out constraint processing on the homography matrix and the rotation matrix between every two adjacent background images so as to obtain homography matrices and rotation matrices between the plurality of background images under the focal length level. The constraint processing of the homography matrix and the rotation matrix between every two adjacent background images belongs to the prior art, and is not repeated here.
For example, a homography matrix, such as a first homography matrix and a first rotation matrix, between two adjacent images of the background at the first focus level is obtained from the image of the background at the first position and the image of the background at the second position at the first focus level. And obtaining homography matrixes, such as a second homography matrix and a second rotation matrix, between two adjacent background images under the first focal length level according to the background images under the second position and the third position under the first focal length level. And then, carrying out constraint processing on the first homography matrix and the second homography matrix and constraint processing on the first rotation matrix and the second rotation matrix, and obtaining homography matrixes and rotation matrixes among images of three backgrounds under the first focal length level.
Similarly, homography and rotation matrices between images of three backgrounds at the second focus level are obtained according to the method described above.
S301b2, acquiring an internal reference matrix of the PTZ camera under the focal length grade according to the homography matrix and the rotation matrix, wherein the internal reference matrix comprises a focal length corresponding to the focal length grade and principal point coordinates of an image of a background under the focal length grade.
For example, according to the homography matrix and the rotation matrix between the three images of the background at the first focal length level acquired in S301b1, an internal reference matrix of the PTZ camera at the first focal length level is acquired, where the internal reference matrix includes the focal length corresponding to the first focal length level and principal point coordinates of the image of the background at the first focal length level.
Similarly, according to the homography matrix and the rotation matrix between the three images of the background at the second focal length level acquired in S301b1, an internal reference matrix of the PTZ camera at the second focal length level is acquired, where the internal reference matrix includes the focal length corresponding to the second focal length level and principal point coordinates of the image of the background at the second focal length level.
S301b3, correcting principal point coordinates of an image of a background in a back focal length grade in two adjacent focal length grades according to an internal reference matrix of the PTZ camera in the two adjacent focal length grades through a formula III.
For example, according to the focal length of the first focal length level corresponding to the second focal length level acquired in S301b2 and the principal point coordinates of the image of the background at the corresponding focal length level, the principal point coordinates of the image of the background at the second focal length level are corrected using formula three.
In the third formula, the formula (III),indicating a focus level of Z i When the background image is used, the pixel coordinate value of any characteristic point of the background image is obtained; />Indicating a focus level of Z i-1 At the time, the pixel coordinate value of any feature point of the background image is thatAnd->Is at focal length level Z i And a focal length level of Z i-1 The feature points matched with the two adjacent background images are obtained; f (Z) i ) Indicating a focus level of Z i A focal length value of the PTZ camera at that time; f (Z) i-1 ) Indicating a focus level of Z i-1 A focal length value of the PTZ camera at that time; (p) x ,p y ) And the principal point coordinates of the background image at the last focal length level in the two corrected adjacent focal length levels are shown.
And S301b4, obtaining target principal point coordinates according to the corrected principal point coordinates of the background images under different focal length levels.
And according to the corrected sizes of the background images under different focal length levels, averaging the principal point coordinates of the corrected background images under different focal length levels, and obtaining target principal point coordinates of the background images.
For example, the size of the background image at the first focal length level and the principal point coordinates of the background image at the corrected second focal length level are averaged to obtain the target principal point coordinates of the background image.
S301b5, performing panoramic stitching on the images of the multiple backgrounds according to the internal reference matrixes respectively corresponding to the different focal length grades, and obtaining the images of the panoramic backgrounds.
And (3) carrying out panoramic stitching on the images of the multiple backgrounds under different focal length grades according to the acquired internal reference matrixes with different focal length grades in S301b2 to obtain images of panoramic backgrounds.
S302, carrying out three-dimensional modeling according to the image of the panoramic background to obtain a three-dimensional image of the panoramic background.
According to the image of the panoramic background in the target scene acquired in S301, performing three-dimensional modeling on the image of the panoramic background to acquire a three-dimensional image of the panoramic background, where performing three-dimensional modeling processing on the image of the panoramic background belongs to the prior art, and is not described herein. For example, as shown in fig. 3, fig. 3 is a three-dimensional schematic diagram of a panoramic background according to an embodiment of the present application.
S303, acquiring a panoramic feature point set of a three-dimensional image of a panoramic background in the target scene.
According to the feature point detection algorithm, a feature point set of images of a plurality of backgrounds is extracted from the images of a plurality of backgrounds in the target scene acquired in S301 a. And then, carrying out spherical panorama conversion on each feature point of the extracted images of the multiple backgrounds to obtain a feature point set of the images of the panoramic backgrounds in the target scene. And then carrying out three-dimensional conversion on the feature point set of the image of the panoramic background in the target scene to obtain the panoramic feature point set of the three-dimensional image of the panoramic background in the target scene. Wherein, the feature point detection algorithm belongs to the prior art, and is not described herein.
S304, acquiring a target image in a target scene shot by the PTZ camera.
The specific implementation process of S304 may be referred to as related description in the embodiment shown in fig. 1, which is not repeated here.
S305, determining a position of the target image mapped to the panoramic background image according to a rotation angle of the target image captured by the PTZ camera relative to the captured initial image and a focal length of the target image captured by the PTZ camera.
According to the pose, such as the horizontal rotation angle, the vertical rotation angle and the current focal length value, of the target image shot by the PTZ camera relative to the shooting initial image, the corresponding position of the target image shot by the PTZ camera, which is mapped in the image of the panoramic background acquired in S301, is found.
S306, determining a panorama local area characteristic point set corresponding to the position from the panorama characteristic point set of the three-dimensional image of the panorama background.
According to the position of the target image mapped into the image of the panoramic background determined in S305, a panoramic local area feature point set of the target image at the corresponding position in the image of the panoramic background is determined from the panoramic feature point set of the image of the panoramic background acquired in S303.
S307, acquiring a target image feature point set of the target image.
The specific implementation process of S307 may be referred to the related description in the embodiment shown in fig. 1, and will not be described herein.
Alternatively, the execution sequence of S306 and S307 is not sequential, and S306 may be executed first and S307 may be executed first, or S307 may be executed first and S306 may be executed second.
And S308, according to the panorama local area feature point set and the target image feature point set, adjusting feature points in the target image feature point set so that the difference between the positions of the feature points in the target image feature point set and the positions of the feature points in the matched panorama local area feature point set is smaller than the preset difference, and obtaining an adjusted target image.
The characteristic point matching adjustment in the step mainly comprises two stages, namely a motion process and a static process. The video image shot by the PTZ camera is also continuously changed in the motion process, at the moment, the time spent by feature point matching is too much, and the real-time requirement cannot be met, so that the pose of the PTZ camera under different time is acquired through a PTZ camera motion model, and then the video image is directly projected into a three-dimensional scene, because the image content is moving, even if small-amplitude texture errors exist, the user experience is not influenced. The stationary process uses a feature point matching calibration algorithm.
In the motion process, taking rotation as an example, the orientation of the PTZ camera, for example, in the rotation process of the PTZ camera, is continuously changed, that is, the pose of the PTZ camera is continuously changed, so that the pose of the PTZ camera needs to be acquired in real time. Namely, the PTZ camera pose is obtained by adopting a motion camera model algorithm. According to the algorithm, the rotation angle of the PTZ camera is calculated according to the following formula IV when the PTZ camera rotates, and the pose of the PTZ camera is updated by collecting time-dependent change data of the rotation angle of the PTZ camera in the rotation process and fitting the rotation angle and a function image of the rotation time through a function. And projecting the image into a three-dimensional scene of the panoramic background image, so as to realize the alignment of textures of each frame of image of the video shot by the PTZ camera and the panoramic background image in the motion change process.
The change relation of the rotation angle of the PTZ camera with time is as follows:
in the fourth formula, t represents the time of rotation after the PTZ camera starts rotating, θ represents the rotation angle of the PTZ camera, and a, B, a, B represent constants.
For example, as shown in fig. 4, fig. 4 is a schematic diagram of target image feature point distortion adjustment matching according to an embodiment of the present application. The PTZ camera rotation process is described as an example. When the PTZ camera rotates, the content of the PTZ camera real-time texture changes, assuming that the content of the real-time texture display before the rotation is an initial image of the a position (initial position) in the scene, the initial image of the real-time texture display becomes a real-time image of the C position, such as a target image, after a certain angle of rotation. When the PTZ camera starts to rotate, the pose of the PTZ camera from the A position to the C position can be calculated according to the rotation time, and the pose of the PTZ camera is continuously updated. When the pose of the PTZ camera changes, the real-time texture is remapped to the image of the panoramic background. After rotation is stopped, namely the PTZ camera is positioned at the C position, the PTZ camera is positioned at a static state, and then feature matching and local distortion are carried out on feature points of a target image at the C position and region feature points of a C position region of an image of the panoramic background, so that more accurate registration of real-time texture and panoramic texture is realized in the static state.
S309, projecting the adjusted target image into the three-dimensional image of the panoramic background according to the real-time texture projection principle, and obtaining a fused three-dimensional image.
The specific implementation process of S308 to S309 may be referred to the related description in the embodiment shown in fig. 1, and will not be described herein.
S310, displaying the fused three-dimensional image.
The fused three-dimensional image is displayed in the image of the three-dimensional background, so that a user can watch the image or video content shot by the PTZ camera at any view angle. For example, as shown in fig. 5, fig. 5 is a schematic perspective view of a fused three-dimensional image according to an embodiment of the present application.
In the embodiment, a plurality of images in a target scene shot by a PTZ camera at different positions are acquired; calculating a loss value according to a plurality of images photographed under the position and the loss function; and when the loss value is smaller than a preset threshold value, determining the image of the background obtained by the corresponding loss value as the image of the background of the position, so that the acquired images of the multiple backgrounds are more accurate. Then, according to the images of the multiple backgrounds under the same focal length level, obtaining homography matrixes and rotation matrixes among the images of the multiple backgrounds under the focal length level; acquiring an internal reference matrix of the PTZ camera under the focal length grade according to the homography matrix and the rotation matrix, wherein the internal reference matrix comprises a focal length corresponding to the focal length grade and principal point coordinates of an image of a background under the focal length grade; correcting principal point coordinates of the background image in the back one of the two adjacent focal length levels according to the internal reference matrix of the PTZ camera in the two adjacent focal length levels; obtaining target principal point coordinates according to principal point coordinates of the background images under different focal length levels; respectively corresponding focal lengths according to different focal length grades; according to focal lengths and target principal point coordinates respectively corresponding to different focal length grades, panoramic stitching is carried out on images of multiple backgrounds to obtain images of panoramic backgrounds, then three-dimensional modeling processing is carried out on the images of the panoramic backgrounds to obtain three-dimensional images of the panoramic backgrounds in a target scene; then obtaining a panoramic feature point set of a three-dimensional image of a panoramic background in a target scene; acquiring a target image in a target scene shot by a PTZ camera; determining a panoramic local area characteristic point set corresponding to the target image from the panoramic characteristic point set according to the target image; acquiring a target image feature point set of a target image; according to the panorama local area feature point set and the target image feature point set, adjusting feature points in the target image feature point set so that the difference between the positions of the feature points in the target image feature point set and the positions of the feature points in the matched panorama local area feature point set is smaller than the preset difference, and obtaining an adjusted target image; and projecting the adjusted target image into the three-dimensional image of the panoramic background according to the real-time texture projection principle to obtain a fused three-dimensional image. Therefore, the method can realize the accurate matching of the target image in the target scene shot by the PTZ camera and the acquired three-dimensional image of the panoramic background in the target scene, and solve the problem of real-time alignment in the prior art.
Those of ordinary skill in the art will appreciate that: all or part of the steps for implementing the method embodiments described above may be performed by hardware associated with program instructions. The foregoing program may be stored in a computer readable storage medium. The program, when executed, performs steps including the method embodiments described above; and the aforementioned storage medium includes: read-Only Memory (ROM), random access Memory (Random Access Memory, RAM), magnetic or optical disk, and the like, which can store program codes.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present application, and not for limiting the same; although the application has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some or all of the technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit of the application.
Claims (7)
1. A method for spherical stitching and real-time alignment of PTZ moving images, the method comprising:
Acquiring a three-dimensional image of a panoramic background in a target scene and a panoramic feature point set;
acquiring target images in a target scene shot by the omnibearing moving and zooming lens control PTZ camera;
determining a panorama local area characteristic point set corresponding to the target image from the panorama characteristic point set according to the target image;
acquiring a target image feature point set of the target image;
according to the panorama local area feature point set and the target image feature point set, twisting and adjusting feature points in the target image feature point set, so that the difference between the positions of the feature points in the target image feature point set and the positions of the feature points in the matched panorama local area feature point set is smaller than the preset difference, and an adjusted target image is obtained;
and projecting the adjusted target image into the three-dimensional image of the panoramic background according to a real-time texture projection principle to obtain a fused three-dimensional image.
2. The method of claim 1, wherein the determining a panorama local area feature point set corresponding to the image from the panorama feature point set according to the target image comprises:
Determining the position of the target image mapped into the panoramic background image according to the rotation angle of the target image shot by the PTZ camera relative to the initial image shot and the focal length of the target image shot by the PTZ camera;
and determining the panorama local area characteristic point set corresponding to the position from the panorama characteristic point set of the three-dimensional image of the panorama background.
3. The method of claim 1, wherein the acquiring a three-dimensional image of a panoramic background in the target scene comprises:
acquiring an image of a panoramic background in the target scene;
and carrying out three-dimensional modeling according to the image of the panoramic background to obtain a three-dimensional image of the panoramic background.
4. A method according to claim 3, wherein said obtaining an image of a panoramic background in said target scene comprises:
acquiring images of a plurality of backgrounds in a target scene;
and carrying out panoramic stitching on the images with the multiple backgrounds to obtain an image with a panoramic background.
5. The method of claim 4, wherein the plurality of background images are obtained from images captured by the PTZ camera at different focal lengths, wherein the panoramic stitching the plurality of background images to obtain a panoramic background image comprises:
Acquiring homography matrixes and rotation matrixes among the images of the multiple backgrounds under the same focal length level according to the images of the multiple backgrounds under the same focal length level; acquiring an internal reference matrix of the PTZ camera under the focal length grade according to the homography matrix and the rotation matrix, wherein the internal reference matrix comprises a focal length corresponding to the focal length grade and principal point coordinates of an image of a background under the focal length grade;
correcting principal point coordinates of an image of a background in a back focal length grade in two adjacent focal length grades according to an internal reference matrix of the PTZ camera in the two adjacent focal length grades by the following formula;
in the method, in the process of the invention,indicating a focus level of Z i When the background image is used, the pixel coordinate value of any characteristic point of the background image is obtained;indicating a focus level of Z i-1 When the background image is used, the pixel coordinate value of any characteristic point of the background image is obtained; f (Z) i ) Indicating a focus level of Z i A focal length value of the PTZ camera at that time; f (Z) i-1 ) Indicating a focus level of Z i-1 A focal length value of the PTZ camera at that time; (p) x ,p y ) Representing principal point coordinates of an image of the background at a rear focal length level of the corrected adjacent two focal length levels;
obtaining target principal point coordinates according to the corrected principal point coordinates of the background images under different focal length levels;
And carrying out panoramic stitching on the images of the multiple backgrounds obtained under the different focal length grades according to the focal lengths respectively corresponding to the different focal length grades and the target principal point coordinates to obtain an image of a panoramic background.
6. The method of claim 4, wherein the capturing images of the plurality of backgrounds in the target scene comprises:
acquiring a plurality of images in a target scene shot by a PTZ camera at different positions;
calculating a loss value according to a plurality of images photographed under a position and a loss function aiming at the position;
if the loss value is larger than the preset threshold, the loss value is used as an iteration error, and the loss value is recalculated according to a plurality of images shot at the position and the loss function;
if the loss value is smaller than a preset threshold value, determining an image of the background obtained by the loss value as the image of the background of the position;
the loss function is:
wherein I is K Representing a kth image of a plurality of images in a target scene photographed at the position; i represents an image of the background at the location; infinity represents an element-by-element product; w (W) K A two-dimensional matrix of weights representing pixel differences of the K-th image at the location and the image of the background, each element of the matrix being Representing the pixel value of the kth image at the same position in the ith row and jth column, ε represents the iteration error, +.>A weight value representing a pixel difference value between a kth image at the position and an image of the background at an ith row and a jth column; pixels I of the ith row and jth column of image I of the background at said location ij The initial value is a weighted average of a plurality of image pixels in the target scene taken at that position, +.>
7. The method of any one of claims 1-6, further comprising:
and displaying the fused three-dimensional image.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010411353.6A CN113674145B (en) | 2020-05-15 | 2020-05-15 | Spherical surface splicing and real-time alignment method for PTZ (pan-tilt-zoom) moving image |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010411353.6A CN113674145B (en) | 2020-05-15 | 2020-05-15 | Spherical surface splicing and real-time alignment method for PTZ (pan-tilt-zoom) moving image |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113674145A CN113674145A (en) | 2021-11-19 |
| CN113674145B true CN113674145B (en) | 2023-08-18 |
Family
ID=78537506
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010411353.6A Active CN113674145B (en) | 2020-05-15 | 2020-05-15 | Spherical surface splicing and real-time alignment method for PTZ (pan-tilt-zoom) moving image |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113674145B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110197455A (en) * | 2019-06-03 | 2019-09-03 | 北京石油化工学院 | Acquisition methods, device, equipment and the storage medium of two-dimensional panoramic image |
| CN110782394A (en) * | 2019-10-21 | 2020-02-11 | 中国人民解放军63861部队 | Panoramic video rapid splicing method and system |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080198159A1 (en) * | 2007-02-16 | 2008-08-21 | Matsushita Electric Industrial Co., Ltd. | Method and apparatus for efficient and flexible surveillance visualization with context sensitive privacy preserving and power lens data mining |
-
2020
- 2020-05-15 CN CN202010411353.6A patent/CN113674145B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110197455A (en) * | 2019-06-03 | 2019-09-03 | 北京石油化工学院 | Acquisition methods, device, equipment and the storage medium of two-dimensional panoramic image |
| CN110782394A (en) * | 2019-10-21 | 2020-02-11 | 中国人民解放军63861部队 | Panoramic video rapid splicing method and system |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113674145A (en) | 2021-11-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108564617B (en) | Three-dimensional reconstruction method and device for multi-view camera, VR camera and panoramic camera | |
| US8599317B2 (en) | Scene recognition methods for virtual insertions | |
| CN112085659B (en) | Panorama splicing and fusing method and system based on dome camera and storage medium | |
| US20030012410A1 (en) | Tracking and pose estimation for augmented reality using real features | |
| CN108519102A (en) | A kind of binocular vision speedometer calculation method based on reprojection | |
| CN108776976A (en) | A kind of while positioning and the method, system and storage medium for building figure | |
| CN111242988A (en) | Method for tracking target by using double pan-tilt coupled by wide-angle camera and long-focus camera | |
| CN109120901B (en) | Method for switching pictures among cameras | |
| CN110827321A (en) | Multi-camera cooperative active target tracking method based on three-dimensional information | |
| CN107038714A (en) | Many types of visual sensing synergistic target tracking method | |
| US20230118280A1 (en) | Use of multiple registrations for augmented reality system for viewing an event | |
| Shum et al. | Tracking the translational and rotational movement of the ball using high-speed camera movies | |
| CN116309686A (en) | Video positioning and speed measuring method, device and equipment for swimmers and storage medium | |
| CN112184807A (en) | Floor type detection method and system for golf balls and storage medium | |
| JP2003179800A (en) | Device for generating multi-viewpoint image, image processor, method and computer program | |
| CN114882106A (en) | Pose determination method and device, equipment and medium | |
| CN114037923A (en) | Target activity hotspot graph drawing method, system, equipment and storage medium | |
| CN112465702B (en) | Synchronous self-adaptive splicing display processing method for multi-channel ultrahigh-definition video | |
| CN110910489B (en) | Monocular vision-based intelligent court sports information acquisition system and method | |
| CN113763544A (en) | Image determination method, image determination device, electronic equipment and computer-readable storage medium | |
| CN113674145B (en) | Spherical surface splicing and real-time alignment method for PTZ (pan-tilt-zoom) moving image | |
| CN110544203A (en) | Motion least square method and line constraint combined parallax image splicing method | |
| KR101132976B1 (en) | Mobile device with a plurality of camera, method for display using the sane | |
| CN113763545A (en) | Image determination method, image determination device, electronic equipment and computer-readable storage medium | |
| Garau et al. | Unsupervised continuous camera network pose estimation through human mesh recovery |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |