CN110310374B - Automatic tripod eliminating method and device in panoramic photo - Google Patents

Automatic tripod eliminating method and device in panoramic photo Download PDF

Info

Publication number
CN110310374B
CN110310374B CN201910489823.8A CN201910489823A CN110310374B CN 110310374 B CN110310374 B CN 110310374B CN 201910489823 A CN201910489823 A CN 201910489823A CN 110310374 B CN110310374 B CN 110310374B
Authority
CN
China
Prior art keywords
tripod
template
region
image
interest
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN201910489823.8A
Other languages
Chinese (zh)
Other versions
CN110310374A (en
Inventor
沈峘
张佩泽
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Hongzhong Electronic Technology Co ltd
Original Assignee
Nanjing Hongzhong Electronic Technology Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Hongzhong Electronic Technology Co ltd filed Critical Nanjing Hongzhong Electronic Technology Co ltd
Priority to CN201910489823.8A priority Critical patent/CN110310374B/en
Publication of CN110310374A publication Critical patent/CN110310374A/en
Application granted granted Critical
Publication of CN110310374B publication Critical patent/CN110310374B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T19/00—Manipulating 3D models or images for computer graphics
    • G06T19/20—Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
    • G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/80—Camera processing pipelines; Components thereof

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Architecture (AREA)
  • Computer Graphics (AREA)
  • Computer Hardware Design (AREA)
  • General Engineering & Computer Science (AREA)
  • Software Systems (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)

Abstract

The invention discloses an automatic tripod eliminating method in a panoramic photo, which comprises the following steps: intercepting an interested area; locating a tripod image in the region of interest; replacing the tripod image; and mapping the replaced region of interest. The step of intercepting the region of interest is to transform the original image of the panoramic photo and then acquire a deformation-free region containing complete tripod information; said step of locating the tripod image in the region of interest is marking the tripod position in the region of interest; the step of replacing the tripod image is populating the tripod marker locations with background information about the tripod; the step of mapping the replaced region of interest is mapping the region of interest into the panoramic image to obtain a complete panoramic image. The invention also discloses an automatic tripod eliminating device in the panoramic photo. The method is full-automatic processing, does not need personnel participation, reduces the use threshold of the panoramic camera, and can promote the popularization of the panoramic camera.

Description

Automatic tripod eliminating method and device in panoramic photo
Technical Field
The invention relates to the technical field of image processing, in particular to a method and a device for automatically eliminating a tripod in a panoramic photo.
Background
Because the spatial information can be really recorded in 360-degree dead angle-free mode, the panoramic image is well appreciated and used in a plurality of spatial display applications. Based on the full-recording characteristic of the panorama, in order to avoid a photographer from appearing in a picture, a panoramic camera is generally placed on a tripod, and the photographer shoots in a hidden manner. In contrast, in a tripod existing in a shot picture, a photographer manually removes the tripod at a computer end by using a professional image processing tool such as Photoshop at a later stage.
However, for a photographer who has no experience of using a professional image processing tool, the tripod removing work is difficult to be unfolded, and the original processing mode needs to export the panoramic picture from the camera and copy the panoramic picture into a computer for processing, so that the operation process is inconvenient.
Disclosure of Invention
The invention aims to provide a method and a device for automatically eliminating a tripod in a panoramic photo, which aim to solve the problems in the prior art, reduce the use threshold of a panoramic camera and promote the popularization of the panoramic camera.
In order to realize the purpose, the invention adopts the technical scheme that:
the automatic tripod elimination method in the panoramic photo comprises the following steps: intercepting an interested area; locating a tripod image in the region of interest; replacing the tripod image; and mapping the replaced region of interest.
Preferably, the method further comprises:
s1, intercepting an interested area from a panoramic image, wherein the interested area comprises a complete tripod image;
s2, searching for a tripod image in the region of interest, and marking position information of the tripod image;
s3, filling pixels of the positions where the marked tripod images are located by using effective pixels around the tripod images;
and S4, mapping the processed region of interest to the panoramic image to obtain the processed panoramic image.
Preferably, S1 further comprises: and projecting the panoramic image onto the spherical surface, and carrying out re-projection on the spherical surface to obtain a region of interest containing a complete tripod image without deformation.
Preferably, S2 further comprises: setting an initial template, wherein the size of the initial template is consistent with that of the region of interest; and calculating a new template matched with the position of the current tripod image, wherein the position information of the new template is the position information of the tripod image.
Preferably, S3 further comprises:
s31, calculating gradient values of edge points of all areas to be filled, and arranging the gradient values from large to small;
s32, filling from the edge point with the maximum gradient value;
s33, updating the template, wherein the filled edge point value is updated to be 0;
and S34, counting the number of the non-0 values in the template, repeating S31-S32 if the counted number of the non-0 values is greater than 0, and ending the filling if all the values of the template are 0.
The automatic tripod eliminating device in the panoramic photo comprises an interested area intercepting module, a tripod positioning module, a tripod replacing module and an interested area mapping module.
Preferably, the region of interest intercepting module is configured to project the panoramic image onto a spherical surface, and perform re-projection from the spherical surface to obtain a region of interest containing a complete tripod image without deformation;
the tripod positioning module is used for marking the position information of the tripod image in the region of interest;
the tripod replacement module is used for filling the marked tripod image area with background information around the tripod image;
the region of interest mapping module is used for mapping the processed image to the original panoramic image so as to obtain a final complete panoramic image.
Compared with the prior art, the invention has the beneficial effects that: the automatic tripod eliminating method and device in the panoramic photo are fully automatically operated, do not need later participation of a user, can realize one-key elimination, reduce the use threshold of the panoramic camera, and can promote the popularization of the panoramic camera.
Drawings
FIG. 1 is a schematic flow chart of a method for automatically eliminating a tripod in a panoramic photo according to an embodiment;
FIG. 2 is a schematic view of a panoramic image before elimination of a tripod according to an embodiment;
FIG. 3 is a schematic diagram of a re-projection of a panoramic image artwork according to an embodiment;
FIG. 4 is a schematic view of a region of interest containing a tripod image according to an embodiment;
FIG. 5 is a schematic diagram of an initial template according to an embodiment;
FIG. 6 is a diagram of a new template according to an embodiment;
FIG. 7 is a schematic diagram of tripod image edge points according to an embodiment;
FIG. 8 is a schematic view of a panoramic image with tripods eliminated, according to an embodiment
Detailed Description
The technical solutions of the present invention will be described clearly and completely with reference to the accompanying drawings, and it is to be understood that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be obtained by a person skilled in the art without making any creative effort based on the embodiments in the present invention, belong to the protection scope of the present invention.
The invention provides an automatic tripod removing method in a panoramic photo. As shown in fig. 1, the method includes: intercepting a region of interest; locating a tripod image in the region of interest; replacing the tripod image; and mapping the replaced region of interest. The step of intercepting the region of interest is to transform the original image of the panoramic photo and then acquire a deformation-free region containing complete tripod information; said step of locating the tripod image in the region of interest is marking the tripod position in the region of interest; the step of replacing the tripod image is populating the tripod marker locations with background information about the tripod; the step of mapping the replaced region of interest is mapping the region of interest into the panoramic image to obtain a complete panoramic image. The automatic tripod eliminating method provided by the invention is a full-automatic method, and personnel participation is not needed, so that the use threshold of the panoramic camera is reduced, and the popularization of the panoramic camera can be promoted.
In a specific implementation, the tripod elimination method of the present invention can be divided into four steps:
the first step is as follows: a region of interest is cut out of the panoramic image, the region of interest containing the complete tripod graphic.
Panoramic images are typically scaled by 2: the latitude and longitude image of 1 indicates that the tripod at the bottom has completely failed to recognize the shape of the tripod due to deformation, as shown by the black frame marked area at the lower part of fig. 2. Therefore, re-projection is required to obtain a complete, cognitive tripod shape. The schematic diagram of the reprojection is shown in fig. 3, and the point Q is the projection of a point Q on the three-dimensional spherical surface in the two-dimensional plane. Assuming that the region of interest that can fully contain the tripod is of size L x L, and the point q coordinate on the region of interest is (u, v) under the coordinate system uo' v, then under the coordinate system xoy, the q point coordinate becomes (x, y),
Figure BDA0002086617890000031
the value of the longitude theta of the point Q under the three-dimensional coordinate system XOY can be calculated according to the coordinates (x, y),
Figure BDA0002086617890000032
assuming that the width of the panorama is W and the height is H, the latitude of the point Q
Figure BDA0002086617890000033
Is calculated as
Figure BDA0002086617890000034
Longitude and latitude coordinates of visible point Q
Figure BDA0002086617890000035
The calculation is completed.
Suppose a point q on the panoramic image 0 The coordinate is (x) 0 ,y 0 ) From the panoramic image plane coordinates (x) 0 ,y 0 ) With longitude and latitude coordinates
Figure BDA0002086617890000036
The mapping relation between the two can be calculated
Figure BDA0002086617890000037
Extraction of (x) 0 ,y 0 ) The pixel value at the location is filled in to the point (u, v) if (x) 0 ,y 0 ) The decimal pixel value can be obtained by interpolation methods such as nearest neighbor interpolation, bilinear interpolation or bicubic spline interpolation. To this end, a map of the region of interest containing the tripod may be taken, as shown in fig. 4.
The second step is that: a tripod is searched for in the region of interest and positional information of the tripod is marked.
The tripod is relatively fixed in shape and therefore an initial template can be set, as shown in fig. 5, having a size L × L corresponding to the size of the region of interest. The white area is marked as 1 and is marked as an effective tripod area; the black area is marked as 0 and is marked as background area.
In fact, the tripod pose is arbitrary, so the initial template is not fully versatile and further calculations are required to calculate a new template that matches the current tripod position. Typically, the tripod is black, so that if a new template is fitted to the tripod, the sum of the pixel grey values of the regions marked by the template is minimal.
The calculation steps based on this analysis of the new template are as follows:
assuming that the coordinates of a point on the region of interest are (u, v), the gray value of the point is I (u, v), and the value of the initial template corresponding to the point is mask0 (u, v), then
1. Calculating the product sum of the initial template and the region of interest, i.e.
Figure BDA0002086617890000041
2. Setting the step length of the rotation angle as step, rotating the initial template by step angle to obtain a first new template mask1, wherein the size of the new template is consistent with that of the initial template and is L multiplied by L. One point value on the template is marked as mask1 (u, v), and calculation is carried out
Figure BDA0002086617890000042
3. Judging the rotation times N, if N is less than 360/step, repeating the step two, and calculating to obtain new templates and corresponding product sum values under different rotation angles;
4. and if N is more than or equal to 360/step, ending to obtain new templates mask1 and mask2 \8230, wherein corresponding product sum values are sum1 and sum2 \8230, and sum N is calculated by combining sum0 of the initial template to obtain the minimum value in N +1 values, and the template is the template which is most matched with the current tripod.
The matched template map is shown in FIG. 6.
The third step: the pixels of the tripod position of the marker are filled with active pixels around the tripod.
As shown in fig. 7, a point of the edge of the region to be filled is denoted as P (u, v), and the calculation steps are as follows:
1. calculating gradient values of edge points of all areas to be filled, and arranging the gradient values from large to small;
calculating gradient values of the edge points in the x direction and the y direction by using a sobel operator, a prewitt operator and the like, and respectively recording the gradient values as gradx and grady, wherein the gradient strength is calculated as
Figure BDA0002086617890000051
And arranging the gradient strength values of all the edge points in a descending order, wherein the calculation order of the edge points to be filled is consistent with the gradient strength arrangement order.
2. Filling from the edge point with the maximum gradient value
During filling, a reference image sub-area which takes the point P as the center and has the size of w multiplied by w is selected, the reference image sub-area is searched in the region of interest, the most similar target image sub-area is searched, and the pixel value of the center point of the most similar sub-area is filled to the point P. The objective function of the search is defined as follows:
Figure BDA0002086617890000052
in the formula, f (x, y) represents a gray value at a point (x, y) in a sub-region of the reference image, g (x, y) represents a gray value at a point (x, y) in a sub-region of the target image, submask (x, y) represents a template value corresponding to the sub-region of the reference image, parameter c represents the number of points which effectively participate in calculation, α is a weight value, and the closer the sub-region of the target image is to the sub-region of the reference image, the larger the weight value is, the smaller the value of α is. And after searching, taking the image sub-area with the minimum objective function value as the target image sub-area most similar to the reference image sub-area, and filling the pixel value of the center point of the target image sub-area into the center point of the reference image sub-area, namely calculating to obtain P (u, v).
The pixel values at the remaining lower priority boundaries are filled in the same way.
3. Updating the template, wherein the filled edge point value is updated to be 0; counting the number of non-0 values in the template, and if the number is greater than 0, repeating the step 1-2; and if all values of the template are 0, finishing filling.
The fourth step: and mapping the modified region of interest to a panoramic image to obtain a beautified panoramic image.
The calculation process of the fourth step is opposite to that of the first step. Assuming that the planar coordinates of the points to be mapped in the panoramic image are (x 0, y 0), the relationship between the planar coordinates of the panoramic image and the longitude and latitude coordinates can be obtained
Figure BDA0002086617890000061
According to longitude and latitude
Figure BDA0002086617890000062
After the value of (b) is calculated and projected into the coordinate system xoy, the coordinate of the point is
Figure BDA0002086617890000063
The coordinates of the mapped point in the coordinate system uo' v can be further calculated as
Figure BDA0002086617890000064
Filling in pixel values at coordinate point (u, v) locations to the panoramic image (x) 0 ,y 0 ) And repeating the calculation process to obtain a complete beautified panoramic image through calculation, wherein the effect is shown in fig. 8.
The invention also provides an automatic tripod elimination device in the panoramic photo, which comprises an interested region intercepting module, a tripod positioning module, a tripod replacing module and an interested region mapping module.
Each module works as follows:
the interesting region intercepting module is used for projecting the panoramic image onto the spherical surface and carrying out re-projection on the panoramic image from the spherical surface to obtain an interesting region which is free of deformation and contains complete tripod information;
the tripod positioning module is used for marking the position information of the tripod in the region of interest;
the tripod replacing module fills the tripod area by using background information around the tripod;
and the region-of-interest mapping module is used for mapping the repaired and beautified image to the original panoramic image so as to obtain a final complete panoramic image.
Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and alterations can be made in these embodiments without departing from the principles and spirit of the invention, the scope of which is defined in the appended claims and their equivalents.

Claims (2)

1. An automatic tripod elimination method in a panoramic photo is characterized by comprising the following steps: intercepting a region of interest; locating a tripod image in the region of interest; replacing the tripod image; mapping the replaced interested region;
the method further comprises:
s1, intercepting an interested area from a panoramic image, wherein the interested area comprises a complete tripod image; further comprising: projecting the panoramic image onto a spherical surface, and carrying out re-projection on the spherical surface to obtain a region of interest containing a complete tripod image without deformation;
s2, searching for a tripod image in the region of interest, and marking position information of the tripod image; further comprising: setting an initial template, wherein the size of the initial template is consistent with that of the region of interest; calculating a new template matched with the position of the current tripod image, wherein the position information of the new template is the position information of the tripod image;
the calculation steps of the new template are as follows:
assuming that the coordinates of a point on the region of interest are (u, v), the gray value of the point is I (u, v), and the value of the initial template corresponding to the point is mask0 (u, v), then
1. Calculating outProduct of the initial template and the region of interest, i.e.
Figure FDA0003935179580000011
2. Setting the step length of the rotation angle as step, rotating the initial template by step angle to obtain a first new template mask1, wherein the size of the new template is consistent with that of the initial template and is L multiplied by L; marking a point value on the template as mask1 (u, v), and calculating
Figure FDA0003935179580000012
3. Judging the rotation times N, if N is less than 360/step, repeating the step 2, and calculating to obtain new templates and corresponding product sum values under different rotation angles;
4. if N is more than or equal to 360/step, ending to obtain new templates mask1 and mask2 \8230, wherein the corresponding product sum values are sum1 and sum2 \8230, and sum is calculated by combining sum0 of the initial template to obtain the minimum value in N +1 values, and the template is the template most matched with the current tripod;
s3, filling pixels of the positions where the marked tripod images are located with effective pixels around the tripod images; further comprising:
s31, calculating gradient values of edge points of all areas to be filled, and arranging the gradient values from large to small;
s32, filling from the edge point with the maximum gradient value;
s33, updating the template, wherein the filled edge point value is updated to be 0;
s34, counting the number of the non-0 values in the template, repeating S31-S32 if the counted number of the non-0 values is larger than 0, and ending filling if all values of the template are 0;
and S4, mapping the processed region of interest to the panoramic image to obtain the processed panoramic image.
2. The automatic tripod eliminating device in the panoramic photo is characterized by comprising an interested area intercepting module, a tripod positioning module, a tripod replacing module and an interested area mapping module;
the interesting region intercepting module is used for projecting the panoramic image onto a spherical surface and carrying out re-projection on the spherical surface to obtain an undeformed interesting region containing a complete tripod image; further comprising: projecting the panoramic image onto a spherical surface, and carrying out re-projection on the spherical surface to obtain a region of interest containing a complete tripod image without deformation;
the tripod positioning module is used for marking the position information of the tripod image in the region of interest; further comprising: setting an initial template, wherein the size of the initial template is consistent with that of the region of interest; calculating a new template matched with the position of the current tripod image, wherein the position information of the new template is the position information of the tripod image;
the calculation steps of the new template are as follows:
assuming that the coordinates of a point on the region of interest are (u, v), the gray value of the point is I (u, v), and the value of the initial template corresponding to the point is mask0 (u, v), then
1. Calculating the product sum of the initial template and the region of interest, i.e.
Figure FDA0003935179580000021
2. Setting the step length of the rotation angle as step, rotating the initial template by step angle to obtain a first new template mask1, wherein the size of the new template is consistent with that of the initial template and is L multiplied by L; marking a point value on the template as mask1 (u, v), and calculating
Figure FDA0003935179580000022
3. Judging the rotation times N, if N is less than 360/step, repeating the step 2, and calculating to obtain new templates and corresponding product sum values under different rotation angles;
4. if N is more than or equal to 360/step, ending, obtaining new templates of mask1 and mask2 \8230, mask N, wherein corresponding product sum values are sum1 and sum2 \8230andsum N, calculating to obtain the minimum value in N +1 values by combining sum0 of the initial template, and obtaining the template which is the template most matched with the current tripod;
the tripod replacement module is used for filling the marked tripod image area with background information around the tripod image; further comprising:
s31, calculating gradient values of edge points of all areas to be filled, and arranging the gradient values from large to small;
s32, filling from the edge point with the maximum gradient value;
s33, updating the template, wherein the filled edge point value is updated to be 0;
s34, counting the number of the non-0 values in the template, repeating S31-S32 if the counted number of the non-0 values is larger than 0, and finishing filling if all the values of the template are 0;
the region of interest mapping module is used for mapping the processed image to the original panoramic image so as to obtain a final complete panoramic image.
CN201910489823.8A 2019-06-06 2019-06-06 Automatic tripod eliminating method and device in panoramic photo Active CN110310374B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910489823.8A CN110310374B (en) 2019-06-06 2019-06-06 Automatic tripod eliminating method and device in panoramic photo

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910489823.8A CN110310374B (en) 2019-06-06 2019-06-06 Automatic tripod eliminating method and device in panoramic photo

Publications (2)

Publication Number Publication Date
CN110310374A CN110310374A (en) 2019-10-08
CN110310374B true CN110310374B (en) 2023-01-10

Family

ID=68075696

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910489823.8A Active CN110310374B (en) 2019-06-06 2019-06-06 Automatic tripod eliminating method and device in panoramic photo

Country Status (1)

Country Link
CN (1) CN110310374B (en)

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB201217721D0 (en) * 2012-10-03 2012-11-14 Holition Ltd Video image processing
CN103873741A (en) * 2014-04-02 2014-06-18 北京奇艺世纪科技有限公司 Method and device for substituting area of interest in video

Also Published As

Publication number Publication date
CN110310374A (en) 2019-10-08

Similar Documents

Publication Publication Date Title
CN108665536B (en) Three-dimensional and live-action data visualization method and device and computer readable storage medium
CN109348119B (en) Panoramic monitoring system
US10922844B2 (en) Image positioning method and system thereof
US8818101B1 (en) Apparatus and method for feature matching in distorted images
EP2477152B1 (en) Image processing device, image processing method, image processing program, and recording medium
KR100796849B1 (en) Method for photographing panorama mosaics picture in mobile device
JP4512584B2 (en) Panorama video providing method and apparatus with improved image matching speed and blending method
CN106060493A (en) Multi-source projection seamless edge stitching method and system
CN107403409B (en) Image processing method and device before image splicing and image splicing method and device
WO2022105415A1 (en) Method, apparatus and system for acquiring key frame image, and three-dimensional reconstruction method
CN111292413A (en) Image model processing method and device, storage medium and electronic device
EP1903498B1 (en) Creating a panoramic image by stitching a plurality of images
CN110288511B (en) Minimum error splicing method and device based on double camera images and electronic equipment
WO2020220832A1 (en) Method and apparatus for achieving projection picture splicing, and projection system
CN105128539A (en) Optical projection laser marking method and laser marking system
CN110838086A (en) Outdoor image splicing method based on correlation template matching
CN112862683A (en) Adjacent image splicing method based on elastic registration and grid optimization
CN110278366B (en) Panoramic image blurring method, terminal and computer readable storage medium
CN106846250B (en) Super-resolution reconstruction method based on multi-scale filtering
CN111343360B (en) Correction parameter obtaining method
CN111131801A (en) Projector correction system and method and projector
CN106023170A (en) Binocular 3D distortion rectification method based on GPU
CN110310374B (en) Automatic tripod eliminating method and device in panoramic photo
CN109801339B (en) Image processing method, apparatus and storage medium
CN115753019A (en) Pose adjusting method, device and equipment of acquisition equipment and readable storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant