CN111447359B - Digital zoom method, system, electronic device, medium, and digital imaging device - Google Patents
Digital zoom method, system, electronic device, medium, and digital imaging device Download PDFInfo
- Publication number
- CN111447359B CN111447359B CN202010197617.2A CN202010197617A CN111447359B CN 111447359 B CN111447359 B CN 111447359B CN 202010197617 A CN202010197617 A CN 202010197617A CN 111447359 B CN111447359 B CN 111447359B
- Authority
- CN
- China
- Prior art keywords
- image
- zoom
- magnification
- frames
- zooming
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
Abstract
The invention discloses a digital zooming method, a digital zooming system, an electronic device, a medium and a digital imaging device. The digital zooming method comprises the following steps: continuously acquiring a plurality of frames of original images according to a zooming instruction, wherein the zooming instruction comprises zooming magnification; cutting each frame of original image according to the zooming magnification to obtain an interested area; upsampling a multi-frame region of interest to obtain a first intermediate image; and performing super-resolution reconstruction on a plurality of frames of the first intermediate image to acquire a zoom image. The method and the device continuously acquire the multi-frame original images according to the zooming instruction, and sequentially perform cutting, up-sampling and super-resolution reconstruction on the acquired multi-frame original images to acquire the zooming images.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a digital zoom method, a digital zoom system, an electronic device, a storage medium, and a digital imaging device.
Background
Zooming refers to the change of the focal length of the imaging lens, which is visually represented by zooming in or out of an object to be imaged, and also represented by zooming in or out of the object to be imaged. The zoom capability is a very important index of the imaging apparatus, which determines the shooting breadth and depth of the photographing apparatus. Zooming can be divided into optical zooming and digital zooming according to implementation modes. Optical zooming is the most traditional method of zooming, which uses different combinations of concave and convex lenses to achieve a change in focal length. The digital zoom is also called digital zoom, which is realized by cutting and interpolation algorithm and belongs to the category of image processing.
The physical structure of optical zooming limits its application to portable photographing devices, and currently, zooming of almost all portable photographing devices is achieved by digital zooming. However, digital zooming is different from optical zooming, which is a kind of lossy zooming, and the zooming result tends to be much worse than optical zooming.
The traditional digital zooming based on a single-frame image is completely realized by depending on an interpolation algorithm, but according to a sampling law, the interpolation algorithm cannot restore original information of the image, cannot generate more information to enable the image to be clearer, can cause obvious image noise, and can introduce an obvious mosaic phenomenon when the digital zooming with high magnification is used. Therefore, the zoom limit of such conventional digital zoom is generally about 5 times.
In order to achieve a better digital zoom effect, some machine learning-based super-resolution algorithms are applied to the digital zoom, but the machine learning-based super-resolution algorithms cannot be widely applied to the digital zoom due to immature learning framework or lack of stability, reliability, and the like.
Disclosure of Invention
The technical problem to be solved by the present invention is to provide a digital zooming method, a digital zooming system, an electronic device, a storage medium and a digital imaging device, so as to overcome the defect of limited digital zooming capability in the prior art.
The invention solves the technical problems through the following technical scheme:
a digital zoom method, comprising:
continuously acquiring a plurality of frames of original images according to a zooming instruction, wherein the zooming instruction comprises zooming magnification;
cutting each frame of original image according to the zooming magnification to obtain an interested area;
upsampling a multi-frame region of interest to obtain a first intermediate image;
and performing super-resolution reconstruction on a plurality of frames of the first intermediate image to acquire a zoom image.
Preferably, the method further comprises, after the step of cropping each frame of original image according to the zoom magnification to obtain the region of interest:
judging whether the zoom magnification is smaller than a preset threshold value or not;
and if not, performing the step of up-sampling the multi-frame interested region to acquire the first intermediate image.
Preferably, the step of upsampling the plurality of frames of interest regions to acquire the first intermediate image further comprises:
acquiring a first multiplying power and a second multiplying power according to the zooming multiplying power, wherein the zooming multiplying power is the product of the first multiplying power and the second multiplying power, the minimum value of the first multiplying power is 1, and the second multiplying power is an integer larger than 1;
the step of upsampling the plurality of frames of interest regions to acquire a first intermediate image comprises:
taking the first multiplying power as an up-sampling multiplying power to up-sample each frame of region of interest to acquire a first intermediate image;
the step of performing super-resolution reconstruction on a plurality of frames of the first intermediate image to obtain a zoom image comprises the following steps:
and performing super-resolution reconstruction on the plurality of frames of the first intermediate image by taking the second magnification as a super-resolution reconstruction magnification to acquire a zoom image.
Preferably, the step of performing super-resolution reconstruction on a plurality of frames of the first intermediate image by using the second magnification as a super-resolution reconstruction magnification to acquire a zoom image includes:
determining one frame of the plurality of frames of the first intermediate image as a reference image;
aligning the plurality of frames of first intermediate images according to the reference image;
performing pixel amplification on the reference image according to the second multiplying power;
filling pixel values of other first intermediate images except the reference image into the pixel-amplified reference image according to the alignment result of the plurality of frames of first intermediate images;
determining whether the padded reference image comprises an unpadded pixel;
if yes, filling the unfilled pixels according to median filtering;
and/or the presence of a gas in the gas,
after the step of super-resolution reconstructing the plurality of frames of the first intermediate image using the second magnification as a super-resolution reconstruction magnification to acquire a zoom image, the method further includes:
and carrying out iterative correction on the zoom image to obtain a final zoom image.
Preferably, when the step of determining whether the zoom magnification is smaller than the preset threshold is yes:
upsampling a plurality of frames of interest regions to acquire a second intermediate image;
and fusing the second intermediate images of the plurality of frames to obtain a zoom image.
Preferably, the step of upsampling the multiple frames of interest regions to acquire the second intermediate image comprises:
taking the zoom magnification as an up-sampling magnification to up-sample each frame of region of interest to acquire a second intermediate image;
and/or the presence of a gas in the gas,
after the step of fusing the plurality of frames of the second intermediate images to obtain the zoom image, the method further comprises:
and carrying out iterative correction on the zoom image to obtain a final zoom image.
Preferably, the step of cropping each frame of original image according to the zoom magnification to obtain the region of interest includes:
and cutting each frame of original image by taking the center of each frame of original image as a reference point according to the zooming magnification to acquire an interested area.
A digital zoom system, comprising:
the first acquisition module is used for continuously acquiring multiple frames of original images according to a zooming instruction, wherein the zooming instruction comprises zooming magnification;
the cutting module is used for cutting each frame of original image according to the zooming multiplying power so as to obtain an interested area;
the first up-sampling module is used for up-sampling a plurality of frames of interested areas to acquire a first intermediate image;
and the super-resolution reconstruction module is used for carrying out super-resolution reconstruction on the plurality of frames of the first intermediate image so as to obtain a zoom image.
Preferably, the digital zoom system further comprises:
the judging module is used for judging whether the zoom magnification is smaller than a preset threshold value or not;
if not, the first up-sampling module is called.
Preferably, the digital zoom system further comprises:
the second obtaining module is used for obtaining a first multiplying power and a second multiplying power according to the zooming multiplying power, wherein the zooming multiplying power is the product of the first multiplying power and the second multiplying power, the minimum value of the first multiplying power is 1, and the second multiplying power is an integer larger than 1;
the first up-sampling module is specifically configured to up-sample each frame of region of interest by using the first magnification as an up-sampling magnification to obtain a first intermediate image;
the super-resolution reconstruction module is specifically configured to perform super-resolution reconstruction on a plurality of frames of the first intermediate image by using the second magnification as a super-resolution reconstruction magnification to obtain a zoom image.
Preferably, the super-resolution reconstruction module includes:
a determining unit, configured to determine one of the plurality of frames of the first intermediate image as a reference image;
an alignment unit configured to align the plurality of frames of first intermediate images based on the reference image;
the amplification unit is used for amplifying the pixels of the reference image according to the second multiplying power;
a first padding unit for padding pixel values of other first intermediate images except the reference image into the pixel-augmented reference image according to an alignment result of a plurality of frames of first intermediate images;
a judging unit for judging whether the padded reference image includes an unfilled pixel;
if so, calling a second filling unit for filling the unfilled pixels according to median filtering;
and/or the presence of a gas in the gas,
the digital zoom system further includes:
and the first iterative correction module is used for iteratively correcting the zoom image acquired by the super-resolution reconstruction module to obtain a final zoom image.
Preferably, the digital zoom system further comprises:
the second up-sampling module is used for up-sampling the multi-frame interested area to acquire a second intermediate image when the judgment module judges that the multi-frame interested area is positive;
and the fusion module is used for fusing the second intermediate images of the plurality of frames to obtain a zoom image.
Preferably, the second upsampling module is specifically configured to upsample each frame of the region of interest by using the zoom magnification as an upsampling magnification to obtain a second intermediate image;
and/or the presence of a gas in the gas,
the digital zoom system further includes:
and the second iterative correction module is used for iteratively correcting the zoom images acquired by the fusion module to obtain final zoom images.
Preferably, the cropping module is specifically configured to crop each frame of the original image according to the zoom magnification by using a center of each frame of the original image as a reference point to obtain the region of interest.
An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing any of the above digital zoom methods when executing the computer program.
A computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of any of the above-described digital zoom methods.
A digital imaging apparatus comprising a digital zoom system as described in any of the above.
The positive progress effects of the invention are as follows: the method and the device continuously acquire the multi-frame original images according to the zooming instruction, and sequentially perform cutting, up-sampling and super-resolution reconstruction on the acquired multi-frame original images to acquire the zooming images.
Drawings
Fig. 1 is a flowchart of a digital zooming method according to embodiment 1 of the present invention.
Fig. 2 is a detailed flowchart of a digital zooming method according to embodiment 1 of the present invention.
Fig. 3 is a flowchart of step S1041 in the digital zooming method according to embodiment 1 of the invention.
Fig. 4 is a schematic diagram of image augmentation and padding in the digital zoom method according to embodiment 1 of the present invention.
Fig. 5 is a flowchart of a digital zoom method according to embodiment 2 of the present invention.
Fig. 6 is a block diagram of a digital zoom system according to embodiment 3 of the present invention.
Fig. 7 is a block diagram of a digital zoom system according to embodiment 4 of the present invention.
Fig. 8 is a schematic structural diagram of an electronic device according to embodiment 5 of the present invention.
Detailed Description
The invention is further illustrated by the following examples, which are not intended to limit the scope of the invention.
Example 1
The present embodiment provides a digital zoom method, and referring to fig. 1, the digital zoom method of the present embodiment includes:
s101, continuously acquiring multiple frames of original images according to a zooming instruction;
s102, cutting each frame of original image according to zooming magnification to obtain an interested area;
s103, upsampling the multi-frame region of interest to obtain a first intermediate image;
and S104, performing super-resolution reconstruction on the multi-frame first intermediate image to acquire a zoom image.
In this embodiment, the zoom instruction includes a zoom magnification, where a minimum value of the zoom magnification is 1, the device that acquires multiple frames of original images according to the zoom instruction may be, for example, a digital imaging device, and the multiple frames of original images acquired according to the zoom instruction may be images acquired by the digital imaging device in real time. Specifically, when the digital imaging device is a photographing device, the zoom magnification of the photographing device is set, and the multiple frames of original images acquired according to the zoom instruction may be images acquired by the photographing device in real time before the photographing device performs the photographing action, or images acquired by the photographing device in real time when the photographing device performs the photographing action.
In the present embodiment, a partial image is cropped from an original image as a region of interest according to a zoom magnification, wherein a reference point for cropping the original image can be set in a customized manner according to an actual application. In addition, in this embodiment, the method for upsampling the region of interest may be set according to practical applications, and for example, any one of interpolation (e.g., bilinear interpolation, etc.), Deconvolution (Deconvolution), and pooling may be selected to implement upsampling.
Referring to fig. 2, after step S102 in this embodiment, the method may further include:
and S105, acquiring a first magnification and a second magnification according to the zooming magnification.
In this embodiment, the zoom magnification is a product of a first magnification and a second magnification, wherein the minimum value of the first magnification is 1, and the second magnification is an integer greater than 1, for example, when the zoom magnification is 2.5, the first magnification 1.25 and the second magnification 2 can be obtained.
Further, in the present embodiment, the maximum value is preferred according to the second magnification acquired by the zoom magnification to maximize the advantage of the super-resolution reconstruction on the integral multiple magnification as much as possible. For example, when the zoom magnification is 4, the first magnification may be 1 and the second magnification may be 4, the first magnification may also be 2 and the second magnification may be 2, the first magnification may also be 4 and the second magnification may be 1, and in order to maximize the advantage of the super-resolution reconstruction in the integral multiple magnification, it is preferable that the first magnification is 1 and the second magnification is 4 in this embodiment.
Step S103 may specifically include:
and S1031, taking the first magnification as an up-sampling magnification to up-sample each frame of the region of interest so as to acquire a first intermediate image.
Step S104 may specifically include:
and S1041, performing super-resolution reconstruction on the multiple frames of first intermediate images by taking the second magnification as a super-resolution reconstruction magnification to acquire a zoom image.
Referring to fig. 3, step S1041 in this embodiment may specifically include:
s10411, determining one frame of the multiple frames of first intermediate images as a reference image;
s10412, aligning the multiple frames of first intermediate images according to the reference image;
s10413, performing pixel amplification on the reference image according to the second multiplying power;
s10414, filling the pixel values of other first intermediate images except the reference image into the pixel-amplified reference image according to the alignment result of the first intermediate images of the plurality of frames;
s10415, judging whether the padded reference image comprises an unfilled pixel;
if yes, go to step S10416;
and S10416, filling the unfilled pixels according to the median filtering.
Specifically, in step S10412, aligning the multiple frames of the first intermediate images, that is, implementing Image Alignment/Image Registration of the multiple frames of the first images, and by aligning the multiple frames of the first intermediate images, pixel points belonging to the same object in the multiple frames of the first images may be in one-to-one correspondence.
In step S10413, performing pixel amplification on the reference image, where a pixel amplification manner may be set by a user according to an actual application, for example, zero pixel amplification may be used to implement the pixel amplification, specifically, amplification of the reference image may be implemented based on nearest neighbor interpolation first, then 0 is used to fill interpolation data, referring to fig. 4, after amplifying one pixel point in the reference image, 3 × 3 pixel points are formed, where a pixel value of a pixel point located at the upper left corner is taken from pixel points in the reference image.
In step S10414, the pixel values of the reference image may be filled in by comparing, so as to avoid blurring of the image due to the motion, and further, according to the alignment result, if there are several frames of the pixel points of the first intermediate image corresponding to the same filled region in the pixel-augmented reference image, the filled value of the same filled region is determined by the pixel values of the pixel points in the involved several frames of the first intermediate image, for example, in fig. 4, the pixel points of the several first intermediate image all correspond to the pixel point located at the central position among the 3 × 3 pixel points, and then the pixel point located at the central position may be filled in according to the determined filled value. Specifically, the padding value may be calculated according to the following formula:
wherein I is a padding value, h represents an adjustable parameter, IkRepresenting the pixel value, diffk＝Ik-I0K denotes a picture frame number, k-0 denotes a reference picture, and flagkIndicating whether the frame image is in a filled position, if so, taking the value as 1, otherwise, taking the value as 0, and further, taking the flag0＝1。
If it is determined in step S10415 that the motion is correct, padding is performed again in step S10416 to obtain the zoom image, specifically, pixel values of the padded reference image may be padded to avoid blurring of the image due to motion, and if it is determined that the motion is not correct, the zoom image may be obtained in step S10414.
After step S104, iterative correction may be performed on the zoom image to obtain a final zoom image. Specifically, the zoom image acquired in step S104 may be used as an estimated value of the high resolution image (HR), a corresponding low resolution image (LR) may be generated by an imaging model, and an error with the original data may be calculated to finally minimize the error. In this embodiment, the number of iterations may be set in a customized manner according to the computational complexity requirement, and may be set to a value within 10 in general, so as to correct the error introduced by the upsampling or the alignment.
Specifically, the iterative correction implements the formula:
wherein n is the number of iterations,
is an estimate of the high resolution image, beta is the iteration step, D, PSF, WrapkRespectively down-sampling, the point spread function of the imaging system and motion-induced image deformation.
Specifically, in the present embodiment, the imaging model of the digital imaging apparatus can be expressed as:
LR＝D{PSF*Wrap{HR}}+noise
where HR is a high resolution image, Wrap { } denotes image deformation due to motion, denotes convolution operation, PSF is a point spread function of the imaging system, D { } denotes the down-sampling process, and noise is noise. Further, in the present embodiment, the first intermediate image is selected as the LR representing the original datakIn the iterative correction of the embodiment, a regularization term is also introduced
To make the iteration more robust, wherein
Is the gradient operator and λ is the regularization parameter. The purpose of the iterative correction is to extrapolate the high-resolution image HR back from the known multi-frame noisy low-resolution map LR:
in the embodiment, multiple frames of original images are continuously acquired according to the zooming instruction, and the acquired multiple frames of original images are sequentially subjected to cutting, upsampling and super-resolution reconstruction to acquire the zooming image. Further, in the embodiment, the zoom magnification is split into the first magnification and the second magnification, which not only overcomes the defect that the super-resolution reconstruction applicable to integral-multiple sampling cannot be applied to digital zoom with the zoom magnification accurate to a decimal place, but also fully utilizes the advantage of the super-resolution reconstruction on integral-multiple magnification. In addition, the introduction of iterative correction can also avoid image blurring caused by up-sampling, image alignment, point spread function of a digital imaging device, and the like.
Example 2
On the basis of embodiment 1, the present embodiment provides a digital zoom method, and referring to fig. 5, the digital zoom method of the present embodiment may further include, after step S102:
s201, judging whether the zoom magnification is smaller than a preset threshold value;
if yes, go to step S202; if not, executing step S105;
s202, upsampling the multi-frame region of interest to obtain a second intermediate image;
and S203, fusing the second intermediate images of the frames to acquire a zoom image.
Specifically, in this embodiment, in order to fully utilize the super-resolution reconstruction to improve the digital zoom capability, the zoom ratio may be segmented to divide the zoom ratio into two zoom segments, namely a digital interpolation magnification segment and a super-resolution zoom segment, where the preset threshold may be set by a user according to actual application, for example, the preset threshold may take a value of 2. In this embodiment, step S202 may specifically include a step of upsampling the region of interest of each frame by using the zoom magnification as the upsampling magnification to acquire a second intermediate image, that is, the size of the second intermediate image is the same as that of the original image. Step S203 may specifically include a step of aligning multiple frames of second intermediate images and a step of fusing the aligned multiple frames of second intermediate images, so as to significantly reduce image noise and improve signal-to-noise ratio of the image.
The present embodiment may further include a step of performing iterative correction on the zoom image to obtain a final zoom image after step S203 to correct errors introduced by upsampling or alignment, so as to further improve the quality of the zoom image, unlike in embodiment 1, in which an original image is selected as the LR representing the original datakAnd may not introduce a regularization term here
Namely:
on the basis of utilizing multi-frame image sampling information, different image processing modes are adopted according to different numerical ranges of zoom magnifications, specifically, when the zoom magnifications are small, digital zooming is realized in a multi-frame fusion mode, compared with digital zooming of a single-frame image which completely depends on an interpolation algorithm, improvement of a signal-to-noise ratio can be obtained, the quality of the obtained zoom image is better, when the zoom magnifications are large, digital zooming is realized in a super-resolution reconstruction mode, improvement of resolution and the signal-to-noise ratio can be obtained, and therefore stronger digital zooming capacity can be realized. In addition, due to the introduction of iterative correction, image blurring caused by up-sampling, image alignment, a point spread function of a digital imaging device and the like can be avoided, so that the quality of the zoom image is further improved.
Example 3
The present embodiment provides a digital zoom system, and referring to fig. 6, the digital zoom system of the present embodiment includes:
a first obtaining module 301, configured to continuously obtain multiple frames of original images according to a zoom instruction;
a clipping module 302, configured to clip each frame of original image according to a zoom magnification to obtain an area of interest;
a first upsampling module 303, configured to upsample multiple frames of regions of interest to obtain a first intermediate image;
and the super-resolution reconstruction module 304 is used for performing super-resolution reconstruction on the multi-frame first intermediate image to acquire a zoom image.
In this embodiment, the zoom instruction includes a zoom magnification, where a minimum value of the zoom magnification is 1, the device that acquires multiple frames of original images according to the zoom instruction may be, for example, a digital imaging device, and the multiple frames of original images acquired according to the zoom instruction may be images acquired by the digital imaging device in real time. Specifically, when the digital imaging device is a photographing device, the zoom magnification of the photographing device is set, and the multiple frames of original images acquired according to the zoom instruction may be images acquired by the photographing device in real time before the photographing device performs the photographing action, or images acquired by the photographing device in real time when the photographing device performs the photographing action.
In this embodiment, a partial image is cropped from the original image according to the zoom magnification as the region of interest, where a reference point for cropping the original image can be set according to the actual application in a customized manner, for example, in this embodiment, the cropping module 302 may crop each frame of the original image according to the zoom magnification by using the center of each frame of the original image as the reference point to obtain the region of interest. In addition, in this embodiment, the method for upsampling the region of interest may be set according to practical applications, and for example, any one of interpolation (e.g., bilinear interpolation, etc.), Deconvolution (Deconvolution), and pooling may be selected to implement upsampling.
Referring to fig. 6, the digital zoom system in this embodiment may further include:
a second obtaining module 305, configured to obtain the first magnification and the second magnification according to the zoom magnification.
In this embodiment, the zoom magnification is a product of a first magnification and a second magnification, wherein the minimum value of the first magnification is 1, and the second magnification is an integer greater than 1, for example, when the zoom magnification is 2.5, the first magnification 1.25 and the second magnification 2 can be obtained.
Further, in the present embodiment, the maximum value is preferred according to the second magnification acquired by the zoom magnification to maximize the advantage of the super-resolution reconstruction on the integral multiple magnification as much as possible. For example, when the zoom magnification is 4, the first magnification may be 1 and the second magnification may be 4, the first magnification may also be 2 and the second magnification may be 2, the first magnification may also be 4 and the second magnification may be 1, and in order to maximize the advantage of the super-resolution reconstruction in the integral multiple magnification, it is preferable that the first magnification is 1 and the second magnification is 4 in this embodiment.
Based on this, the first upsampling module 303 is specifically configured to upsample each frame of the region of interest by using the first magnification as the upsampling magnification to acquire a first intermediate image, and the super-resolution reconstruction module 304 is specifically configured to perform super-resolution reconstruction on a plurality of frames of the first intermediate image by using the second magnification as the super-resolution reconstruction magnification to acquire the zoom image.
Referring to fig. 6, the super-resolution reconstruction module 304 in this embodiment may specifically include:
a determining unit 3041 for determining one frame of the plurality of frames of the first intermediate image as a reference image;
an alignment unit 3042 for aligning a plurality of frames of the first intermediate image based on the reference image;
an amplification unit 3043 for performing pixel amplification on the reference image according to the second magnification;
a first padding unit 3044 for padding pixel values of other first intermediate images than the reference image into the pixel-augmented reference image according to the alignment result of the plurality of frames of first intermediate images;
a judging unit 3045 for judging whether the padded reference image includes an unfilled pixel;
if so, a second padding unit 3046 is invoked to pad the non-padded pixels according to median filtering.
Specifically, aligning multiple frames of first intermediate images, that is, implementing image registration of multiple frames of first images, and aligning multiple frames of first intermediate images by the aligning unit 3042, so that pixel points belonging to the same object in the multiple frames of first images are in one-to-one correspondence.
The amplifying unit 3043 performs pixel amplification on the reference image, where the manner of pixel amplification may be customized according to the actual application, for example, zero pixel amplification may be adopted to implement, specifically, amplification of the reference image may be implemented first based on the nearest neighbor interpolation, and then interpolation data is filled with a value of 0.
The first padding unit 3044 may pad pixel values of the reference image to avoid blurring of the image due to the motion, and further, according to the alignment result, if there are several frames of pixel points of the first intermediate image corresponding to the same padding area in the reference image, the padding value of the same padding area is determined by the pixel values of the pixel points in the several frames of the first intermediate image. Specifically, the padding value may be calculated according to the following formula:
wherein I is a padding value, h represents an adjustable parameter, IkRepresenting the pixel value, diffk＝Ik-I0K denotes a picture frame number, k-0 denotes a reference picture, and flagkIndicating whether the frame image is in a filled position, if so, taking the value as 1, otherwise, taking the value as 0, and further, taking the flag0＝1。
If the determination unit 3045 determines that the image is a zoom image, the second padding unit 3046 is called to perform padding again to acquire the zoom image, specifically, the pixel values of the padded reference image may be padded to avoid blurring of the image due to motion, and if the determination is no, the zoom image may be acquired after the second padding unit 3046 is called.
Referring to fig. 6, the digital zoom system of the present embodiment may further include a first iterative correction module 306, configured to perform iterative correction on the zoom image acquired by the super-resolution reconstruction module 304 to obtain a final zoom image. Specifically, the zoom image acquired by the super-resolution reconstruction module 304 may be used as an estimated value of a high-resolution image (HR), a corresponding low-resolution image (LR) may be generated by an imaging model, and an error with the original data may be calculated, so that the error is minimized. In this embodiment, the number of iterations may be set in a customized manner according to the computational complexity requirement, and may be set to a value within 10 in general, so as to correct the error introduced by the upsampling or the alignment.
Specifically, the iterative correction implements the formula:
wherein n is the number of iterations,
is an estimate of the high resolution image, beta is the iteration step, D, PSF, WrapkRespectively down-sampling, the point spread function of the imaging system and motion-induced image deformation.
Specifically, in the present embodiment, the imaging model of the digital imaging apparatus can be expressed as:
LR＝D{PSF*Wrap{HR}}+noise
where HR is a high resolution image, Wrap { } denotes image deformation due to motion, denotes convolution operation, PSF is a point spread function of the imaging system, D { } denotes the down-sampling process, and noise is noise. Further, in the present embodiment, the first intermediate image is selected as the LR representing the original datakIn the iterative correction of the embodiment, a regularization term is also introduced
To make the iteration more robust, wherein
LadderThe degree operator, λ, is the regularization parameter. The purpose of the iterative correction is to extrapolate the high-resolution image HR back from the known multi-frame noisy low-resolution map LR:
in the embodiment, multiple frames of original images are continuously acquired according to the zooming instruction, and the acquired multiple frames of original images are sequentially subjected to cutting, upsampling and super-resolution reconstruction to acquire the zooming image. Further, in the embodiment, the zoom magnification is split into the first magnification and the second magnification, which not only overcomes the defect that the super-resolution reconstruction applicable to integral-multiple sampling cannot be applied to digital zoom with the zoom magnification accurate to a decimal place, but also fully utilizes the advantage of the super-resolution reconstruction on integral-multiple magnification. In addition, the introduction of iterative correction can also avoid image blurring caused by up-sampling, image alignment, point spread function of a digital imaging device, and the like.
Example 4
On the basis of embodiment 3, the present embodiment provides a digital zoom system, and referring to fig. 7, the digital zoom system of the present embodiment may further include:
a determining module 401, configured to determine whether a zoom magnification is smaller than a preset threshold;
if so, the second upsampling module 402 is invoked; if not, the second obtaining module 305 is called;
a second upsampling module 402, configured to upsample multiple frames of regions of interest to obtain a second intermediate image;
and a fusion module 403, configured to fuse the multiple frames of second intermediate images to obtain a zoom image.
Specifically, in this embodiment, in order to fully utilize the super-resolution reconstruction to improve the digital zoom capability, the zoom ratio may be segmented to divide the zoom ratio into two zoom segments, namely a digital interpolation magnification segment and a super-resolution zoom segment, where the preset threshold may be set by a user according to actual application, for example, the preset threshold may take a value of 2. In this embodiment, the second upsampling module 402 may specifically be configured to upsample the region of interest of each frame by using the zoom magnification as the upsampling magnification to obtain a second intermediate image, that is, the size of the second intermediate image is the same as that of the original image. The fusion module 403 may be specifically configured to align multiple frames of second intermediate images and fuse the aligned multiple frames of second intermediate images, so as to significantly reduce image noise and improve the signal-to-noise ratio of the image.
Referring to fig. 7, the digital zoom system of this embodiment may further include a second iterative correction module 404, configured to perform iterative correction on the zoom image obtained by the fusion module 403 to obtain a final zoom image, so as to correct errors introduced by upsampling or alignment, and further improve the quality of the zoom image, unlike in embodiment 3, in this embodiment, an original image is selected as the LR representing the original datakAnd may not introduce a regularization term here
Namely:
on the basis of utilizing multi-frame image sampling information, different image processing modes are adopted according to different numerical ranges of zoom magnifications, specifically, when the zoom magnifications are small, digital zooming is realized in a multi-frame fusion mode, compared with digital zooming of a single-frame image which completely depends on an interpolation algorithm, improvement of a signal-to-noise ratio can be obtained, the quality of the obtained zoom image is better, when the zoom magnifications are large, digital zooming is realized in a super-resolution reconstruction mode, improvement of resolution and the signal-to-noise ratio can be obtained, and therefore stronger digital zooming capacity can be realized. In addition, due to the introduction of iterative correction, image blurring caused by up-sampling, image alignment, a point spread function of a digital imaging device and the like can be avoided, so that the quality of the zoom image is further improved.
Example 5
The present embodiment provides an electronic device, which may be represented in the form of a computing device (for example, may be a server device), including a memory, a processor, and a computer program stored in the memory and running on the processor, where the processor executes the computer program to implement the digital zoom method provided in embodiment 1 or 2.
Fig. 8 shows a schematic diagram of a hardware structure of the present embodiment, and as shown in fig. 8, the electronic device 9 specifically includes:
at least one processor 91, at least one memory 92, and a bus 93 for connecting the various system components (including the processor 91 and the memory 92), wherein:
the bus 93 includes a data bus, an address bus, and a control bus.
The processor 91 executes various functional applications and data processing, such as the digital zoom method provided in embodiment 1 or 2 of the present invention, by executing the computer program stored in the memory 92.
The electronic device 9 may further communicate with one or more external devices 94 (e.g., a keyboard, a pointing device, etc.). Such communication may be through an input/output (I/O) interface 95. Also, the electronic device 9 may communicate with one or more networks (e.g., a Local Area Network (LAN), a Wide Area Network (WAN), and/or a public network, such as the Internet) via the network adapter 96. The network adapter 96 communicates with the other modules of the electronic device 9 via the bus 93. It should be understood that although not shown in the figures, other hardware and/or software modules may be used in conjunction with the electronic device 9, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems, etc.
It should be noted that although in the above detailed description several units/modules or sub-units/modules of the electronic device are mentioned, such a division is merely exemplary and not mandatory. Indeed, the features and functionality of two or more of the units/modules described above may be embodied in one unit/module, according to embodiments of the application. Conversely, the features and functions of one unit/module described above may be further divided into embodiments by a plurality of units/modules.
Example 6
The present embodiment provides a computer-readable storage medium on which a computer program is stored, which when executed by a processor implements the steps of the digital zoom method provided in embodiment 1 or 2.
More specific examples, among others, that the readable storage medium may employ may include, but are not limited to: a portable disk, a hard disk, random access memory, read only memory, erasable programmable read only memory, optical storage device, magnetic storage device, or any suitable combination of the foregoing.
In a possible implementation, the invention may also be implemented in the form of a program product comprising program code for causing a terminal device to perform the steps of implementing the digital zoom method as described in embodiment 1 or 2, when said program product is run on said terminal device.
Where program code for carrying out the invention is written in any combination of one or more programming languages, the program code may be executed entirely on the user device, partly on the user device, as a stand-alone software package, partly on the user device and partly on a remote device or entirely on the remote device.
Example 7
The present embodiment provides a digital imaging device, which includes the digital zoom system provided in embodiment 3 or 4, where the digital imaging device of this embodiment may continuously obtain multiple frames of original images according to a zoom instruction, and sequentially perform cropping, upsampling, and super-resolution reconstruction on the obtained multiple frames of original images to obtain a zoom image, and compared with digital zooming on a single frame of image that completely depends on an interpolation algorithm, better image resolution and better image quality may be obtained, so that a stronger digital zoom capability may be achieved. Further, in the embodiment, the zoom magnification is split into the first magnification and the second magnification, which not only overcomes the defect that the super-resolution reconstruction applicable to integral-multiple sampling cannot be applied to digital zoom with the zoom magnification accurate to a decimal place, but also fully utilizes the advantage of the super-resolution reconstruction on integral-multiple magnification. In addition, the introduction of iterative correction can also avoid image blurring caused by up-sampling, image alignment, point spread function of a digital imaging device, and the like.
While specific embodiments of the invention have been described above, it will be appreciated by those skilled in the art that this is by way of example only, and that the scope of the invention is defined by the appended claims. Various changes and modifications to these embodiments may be made by those skilled in the art without departing from the spirit and scope of the invention, and these changes and modifications are within the scope of the invention.
Claims (15)
1. A digital zoom method, comprising:
continuously acquiring a plurality of frames of original images according to a zooming instruction, wherein the zooming instruction comprises zooming magnification;
cutting each frame of original image according to the zooming magnification to obtain an interested area;
upsampling a multi-frame region of interest to obtain a first intermediate image;
performing super-resolution reconstruction on a plurality of frames of the first intermediate image to obtain a zoom image;
the method further comprises the following steps of up-sampling the plurality of frames of interest regions to acquire a first intermediate image:
acquiring a first multiplying power and a second multiplying power according to the zooming multiplying power, wherein the zooming multiplying power is the product of the first multiplying power and the second multiplying power, the minimum value of the first multiplying power is 1, and the second multiplying power is an integer larger than 1;
the step of upsampling the plurality of frames of interest regions to acquire a first intermediate image comprises:
taking the first multiplying power as an up-sampling multiplying power to up-sample each frame of region of interest to acquire a first intermediate image;
the step of performing super-resolution reconstruction on a plurality of frames of the first intermediate image to obtain a zoom image comprises the following steps:
and performing super-resolution reconstruction on the plurality of frames of the first intermediate image by taking the second magnification as a super-resolution reconstruction magnification to acquire a zoom image.
2. The digital zooming method of claim 1, further comprising, after the step of cropping each frame of original image according to the zoom magnification to obtain a region of interest:
judging whether the zoom magnification is smaller than a preset threshold value or not;
and if not, performing the step of up-sampling the multi-frame interested region to acquire the first intermediate image.
3. The digital zooming method of claim 1, wherein said step of super-resolution reconstructing a plurality of frames of said first intermediate image using said second magnification as a super-resolution reconstruction magnification to obtain a zoom image comprises:
determining one frame of the plurality of frames of the first intermediate image as a reference image;
aligning the plurality of frames of first intermediate images according to the reference image;
performing pixel amplification on the reference image according to the second multiplying power;
filling pixel values of other first intermediate images except the reference image into the pixel-amplified reference image according to the alignment result of the plurality of frames of first intermediate images;
determining whether the padded reference image comprises an unpadded pixel;
if yes, filling the unfilled pixels according to median filtering;
and/or the presence of a gas in the gas,
after the step of super-resolution reconstructing the plurality of frames of the first intermediate image using the second magnification as a super-resolution reconstruction magnification to acquire a zoom image, the method further includes:
and carrying out iterative correction on the zoom image to obtain a final zoom image.
4. The digital zooming method of claim 2, wherein when the step of determining whether the zoom magnification is smaller than a preset threshold is determined as yes:
upsampling a plurality of frames of interest regions to acquire a second intermediate image;
and fusing the second intermediate images of the plurality of frames to obtain a zoom image.
5. The digital zoom method of claim 4, wherein the step of upsampling the plurality of frames of interest to obtain the second intermediate image comprises:
taking the zoom magnification as an up-sampling magnification to up-sample each frame of region of interest to acquire a second intermediate image;
and/or the presence of a gas in the gas,
after the step of fusing the plurality of frames of the second intermediate images to obtain the zoom image, the method further comprises:
and carrying out iterative correction on the zoom image to obtain a final zoom image.
6. The digital zooming method as claimed in claim 1, wherein said step of cropping each frame of original image according to said zoom magnification to obtain a region of interest comprises:
and cutting each frame of original image by taking the center of each frame of original image as a reference point according to the zooming magnification to acquire an interested area.
7. A digital zoom system, comprising:
the first acquisition module is used for continuously acquiring multiple frames of original images according to a zooming instruction, wherein the zooming instruction comprises zooming magnification;
the cutting module is used for cutting each frame of original image according to the zooming multiplying power so as to obtain an interested area;
the first up-sampling module is used for up-sampling a plurality of frames of interested areas to acquire a first intermediate image;
the super-resolution reconstruction module is used for performing super-resolution reconstruction on the first intermediate images of the multiple frames to acquire a zoom image;
the digital zoom system further includes:
the second obtaining module is used for obtaining a first multiplying power and a second multiplying power according to the zooming multiplying power, wherein the zooming multiplying power is the product of the first multiplying power and the second multiplying power, the minimum value of the first multiplying power is 1, and the second multiplying power is an integer larger than 1;
the first up-sampling module is specifically configured to up-sample each frame of region of interest by using the first magnification as an up-sampling magnification to obtain a first intermediate image;
the super-resolution reconstruction module is specifically configured to perform super-resolution reconstruction on a plurality of frames of the first intermediate image by using the second magnification as a super-resolution reconstruction magnification to obtain a zoom image.
8. The digital zoom system of claim 7, further comprising:
the judging module is used for judging whether the zoom magnification is smaller than a preset threshold value or not;
if not, the first up-sampling module is called.
9. The digital zoom system of claim 7, wherein the super-resolution reconstruction module comprises:
a determining unit, configured to determine one of the plurality of frames of the first intermediate image as a reference image;
an alignment unit configured to align the plurality of frames of first intermediate images based on the reference image;
the amplification unit is used for amplifying the pixels of the reference image according to the second multiplying power;
a first padding unit for padding pixel values of other first intermediate images except the reference image into the pixel-augmented reference image according to an alignment result of a plurality of frames of first intermediate images;
a judging unit for judging whether the padded reference image includes an unfilled pixel;
if so, calling a second filling unit for filling the unfilled pixels according to median filtering;
and/or the presence of a gas in the gas,
the digital zoom system further includes:
and the first iterative correction module is used for iteratively correcting the zoom image acquired by the super-resolution reconstruction module to obtain a final zoom image.
10. The digital zoom system of claim 8, further comprising:
the second up-sampling module is used for up-sampling the multi-frame interested area to acquire a second intermediate image when the judgment module judges that the multi-frame interested area is positive;
and the fusion module is used for fusing the second intermediate images of the plurality of frames to obtain a zoom image.
11. The digital zoom system of claim 10, wherein the second upsampling module is specifically configured to upsample each frame of interest using the zoom magnification as an upsampling magnification to obtain a second intermediate image;
and/or the presence of a gas in the gas,
the digital zoom system further includes:
and the second iterative correction module is used for iteratively correcting the zoom images acquired by the fusion module to obtain final zoom images.
12. The digital zoom system of claim 7, wherein the cropping module is specifically configured to crop each frame of the original image according to the zoom magnification with a center of each frame of the original image as a reference point to obtain the region of interest.
13. An electronic device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the digital zoom method of any of claims 1 to 6 when executing the computer program.
14. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the digital zoom method according to any one of claims 1 to 6.
15. A digital imaging device comprising a digital zoom system as claimed in any one of claims 7-12.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010197617.2A CN111447359B (en) | 2020-03-19 | 2020-03-19 | Digital zoom method, system, electronic device, medium, and digital imaging device |
| PCT/CN2021/079929 WO2021185130A1 (en) | 2020-03-19 | 2021-03-10 | Digital zoom method and system, electronic device, medium, and digital imaging device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010197617.2A CN111447359B (en) | 2020-03-19 | 2020-03-19 | Digital zoom method, system, electronic device, medium, and digital imaging device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111447359A CN111447359A (en) | 2020-07-24 |
| CN111447359B true CN111447359B (en) | 2021-07-02 |
Family
ID=71654153
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010197617.2A Active CN111447359B (en) | 2020-03-19 | 2020-03-19 | Digital zoom method, system, electronic device, medium, and digital imaging device |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN111447359B (en) |
| WO (1) | WO2021185130A1 (en) |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111447359B (en) * | 2020-03-19 | 2021-07-02 | 展讯通信（上海）有限公司 | Digital zoom method, system, electronic device, medium, and digital imaging device |
| CN114390213B (en) * | 2020-10-22 | 2023-04-18 | 华为技术有限公司 | Shooting method and equipment |
| CN112954195A (en) * | 2021-01-27 | 2021-06-11 | 维沃移动通信有限公司 | Focusing method, focusing device, electronic equipment and medium |
| WO2022199594A1 (en) * | 2021-03-26 | 2022-09-29 | 华为技术有限公司 | Method for performing remote video, and related device |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102084327A (en) * | 2009-05-19 | 2011-06-01 | 索尼公司 | Digital image processing device and associated methodology of performing touch-based image scaling |
| CN102170571A (en) * | 2010-06-22 | 2011-08-31 | 上海盈方微电子有限公司 | Digital still camera framework for supporting two-channel CMOS (Complementary Metal Oxide Semiconductor) sensor |
| CN103986867A (en) * | 2014-04-24 | 2014-08-13 | 宇龙计算机通信科技(深圳)有限公司 | Image shooting terminal and image shooting method |
| CN108391035A (en) * | 2018-03-26 | 2018-08-10 | 华为技术有限公司 | A kind of image pickup method, device and equipment |
| CN108694700A (en) * | 2017-04-10 | 2018-10-23 | 三星电子株式会社 | System and method for deep learning image super-resolution |
| CN109495681A (en) * | 2017-09-12 | 2019-03-19 | 天津三星通信技术研究有限公司 | The method and apparatus for obtaining image |
| CN109886875A (en) * | 2019-01-31 | 2019-06-14 | 深圳市商汤科技有限公司 | Image super-resolution rebuilding method and device, storage medium |
| CN110490196A (en) * | 2019-08-09 | 2019-11-22 | Oppo广东移动通信有限公司 | Subject detection method and apparatus, electronic equipment, computer readable storage medium |
| CN110536057A (en) * | 2019-08-30 | 2019-12-03 | Oppo广东移动通信有限公司 | Image processing method and device, electronic equipment, computer readable storage medium |
| CN110650295A (en) * | 2019-11-26 | 2020-01-03 | 展讯通信（上海）有限公司 | Image processing method and device |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US8477217B2 (en) * | 2008-06-30 | 2013-07-02 | Sony Corporation | Super-resolution digital zoom |
| US8988538B2 (en) * | 2012-07-02 | 2015-03-24 | Canon Kabushiki Kaisha | Image pickup apparatus and lens apparatus |
| US9154698B2 (en) * | 2013-06-19 | 2015-10-06 | Qualcomm Technologies, Inc. | System and method for single-frame based super resolution interpolation for digital cameras |
| JP2017010115A (en) * | 2015-06-17 | 2017-01-12 | 日本電信電話株式会社 | Image resolution conversion device, image resolution conversion method, and computer program |
| CN111447359B (en) * | 2020-03-19 | 2021-07-02 | 展讯通信（上海）有限公司 | Digital zoom method, system, electronic device, medium, and digital imaging device |
-
2020
- 2020-03-19 CN CN202010197617.2A patent/CN111447359B/en active Active
-
2021
- 2021-03-10 WO PCT/CN2021/079929 patent/WO2021185130A1/en active Application Filing
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102084327A (en) * | 2009-05-19 | 2011-06-01 | 索尼公司 | Digital image processing device and associated methodology of performing touch-based image scaling |
| CN102170571A (en) * | 2010-06-22 | 2011-08-31 | 上海盈方微电子有限公司 | Digital still camera framework for supporting two-channel CMOS (Complementary Metal Oxide Semiconductor) sensor |
| CN103986867A (en) * | 2014-04-24 | 2014-08-13 | 宇龙计算机通信科技(深圳)有限公司 | Image shooting terminal and image shooting method |
| CN108694700A (en) * | 2017-04-10 | 2018-10-23 | 三星电子株式会社 | System and method for deep learning image super-resolution |
| CN109495681A (en) * | 2017-09-12 | 2019-03-19 | 天津三星通信技术研究有限公司 | The method and apparatus for obtaining image |
| CN108391035A (en) * | 2018-03-26 | 2018-08-10 | 华为技术有限公司 | A kind of image pickup method, device and equipment |
| CN109886875A (en) * | 2019-01-31 | 2019-06-14 | 深圳市商汤科技有限公司 | Image super-resolution rebuilding method and device, storage medium |
| CN110490196A (en) * | 2019-08-09 | 2019-11-22 | Oppo广东移动通信有限公司 | Subject detection method and apparatus, electronic equipment, computer readable storage medium |
| CN110536057A (en) * | 2019-08-30 | 2019-12-03 | Oppo广东移动通信有限公司 | Image processing method and device, electronic equipment, computer readable storage medium |
| CN110650295A (en) * | 2019-11-26 | 2020-01-03 | 展讯通信（上海）有限公司 | Image processing method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111447359A (en) | 2020-07-24 |
| WO2021185130A1 (en) | 2021-09-23 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111447359B (en) | Digital zoom method, system, electronic device, medium, and digital imaging device | |
| US9202258B2 (en) | Video retargeting using content-dependent scaling vectors | |
| CN111402139B (en) | Image processing method, apparatus, electronic device, and computer-readable storage medium | |
| KR20130001213A (en) | Method and system for generating an output image of increased pixel resolution from an input image | |
| CN112767290B (en) | Image fusion method, image fusion device, storage medium and terminal device | |
| Jeong et al. | Multi-frame example-based super-resolution using locally directional self-similarity | |
| CN111652800B (en) | Single image super-resolution method and computer readable storage medium | |
| JPWO2018225133A1 (en) | Image processing apparatus, image processing method, and image processing program | |
| Xu et al. | Exploiting raw images for real-scene super-resolution | |
| US20220343463A1 (en) | Changing the size of images by means of a neural network | |
| US20150324953A1 (en) | Method and apparatus for performing single-image super-resolution | |
| Zhang et al. | Optical flow reusing for high-efficiency space-time video super resolution | |
| KR101140953B1 (en) | Method and apparatus for correcting distorted image | |
| WO2021211771A1 (en) | Systems and methods for optical flow estimation | |
| GB2475716A (en) | Providing a super-resolution image | |
| KR20130104410A (en) | Apparatus and method for super-resolution based on error model of single image | |
| KR101152525B1 (en) | Methods of removing motion blur and apparatuses for the same | |
| Xie et al. | Bidirectionally aligned sparse representation for single image super-resolution | |
| Sv et al. | Detail warping based video super-resolution using image guides | |
| CN114092562A (en) | Noise model calibration method, image denoising method, device, equipment and medium | |
| CN113592844A (en) | Image distortion correction method and device, terminal and storage medium | |
| JP2013069012A (en) | Multi-lens imaging apparatus | |
| CN108198134B (en) | Single-image super-resolution reconstruction method based on combined average extreme learning machine | |
| CN114902237A (en) | Image processing method and device and electronic equipment | |
| CN112581362A (en) | Image processing method and device for adjusting image details |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |