CN113313707A - Original image processing method, device, equipment and readable storage medium - Google Patents
Original image processing method, device, equipment and readable storage medium Download PDFInfo
- Publication number
- CN113313707A CN113313707A CN202110716617.3A CN202110716617A CN113313707A CN 113313707 A CN113313707 A CN 113313707A CN 202110716617 A CN202110716617 A CN 202110716617A CN 113313707 A CN113313707 A CN 113313707A
- Authority
- CN
- China
- Prior art keywords
- image
- registered
- frame
- reference frame
- sub
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003860 storage Methods 0.000 title claims description 12
- 238000003672 processing method Methods 0.000 title abstract description 9
- 230000004927 fusion Effects 0.000 claims abstract description 27
- 238000000034 method Methods 0.000 claims description 73
- 230000009466 transformation Effects 0.000 claims description 60
- 239000011159 matrix material Substances 0.000 claims description 48
- 239000013598 vector Substances 0.000 claims description 43
- 238000012545 processing Methods 0.000 claims description 25
- 238000013519 translation Methods 0.000 claims description 11
- 238000007781 pre-processing Methods 0.000 claims description 9
- 238000009825 accumulation Methods 0.000 claims description 8
- 230000003628 erosive effect Effects 0.000 claims description 8
- 238000012937 correction Methods 0.000 claims description 7
- 238000012935 Averaging Methods 0.000 claims description 6
- 230000000877 morphologic effect Effects 0.000 claims description 6
- 238000003709 image segmentation Methods 0.000 claims description 3
- 230000000694 effects Effects 0.000 abstract description 6
- 230000008569 process Effects 0.000 description 35
- 238000004364 calculation method Methods 0.000 description 8
- 230000011218 segmentation Effects 0.000 description 8
- 238000010586 diagram Methods 0.000 description 7
- 238000005192 partition Methods 0.000 description 6
- 238000007499 fusion processing Methods 0.000 description 5
- 230000000903 blocking effect Effects 0.000 description 4
- 238000012216 screening Methods 0.000 description 4
- 230000009286 beneficial effect Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000010339 dilation Effects 0.000 description 2
- 230000002349 favourable effect Effects 0.000 description 2
- PCHJSUWPFVWCPO-UHFFFAOYSA-N gold Chemical compound [Au] PCHJSUWPFVWCPO-UHFFFAOYSA-N 0.000 description 2
- 239000010931 gold Substances 0.000 description 2
- 229910052737 gold Inorganic materials 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000005457 optimization Methods 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 238000000844 transformation Methods 0.000 description 2
- 238000011426 transformation method Methods 0.000 description 2
- 238000004422 calculation algorithm Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/0002—Inspection of images, e.g. flaw detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/20—Image enhancement or restoration using local operators
- G06T5/30—Erosion or dilatation, e.g. thinning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/155—Segmentation; Edge detection involving morphological operators
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/30—Determination of transform parameters for the alignment of images, i.e. image registration
- G06T7/32—Determination of transform parameters for the alignment of images, i.e. image registration using correlation-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20016—Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Quality & Reliability (AREA)
- Image Processing (AREA)
Abstract
The invention provides an original image processing method, which comprises the following steps: acquiring multiple continuous original images; determining a reference frame image and a frame image to be registered according to a plurality of frames of original images; determining the rotation angle of a camera of the frame image to be registered relative to the camera of the reference frame image according to the reference frame image and the frame image to be registered; determining a registration mode of the frame image to be registered and the reference frame image according to the rotation angle, and performing registration according to the registration mode to form an alignment frame image corresponding to the frame image to be registered; and fusing the alignment frame image and the reference frame image to obtain a processed original image. The invention can accurately match multiple frames of original images and obtain good fusion effect.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method, an apparatus, a device, and a readable storage medium for processing an original image.
Background
Common image denoising methods are mostly applied to images in an RGB format or a YUV format. Images in RGB format or YUV format are usually obtained by performing nonlinear local processing transformation on original images. In the nonlinear local processing transformation process, the noise model of the original image is destroyed, thereby causing the noise reduction algorithm to be extremely complex. In order to accurately denoise an image, an original image can be used for denoising. When the original image is used for denoising, in order to obtain more image information, a plurality of frames of images can be used for fusion. However, because global or local motion usually exists between multiple frames of images, noise is introduced during the fusion process, so that matching accuracy cannot be guaranteed, and the fusion effect is affected.
Disclosure of Invention
The original image processing method, the device and the equipment and the readable storage medium provided by the invention can accurately match a plurality of frames of original images and obtain a good fusion effect.
In a first aspect, the present invention provides a method for processing an original image, the method comprising:
acquiring multiple continuous original images;
determining a reference frame image and a frame image to be registered according to a plurality of frames of original images;
determining the rotation angle of a camera of the frame image to be registered relative to the camera of the reference frame image according to the reference frame image and the frame image to be registered;
determining a registration mode of the frame image to be registered and the reference frame image according to the rotation angle, and performing registration according to the registration mode to form an alignment frame image corresponding to the frame image to be registered;
and fusing the alignment frame image and the reference frame image to obtain a processed original image.
Optionally, determining, according to the rotation angle, a registration manner of the frame image to be registered and the reference frame image includes:
when the rotation angle is smaller than a preset threshold value, registering the frame image to be registered to the reference frame image in a translation transformation mode;
and when the rotation angle is not less than a preset threshold value, registering the frame image to be registered to the reference frame image in a homography transformation mode.
Optionally, determining a registration manner of the frame image to be registered and the reference frame image according to the rotation angle, and performing registration according to the registration manner includes:
carrying out interlaced alternate dereferencing on pixels of an original image to split the reference frame image into reference frame sub-images corresponding to four channels, and splitting the frame image to be registered into frame sub-images to be registered corresponding to the four channels;
preprocessing the reference frame sub-image and the frame sub-image to be registered to realize the enhancement of the reference frame sub-image and the frame sub-image to be registered;
and constructing a reference frame pyramid for the reference frame sub-images, constructing a frame pyramid to be registered for the frame sub-images to be registered, and performing registration based on the reference frame pyramid and the frame pyramid to be registered.
Optionally, registering based on the reference frame pyramid and the frame pyramid to be registered includes:
registering the frame pyramid to be registered corresponding to the four channels to the reference frame pyramid so as to determine registration transformation sub-matrixes corresponding to the four channels;
and averaging the registration transformation sub-matrixes corresponding to the four channels to obtain a registration transformation matrix.
Optionally, when the rotation angle is smaller than a predetermined threshold, performing registration based on the reference frame pyramid and the frame pyramid to be registered includes:
acquiring an offset vector by adopting a template matching method for the topmost pyramid image of the frame to be registered and the topmost pyramid image of the reference frame;
and solving and correcting layer by layer from the top layer of the pyramid of the frame to be registered and the bottom layer of the pyramid of the reference frame according to the offset vector so as to obtain the offset vector between the sub-image of the frame to be registered and the sub-image of the reference frame.
Optionally, when the rotation angle is not less than a predetermined threshold, performing registration based on the reference frame pyramid and the frame pyramid to be registered includes:
dividing each layer of images of the frame pyramid to be registered and the reference frame pyramid to obtain frame sub image blocks to be registered and reference frame sub image blocks;
determining the characteristic points of the frame sub image blocks to be registered and the reference frame sub image blocks in each layer of images;
determining all feature points of each frame sub-image block to be registered of the frame sub-image to be registered and all feature points of each frame sub-image block to be registered of the reference frame sub-image according to the feature points in each layer of image;
matching all the feature points of the reference frame sub image blocks according to all the feature points of the frame sub image blocks to be registered so as to determine matching point pairs in the frame sub image blocks to be registered and the corresponding reference frame sub image blocks;
and selecting a preset proportion of matching point pairs with the best quality from the frame sub image blocks to be registered and the corresponding reference frame sub image blocks to determine the homography matrix.
Optionally, fusing the alignment frame image and the reference frame image comprises:
determining motion pixels in the aligned frame images according to a noise model;
performing morphological erosion and expansion on a region corresponding to the motion pixel in the reference frame image;
and performing accumulation fusion on the regions except the motion pixels in the aligned frame image and the reference frame image.
Optionally, determining the reference frame image and the frame image to be registered according to a plurality of frames of the original images includes:
determining a reference frame image according to focusing information corresponding to a plurality of frames of original images, and taking the original images except the reference frame as frame images to be registered; or,
and determining a reference frame image according to the high-frequency information of the original images of the plurality of frames, and taking the original images except the reference frame as frame images to be registered.
In a second aspect, the present invention provides an original image processing apparatus, comprising:
the image acquisition module is used for acquiring multiple continuous original images;
the type determining module is used for determining a reference frame image and a frame image to be registered according to a plurality of frames of the original images;
an angle determining module, configured to determine, according to the reference frame image and the frame image to be registered, a rotation angle of a camera of the frame image to be registered relative to a camera of the reference frame image;
the image registration module is used for determining a registration mode of the frame image to be registered and the reference frame image according to the rotation angle, and performing registration according to the registration mode to form an alignment frame image corresponding to the frame image to be registered;
and the image fusion module is used for fusing the alignment frame image and the reference frame image to obtain a processed original image.
Optionally, the image registration module comprises:
the translation registration submodule is used for registering the frame image to be registered to the reference frame image in a translation transformation mode when the rotation angle is smaller than a preset threshold value;
and the homography registration submodule is used for registering the frame image to be registered to the reference frame image in a homography transformation mode when the rotation angle is not less than a preset threshold value.
Optionally, the image registration module comprises:
the image splitting submodule is used for carrying out interlaced alternate value taking on the pixels of the original image so as to split the reference frame image into reference frame sub-images corresponding to four channels and split the frame image to be registered into frame sub-images to be registered corresponding to the four channels;
the preprocessing submodule is used for preprocessing the reference frame sub-image and the frame sub-image to be registered so as to enhance the reference frame sub-image and the frame sub-image to be registered;
and the image registration submodule is used for constructing a reference frame pyramid for the reference frame sub-images, constructing a frame pyramid to be registered for the frame sub-images to be registered, and performing registration based on the reference frame pyramid and the frame pyramid to be registered.
Optionally, the image registration module comprises:
the submatrix determining submodule is used for registering the pyramid of the frame to be registered corresponding to the four channels to the pyramid of the reference frame so as to determine a registration transformation submatrix corresponding to the four channels;
and the transformation matrix determining submodule is used for averaging the registration transformation submatrices corresponding to the four channels to obtain a registration transformation matrix.
Optionally, translating the registration sub-module comprises:
the top-level offset unit is used for acquiring offset vectors by adopting a template matching device for the Gaussian pyramid top-level image of the frame sub-image to be registered and the Gaussian pyramid top-level image of the reference frame sub-image;
and the offset correction unit is used for solving and correcting the frame pyramid to be registered and the reference frame pyramid from the top layer to the bottom layer by layer according to the offset vector so as to obtain the offset vector between the frame sub-image to be registered and the reference frame sub-image.
Optionally, the homography registration sub-module comprises:
the image segmentation unit is used for segmenting each layer of images of the frame pyramid to be registered and the reference frame pyramid to obtain frame sub image blocks to be registered and reference frame sub image blocks;
the pyramid characteristic point unit is used for determining characteristic points of the frame sub image blocks to be registered and the reference frame sub image blocks in each layer of images;
the bottom layer feature point unit is used for determining all feature points of each frame sub-image block to be registered of the frame sub-image to be registered and all feature points of each frame sub-image block to be registered of the reference frame sub-image according to the feature points in each layer of image;
the matching point pair unit is used for matching all the feature points of the reference frame sub image blocks according to all the feature points of the frame sub image blocks to be registered so as to determine matching point pairs in the frame sub image blocks to be registered and the corresponding reference frame sub image blocks;
and the matrix determining unit is used for selecting a preset proportion of matching point pairs with the best quality from the frame sub image blocks to be registered and the corresponding reference frame sub image blocks to determine the homography matrix.
Optionally, the image fusion module comprises:
a moving pixel determination module for determining moving pixels in the aligned frame images according to a noise model;
the reference frame image processing module is used for performing morphological erosion and expansion on a region corresponding to the motion pixel in the reference frame image;
and the accumulation fusion module is used for carrying out accumulation fusion on the areas except the motion pixels in the alignment frame image and the reference frame image.
Optionally, the type determining module includes:
the first type determining submodule is used for determining a reference frame image according to focusing information corresponding to a plurality of frames of original images, and taking the original images except the reference frame as frame images to be registered; or,
and the second type determining submodule is used for determining a reference frame image according to the high-frequency information of the original images of the plurality of frames, and taking the original images except the reference frame as frame images to be registered.
In a third aspect, the present invention provides an apparatus comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the methods described above.
In a fourth aspect, the invention provides a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement a method as in any one of the above.
In the technical scheme provided by the invention, a reference frame image is selected from a plurality of frame images, the registration mode of the frame image to be registered is determined according to the rotation angle between the frame image to be registered and the reference frame image, and the most appropriate registration mode is adopted for different frames to be registered. Due to the fact that the most appropriate registration mode is adopted for different frames to be registered, introduction of noise can be avoided, matching accuracy is improved, and images with good fusion effects are obtained.
Drawings
FIG. 1 is a flowchart illustrating a method for processing an original image according to an embodiment of the present invention;
FIG. 2 is a flowchart of image registration in a raw image processing method according to another embodiment of the present invention;
FIG. 3 is a flowchart of image registration in a raw image processing method according to another embodiment of the present invention;
FIG. 4 is a flowchart of obtaining a transformation matrix in an original image processing method according to another embodiment of the present invention;
FIG. 5 is a flowchart illustrating an offset vector obtaining method according to another embodiment of the present invention;
FIG. 6 is a flowchart of homography matrix acquisition in a method of raw image processing according to another embodiment of the present invention;
FIG. 7 is a flowchart illustrating image fusion performed in an original image processing method according to another embodiment of the present invention;
FIG. 8 is a diagram of an original image processing apparatus according to another embodiment of the present invention;
FIG. 9 is a diagram illustrating an image registration module of an original image processing apparatus according to another embodiment of the present invention;
FIG. 10 is a diagram illustrating an image registration module of an original image processing apparatus according to another embodiment of the present invention;
FIG. 11 is a diagram illustrating an image registration module of an original image processing apparatus according to another embodiment of the present invention;
FIG. 12 is a schematic view of a translational registration sub-module of an original image processing apparatus according to another embodiment of the present invention;
FIG. 13 is a schematic diagram of a homography registration sub-module of an original image processing apparatus according to another embodiment of the present invention;
FIG. 14 is a diagram illustrating an image fusion module of an original image processing apparatus according to another embodiment of the present invention;
fig. 15 is a schematic diagram of arrangement of four channel pixels of a bayer-format raw image.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
An embodiment of the present invention provides an original image processing method, as shown in fig. 1, the method includes:
in some embodiments, the RAW image refers to a RAW image obtained using an image sensor, such as a RAW domain image that may be in bayer format. The multi-frame continuous original image is an original image of two or more frames continuously shot for the same scene. The original images of multiple frames may be those taken in real time or those read from a storage device.
in some embodiments, a reference image is selected from a plurality of original images, and images other than the reference image are fused to the reference image in the subsequent registration and fusion process. In general, since the reference frame image is a basic image of image processing, it is necessary to ensure that information in the reference frame image is comprehensive. For example, a picture having the most high frequency information may be selected as the reference frame picture.
in some embodiments, the transformation angle of the camera generally represents the motion state between different images, and therefore, in this step, the difference between the camera state of the frame image to be registered at the time of shooting and the camera state of the reference frame image at the time of shooting is determined first, so as to determine the registration mode subsequently. For example, camera motion parameters provided by a gyroscope module of the camera may be employed to determine the motion state of the camera.
in some embodiments, since the motion state of the camera generally has a large influence on the inter-frame motion state of the image, in this step, the manner of registering different frame images to be registered to the reference frame image is determined by distinguishing the rotation angle of the camera. In the determination process, the registration mode is determined through a predetermined threshold value and the camera rotation angle corresponding to the frame image with registration and the reference frame image. For example, when the camera rotation angle is less than a predetermined threshold, a translation transformation model is used for registration. And when the rotation angle of the camera is larger than a predetermined threshold value, adopting a homography change model for registration. The predetermined threshold is an empirically determined threshold, and may be set to 0 ° to 10 °, for example. Through registration, a registration matrix can be obtained, and the image of the frame to be registered and the registration matrix are subjected to convolution operation to obtain an aligned frame.
And 500, fusing the alignment frame image and the reference frame image to obtain a processed original image.
In some embodiments, the features of the alignment frame image and the features of the reference frame image are already aligned, and in this step, the original image with good quality is obtained by fusing the alignment frame image and the reference frame image.
In the technical scheme provided by this embodiment, a reference frame image is selected from a plurality of frame images, a registration mode of the frame image to be registered is determined according to a rotation angle between the frame image to be registered and the reference frame image, and different frames to be registered are registered by adopting an optimal registration mode. Due to the fact that the most appropriate registration mode is adopted for different frames to be registered, introduction of noise can be avoided, matching accuracy is improved, and images with good fusion effects are obtained.
As an alternative embodiment, as shown in fig. 2, step 400 includes:
in some embodiments, when the rotation angle is small, the captured image may be generally considered as a translational motion relationship approximately, and therefore, in this step, the frame image to be registered is registered to the reference frame image by using a translational transformation method, so that registration can be achieved without performing a large amount of operations, and the calculation speed of registration can be increased.
And step 420, when the rotation angle is not less than a preset threshold value, registering the frame image to be registered to the reference frame image in a homography transformation mode.
In some embodiments, when the rotation angle is large, since the rotation relationship between the frame image to be registered and the reference frame image cannot be ignored, it is necessary to perform homography transformation on the frame image to be registered to the reference frame image, so that the registration can be performed more accurately.
In the embodiment, different registration transformations are performed on different frames to be registered, so that both the calculation speed and the calculation accuracy can be taken into consideration, and thus, a good fusion image can be obtained quickly.
As an alternative embodiment, as shown in fig. 3, step 400 includes:
in some embodiments, as shown in fig. 15, the original image usually includes four channels, that is, four channels of R, Gr, Gb and B, the arrangement of the four channels is arranged as in fig. 15, in this step, information of the four channels is extracted respectively to form four sub-images, that is, the reference frame image forms four reference frame sub-images, and the frame image to be registered forms four frame sub-images to be registered.
in some embodiments, in order to facilitate the subsequent registration process of the image, in this step, the image is preprocessed to enhance the image, so as to highlight the whole or local features in the image, improve the image quality, enrich the image information, and provide a basis for the subsequent registration process. For example, available methods include, but are not limited to, efficient lightweight image enhancement networks, histogram equalization, gamma transformation, etc., depending on image brightness and noise information.
And 450, constructing a reference frame pyramid for the reference frame sub-images, constructing a frame pyramid to be registered for the frame sub-images to be registered, and performing registration based on the reference frame pyramid and the frame pyramid to be registered.
In some embodiments, in the registration process, since the identification of part of the feature points or the feature point matching is associated with the image scale, in this step, a frame pyramid to be registered is constructed for the frame sub-image to be registered, and a reference frame sub-image is constructed for the reference frame sub-image. Those skilled in the art will appreciate that for a single reference frame image, there are four reference frame sub-images corresponding to the four channels, respectively, and thus four reference frame pyramids corresponding to the four channels, respectively, are formed; similarly, for a single frame image to be registered, four frame pyramids to be registered corresponding to the four channels are formed. In the subsequent frame registration process, the pyramid of the frame to be registered under the same channel should be registered to the pyramid of the reference frame, for example, the pyramid corresponding to the R channel of the pyramid of the frame to be registered is adopted to register to the pyramid corresponding to the R channel of the image of the reference frame.
In the embodiment, the reference frame image and the frame image to be registered are preprocessed, the image is enhanced, the overall or local features in the image are highlighted, and a pyramid is constructed, so that the method is favorable for matching more features on multiple scales, and conditions are provided for subsequent registration.
As an alternative embodiment, as shown in fig. 4, step 450 includes:
in some embodiments, the moving directions and distances of the images in the four channels are generally similar, so in this step, the pyramid of the frame to be registered of the four channels is registered to the pyramid of the reference frame, and thus, the registration transformation sub-matrices corresponding to the four channels are solved.
In some embodiments, in order to obtain a more accurate registration transformation matrix, in this step, the registration transformation sub-matrices of the multiple channels are averaged.
In this embodiment, since the motion information in the multiple channels is usually similar, the registration transformation sub-matrices in the multiple channels are averaged to obtain a registration transformation matrix, so as to obtain a relatively accurate registration transformation matrix.
As an alternative embodiment, as shown in fig. 5, step 410 includes:
in some embodiments, when the camera deflection angle is small, the relationship between the frame to be registered and the reference frame is generally an approximately translational relationship, and the scale of the topmost image of the pyramid is extremely small compared with the scale of the bottom image, so in this step, the offset vector is obtained by adopting a template matching manner.
In some embodiments, the offset vector estimated from the highest-layer image is used as an initial value in the next-layer image, and the offset vector is corrected by a finer search optimization method to obtain a more accurate offset vector; and by analogy, obtaining the offset vector on the bottom original resolution map. Those skilled in the art will appreciate that when solving for an initial value in a next layer, the ratio of the offset vector to the sampling rate of the previous layer obtains the initial value for the next layer. Since the initial value is an estimated value and is not always accurate, the offset vector needs to be corrected, for example, in the correction process, the image to be registered can be moved on the reference frame image, and the determined offset vector is the offset vector of the next layer when the number of pixel values of the overlapped area of the two images is 0 at most.
In this embodiment, the offset vector of the top image is obtained by template matching at the top of the pyramid, and then, the offset vector between the frame image to be registered and the reference frame image can be obtained by calculation and correction layer by layer, so that the registration matrix of the image with a small rotation angle can be rapidly calculated. In the solving process of the matrix, the following equation is satisfied between the sub-image of the frame to be registered and the sub-image of the reference frame:
in the above formula, x 'and y' are pixel coordinates in the sub-image of the frame to be registered, x and y are pixel coordinates in the sub-image of the reference frame, and Tx and Ty are offset vectors of the sub-image of the frame to be registered relative to the sub-image of the reference frame.
As an alternative implementation, as shown in fig. 6, step 420 includes:
in some embodiments, when the deflection angle between the frame image to be registered and the reference frame image is large, the transformation modes of different parts of the image may be different, so in this step, each layer of image of the pyramid is segmented, and in the segmented image, in the subsequent solving process, the matching feature point with the best matching quality is selected from each block of image to calculate the homography matrix, so that the registration result is more accurate. For example, the segmentation may be performed by a semantic segmentation blocking strategy, in which an image is segmented into a plurality of regions roughly labeled as a static background (buildings, street scenes, flowers, sky, etc.) and a moving foreground (people, cars, living things, etc.) by using a lightweight semantic segmentation depth learning model, and the number and size of specific blocks are determined according to the labeling information and the image resolution, and the blocks are matrix blocks of appropriate size including labels, and generally have different sizes; the other is a fixed number and size of blocking strategy, which equally divides the image into M blocks, where M can be set to 9, but not limited to.
in some embodiments, the identification of feature points is performed in each layer of frame image blocks to be registered and reference frame image blocks, for example, the feature points may be fast feature points and corresponding feature descriptor ORB feature points. The identification of the feature points is usually related to the image scale, so that the identification of the feature points in each layer of image is beneficial to identifying all the feature points.
in some embodiments, since the identification of the feature points is related to the scale, the feature points in each layer of image are mapped to the pyramid bottom layer image, so as to obtain all the feature points in the reference frame sub-image and the frame sub-image to be registered.
Step 424, matching all feature points of the reference frame sub image blocks according to all feature points of the frame sub image blocks to be registered, so as to determine matching point pairs in the frame sub image blocks to be registered and the corresponding reference frame sub image blocks;
in some embodiments, since the image is partitioned, the motion manner of different blocks may be different during the mapping process, and in order to ensure matching of different blocks, feature points of the frame partition image block to be registered are matched with feature points in the corresponding reference frame image block, so that each pair of the corresponding frame partition image block to be registered and the reference frame partition image block has at least one matching point pair. In the matching process, hamming distance can be used for feature point matching.
In some embodiments, when more than one matching point pair exists between a pair of corresponding frame partial images to be registered and reference frame partial image blocks, if the quality of some matching point pairs is poor, the matching accuracy of the whole image is affected, and therefore in this step, the matching point pair with the best matching quality is selected to determine the homography matrix, so that an accurate homography matrix can be obtained. In the selecting process, for example, the matching point pairs may be sorted according to the matching quality, and the top 30% or 50% of the matching point pairs may be selected. In the selection process, each sub-image block is selected independently, and the selected matching point pair is the matching point pair with the best matching quality, so that each sub-image block can obtain good matching quality. In the selection process, for example, the following method may be adopted to select, and the implementation of screening the best K matching point pairs from each block of image may be based on the marking information of image semantic segmentation, and the method of making difference between block image frames is used to screen out incorrect feature matching information caused by excessive scene motion.
In the embodiment, a gold tower mode is adopted, so that the condition that the feature points are not selected due to the influence of the scale is avoided, and the feature points with the best matching quality are obtained by screening the matching quality of the feature points, so that each region of the frame image to be registered can be well matched with the corresponding region of the reference frame image. In the solving process, the frame sub-image to be registered and the reference frame sub-image have the following relationship:
in the formula, x 'and y' are pixel coordinates in the frame sub-image to be registered, and x and y are pixel coordinates in the reference frame sub-image; the matrix H is a homography matrix obtained by final solution, wherein H1-H9 are homography matrix elements obtained by solution.
As an alternative embodiment, as shown in fig. 7, step 500 includes:
in some embodiments, a motion pixel generally refers to a pixel corresponding to a location where the difference between an aligned frame and a reference frame is greater than a predetermined pixel value. In the aligned frame image, the pixels corresponding to the reference frame image should generally correspond, and the pixels where motion occurs locally will generate a difference value larger than a predetermined pixel value.
in some embodiments, erosion and dilation can more clearly distinguish and process regions of moving pixels, thereby enabling more accurate pixel determination of moving regions.
In some embodiments, since the motion pixels are different pixels in the alignment frame and image and the reference frame image, they, if fused, can result in the introduction of noise into the reference frame image. In this step, the other pixels in the frame image and the reference frame image are fused, so that the picture quality can be improved.
In the embodiment, the method obtains the area where the motion pixel is located through processing the motion pixel, and avoids noise caused by fusion of the motion pixel in the fusion process, so that high picture quality can be obtained.
As an optional implementation manner, determining, according to multiple frames of the original images, a reference frame image and a frame image to be registered includes:
determining a reference frame image according to focusing information corresponding to a plurality of frames of original images, and taking the original images except the reference frame as frame images to be registered; in some embodiments, the focusing information may be a parameter indicating whether focusing is clear, and in the selecting process, an image with the clearest focusing is selected as the reference frame. Or,
and determining a reference frame image according to the high-frequency information of the original images of the plurality of frames, and taking the original images except the reference frame as frame images to be registered. In some embodiments, since focusing is usually performed first before shooting, generally, in a continuous image, an image arranged earlier usually has more high-frequency information, and in order to increase the speed of selecting the reference frame, a selection can be directly performed from the earlier image, for example, from the first 1/2 image in the continuous image of multiple frames.
An embodiment of the present invention further provides an original image processing apparatus, as shown in fig. 8, the apparatus includes:
the image acquisition module is used for acquiring multiple continuous original images;
in some embodiments, the RAW image refers to a RAW image obtained using an image sensor, such as a RAW domain image that may be in bayer format. The multi-frame continuous original image is an original image of two or more frames continuously shot for the same scene. The original images of multiple frames may be those taken in real time or those read from a storage device.
The type determining module is used for determining a reference frame image and a frame image to be registered according to a plurality of frames of the original images;
in some embodiments, a reference image is selected from a plurality of original images, and images other than the reference image are fused to the reference image in the subsequent registration and fusion process. In general, since the reference frame image is a basic image of image processing, it is necessary to ensure that information in the reference frame image is comprehensive. For example, a picture having the most high frequency information may be selected as the reference frame picture.
An angle determining module, configured to determine, according to the reference frame image and the frame image to be registered, a rotation angle of a camera of the frame image to be registered relative to a camera of the reference frame image;
in some embodiments, the transformation angle of the camera generally represents the motion state between different images, and therefore, in this step, the difference between the camera state of the frame image to be registered at the time of shooting and the camera state of the reference frame image at the time of shooting is determined first, so as to determine the registration mode subsequently. For example, camera motion parameters provided by a gyroscope module of the camera may be employed to determine the motion state of the camera.
The image registration module is used for determining a registration mode of the frame image to be registered and the reference frame image according to the rotation angle, and performing registration according to the registration mode to form an alignment frame image corresponding to the frame image to be registered;
in some embodiments, since the motion state of the camera generally has a large influence on the inter-frame motion state of the image, in this step, the manner of registering different frame images to be registered to the reference frame image is determined by distinguishing the rotation angle of the camera. In the determination process, the registration mode is determined through a predetermined threshold value and the camera rotation angle corresponding to the frame image with registration and the reference frame image. For example, when the camera rotation angle is less than a predetermined threshold, a translation transformation model is used for registration. And when the rotation angle of the camera is larger than a predetermined threshold value, adopting a homography change model for registration. The predetermined threshold is an empirically determined threshold, and may be set to 0 ° to 10 °, for example. Through registration, a registration matrix can be obtained, and the image of the frame to be registered and the registration matrix are subjected to convolution operation to obtain an aligned frame.
And the image fusion module is used for fusing the alignment frame image and the reference frame image to obtain a processed original image.
In some embodiments, the features of the alignment frame image and the features of the reference frame image are already aligned, and in this step, the original image with good quality is obtained by fusing the alignment frame image and the reference frame image.
In the technical scheme provided by this embodiment, a reference frame image is selected from a plurality of frame images, a registration mode of the frame image to be registered is determined according to a rotation angle between the frame image to be registered and the reference frame image, and different frames to be registered are registered by adopting an optimal registration mode. Due to the fact that the most appropriate registration mode is adopted for different frames to be registered, introduction of noise can be avoided, matching accuracy is improved, and images with good fusion effects are obtained.
As an alternative embodiment, as shown in fig. 9, the image registration module includes:
the translation registration submodule is used for registering the frame image to be registered to the reference frame image in a translation transformation mode when the rotation angle is smaller than a preset threshold value;
in some embodiments, when the rotation angle is small, the captured image may be generally considered as a translational motion relationship approximately, and therefore, in this step, the frame image to be registered is registered to the reference frame image by using a translational transformation method, so that registration can be achieved without performing a large amount of operations, and the calculation speed of registration can be increased.
And the homography registration submodule is used for registering the frame image to be registered to the reference frame image in a homography transformation mode when the rotation angle is not less than a preset threshold value.
In some embodiments, when the rotation angle is large, since the rotation relationship between the frame image to be registered and the reference frame image cannot be ignored, it is necessary to perform homography transformation on the frame image to be registered to the reference frame image, so that the registration can be performed more accurately.
In the embodiment, different registration transformations are performed on different frames to be registered, so that both the calculation speed and the calculation accuracy can be taken into consideration, and thus, a good fusion image can be obtained quickly.
As an alternative embodiment, as shown in fig. 10, the image registration module includes:
the image splitting submodule is used for carrying out interlaced alternate value taking on the pixels of the original image so as to split the reference frame image into reference frame sub-images corresponding to four channels and split the frame image to be registered into frame sub-images to be registered corresponding to the four channels;
in some embodiments, as shown in fig. 15, the original image usually includes four channels, that is, four channels of R, Gr, Gb and B, the arrangement of the four channels is arranged as in fig. 15, in this step, information of the four channels is extracted respectively to form four sub-images, that is, the reference frame image forms four reference frame sub-images, and the frame image to be registered forms four frame sub-images to be registered.
The preprocessing submodule is used for preprocessing the reference frame sub-image and the frame sub-image to be registered so as to enhance the reference frame sub-image and the frame sub-image to be registered;
in some embodiments, in order to facilitate the subsequent registration process of the image, in this step, the image is preprocessed to enhance the image, so as to highlight the whole or local features in the image, improve the image quality, enrich the image information, and provide a basis for the subsequent registration process. For example, available methods include, but are not limited to, efficient lightweight image enhancement networks, histogram equalization, gamma transformation, etc., depending on image brightness and noise information.
And the image registration submodule is used for constructing a reference frame pyramid for the reference frame sub-images, constructing a frame pyramid to be registered for the frame sub-images to be registered, and performing registration based on the reference frame pyramid and the frame pyramid to be registered.
In some embodiments, in the registration process, since the identification of part of the feature points or the feature point matching is associated with the image scale, in this step, a frame pyramid to be registered is constructed for the frame sub-image to be registered, and a reference frame sub-image is constructed for the reference frame sub-image. Those skilled in the art will appreciate that for a single reference frame image, there are four reference frame sub-images corresponding to the four channels, respectively, and thus four reference frame pyramids corresponding to the four channels, respectively, are formed; similarly, for a single frame image to be registered, four frame pyramids to be registered corresponding to the four channels are formed. In the subsequent frame registration process, the pyramid of the frame to be registered under the same channel should be registered to the pyramid of the reference frame, for example, the pyramid corresponding to the R channel of the pyramid of the frame to be registered is adopted to register to the pyramid corresponding to the R channel of the image of the reference frame.
In the embodiment, the reference frame image and the frame image to be registered are preprocessed, the image is enhanced, the overall or local features in the image are highlighted, and a pyramid is constructed, so that the method is favorable for matching more features on multiple scales, and conditions are provided for subsequent registration.
As an alternative embodiment, as shown in fig. 11, the image registration module includes:
the submatrix determining submodule is used for registering the pyramid of the frame to be registered corresponding to the four channels to the pyramid of the reference frame so as to determine a registration transformation submatrix corresponding to the four channels;
in some embodiments, the moving directions and distances of the images in the four channels are generally similar, so in this step, the pyramid of the frame to be registered of the four channels is registered to the pyramid of the reference frame, and thus, the registration transformation sub-matrices corresponding to the four channels are solved.
And the transformation matrix determining submodule is used for averaging the registration transformation submatrices corresponding to the four channels to obtain a registration transformation matrix.
In some embodiments, in order to obtain a more accurate registration transformation matrix, in this step, the registration transformation sub-matrices of the multiple channels are averaged.
In this embodiment, since the motion information in the multiple channels is usually similar, the registration transformation sub-matrices in the multiple channels are averaged to obtain a registration transformation matrix, so as to obtain a relatively accurate registration transformation matrix.
As an alternative embodiment, as shown in fig. 12, the translational registration sub-module includes:
the top-level offset unit is used for acquiring offset vectors by adopting a template matching device for the Gaussian pyramid top-level image of the frame sub-image to be registered and the Gaussian pyramid top-level image of the reference frame sub-image;
in some embodiments, when the camera deflection angle is small, the relationship between the frame to be registered and the reference frame is generally an approximately translational relationship, and the scale of the topmost image of the pyramid is extremely small compared with the scale of the bottom image, so in this step, the offset vector is obtained by adopting a template matching manner.
And the offset correction unit is used for solving and correcting the frame pyramid to be registered and the reference frame pyramid from the top layer to the bottom layer by layer according to the offset vector so as to obtain the offset vector between the frame sub-image to be registered and the reference frame sub-image.
In some embodiments, the offset vector estimated from the highest-layer image is used as an initial value in the next-layer image, and the offset vector is corrected by a finer search optimization method to obtain a more accurate offset vector; and by analogy, obtaining the offset vector on the bottom original resolution map. Those skilled in the art will appreciate that when solving for an initial value in a next layer, the ratio of the offset vector to the sampling rate of the previous layer obtains the initial value for the next layer. Since the initial value is an estimated value and is not always accurate, the offset vector needs to be corrected, for example, in the correction process, the image to be registered can be moved on the reference frame image, and the determined offset vector is the offset vector of the next layer when the number of pixel values of the overlapped area of the two images is 0 at most.
In this embodiment, the offset vector of the top image is obtained by template matching at the top of the pyramid, and then, the offset vector between the frame image to be registered and the reference frame image can be obtained by calculation and correction layer by layer, so that the registration matrix of the image with a small rotation angle can be rapidly calculated. In the solving process of the matrix, the following equation is satisfied between the sub-image of the frame to be registered and the sub-image of the reference frame:
in the above formula, x 'and y' are pixel coordinates in the sub-image of the frame to be registered, x and y are pixel coordinates in the sub-image of the reference frame, and Tx and Ty are offset vectors of the sub-image of the frame to be registered relative to the sub-image of the reference frame.
As an alternative embodiment, as shown in fig. 13, the homography registration sub-module includes:
the image segmentation unit is used for segmenting each layer of images of the frame pyramid to be registered and the reference frame pyramid to obtain frame sub image blocks to be registered and reference frame sub image blocks;
in some embodiments, when the deflection angle between the frame image to be registered and the reference frame image is large, the transformation modes of different parts of the image may be different, so in this step, each layer of image of the pyramid is segmented, and in the segmented image, in the subsequent solving process, the matching feature point with the best matching quality is selected from each block of image to calculate the homography matrix, so that the registration result is more accurate. For example, the segmentation may be performed by a semantic segmentation blocking strategy, in which an image is segmented into a plurality of regions roughly labeled as a static background (buildings, street scenes, flowers, sky, etc.) and a moving foreground (people, cars, living things, etc.) by using a lightweight semantic segmentation depth learning model, and the number and size of specific blocks are determined according to the labeling information and the image resolution, and the blocks are matrix blocks of appropriate size including labels, and generally have different sizes; the other is a fixed number and size of blocking strategy, which equally divides the image into M blocks, where M can be set to 9, but not limited to.
The pyramid characteristic point unit is used for determining characteristic points of the frame sub image blocks to be registered and the reference frame sub image blocks in each layer of images;
in some embodiments, the identification of feature points is performed in each layer of frame image blocks to be registered and reference frame image blocks, for example, the feature points may be fast feature points and corresponding feature descriptor ORB feature points. The identification of the feature points is usually related to the image scale, so that the identification of the feature points in each layer of image is beneficial to identifying all the feature points.
The bottom layer feature point unit is used for determining all feature points of each frame sub-image block to be registered of the frame sub-image to be registered and all feature points of each frame sub-image block to be registered of the reference frame sub-image according to the feature points in each layer of image;
in some embodiments, since the identification of the feature points is related to the scale, the feature points in each layer of image are mapped to the pyramid bottom layer image, so as to obtain all the feature points in the reference frame sub-image and the frame sub-image to be registered.
The matching point pair unit is used for matching all the feature points of the reference frame sub image blocks according to all the feature points of the frame sub image blocks to be registered so as to determine matching point pairs in the frame sub image blocks to be registered and the corresponding reference frame sub image blocks;
in some embodiments, since the image is partitioned, the motion manner of different blocks may be different during the mapping process, and in order to ensure matching of different blocks, feature points of the frame partition image block to be registered are matched with feature points in the corresponding reference frame image block, so that each pair of the corresponding frame partition image block to be registered and the reference frame partition image block has at least one matching point pair. In the matching process, hamming distance can be used for feature point matching.
And the matrix determining unit is used for selecting a preset proportion of matching point pairs with the best quality from the frame sub image blocks to be registered and the corresponding reference frame sub image blocks to determine the homography matrix.
In some embodiments, when more than one matching point pair exists between a pair of corresponding frame partial images to be registered and reference frame partial image blocks, if the quality of some matching point pairs is poor, the matching accuracy of the whole image is affected, and therefore in this step, the matching point pair with the best matching quality is selected to determine the homography matrix, so that an accurate homography matrix can be obtained. In the selecting process, for example, the matching point pairs may be sorted according to the matching quality, and the top 30% or 50% of the matching point pairs may be selected. In the selection process, each sub-image block is selected independently, and the selected matching point pair is the matching point pair with the best matching quality, so that each sub-image block can obtain good matching quality. In the selection process, for example, the following method may be adopted to select, and the implementation of screening the best K matching point pairs from each block of image may be based on the marking information of image semantic segmentation, and the method of making difference between block image frames is used to screen out incorrect feature matching information caused by excessive scene motion.
In the embodiment, a gold tower mode is adopted, so that the condition that the feature points are not selected due to the influence of the scale is avoided, and the feature points with the best matching quality are obtained by screening the matching quality of the feature points, so that each region of the frame image to be registered can be well matched with the corresponding region of the reference frame image. In the solving process, the frame sub-image to be registered and the reference frame sub-image have the following relationship:
in the formula, x 'and y' are pixel coordinates in the frame sub-image to be registered, and x and y are pixel coordinates in the reference frame sub-image; the matrix H is a homography matrix obtained by final solution, wherein H1-H9 are homography matrix elements obtained by solution.
As an alternative embodiment, as shown in fig. 14, the image fusion module includes:
a moving pixel determination module for determining moving pixels in the aligned frame images according to a noise model;
in some embodiments, a motion pixel generally refers to a pixel corresponding to a location where the difference between an aligned frame and a reference frame is greater than a predetermined pixel value. In the aligned frame image, the pixels corresponding to the reference frame image should generally correspond, and the pixels where motion occurs locally will generate a difference value larger than a predetermined pixel value.
The reference frame image processing module is used for performing morphological erosion and expansion on a region corresponding to the motion pixel in the reference frame image;
in some embodiments, erosion and dilation can more clearly distinguish and process regions of moving pixels, thereby enabling more accurate pixel determination of moving regions.
And the accumulation fusion module is used for carrying out accumulation fusion on the areas except the motion pixels in the alignment frame image and the reference frame image.
In some embodiments, since the motion pixels are different pixels in the alignment frame and image and the reference frame image, they, if fused, can result in the introduction of noise into the reference frame image. In this step, the other pixels in the frame image and the reference frame image are fused, so that the picture quality can be improved.
In the embodiment, the method obtains the area where the motion pixel is located through processing the motion pixel, and avoids noise caused by fusion of the motion pixel in the fusion process, so that high picture quality can be obtained.
As an optional implementation, the type determining module includes:
the first type determining submodule is used for determining a reference frame image according to focusing information corresponding to a plurality of frames of original images, and taking the original images except the reference frame as frame images to be registered; in some embodiments, the focusing information may be a parameter indicating whether focusing is clear, and in the selecting process, an image with the clearest focusing is selected as the reference frame. Or,
and the second type determining submodule is used for determining a reference frame image according to the high-frequency information of the original images of the plurality of frames, and taking the original images except the reference frame as frame images to be registered. In some embodiments, since focusing is usually performed first before shooting, generally, in a continuous image, an image arranged earlier usually has more high-frequency information, and in order to increase the speed of selecting the reference frame, a selection can be directly performed from the earlier image, for example, from the first 1/2 image in the continuous image of multiple frames.
The present invention provides an apparatus, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the methods described above.
The present invention provides a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement a method as in any one of the above.
It will be understood by those skilled in the art that all or part of the processes of the embodiments of the methods described above may be implemented by a computer program, which may be stored in a computer-readable storage medium, and when executed, may include the processes of the embodiments of the methods described above. The storage medium may be a magnetic disk, an optical disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), or the like.
The above description is only for the specific embodiment of the present invention, but the scope of the present invention is not limited thereto, and any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope of the present invention are included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (18)
1. A method of raw image processing, the method comprising:
acquiring multiple continuous original images;
determining a reference frame image and a frame image to be registered according to a plurality of frames of original images;
determining the rotation angle of a camera of the frame image to be registered relative to the camera of the reference frame image according to the reference frame image and the frame image to be registered;
determining a registration mode of the frame image to be registered and the reference frame image according to the rotation angle, and performing registration according to the registration mode to form an alignment frame image corresponding to the frame image to be registered;
and fusing the alignment frame image and the reference frame image to obtain a processed original image.
2. The method according to claim 1, wherein determining the registration manner of the frame image to be registered and the reference frame image according to the rotation angle comprises:
when the rotation angle is smaller than a preset threshold value, registering the frame image to be registered to the reference frame image in a translation transformation mode;
and when the rotation angle is not less than a preset threshold value, registering the frame image to be registered to the reference frame image in a homography transformation mode.
3. The method according to claim 2, wherein a registration manner of the frame image to be registered and the reference frame image is determined according to the rotation angle, and the registering according to the registration manner comprises:
carrying out interlaced alternate dereferencing on pixels of an original image to split the reference frame image into reference frame sub-images corresponding to four channels, and splitting the frame image to be registered into frame sub-images to be registered corresponding to the four channels;
preprocessing the reference frame sub-image and the frame sub-image to be registered to realize the enhancement of the reference frame sub-image and the frame sub-image to be registered;
and constructing a reference frame pyramid for the reference frame sub-images, constructing a frame pyramid to be registered for the frame sub-images to be registered, and performing registration based on the reference frame pyramid and the frame pyramid to be registered.
4. The method of claim 3, wherein registering based on the reference frame pyramid and a frame pyramid to be registered comprises:
registering the frame pyramid to be registered corresponding to the four channels to the reference frame pyramid so as to determine registration transformation sub-matrixes corresponding to the four channels;
and averaging the registration transformation sub-matrixes corresponding to the four channels to obtain a registration transformation matrix.
5. The method of claim 3, wherein registering based on the reference frame pyramid and a frame pyramid to be registered when the rotation angle is less than a predetermined threshold comprises:
acquiring an offset vector by adopting a template matching method for the topmost pyramid image of the frame to be registered and the topmost pyramid image of the reference frame;
and solving and correcting layer by layer from the top layer of the pyramid of the frame to be registered and the bottom layer of the pyramid of the reference frame according to the offset vector so as to obtain the offset vector between the sub-image of the frame to be registered and the sub-image of the reference frame.
6. The method of claim 3, wherein registering based on the reference frame pyramid and a frame pyramid to be registered when the rotation angle is not less than a predetermined threshold comprises:
dividing each layer of images of the frame pyramid to be registered and the reference frame pyramid to obtain frame sub image blocks to be registered and reference frame sub image blocks;
determining the characteristic points of the frame sub image blocks to be registered and the reference frame sub image blocks in each layer of images;
determining all feature points of each frame sub-image block to be registered of the frame sub-image to be registered and all feature points of each frame sub-image block to be registered of the reference frame sub-image according to the feature points in each layer of image;
matching all the feature points of the reference frame sub image blocks according to all the feature points of the frame sub image blocks to be registered so as to determine matching point pairs in the frame sub image blocks to be registered and the corresponding reference frame sub image blocks;
and selecting a preset proportion of matching point pairs with the best quality from the frame sub image blocks to be registered and the corresponding reference frame sub image blocks to determine the homography matrix.
7. The method of any of claims 1-6, wherein fusing the aligned frame image with the reference frame image comprises:
determining motion pixels in the aligned frame images according to a noise model;
performing morphological erosion and expansion on a region corresponding to the motion pixel in the reference frame image;
and performing accumulation fusion on the regions except the motion pixels in the aligned frame image and the reference frame image.
8. The method according to any one of claims 1 to 6, wherein determining a reference frame image and a frame image to be registered according to a plurality of frames of the original images comprises:
determining a reference frame image according to focusing information corresponding to a plurality of frames of original images, and taking the original images except the reference frame as frame images to be registered; or,
and determining a reference frame image according to the high-frequency information of the original images of the plurality of frames, and taking the original images except the reference frame as frame images to be registered.
9. An original image processing apparatus, characterized in that the apparatus comprises:
the image acquisition module is used for acquiring multiple continuous original images;
the type determining module is used for determining a reference frame image and a frame image to be registered according to a plurality of frames of the original images;
an angle determining module, configured to determine, according to the reference frame image and the frame image to be registered, a rotation angle of a camera of the frame image to be registered relative to a camera of the reference frame image;
the image registration module is used for determining a registration mode of the frame image to be registered and the reference frame image according to the rotation angle, and performing registration according to the registration mode to form an alignment frame image corresponding to the frame image to be registered;
and the image fusion module is used for fusing the alignment frame image and the reference frame image to obtain a processed original image.
10. The apparatus of claim 9, wherein the image registration module comprises:
the translation registration submodule is used for registering the frame image to be registered to the reference frame image in a translation transformation mode when the rotation angle is smaller than a preset threshold value;
and the homography registration submodule is used for registering the frame image to be registered to the reference frame image in a homography transformation mode when the rotation angle is not less than a preset threshold value.
11. The apparatus of claim 10, wherein the image registration module comprises:
the image splitting submodule is used for carrying out interlaced alternate value taking on the pixels of the original image so as to split the reference frame image into reference frame sub-images corresponding to four channels and split the frame image to be registered into frame sub-images to be registered corresponding to the four channels;
the preprocessing submodule is used for preprocessing the reference frame sub-image and the frame sub-image to be registered so as to enhance the reference frame sub-image and the frame sub-image to be registered;
and the image registration submodule is used for constructing a reference frame pyramid for the reference frame sub-images, constructing a frame pyramid to be registered for the frame sub-images to be registered, and performing registration based on the reference frame pyramid and the frame pyramid to be registered.
12. The apparatus of claim 11, wherein the image registration module comprises:
the submatrix determining submodule is used for registering the pyramid of the frame to be registered corresponding to the four channels to the pyramid of the reference frame so as to determine a registration transformation submatrix corresponding to the four channels;
and the transformation matrix determining submodule is used for averaging the registration transformation submatrices corresponding to the four channels to obtain a registration transformation matrix.
13. The apparatus of claim 11, wherein translating the registration sub-module comprises:
the top-level offset unit is used for acquiring offset vectors by adopting a template matching device for the Gaussian pyramid top-level image of the frame sub-image to be registered and the Gaussian pyramid top-level image of the reference frame sub-image;
and the offset correction unit is used for solving and correcting the frame pyramid to be registered and the reference frame pyramid from the top layer to the bottom layer by layer according to the offset vector so as to obtain the offset vector between the frame sub-image to be registered and the reference frame sub-image.
14. The apparatus of claim 11, wherein the homography registration sub-module comprises:
the image segmentation unit is used for segmenting each layer of images of the frame pyramid to be registered and the reference frame pyramid to obtain frame sub image blocks to be registered and reference frame sub image blocks;
the pyramid characteristic point unit is used for determining characteristic points of the frame sub image blocks to be registered and the reference frame sub image blocks in each layer of images;
the bottom layer feature point unit is used for determining all feature points of each frame sub-image block to be registered of the frame sub-image to be registered and all feature points of each frame sub-image block to be registered of the reference frame sub-image according to the feature points in each layer of image;
the matching point pair unit is used for matching all the feature points of the reference frame sub image blocks according to all the feature points of the frame sub image blocks to be registered so as to determine matching point pairs in the frame sub image blocks to be registered and the corresponding reference frame sub image blocks;
and the matrix determining unit is used for selecting a preset proportion of matching point pairs with the best quality from the frame sub image blocks to be registered and the corresponding reference frame sub image blocks to determine the homography matrix.
15. The apparatus according to any one of claims 9-14, wherein the image fusion module comprises:
a moving pixel determination module for determining moving pixels in the aligned frame images according to a noise model;
the reference frame image processing module is used for performing morphological erosion and expansion on a region corresponding to the motion pixel in the reference frame image;
and the accumulation fusion module is used for carrying out accumulation fusion on the areas except the motion pixels in the alignment frame image and the reference frame image.
16. The apparatus of any of claims 9-14, wherein the type determination module comprises:
the first type determining submodule is used for determining a reference frame image according to focusing information corresponding to a plurality of frames of original images, and taking the original images except the reference frame as frame images to be registered; or,
and the second type determining submodule is used for determining a reference frame image according to the high-frequency information of the original images of the plurality of frames, and taking the original images except the reference frame as frame images to be registered.
17. An apparatus, characterized in that the apparatus comprises:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 9.
18. A computer readable storage medium, wherein the computer readable storage medium stores computer instructions which, when executed by a processor, implement the method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110716617.3A CN113313707B (en) | 2021-06-25 | 2021-06-25 | Original image processing method, device, equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110716617.3A CN113313707B (en) | 2021-06-25 | 2021-06-25 | Original image processing method, device, equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113313707A true CN113313707A (en) | 2021-08-27 |
| CN113313707B CN113313707B (en) | 2024-07-26 |
Family
ID=77380531
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110716617.3A Active CN113313707B (en) | 2021-06-25 | 2021-06-25 | Original image processing method, device, equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113313707B (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023245383A1 (en) * | 2022-06-20 | 2023-12-28 | 北京小米移动软件有限公司 | Method for aligning multiple image frames, apparatus for aligning multiple image frames, and storage medium |
Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100079667A1 (en) * | 2008-08-21 | 2010-04-01 | Vestel Elektronik Sanayi Ve Ticaret A.S. | Method and apparatus for increasing the frame rate of a video signal |
| CN104978743A (en) * | 2015-06-12 | 2015-10-14 | 中国电子科技集团公司第三十八研究所 | Multi-core parallel SAR (synthetic aperture radar) image change information real-time extraction method and extraction device |
| JP2016013610A (en) * | 2014-07-03 | 2016-01-28 | セイコーエプソン株式会社 | Robot, and control method |
| CN108053432A (en) * | 2017-11-14 | 2018-05-18 | 华南理工大学 | The method for registering of indoor sparse cloud scene based on local I CP |
| CN108305277A (en) * | 2017-12-26 | 2018-07-20 | 中国航天电子技术研究院 | A kind of heterologous image matching method based on straightway |
| CN110874864A (en) * | 2019-10-25 | 2020-03-10 | 深圳奥比中光科技有限公司 | Method, device, electronic equipment and system for obtaining three-dimensional model of object |
| CN113643333A (en) * | 2021-08-25 | 2021-11-12 | Oppo广东移动通信有限公司 | Image registration method and device, electronic equipment and computer-readable storage medium |
| CN115471528A (en) * | 2022-07-28 | 2022-12-13 | 上海奕瑞光电子科技股份有限公司 | Registration method and device for upper and lower layer images of dual-energy detector, storage medium and terminal |
-
2021
- 2021-06-25 CN CN202110716617.3A patent/CN113313707B/en active Active
Patent Citations (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20100079667A1 (en) * | 2008-08-21 | 2010-04-01 | Vestel Elektronik Sanayi Ve Ticaret A.S. | Method and apparatus for increasing the frame rate of a video signal |
| JP2016013610A (en) * | 2014-07-03 | 2016-01-28 | セイコーエプソン株式会社 | Robot, and control method |
| CN104978743A (en) * | 2015-06-12 | 2015-10-14 | 中国电子科技集团公司第三十八研究所 | Multi-core parallel SAR (synthetic aperture radar) image change information real-time extraction method and extraction device |
| CN108053432A (en) * | 2017-11-14 | 2018-05-18 | 华南理工大学 | The method for registering of indoor sparse cloud scene based on local I CP |
| CN108305277A (en) * | 2017-12-26 | 2018-07-20 | 中国航天电子技术研究院 | A kind of heterologous image matching method based on straightway |
| CN110874864A (en) * | 2019-10-25 | 2020-03-10 | 深圳奥比中光科技有限公司 | Method, device, electronic equipment and system for obtaining three-dimensional model of object |
| CN113643333A (en) * | 2021-08-25 | 2021-11-12 | Oppo广东移动通信有限公司 | Image registration method and device, electronic equipment and computer-readable storage medium |
| CN115471528A (en) * | 2022-07-28 | 2022-12-13 | 上海奕瑞光电子科技股份有限公司 | Registration method and device for upper and lower layer images of dual-energy detector, storage medium and terminal |
Non-Patent Citations (2)
| Title |
|---|
| 李莉: "图像配准算法研究", 《中国博士学位论文全文数据库（电子期刊）》, pages 31 - 59 * |
| 谢妤婵: "直线特征提取与目标特征自动配准", 《中国博士学位论文全文数据库（电子期刊）》, pages 53 - 69 * |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023245383A1 (en) * | 2022-06-20 | 2023-12-28 | 北京小米移动软件有限公司 | Method for aligning multiple image frames, apparatus for aligning multiple image frames, and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113313707B (en) | 2024-07-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN109785291B (en) | Lane line self-adaptive detection method | |
| CN109410207B (en) | NCC (non-return control) feature-based unmanned aerial vehicle line inspection image transmission line detection method | |
| US9070042B2 (en) | Image processing apparatus, image processing method, and program thereof | |
| US8385595B2 (en) | Motion detection method, apparatus and system | |
| CN110930301B (en) | Image processing method, device, storage medium and electronic equipment | |
| CN108229475B (en) | Vehicle tracking method, system, computer device and readable storage medium | |
| US9202263B2 (en) | System and method for spatio video image enhancement | |
| CN106934806B (en) | It is a kind of based on text structure without with reference to figure fuzzy region dividing method out of focus | |
| CN107689050B (en) | Depth image up-sampling method based on color image edge guide | |
| JP2010045613A (en) | Image identifying method and imaging device | |
| CN105931213B (en) | The method that high dynamic range video based on edge detection and frame difference method removes ghost | |
| CN109376641B (en) | Moving vehicle detection method based on unmanned aerial vehicle aerial video | |
| CN112215925A (en) | Self-adaptive follow-up tracking multi-camera video splicing method for coal mining machine | |
| JP2018124890A (en) | Image processing apparatus, image processing method, and image processing program | |
| CN111738211B (en) | PTZ camera moving object detection and recognition method based on dynamic background compensation and deep learning | |
| WO2017081839A1 (en) | Moving body tracking method, moving body tracking device, and program | |
| CN111028263A (en) | Moving object segmentation method and system based on optical flow color clustering | |
| US20110085026A1 (en) | Detection method and detection system of moving object | |
| JP4296617B2 (en) | Image processing apparatus, image processing method, and recording medium | |
| CN113313707B (en) | Original image processing method, device, equipment and readable storage medium | |
| CN112465702B (en) | Synchronous self-adaptive splicing display processing method for multi-channel ultrahigh-definition video | |
| CN113096016A (en) | Low-altitude aerial image splicing method and system | |
| CN108470326B (en) | Image completion method and device | |
| CN111160340A (en) | Moving target detection method and device, storage medium and terminal equipment | |
| US9380285B2 (en) | Stereo image processing method, stereo image processing device and display device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |