CN110930440B

CN110930440B - Image alignment method, device, storage medium and electronic equipment

Info

Publication number: CN110930440B
Application number: CN201911252973.3A
Authority: CN
Inventors: 贾玉虎
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2019-12-09
Filing date: 2019-12-09
Publication date: 2023-06-27
Anticipated expiration: 2039-12-09
Also published as: CN110930440A

Abstract

The application discloses an image alignment method, an image alignment device, a storage medium and electronic equipment, wherein the image alignment method comprises the following steps: acquiring a first image and a second image of the same shooting scene; performing a first superimposed translation of the second image relative to the first image based on a plurality of preset vectors to determine a pair Ji Xiangliang; acquiring a history pair Ji Xiangliang of a previous frame image in the current image stream; determining a common reference direction for the historical alignment vector and the alignment vector, calculating an alignment component of the pair Ji Xiangliang in the common reference direction; and according to the alignment component, controlling the second image to perform second superposition translation relative to the first image so as to align the second image with the first image. The image alignment scheme provided by the embodiment can ensure that the overlapping moving directions of every two adjacent second alignment processes of the image streams are similar or identical, and prevent the image streams from shaking.

Description

Image alignment method, device, storage medium and electronic equipment

Technical Field

The application belongs to the technical field of images, and particularly relates to an image alignment method, an image alignment device, a storage medium and electronic equipment.

Background

Video composed of a composite image of High Dynamic Range (HDR) may provide more Dynamic Range and image detail than normal video. For a composite image with a high dynamic range of each frame in a video, before the composite processing is performed on two frames of images of the same shooting scene, the two frames of images need to be aligned.

In the related art, for a composite image of a high dynamic range of each frame in a video, before the composite processing is performed on two frames of images of the same shooting scene, the two frames of images are aligned based on an affine transformation matrix. The alignment mode may cause different or opposite alignment directions of two adjacent frames of composite images in the video, so that the video is dithered.

Disclosure of Invention

The embodiment of the application provides an image alignment method, an image alignment device, a storage medium and electronic equipment, which can ensure that the superposition moving directions of every two adjacent second alignment processes of image streams are similar or identical and prevent the image streams from shaking.

In a first aspect, an embodiment of the present application provides an image alignment method, including:

acquiring a first image and a second image of the same shooting scene;

performing a first superimposed translation of the second image relative to the first image based on a plurality of preset vectors to determine a pair Ji Xiangliang;

Acquiring a history pair Ji Xiangliang of a previous frame image in the current image stream;

determining a common reference direction for the historical alignment vector and the alignment vector, calculating an alignment component of the pair Ji Xiangliang in the common reference direction;

and according to the alignment component, controlling the second image to perform second superposition translation relative to the first image so as to align the second image with the first image.

In a second aspect, an embodiment of the present application provides an image alignment apparatus, including:

the first acquisition module is used for acquiring a first image and a second image of the same shooting scene;

a first processing module for performing a first superimposed translation of the second image relative to the first image based on a plurality of preset vectors to determine a pair Ji Xiangliang;

a second obtaining module, configured to obtain a history pair Ji Xiangliang of a previous frame image in the current image stream;

a first determining module for determining a common reference direction of the historical alignment vector and the alignment vector, calculating an alignment component of the pair Ji Xiangliang in the common reference direction;

and the second processing module is used for controlling the second image to perform second superposition translation relative to the first image according to the alignment component so as to align the second image with the first image.

In a third aspect, embodiments of the present application provide a storage medium having a computer program stored thereon, which when run on a computer causes the computer to perform an image alignment method as provided in any of the embodiments of the present application.

In a fourth aspect, an electronic device provided in an embodiment of the present application includes a processor and a memory, where the memory has a computer program, and the processor is configured to execute an image alignment method as provided in any embodiment of the present application by calling the computer program.

According to the image alignment scheme provided by the embodiment of the application, after the alignment vector is determined, the electronic device does not perform second overlapping movement on the second image according to the pair Ji Xiangliang, but acquires the historical alignment vector of the last frame of image in the current image stream, determines the common reference direction of the historical alignment vector and the alignment vector, and then performs second overlapping movement on the second image according to the alignment component of the alignment vector in the common reference direction, so that the overlapping movement directions of every two adjacent second alignment processes of the image stream are similar or identical, and the image stream shaking is prevented.

Drawings

The technical solution of the present application and the advantageous effects thereof will be made apparent from the following detailed description of the specific embodiments of the present application with reference to the accompanying drawings.

Fig. 1 is a schematic flow chart of an image alignment method according to an embodiment of the present application.

Fig. 2 is a schematic view of capturing a current image stream according to an embodiment of the present application.

Fig. 3 is a schematic diagram illustrating determination of a common reference direction according to an embodiment of the present application.

Fig. 4 is a second superimposed translational schematic provided in an embodiment of the present application.

Fig. 5 is a second flowchart of an image alignment method according to an embodiment of the present application.

Fig. 6 is a schematic diagram of multiple superimposed translations provided by an embodiment of the present application.

Fig. 7 is a third flow chart of an image alignment method according to an embodiment of the present application.

Fig. 8 is a schematic diagram of image downsampling in the image alignment method according to the embodiment of the present application.

Fig. 9 is a schematic diagram of a first superimposed translation provided in an embodiment of the present application.

Fig. 10 is a fourth flowchart of an image alignment method provided in an embodiment of the present application.

Fig. 11 is a schematic structural view of an image alignment apparatus provided in an embodiment of the present application.

Fig. 12 is a first structural schematic diagram of an electronic device provided in an embodiment of the present application.

Fig. 13 is a schematic diagram of the structure of an image processing circuit provided in the embodiment of the present application.

Fig. 14 is a second structural schematic diagram of the electronic device provided in the embodiment of the present application.

Detailed Description

The following description is based on the illustrated embodiments of the present application and should not be taken as limiting other embodiments not described in detail herein. The term "module" as used herein may be considered as a software object executing on the computing system. The various modules, engines, and services herein may be viewed as implementation objects on the computing system.

The embodiment of the application provides an image alignment method, and an execution subject of the image alignment method may be the image alignment apparatus provided in the embodiment of the application or an electronic device integrated with the image alignment apparatus. The electronic device comprises a camera assembly, and the camera assembly comprises at least one camera. The electronic device may be a smart phone, tablet, palm top (PDA, personal Digital Assistant), etc.

Specific analysis is described below.

Referring to fig. 1, fig. 1 is a first flowchart of an image alignment method according to an embodiment of the present application, where the image alignment method may include the following steps:

101. a first image and a second image of the same shooting scene are acquired.

In some photographing modes, such as continuous photographing, video recording, time-lapse photographing, etc., the electronic device needs to continuously output multiple frames of images to constitute an image stream. Each frame of image in the image stream can be synthesized by adopting a plurality of frames of original images.

The original image corresponding to each frame of image in the image stream can be subjected to image alignment processing according to the scheme of the embodiment of the application.

For example, the electronic device may acquire multiple frames of images to be processed of the same shooting scene through one or more cameras, select any one frame of image (or select the frame of image with the highest definition) from the multiple frames of images to be processed as a reference image, record the reference image as a first image, and record the remaining images (or take one or more frames of images with better definition in the remaining images) as images to be aligned as a second image. In the scheme, each camera is exposed once to obtain one frame of image.

The first image and the second image represent the same shooting scene, and the sizes of the first image and the second image are the same. The shooting scene in the scheme refers to a scene to be shot by a user through the camera, namely, the scene aimed at by the camera is the shooting scene. It should be noted that, in the embodiment of the present application, the shooting scene is not a specific scene, but a scene aligned in real time along with the direction of the camera.

102. The second image is first superimposed translated relative to the first image based on a plurality of preset vectors to determine an alignment vector.

In this embodiment of the present application, after acquiring a first image and a second image of the same shooting scene, the electronic device may perform a first superposition translation on the second image with respect to the first image based on a plurality of preset vectors, to obtain an alignment vector. Alternatively, the electronic device may also perform a plurality of first superimposed translations of the preprocessed second image with respect to the preprocessed first image based on the plurality of preset vectors, to obtain the alignment vector.

Wherein the preprocessing may include at least one downsampling process, and the preprocessing may further include a binarization process. The plurality of preset vectors are preset in the electronic equipment and have different directions. When the first image is subjected to the binarization processing and the downsampling processing, the electronic device may perform the binarization processing on the first image and then perform the downsampling processing on the image obtained by the binarization processing. The electronic device may perform downsampling on the first image, and then perform binarization on the image obtained by downsampling.

The first superimposed translation means that the second image is translated relative to the first image according to a plurality of preset vectors, after each translation, pixel value differences of pixel points of an overlapped part between the second image and the first image are obtained, and an alignment vector is determined based on a result obtained after each translation.

103. And acquiring a historical alignment vector of the previous frame of image in the current image stream.

In this embodiment of the present application, after obtaining the alignment vector, the electronic device may obtain the historical alignment vector of the previous frame image in the current image stream. The historical alignment vector is an alignment vector determined by performing image alignment operation on a previous frame of image in the current image stream by using a plurality of frames of images to be processed.

For example, referring to fig. 2, fig. 2 is a schematic view of capturing a current image stream according to an embodiment of the present application. The electronic device exposes 3 frames of images with exposure parameters of 0EV, -1EV and 1EV to be used as images to be processed in the current alignment processing, for example, a 0EV image is used as a first image, and the rest of-1 EV images and 1EV images are used as second images. For an exposure time of T _i Is T for 0EV image and exposure time _i The electronic equipment obtains the exposure time of T for the-1 EV image _i After the alignment vectors of-1 EV image and 0EV image, the historical alignment vector of the last frame image in the current image stream is obtained, namely the exposure time is T _i-1 A vector that translates when aligned with a 0EV image. Where i is a positive integer greater than 1. Similarly, for an exposure time of T _i Is T for 0EV image and exposure time _i The historical alignment vector of the previous frame image in the current image stream is the exposure time T _i-1 A vector that translates when aligned with a 0EV image.

It will be appreciated that as an exception, for a first frame image in an image stream (e.g., a video stream), there is no previous frame image, and therefore, after the alignment vector is calculated at 102, the second image is controlled to undergo a second superimposed translation relative to the first image directly in accordance with the alignment vector to align the second image with the first image.

104. A common reference direction of the historical alignment vector and the alignment vector is determined, and an alignment component of the alignment vector in the common reference direction is calculated.

In this embodiment of the present application, after acquiring the historical alignment vector of the previous frame of image in the current image stream, the electronic device may determine a preset direction corresponding to the historical alignment vector and a preset direction corresponding to the alignment vector from one or more preset directions, take the preset directions corresponding to both the historical alignment vector and the alignment vector as a common reference direction, and then calculate an alignment component of the pair Ji Xiangliang in the common reference direction. One or more preset directions are preset in the electronic equipment. The direction of the preset direction is not particularly limited in the embodiments of the present application. It should be noted that the common reference direction may be one or more.

For example, referring to fig. 3, fig. 3 is a schematic diagram illustrating determination of a common reference direction according to an embodiment of the present application. The electronic device is provided with 4 preset directions, noted as: the included angle between any two adjacent directions in the four directions is 90 degrees on a two-dimensional plane. The alignment vector obtained by the electronic device is P2, and the acquired historical alignment vector is P1. The electronic device may determine that the preset directions corresponding to the pair Ji Xiangliang P2 are the N direction and the E direction from the 4 preset directions, the preset directions corresponding to the history pair Ji Xiangliang P1 are the N direction and the W direction, and the N directions corresponding to the history pair Ji Xiangliang P1 and the pair Ji Xiangliang P2 are both the common reference directions. The electronic device may then calculate an alignment component Y1 of the pair Ji Xiangliang P2 in the common reference direction. If the component of the vector in the preset direction is greater than 0, the preset direction is taken as the direction corresponding to the vector.

When there are a plurality of common reference directions, the combined vector of the plurality of alignment components is an alignment vector.

105. The second image is controlled to undergo a second superimposed translation relative to the first image in accordance with the alignment component to align the second image with the first image.

In this embodiment of the present application, after calculating the alignment component of the alignment vector in the common reference direction, the electronic device may control the second image to overlap with the first image, and then control the second image to perform translation with respect to the first image according to the alignment component, that is, control the second image to perform second superimposed translation with respect to the first image according to the alignment component, so as to achieve alignment between the second image and the first image.

For example, referring to fig. 4, fig. 4 is a schematic diagram of second superimposed translation provided in an embodiment of the present application. Wherein the image L1 is a first image, the image L2 is a second image, and the alignment component obtained from the images L1 and L2 is

The electronic device is in terms of alignment component->

When the image L2 is controlled to carry out second superposition translation relative to the image L1, the image L2 is overlapped with the image L1, and then the alignment component is adopted>

The control image L2 is translated relative to the image L1, i.e. the translated image L2 is aligned with the image L1.

As can be seen from the foregoing, in the image alignment method provided in the embodiment of the present application, after determining the alignment vector, the electronic device does not perform the second overlapping movement on the second image according to the pair Ji Xiangliang, but obtains the historical alignment vector of the previous frame image in the current image stream, determines the common reference direction of the historical alignment vector and the alignment vector, and then performs the second overlapping movement on the second image according to the alignment component of the alignment vector in the common reference direction, so that the overlapping movement directions of every two adjacent second alignment processes of the image stream are similar or identical, and the image stream is prevented from shaking.

Referring to fig. 5, fig. 5 is a second flowchart of an image alignment method according to an embodiment of the present application, where the image alignment method may include the following steps:

201. a first image and a second image of the same shooting scene are acquired.

In the embodiment of the application, in a continuous shooting or video recording process, the electronic device may acquire the first image and the second image of the same shooting scene through one or more cameras.

The first image is a reference image of the image alignment process, and the first image is a frame image. The second image is an image to be processed of the image alignment process, and the second image may be one or more frames of images. The first image and the second image represent the same shooting scene, and the first image and the second image are the same in size. The shooting scene in the scheme refers to a scene to be shot by a user through the camera, namely, the scene aimed at by the camera is the shooting scene. It should be noted that, in the embodiment of the present application, the shooting scene is not a specific scene, but a scene aligned in real time along with the direction of the camera.

202. And respectively carrying out superposition translation on the second image relative to the first image for a plurality of times according to a plurality of preset vectors.

In this embodiment of the present application, after acquiring a first image and a second image of the same shooting scene, the electronic device may translate the second image relative to the first image for multiple times according to multiple preset vectors on the basis that the second image coincides with the first image. The preset vector is preset in the electronic equipment. A preset vector has a direction and a size, and the size of the preset vector is not specifically limited in the embodiments of the present application, for example, the size may be one pixel. Any two preset vectors are different in direction and same in size. The second image is coincident with the first image before each second image translation.

For example, referring to fig. 6, fig. 6 is a schematic diagram of multiple superimposed translations provided in an embodiment of the present application. Wherein, image S1 is as first image, and image S2 is as the second image, and electronic equipment has set up four default vectors in advance, marks as: the first preset vector, the second preset vector, the third preset vector and the fourth preset vector. The electronic device overlaps the image S2 with the image S1, and then translates the image S2 according to the first preset vector, the second preset vector, the third preset vector and the fourth preset vector respectively, namely, four superimposed translations of the image S2 are realized.

203. After each superposition translation, calculating a correlation value or a non-correlation value of the second image and the first image, and taking a preset vector corresponding to the maximum correlation value or the minimum non-correlation value as an alignment vector.

In this embodiment of the present application, after each overlay shift, the electronic device may determine an overlapping area of the second image and the first image, and calculate a correlation value or a non-correlation value of the second image and the first image according to the overlapping area. According to the method, after the superposition and the translation are carried out for a plurality of times, the electronic equipment can obtain a plurality of correlation values or a plurality of uncorrelated values. And then taking a preset vector corresponding to the maximum correlation value in the correlation values as an alignment vector, or taking a preset vector corresponding to the minimum uncorrelated value in the uncorrelated values as an alignment vector.

For example, referring to fig. 6, after the electronic device translates the image S2 according to the first preset vector, an overlapping area of the image S2 and the image S1 may be determined and denoted as a K1K2K3K4 area. Then, the electronic device counts the number of pixels overlapping in position and having the same gray value in the K1K2K3K4 region as the correlation value G1. Similarly, the electronic device may obtain the correlation value G2, the correlation value G3, and the correlation value G4. Finally, the electronic device may use the preset vector corresponding to the maximum correlation value of the 4 correlation values as the alignment vector, and if it is assumed that the maximum correlation value of the 4 correlation values is the correlation value G4, the alignment vector is a fourth preset vector.

The correlation value is the number of pixels of the second image, which have the same gray value as the gray value of the first image and overlap in position. The uncorrelated value is the number of pixels in the second image that are different in gray scale value from the first image and overlap in position.

It should be noted that, the alignment vector obtained through the first superposition translation is a certain preset vector selected from a plurality of preset vectors according to a plurality of correlation values or a plurality of non-correlation values.

In some embodiments, the electronic device may exclusive-or the overlapping area of the second image and the first image after the superposition translation when calculating the correlation value or the uncorrelated value of the second image and the first image; and calculating a correlation value or a non-correlation value of the second image and the first image after superposition and translation according to the exclusive OR operation result. The exclusive or operation means that the gray values of two pixel points at the overlapping position are the same as false and marked as 0; the dissimilarity is true and is noted as 1.

204. And acquiring a historical alignment vector of the previous frame of image in the current image stream.

In this embodiment of the present application, after obtaining the alignment vector, the electronic device may obtain the historical alignment vector of the previous frame image in the current image stream. Wherein the historical alignment vector is a vector that translates when a previous frame of the historical second image in the current image stream is aligned with a previous frame of the historical first image.

205. A common reference direction of the historical alignment vector and the alignment vector is determined, and an alignment component of the alignment vector in the common reference direction is calculated.

In some embodiments, after determining the historical alignment vector and the common reference direction of the alignment vector, the electronic device may determine a number of common reference directions; when there is one of the common reference directions, performing calculation of an alignment component of the pair Ji Xiangliang in the common reference direction; when there are a plurality of common reference directions, the second image is controlled to perform a second superimposed translation with respect to the first image according to the alignment vector to align the second image with the first image.

206. The second image is controlled to undergo a second superimposed translation relative to the first image in accordance with the alignment component to align the second image with the first image.

207. And carrying out synthesis processing on the aligned first image and the aligned second image to obtain a synthesized image with a high dynamic range.

In the embodiment of the application, after the second image is aligned with the first image, the aligned first image and second image are subjected to synthesis processing, so as to obtain a synthesized image with a high dynamic range. The dynamic range of an image refers to the difference between the brightness and darkness of the image.

In some embodiments, when the current shooting mode is a video mode and the historical alignment vector of the previous frame of image in the current image stream is obtained, the electronic device may obtain the historical alignment vector of the previous video frame in the current video stream. After obtaining the composite image with high dynamic range, the electronic device can perform video encoding processing on the composite image to obtain a video corresponding to the shooting scene.

Referring to fig. 7, fig. 7 is a third flow chart of an image alignment method according to an embodiment of the present application, where the image alignment method may include the following steps:

301. a first image and a second image of the same shooting scene are acquired.

In the embodiment of the application, in a continuous shooting or video recording process, the electronic device may acquire the first image and the second image of the same shooting scene through one or more cameras. The first image is a reference image of the image alignment process, and the first image is a frame image. The second image is an image to be processed of the image alignment process, and the second image may be one or more frames of images. The first image and the second image represent the same shooting scene, and the first image and the second image are the same in size.

302. And performing one or more continuous downsampling treatments on the first image according to one or more preset sampling multiples to obtain a third image with one or more sizes, and performing one or more continuous downsampling treatments on the second image to obtain a fourth image with one or more sizes.

In this embodiment of the present invention, after acquiring the first image and the second image, the electronic device may perform a downsampling process on the first image according to a preset sampling multiple to obtain a third image, and perform a downsampling process on the second image to obtain a fourth image. Or the electronic equipment performs continuous repeated downsampling processing on the first image according to various preset sampling multiples to obtain a third image with various sizes, and performs continuous repeated downsampling processing on the second image to obtain a fourth image with various sizes.

The preset sampling multiple of the scheme is preset in the electronic equipment. The plurality of preset sampling multiples are different. It is to be noted that, assuming that the sampling multiple is 4, an image with a pixel resolution of 512×384 is obtained after downsampling an image with a pixel resolution of 2048×1536.

In some embodiments, the electronic device may perform a gaussian filter process on the first image before performing one or more consecutive downsampling processes on the first image according to one or more preset sampling multiples. Similarly, the electronic device may also perform gaussian filtering on the second image before performing downsampling on the second image one or more times in succession.

303. And performing first overlapping translation on the fourth image and the third image with the next size based on the result of the first overlapping translation on the fourth image and the third image with the previous size according to the order of the sizes from small to large until a pair Ji Xiangliang of the second image relative to the first image is obtained, wherein when performing first overlapping translation on the fourth image and the third image with each size, multiple overlapping translations are performed based on a plurality of preset vectors to obtain the result.

Wherein, when the fourth image is subjected to the first superposition translation relative to the third image, the following procedure is executed:

a. respectively carrying out superposition translation on the fourth image for multiple times relative to the third image according to multiple preset vectors;

b. after each superposition translation, calculating a correlation value or a non-correlation value of the fourth image and the third image, and taking a preset vector corresponding to the maximum correlation value or the minimum non-correlation value as an offset vector of the first superposition translation.

Referring to fig. 8, fig. 8 is a schematic diagram of image downsampling in the image alignment method according to the embodiment of the present application. The electronic device performs downsampling processing on the first image and the second image (marked as an image H1 and an image H2) according to the sampling multiple m1 to obtain a third image and a fourth image with one size, which are marked as an image H3 and an image H4. And performing downsampling processing on the image H3 and the image H4 according to the sampling multiple m2 to obtain a third image and a fourth image with the same size, which are marked as an image H5 and an image H6.

On the basis of superposition of the image H5 and the image H6, the electronic device performs first superposition translation on the image H6 relative to the image H5 to obtain a first offset vector, as shown in fig. 9, fig. 9 is a schematic diagram of first superposition translation provided in the embodiment of the present application, where the size of the first offset vector is 1, and the direction is rightward.

Then, based on the position superposition of the image H4 and the image H3, the electronic device translates the image H4 relative to the image H3 by m2 according to the direction of the first offset vector according to the previous result of the first superposition translation, and then translates the image H4 relative to the image H3 to obtain a second offset vector, where the second offset vector has a size of 1 and a downward direction as shown in fig. 9.

Then, based on the position superposition of the image H1 and the image H2, the electronic device translates the image H2 relative to the image H1 by m1 according to the direction of the second offset vector, translates the image H2 relative to the image H1 by m1×m2 according to the direction of the first offset vector, and translates the image H2 relative to the image H1 to obtain a third offset vector according to the first superposition translation, as shown in fig. 9, the size of the third offset vector is 1, and the direction is right.

304. And combining the preset vectors corresponding to the plurality of first superposition translations according to one or more preset sampling multiples to obtain an alignment vector.

In this embodiment of the present application, after the first superposition translation is performed on the second image with respect to the first image, for an offset vector obtained by performing the first superposition translation on the fourth image with respect to the third image, where the electronic device may multiply the offset vector by a product of a preset sampling multiple smaller than the size, and then add the vector obtained by multiplying the offset vector with an offset vector obtained by performing the first superposition translation on the second image with respect to the first image, to obtain the alignment vector. The preset vector corresponding to the first superposition translation refers to an offset vector obtained by the first superposition translation.

For example, as shown in fig. 9, after the first overlay shift is performed on the second image with respect to the first image, that is, after the first overlay shift is performed on the image H2 with respect to the image H1, a third offset vector is obtained, and the electronic device may calculate according to the following formula to obtain the pair Ji Xiangliang: for Ji Xiangliang =m1×m2×first offset vector+m1×second offset vector+third offset vector.

305. And acquiring a historical alignment vector of the previous frame of image in the current image stream.

306. A common reference direction of the historical alignment vector and the alignment vector is determined, and an alignment component of the alignment vector in the common reference direction is calculated.

307. The second image is controlled to undergo a second superimposed translation relative to the first image in accordance with the alignment component to align the second image with the first image.

Referring to fig. 10, fig. 10 is a fourth flowchart of an image alignment method according to an embodiment of the present application, where the image alignment method may include the following steps:

401. a first image and a second image of the same shooting scene are acquired.

402. And binarizing the first image to obtain a first binarized image, and binarizing the second image to obtain a second binarized image.

Only black pixel points and white pixel points exist in the binarized image, namely, only two situations exist in gray values of the pixel points in the binarized image: 1 or 0.

In some embodiments, when the first image is binarized to obtain a first binarized image, the electronic device may obtain a median gray level of a pixel point in the first image; and setting the gray scale of the pixel point with the gray scale value being larger than the gray scale median value in the first image as one, and setting the gray scale of the pixel point with the gray scale value being smaller than or equal to the gray scale median value in the first image as zero to obtain a first binarized image. Similarly, the electronic device may acquire the second binarized image according to the method described above.

In some embodiments, when the first image is binarized to obtain a first binarized image, the electronic device may obtain a gray average value of pixels in the first image; and setting the gray level of the pixel point with the gray level value larger than the gray level average value in the first image as one, and setting the gray level of the pixel point with the gray level value smaller than or equal to the gray level average value in the first image as zero to obtain a first binarized image. Similarly, the electronic device may acquire the second binarized image according to the method described above. In addition, compared with the binarization processing method based on the gray median, the binarization processing method based on the gray mean can save the calculated amount and improve the processing speed.

403. The second binarized image is subjected to first superposition translation relative to the first binarized image based on a plurality of preset vectors to determine an alignment vector.

Wherein when the second binarized image is subjected to first superimposed translation relative to the first binarized image based on a plurality of preset vectors to determine an alignment vector, the following procedure is performed:

c. and performing one or more continuous downsampling treatments on the first binarized image according to one or more preset sampling multiples to obtain a fifth image with one or more sizes, and performing one or more continuous downsampling treatments on the second binarized image to obtain a sixth image with one or more sizes.

d. And performing first overlapping translation on the sixth image and the fifth image with the next size until a pair Ji Xiangliang of the second image relative to the first image is obtained based on the result of the first overlapping translation on the sixth image and the fifth image with the previous size in order from small to large, wherein when performing the first overlapping translation on the sixth image and the fifth image with the respective sizes, multiple overlapping translations are performed based on a plurality of preset vectors to obtain the result.

e. And combining the preset vectors corresponding to the plurality of first superposition translations according to one or more preset sampling multiples to obtain an alignment vector.

It should be noted that, the above flow is different from the flow in the above embodiment only by implementation objects, so the specific implementation process can be referred to the above embodiment, and will not be repeated here.

404. And acquiring a historical alignment vector of the previous frame of image in the current image stream.

405. A common reference direction of the historical alignment vector and the alignment vector is determined, and an alignment component of the alignment vector in the common reference direction is calculated.

406. The second image is controlled to undergo a second superimposed translation relative to the first image in accordance with the alignment component to align the second image with the first image.

Referring to fig. 11, fig. 11 is a schematic structural diagram of an image alignment apparatus according to an embodiment of the present application. The image alignment apparatus 500 may include: a first acquisition module 501, a first processing module 502, a second acquisition module 503, a first determination module 504, and a second processing module 505.

A first obtaining module 501, configured to obtain a first image and a second image of the same shooting scene;

a first processing module 502 configured to perform a first superimposed translation on the second image with respect to the first image based on a plurality of preset vectors to determine a pair Ji Xiangliang;

a second obtaining module 503, configured to obtain a history pair Ji Xiangliang of a previous frame image in the current image stream;

A first determining module 504 for determining a common reference direction of the historical alignment vector and the alignment vector, calculating an alignment component of the pair Ji Xiangliang in the common reference direction;

a second processing module 505 is configured to control the second image to perform a second overlay shift with respect to the first image according to the alignment component to align the second image with the first image.

In some embodiments, after acquiring the first image and the second image of the same shooting scene, the image alignment apparatus 500 may further include:

the third processing module is used for carrying out binarization processing on the first image to obtain a first binarized image, and carrying out binarization processing on the second image to obtain a second binarized image;

the first processing module 502 is further configured to, when performing a first superimposed translation of the second image with respect to the first image based on a plurality of preset vectors to determine an alignment vector:

and performing first superposition translation on the second binarized image relative to the first binarized image based on a plurality of preset vectors to determine an alignment vector.

In some embodiments, when the first image is binarized to obtain a first binarized image, the third processing module may be configured to:

Acquiring a gray average value of pixel points in the first image;

and setting the gray level of the pixel point with the gray level value smaller than or equal to the gray level average value in the first image as zero to obtain a first binarized image.

In some embodiments, when performing a first superimposed translation of the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector, the first processing module 502 may be configured to:

respectively carrying out superposition translation on the second image for a plurality of times relative to the first image according to a plurality of preset vectors;

and after each superposition translation, calculating a correlation value or a non-correlation value of the second image and the first image, and taking a preset vector corresponding to the maximum correlation value or the minimum non-correlation value as an alignment vector.

In some embodiments, when calculating the correlation value or the uncorrelated value of the second image and the first image, the first processing module 502 may be configured to:

performing exclusive OR operation on the overlapped area of the second image and the first image after superposition and translation;

and calculating a correlation value or a non-correlation value of the second image and the first image after superposition and translation according to the exclusive OR operation result.

performing one or more continuous downsampling treatments on the first image according to one or more preset sampling multiples to obtain a third image with one or more sizes, and performing one or more continuous downsampling treatments on the second image to obtain a fourth image with one or more sizes;

performing first superimposed translation on the fourth image and the third image of the next size based on the result of the first superimposed translation on the fourth image and the third image of the previous size in order of the sizes from small to large until a pair Ji Xiangliang of the second image relative to the first image is obtained, wherein when performing first superimposed translation on the fourth image and the third image of each size, multiple superimposed translations are performed based on a plurality of preset vectors to obtain the result;

and combining preset vectors corresponding to the first superposition translation for a plurality of times according to one or more preset sampling multiples to obtain the alignment vector.

In some embodiments, after determining the common reference direction of the historical alignment vector and the alignment vector, the image alignment apparatus 500 may further include:

A second determining module for determining the number of the common reference directions;

a fourth processing module for controlling the second image to perform a second superimposed translation with respect to the first image according to the alignment vector when there are a plurality of the common reference directions to align the second image with the first image;

when there is one of the common reference directions, the first determination module 504 performs the calculation of the alignment component of the pair Ji Xiangliang in the common reference direction.

In some embodiments, after obtaining the second image aligned with the first image, the image alignment apparatus 500 may further include:

and a fifth processing module, configured to perform a synthesis process on the aligned first image and second image, so as to obtain a synthesized image with a high dynamic range.

In some embodiments, when the current shooting mode is a video mode and the historical alignment vector of the previous frame of image in the current image stream is acquired, the second acquiring module 503 may be configured to:

acquiring a history pair Ji Xiangliang of a previous video frame in the current video stream;

after obtaining the composite image with high dynamic range, the image alignment apparatus 500 may further include:

and a sixth processing module, configured to perform video encoding processing on the composite image to obtain a video corresponding to the shooting scene.

As can be seen from the foregoing, in the image alignment apparatus provided in the embodiment of the present application, the first obtaining module 501 obtains the first image and the second image of the same shooting scene, the first processing module 502 performs the first overlay translation on the second image relative to the first image based on the plurality of preset vectors to determine the pair Ji Xiangliang, the second obtaining module 503 obtains the historical alignment vector of the previous frame image in the current image stream, the first determining module 504 determines the common reference direction of the historical alignment vector and the alignment vector, calculates the alignment component of the alignment vector in the common reference direction, and the second processing module 505 controls the second image to overlay and move relative to the first image according to the alignment component, so as to perform the second alignment process, to obtain the second image aligned with the first image, so that the overlay moving direction of each two adjacent second alignment processes of the image stream is close to or the same, and the image stream shake is prevented.

An embodiment of the present application further provides an electronic device, referring to fig. 12, and fig. 12 is a first structural schematic diagram of the electronic device provided in the embodiment of the present application. The electronic device 600 includes a processor 601, a memory 602, and a camera assembly 603. The processor 601 is electrically connected to the memory 602 and the camera module 603.

The processor 601 is a control center of the electronic device 600, connects various parts of the entire electronic device using various interfaces and lines, and performs various functions of the electronic device 600 and processes data by running or loading computer programs stored in the memory 602 and calling data stored in the memory 602, thereby performing overall monitoring of the electronic device 600.

The memory 602 may be used to store software programs and modules, and the processor 601 may execute various functional applications and data processing by executing the computer programs and modules stored in the memory 602. The memory 602 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, a computer program required for at least one function, and the like; the storage data area may store data created according to the use of the electronic device, etc. In addition, the memory 602 may include high-speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, the memory 602 may also include a memory controller to provide access to the memory 602 by the processor 601.

The camera assembly 603 may include image processing circuitry, which may be implemented using hardware and/or software components, and may include various processing units defining an image signal processing (Image Signal Processing) pipeline. The image processing circuit may include at least: a camera, an image signal processor (Image Signal Processor, ISP processor), control logic, image memory, a display, etc. Wherein the camera may comprise at least one or more lenses and an image sensor. The image sensor may include an array of color filters (e.g., bayer filters). The image sensor may acquire light intensity and wavelength information captured with each imaging pixel of the image sensor and provide a set of raw image data that may be processed by an image signal processor.

The image signal processor may process the raw image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the image signal processor may perform one or more image processing operations on the raw image data, collecting statistical information about the image data. Wherein the image processing operations may be performed with the same or different bit depth precision. The original image data can be stored in the image memory after being processed by the image signal processor. The image signal processor may also receive image data from an image memory.

The image memory may be part of a memory device, a storage device, or a separate dedicated memory within an electronic device, and may include DMA (Direct Memory Access ) features.

When image data is received from the image memory, the image signal processor may perform one or more image processing operations, such as temporal filtering. The processed image data may be sent to an image memory for additional processing before being displayed. The image signal processor may also receive processing data from the image memory and process the processing data for image data in the raw domain and in RGB and YCbCr color spaces. The processed image data may be output to a display for viewing by a user and/or further processing by a graphics engine or GPU (Graphics Processing Unit, graphics processor). In addition, the output of the image signal processor may also be sent to an image memory, and the display may read image data from the image memory. In one embodiment, the image memory may be configured to implement one or more frame buffers.

The statistical data determined by the image signal processor may be sent to the control logic. For example, the statistics may include statistics of the image sensor for auto exposure, auto white balance, auto focus, flicker detection, black level compensation, lens shading correction, and the like.

The control logic may include a processor and/or microcontroller that executes one or more routines (e.g., firmware). One or more routines may determine camera control parameters and ISP control parameters based on the received statistics. For example, the control parameters of the camera may include camera flash control parameters, control parameters of the lens (e.g., focal length for focusing or zooming), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balancing and color adjustment (e.g., during RGB processing), and the like.

Referring to fig. 13, fig. 13 is a schematic structural diagram of an image processing circuit according to an embodiment of the present application. As shown in fig. 13, for convenience of explanation, only aspects of the image processing technology related to the embodiment of the present invention are shown.

For example, the image processing circuit may include: the device comprises a camera, an image signal processor, a control logic, an image memory and a display. Wherein the camera may comprise one or more lenses and an image sensor. In some embodiments, the camera may be any of a tele camera or a wide camera.

And transmitting the first image acquired by the camera to an image signal processor for processing. After the image signal processor processes the first image, statistical data of the first image (e.g., brightness of the image, contrast value of the image, color of the image, etc.) may be sent to the control logic. The control logic can determine the control parameters of the camera according to the statistical data, so that the camera can perform operations such as automatic focusing and automatic exposure according to the control parameters. The first image may be stored in the image memory after being processed by the image signal processor. The image signal processor may also read the image stored in the image memory for processing. In addition, the first image can be directly sent to a display for display after being processed by the image signal processor. The display may also read the image in the image memory for display.

In the embodiment of the present application, the processor 601 in the electronic device 600 loads instructions corresponding to the processes of one or more computer programs into the memory 602 according to the following steps, and the processor 601 executes the computer programs stored in the memory 602, so as to implement various functions, as follows:

acquiring a first image and a second image of the same shooting scene;

Referring to fig. 14 together, fig. 14 is a second structural schematic diagram of the electronic device according to the embodiment of the present application. In some implementations, the electronic device 700 may further include: a display 704 and a power supply 705. The display 704 and the power supply 705 are electrically connected to the processor 701, respectively.

The display 704 may be used to display information entered by a user or provided to a user as well as various graphical user interfaces that may be composed of graphics, text, icons, video, and any combination thereof.

The power supply 705 may be used to power various components of the electronic device 700. In some embodiments, the power supply 705 may be logically connected to the processor 701 through a power management system, so as to perform functions of managing charging, discharging, and power consumption management through the power management system.

In the embodiment of the present application, the processor 701 in the electronic device 700 loads the instructions corresponding to the processes of one or more computer programs into the memory 702 according to the following steps, and the processor 701 executes the computer programs stored in the memory 702, so as to implement various functions, as follows:

acquiring a first image and a second image of the same shooting scene;

In some embodiments, after acquiring the first image and the second image of the same shooting scene, the processor 701 may further perform:

binarizing the first image to obtain a first binarized image, and binarizing the second image to obtain a second binarized image;

the processor 701 may perform a first superimposed translation of the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector, where:

In some embodiments, when the first image binarization process is performed to obtain a first binarized image, the processor 701 may perform:

acquiring a gray average value of pixel points in the first image;

In some embodiments, when performing a first superimposed translation of the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector, the processor 701 may perform:

In some embodiments, when calculating the correlation value or the uncorrelated value of the second image and the first image, the processor 701 may perform:

In some embodiments, after determining the common reference direction of the historical alignment vector and the alignment vector, the processor 701 may further perform:

determining a number of the common reference directions;

performing said calculating an alignment component of said pair Ji Xiangliang in said common reference direction when there is one of said common reference directions;

and when the common reference directions are multiple, controlling the second image to perform second superposition translation relative to the first image according to the alignment vector so as to align the second image with the first image.

In some embodiments, after obtaining the second image aligned with the first image, the processor 701 may further perform:

and carrying out synthesis processing on the aligned first image and the aligned second image to obtain a synthesized image with a high dynamic range.

In some embodiments, the current shooting mode is a video mode; the processor 701 may perform, when acquiring the historical alignment vector of the previous frame of image in the current image stream:

after obtaining the composite image with high dynamic range, the processor 701 may also perform:

and carrying out video coding processing on the synthesized image to obtain a video corresponding to the shooting scene.

As can be seen from the foregoing, after determining the alignment vector, the electronic device provided in the embodiment of the present application does not perform the second overlapping movement on the second image according to the pair Ji Xiangliang, but obtains the historical alignment vector of the previous frame image in the current image stream, determines the common reference direction of the historical alignment vector and the alignment vector, and then performs the second overlapping movement on the second image according to the alignment component of the alignment vector in the common reference direction, so that the overlapping movement directions of every two adjacent second alignment processes of the image stream are similar or identical, and the image stream jitter is prevented.

The present application also provides a storage medium storing a computer program, which when executed on a computer, causes the computer to perform the image alignment method in any of the above embodiments, such as: acquiring a first image and a second image of the same shooting scene; performing a first superimposed translation of the second image relative to the first image based on a plurality of preset vectors to determine a pair Ji Xiangliang; acquiring a history pair Ji Xiangliang of a previous frame image in the current image stream; determining a common reference direction for the historical alignment vector and the alignment vector, calculating an alignment component of the pair Ji Xiangliang in the common reference direction; and according to the alignment component, controlling the second image to perform second superposition translation relative to the first image so as to align the second image with the first image.

In the embodiment of the present application, the storage medium may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a random access Memory (Random Access Memory, RAM), or the like.

In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.

It should be noted that, for the image alignment method according to the embodiment of the present application, it will be understood by those skilled in the art that all or part of the flow of implementing the image alignment method according to the embodiment of the present application may be implemented by controlling related hardware through a computer program, where the computer program may be stored in a computer readable storage medium, such as a memory of an electronic device, and executed by at least one processor within the electronic device, and the execution may include, for example, the flow of the embodiment of the image alignment method. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

For the image alignment apparatus of the embodiments of the present application, each functional module may be integrated in one processing chip, or each module may exist separately and physically, or two or more modules may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated module, if implemented as a software functional module and sold or used as a stand-alone product, may also be stored on a computer readable storage medium such as read-only memory, magnetic or optical disk, etc.

The foregoing describes in detail an image alignment method, apparatus, storage medium and electronic device provided in the embodiments of the present application, and specific examples are applied to illustrate principles and implementations of the present application, where the foregoing description of the embodiments is only for helping to understand the method and core ideas of the present application; meanwhile, those skilled in the art will have variations in the specific embodiments and application scope in light of the ideas of the present application, and the present description should not be construed as limiting the present application in view of the above.

Claims

1. An image alignment method, comprising:

acquiring a first image and a second image of the same shooting scene;

acquiring a historical alignment vector of a previous frame image in a current image stream, wherein the exposure time of a first image and a second image in the current image stream is T _i The historical alignment vector is the exposure time T _i-1 And the exposure time is T _i-1 A vector translated when aligned, i is a positive integer greater than 1;

2. The image alignment method according to claim 1, wherein after the first image and the second image of the same shooting scene are acquired, further comprising:

the performing a first superimposed translation of the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector includes:

3. The image alignment method according to claim 2, wherein the binarizing the first image to obtain a first binarized image comprises:

Acquiring a gray average value of pixel points in the first image;

4. The image alignment method of claim 1, wherein the first overlay translating the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector comprises:

5. The image alignment method of claim 4, wherein the calculating a correlation value or a non-correlation value of the second image and the first image comprises:

6. The image alignment method of claim 1, wherein the first overlay translating the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector comprises:

7. The image alignment method of claim 1, further comprising, after determining the common reference direction of the historical alignment vector and the alignment vector:

determining a number of the common reference directions;

8. The image alignment method according to any one of claims 1 to 7, further comprising, after obtaining a second image aligned with the first image:

9. The image alignment method of claim 8, wherein the current shooting mode is a video mode; the obtaining the historical alignment vector of the previous frame of image in the current image stream comprises the following steps:

After the synthetic image with high dynamic range is obtained, the method further comprises:

10. An image alignment apparatus, comprising:

a second acquisition module, configured to acquire a historical alignment vector of a previous frame image in a current image stream, where an exposure time of the first image and the second image in the current image stream is T _i The historical alignment vector is the exposure time T _i-1 And the exposure time is T _i-1 A vector translated when aligned, i is a positive integer greater than 1;

11. A storage medium having stored thereon a computer program which, when run on a computer, causes the computer to perform the image alignment method according to any of claims 1 to 9.

12. An electronic device comprising a processor, a memory, the memory having a computer program, wherein the processor is configured to perform the image alignment method of any of claims 1 to 9 by invoking the computer program.