CN110930440A

CN110930440A - Image alignment method and device, storage medium and electronic equipment

Info

Publication number: CN110930440A
Application number: CN201911252973.3A
Authority: CN
Inventors: 贾玉虎
Original assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Current assignee: Guangdong Oppo Mobile Telecommunications Corp Ltd
Priority date: 2019-12-09
Filing date: 2019-12-09
Publication date: 2020-03-27
Anticipated expiration: 2039-12-09
Also published as: CN110930440B

Abstract

The application discloses an image alignment method, an image alignment device, a storage medium and an electronic device, wherein the image alignment method comprises the following steps: acquiring a first image and a second image of the same shooting scene; performing a first overlay translation of the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector; acquiring a historical alignment vector of a previous frame of image in a current image stream; determining a common reference direction of the historical alignment vector and the alignment vector, and calculating an alignment component of the alignment vector in the common reference direction; and controlling the second image to perform second superposition translation relative to the first image according to the alignment component so as to align the second image with the first image. The image alignment scheme provided by the embodiment can ensure that the superposition moving directions of every two adjacent second alignment treatments of the image stream are close or the same, and prevent the image stream from shaking.

Description

Image alignment method and device, storage medium and electronic equipment

Technical Field

The present application relates to the field of image technologies, and in particular, to an image alignment method and apparatus, a storage medium, and an electronic device.

Background

Compared with a common video, a video composed of a High-Dynamic Range (HDR) synthesized image can provide more Dynamic Range and image details. For each frame of high dynamic range composite image in a video, before the two frames of images of the same shooting scene are subjected to composite processing, the two frames of images need to be subjected to alignment processing.

In the related art, for each frame of high dynamic range synthesized image in a video, before performing synthesis processing from two frames of images of the same shooting scene, the two frames of images are aligned based on an affine transformation matrix. The alignment may cause the alignment direction of the two adjacent frames of the composite image in the video to be different or opposite, which may cause the video to be jittered.

Disclosure of Invention

The embodiment of the application provides an image alignment method, an image alignment device, a storage medium and electronic equipment, which can ensure that the superposition moving directions of every two adjacent second alignment treatments of an image stream are close or the same, and prevent the image stream from shaking.

In a first aspect, an embodiment of the present application provides an image alignment method, including:

acquiring a first image and a second image of the same shooting scene;

performing a first overlay translation of the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector;

acquiring a historical alignment vector of a previous frame of image in a current image stream;

determining a common reference direction of the historical alignment vector and the alignment vector, and calculating an alignment component of the alignment vector in the common reference direction;

and controlling the second image to perform second superposition translation relative to the first image according to the alignment component so as to align the second image with the first image.

In a second aspect, an embodiment of the present application provides an image alignment apparatus, including:

the first acquisition module is used for acquiring a first image and a second image of the same shooting scene;

a first processing module configured to perform a first overlay translation of the second image with respect to the first image based on a plurality of preset vectors to determine an alignment vector;

the second acquisition module is used for acquiring the historical alignment vector of the previous frame of image in the current image stream;

a first determining module, configured to determine a common reference direction of the historical alignment vector and the alignment vector, and calculate an alignment component of the alignment vector in the common reference direction;

and the second processing module is used for controlling the second image to perform second superposition translation relative to the first image according to the alignment component so as to align the second image with the first image.

In a third aspect, a storage medium is provided in an embodiment of the present application, and has a computer program stored thereon, where the computer program is executed on a computer, so as to enable the computer to execute an image alignment method according to any embodiment of the present application.

In a fourth aspect, an electronic device provided in an embodiment of the present application includes a processor and a memory, where the memory has a computer program, and the processor is configured to execute the image alignment method provided in any embodiment of the present application by calling the computer program.

According to the image alignment scheme provided by the embodiment of the application, after the alignment vector is determined, the electronic device does not perform second overlay movement on the second image according to the alignment vector, but obtains the historical alignment vector of the previous frame of image in the current image stream, determines the common reference direction of the historical alignment vector and the alignment vector, and performs second overlay movement on the second image according to the alignment component of the alignment vector in the common reference direction, so that the overlay movement directions of every two adjacent second alignment processes of the image stream are close or identical, and image stream jitter is prevented.

Drawings

The technical solutions and advantages of the present application will become apparent from the following detailed description of specific embodiments of the present application when taken in conjunction with the accompanying drawings.

Fig. 1 is a first flowchart of an image alignment method according to an embodiment of the present disclosure.

Fig. 2 is a schematic diagram of shooting a current image stream according to an embodiment of the present disclosure.

Fig. 3 is a schematic diagram of determining a common reference direction provided in an embodiment of the present application.

Fig. 4 is a schematic diagram of a second superimposed translation provided by an embodiment of the present application.

Fig. 5 is a second flowchart of an image alignment method according to an embodiment of the present application.

Fig. 6 is a schematic diagram of multiple superimposed translations provided by an embodiment of the present application.

Fig. 7 is a third flowchart of an image alignment method according to an embodiment of the present application.

Fig. 8 is a schematic diagram of down-sampling an image in an image alignment method according to an embodiment of the present application.

Fig. 9 is a schematic view of a first superimposed translation provided by an embodiment of the present application.

Fig. 10 is a fourth flowchart illustrating an image alignment method according to an embodiment of the present application.

Fig. 11 is a schematic structural diagram of an image alignment apparatus according to an embodiment of the present application.

Fig. 12 is a schematic view of a first structure of an electronic device according to an embodiment of the present application.

Fig. 13 is a schematic structural diagram of an image processing circuit according to an embodiment of the present application.

Fig. 14 is a second structural schematic diagram of an electronic device provided in the embodiment of the present application.

Detailed Description

The following description is based on illustrated embodiments of the application and should not be taken as limiting the application with respect to other embodiments that are not detailed herein. The term "module" as used herein may be considered a software object executing on the computing system. The various modules, engines, and services herein may be considered as objects of implementation on the computing system.

The embodiment of the application provides an image alignment method, and an execution main body of the image alignment method can be the image alignment device provided by the embodiment of the application or an electronic device integrated with the image alignment device. The electronic equipment comprises a camera shooting assembly, and the camera shooting assembly comprises at least one camera. The electronic device may be a smart phone, a tablet computer, a Personal Digital Assistant (PDA), or the like.

The following is a detailed description of the analysis.

Referring to fig. 1, fig. 1 is a first flowchart of an image alignment method according to an embodiment of the present disclosure, where the image alignment method includes the following steps:

101. a first image and a second image of the same shooting scene are acquired.

In some shooting modes, such as continuous shooting, video recording, and delayed shooting, the electronic device needs to output multiple frames of images continuously to form an image stream. Each frame of image in the image stream can be synthesized by adopting a plurality of frames of original images.

The original image corresponding to each frame of image in the image stream can be subjected to image alignment processing according to the scheme of the embodiment of the application.

For example, the electronic device may acquire multiple frames of images to be processed of the same shooting scene through one or more cameras, select any one frame of image (or select one frame of image with the highest definition) from the multiple frames of images to be processed as a reference image, and record the reference image as a first image, and record the remaining images (or select one or more frames of images with better definition in the remaining images) as images to be aligned as second images. Wherein, in the scheme, each camera is exposed once to obtain a frame of image.

The first image and the second image present the same shooting scene, and the first image and the second image have the same size. The shooting scene in the scheme refers to a scene to be shot by a user through the camera, namely, the scene aligned with the camera is the shooting scene. It should be noted that, in the embodiment of the present application, the shooting scene is not particularly a specific scene, but is a scene aligned in real time according to the orientation of the camera.

102. A first overlay translation of the second image relative to the first image is performed based on a plurality of preset vectors to determine an alignment vector.

In the embodiment of the application, after the first image and the second image of the same shooting scene are acquired, the electronic device may perform first superposition translation on the second image relative to the first image for one time based on a plurality of preset vectors to obtain the alignment vector. Or, the electronic device may also perform a first stacking translation on the preprocessed second image relative to the preprocessed first image for multiple times based on the plurality of preset vectors, so as to obtain the alignment vector.

The preprocessing may include at least one downsampling processing, and the preprocessing may further include a binarization processing. A plurality of preset vectors are preset in the electronic equipment, and the preset vectors have different directions. When the first image is subjected to the binarization processing and the down-sampling processing, the electronic device may perform the binarization processing on the first image and then perform the down-sampling processing on the image obtained by the binarization processing. The electronic device may perform downsampling on the first image, and then perform binarization on the image obtained by the downsampling.

The first superposition translation refers to translating the second image relative to the first image according to a plurality of preset vectors respectively, obtaining pixel value differences of pixel points of an overlapped part between the second image and the first image after each translation, and determining an alignment vector based on a result obtained after each translation.

103. And acquiring a historical alignment vector of a previous frame of image in the current image stream.

In this embodiment of the application, after obtaining the alignment vector, the electronic device may obtain a historical alignment vector of a previous frame of image in the current image stream. The historical alignment vector is determined by performing image alignment operation on a previous frame of image in the current image stream by using a plurality of frames of images to be processed.

For example, please refer to fig. 2, fig. 2 is a schematic diagram illustrating a current image stream according to an embodiment of the present disclosure. The electronic equipment exposure obtains 3 frames of images with exposure parameters of 0EV, -1EV and 1EV as the images to be processed of the alignment processing, for example, the 0EV image is taken as the first image, and the rest-1 EV image and 1EV image are taken as the second image. For an exposure time of T_i0EV image and exposure time of T_iOf the electronic device, the exposure time of the electronic device is T_iAfter aligning the vectors of the-1 EV image and the 0EV image, the obtainedThe historical alignment vector of the previous frame image in the current image stream refers to the exposure time T_i-1The vector of the-1 EV image translated when aligned with the 0EV image. Wherein i is a positive integer greater than 1. Similarly, for an exposure time of T_i0EV image and exposure time of T_iThe 1EV image of (1), the historical alignment vector of the previous frame image in the current image stream, means that the exposure time is T_i-1The vector of 1EV image (1) translated when aligned with the 0EV image.

It will be appreciated that as an exception, for a first frame of image in an image stream (e.g. a video stream), there is no previous frame of image, and therefore, after the alignment vector is calculated in 102, the second image is controlled to perform a second overlay translation with respect to the first image directly in accordance with the alignment vector to align the second image with the first image.

104. Determining a common reference direction of the historical alignment vector and the alignment vector, and calculating an alignment component of the alignment vector in the common reference direction.

In this embodiment of the application, after obtaining the historical alignment vector of the previous frame of image in the current image stream, the electronic device may determine a preset direction corresponding to the historical alignment vector and a preset direction corresponding to the alignment vector from among the one or more preset directions, use the preset directions corresponding to both the historical alignment vector and the alignment vector as a common reference direction, and then calculate an alignment component of the alignment vector in the common reference direction. Wherein one or more preset directions are preset in the electronic device. The orientation of the preset direction is not particularly limited in the embodiments of the present application. It should be noted that the common reference direction may be one or more.

For example, please refer to fig. 3, fig. 3 is a schematic diagram illustrating the determination of the common reference direction according to the embodiment of the present application. The electronic device is provided with 4 preset directions, which are recorded as: the N direction, the S direction, the E direction and the W direction, and on a two-dimensional plane, an included angle between any two adjacent directions in the four directions is 90 degrees. The electronic device obtains an alignment vector P2 and the obtained historical alignment vector P1. The electronic device may determine, from the 4 preset directions, that the preset directions corresponding to the alignment vector P2 are the N direction and the E direction, the preset directions corresponding to the historical alignment vector P1 are the N direction and the W direction, and the N direction corresponding to both the historical alignment vector P1 and the alignment vector P2 is taken as a common reference direction. The electronic device may then calculate an alignment component Y1 of the alignment vector P2 in the common reference direction. And if the component of the vector in the preset direction is greater than 0, taking the preset direction as the direction corresponding to the vector.

It should be noted that, when there are a plurality of common reference directions, the merged vector of the plurality of alignment components is an alignment vector.

105. And controlling the second image to perform a second overlay translation relative to the first image according to the alignment component so as to align the second image with the first image.

In this embodiment, after calculating the alignment component of the alignment vector in the common reference direction, the electronic device may control the second image to coincide with the first image, and then control the second image to translate with respect to the first image according to the alignment component, that is, control the second image to translate with respect to the first image in the second superposition according to the alignment component, thereby achieving the alignment of the second image with the first image.

For example, please refer to fig. 4, fig. 4 is a second schematic view of the stacking translation according to the embodiment of the present application. Wherein the image L1 is used as the first image, the image L2 is used as the second image, and the alignment components obtained from the image L1 and the image L2 are

The electronic device is in accordance with the alignment component

Controlling the second superimposed translation of image L2 with respect to image L1, image L2 is first registered with image L1, and then registered according to the registration component

Control image L2 is translated relative to image L1, i.e., translated image L2 is aligned with image L1.

As can be seen from the above, in the image alignment method provided in this embodiment of the application, after the alignment vector is determined, the electronic device does not perform the second overlay movement on the second image according to the alignment vector, but obtains the historical alignment vector of the previous frame of image in the current image stream, determines the common reference direction between the historical alignment vector and the alignment vector, and then performs the second overlay movement on the second image according to the alignment component of the alignment vector in the common reference direction, so that the overlay movement directions of the second alignment processing every two adjacent times of the image stream are ensured to be close to or the same, and the image stream is prevented from shaking.

Referring to fig. 5, fig. 5 is a second flow chart of the image alignment method according to the embodiment of the present disclosure, where the image alignment method includes the following steps:

201. a first image and a second image of the same shooting scene are acquired.

In the embodiment of the application, in the continuous shooting or video recording process, the electronic equipment can acquire the first image and the second image of the same shooting scene through one or more cameras.

The first image is a reference image for image alignment processing, and the first image is a frame image. The second image is an image to be processed for the image alignment process, and the second image may be one or more frame images. The first image and the second image present the same shooting scene, and the first image and the second image are the same in size. The shooting scene in the scheme refers to a scene to be shot by a user through the camera, namely, the scene aligned with the camera is the shooting scene. It should be noted that, in the embodiment of the present application, the shooting scene is not particularly a specific scene, but is a scene aligned in real time according to the orientation of the camera.

202. And performing superposition translation on the second image relative to the first image for multiple times according to the preset vectors respectively.

In this embodiment of the application, after acquiring a first image and a second image of the same shooting scene, the electronic device may perform multiple translations on the second image with respect to the first image according to a plurality of preset vectors on the basis that the second image is overlapped with the first image. The preset vector is preset in the electronic equipment. A predetermined vector has a direction and a magnitude, and the magnitude of the predetermined vector is not particularly limited in the embodiments of the present application, for example, the magnitude may be one pixel. Any two preset vectors have different directions and the same size. The second image is coincident with the first image before each translation of the second image.

For example, please refer to fig. 6, fig. 6 is a schematic diagram of multiple overlay translations according to an embodiment of the present disclosure. The image S1 is used as a first image, the image S2 is used as a second image, and the electronic device is preset with four preset vectors, which are recorded as: the first preset vector, the second preset vector, the third preset vector and the fourth preset vector. The electronic device coincides the image S2 with the image S1, and then translates the image S2 according to the first preset vector, the second preset vector, the third preset vector and the fourth preset vector, respectively, that is, four times of superimposed translation of the image S2 is achieved.

203. And after each superposition translation, calculating a correlation value or a non-correlation value of the second image and the first image, and taking a preset vector corresponding to the maximum correlation value or the minimum non-correlation value as an alignment vector.

In this embodiment of the application, after each overlay translation, the electronic device may determine an overlapping region of the second image and the first image, and calculate a correlation value or a non-correlation value of the second image and the first image according to the overlapping region. According to the above method, after a plurality of superimposed translations, the electronic device may obtain a plurality of correlation values or a plurality of non-correlation values. And then taking a preset vector corresponding to the maximum correlation value in the plurality of correlation values as an alignment vector, or taking a preset vector corresponding to the minimum non-correlation value in the plurality of non-correlation values as an alignment vector.

For example, referring to fig. 6, after the electronic device translates the image S2 according to the first predetermined vector, the overlapping region of the image S2 and the image S1 can be determined and is denoted as the K1K2K3K4 region. The electronic device then counts the number of pixel points that overlap in position and have the same gray value in the region K1K2K3K4 as a correlation value G1. Similarly, the electronic device may obtain the correlation value G2, the correlation value G3, and the correlation value G4. Finally, the electronic device may use the preset vector corresponding to the largest correlation value of the 4 correlation values as the alignment vector, and if the largest correlation value of the 4 correlation values is assumed to be the correlation value G4, the alignment vector is the fourth preset vector.

Wherein the correlation value is the number of pixel points in the second image that have the same gray scale value as the first image and that overlap in position. The non-correlation value is the number of pixel points in the second image that differ in gray scale value from the first image and overlap in position.

It should be noted that the alignment vector obtained through the first stacking translation is a certain preset vector screened from the plurality of preset vectors according to the plurality of correlation values or the plurality of non-correlation values.

In some embodiments, in calculating the correlation value or the non-correlation value of the second image with the first image, the electronic device may perform an exclusive or operation on an overlapping area of the superimposed translated second image and the first image; and calculating a correlation value or a non-correlation value of the overlapped and translated second image and the first image according to the result of the exclusive-or operation. The XOR operation means that the gray values of two pixels at the overlapping position are the same and are false, and are marked as 0; the difference is true and is noted as 1.

204. And acquiring a historical alignment vector of a previous frame of image in the current image stream.

In this embodiment of the application, after obtaining the alignment vector, the electronic device may obtain a historical alignment vector of a previous frame of image in the current image stream. Wherein the historical alignment vector is a vector of a previous frame of historical second image in the current image stream translated when aligned with the previous frame of historical first image.

205. Determining a common reference direction of the historical alignment vector and the alignment vector, and calculating an alignment component of the alignment vector in the common reference direction.

In some embodiments, after determining the historical alignment vector and the common reference direction of the alignment vector, the electronic device may determine a number of common reference directions; when there is one common reference direction, performing calculation of an alignment component of the alignment vector in the common reference direction; when there are a plurality of common reference directions, the second image is controlled to perform a second superimposed translation with respect to the first image according to the alignment vector to align the second image with the first image.

206. And controlling the second image to perform a second overlay translation relative to the first image according to the alignment component so as to align the second image with the first image.

207. And synthesizing the aligned first image and the second image to obtain a synthesized image with a high dynamic range.

In the embodiment of the application, after the second image is aligned with the first image, the aligned first image and second image are subjected to synthesis processing, so that a synthesized image with a high dynamic range is obtained. The dynamic range of an image refers to the difference between brightness and darkness of the image.

In some embodiments, the current shooting mode is a video recording mode, and when the historical alignment vector of the previous frame of image in the current image stream is obtained, the electronic device may obtain the historical alignment vector of the previous frame of video in the current video stream. After the composite image with the high dynamic range is obtained, the electronic device may perform video coding processing on the composite image to obtain a video corresponding to the shooting scene.

Referring to fig. 7, fig. 7 is a third flowchart of an image alignment method according to an embodiment of the present disclosure, where the image alignment method includes the following steps:

301. a first image and a second image of the same shooting scene are acquired.

In the embodiment of the application, in the continuous shooting or video recording process, the electronic equipment can acquire the first image and the second image of the same shooting scene through one or more cameras. The first image is a reference image for image alignment processing, and the first image is a frame image. The second image is an image to be processed for the image alignment process, and the second image may be one or more frame images. The first image and the second image present the same shooting scene, and the first image and the second image are the same in size.

302. And carrying out one-time or continuous multiple down-sampling processing on the first image according to one or more preset sampling multiples to obtain a third image with one or more sizes, and carrying out one-time or continuous multiple down-sampling processing on the second image to obtain a fourth image with one or more sizes.

In this embodiment, after acquiring the first image and the second image, the electronic device may perform downsampling processing on the first image once according to a preset sampling multiple to obtain a third image, and perform downsampling processing on the second image once to obtain a fourth image. Or the electronic equipment performs continuous multiple downsampling processing on the first image according to multiple preset sampling multiples to obtain third images with multiple sizes, and performs continuous multiple downsampling processing on the second image to obtain fourth images with multiple sizes.

The preset sampling multiple of the scheme is preset in the electronic equipment. The plurality of preset sampling multiples are different from each other. Assuming that the sampling multiple is 4, an image having a pixel resolution of 512 × 384 is obtained by downsampling an image having a pixel resolution of 2048 × 1536.

In some embodiments, the electronic device may perform gaussian filtering on the first image before performing downsampling on the first image one or more times according to one or more preset sampling multiples. Similarly, before the down-sampling process is performed on the second image for one or more consecutive times, the electronic device may perform the gaussian filtering process on the second image.

303. And performing first superposition translation on a fourth image and a third image of a next size based on a first superposition translation result of the fourth image and the third image of the previous size in the order from small to large until an alignment vector of the second image relative to the first image is obtained, wherein when the first superposition translation is performed on the fourth image and the third image of each size, a plurality of times of superposition translation are performed based on a plurality of preset vectors to obtain the result.

When the fourth image is subjected to the first superposition translation relative to the third image, the following process is executed:

a. performing superposition translation on the fourth image for multiple times relative to the third image according to a plurality of preset vectors respectively;

b. and after each superposition translation, calculating a correlation value or a non-correlation value of the fourth image and the third image, and taking a preset vector corresponding to the maximum correlation value or the minimum non-correlation value as an offset vector of the first superposition translation.

Referring to fig. 8, fig. 8 is a schematic view illustrating image down-sampling in an image alignment method according to an embodiment of the present disclosure. The electronic device downsamples the first and second images (denoted as images H1 and H2) by the sampling multiple m1 to obtain a third and fourth image of one size, denoted as images H3 and H4. And performing downsampling processing on the image H3 and the image H4 according to the sampling multiple m2 to obtain a third image and a fourth image of still another size, which are recorded as an image H5 and an image H6.

The electronic device performs a first overlay translation on the image H6 relative to the image H5 on the basis that the image H5 coincides with the image H6 to obtain a first offset vector, as shown in fig. 9, where fig. 9 is a schematic diagram of the first overlay translation provided in the embodiment of the present application, a size of the first offset vector is 1, and a direction of the first offset vector is rightward.

Then, on the basis of the position coincidence of the image H4 and the image H3, the electronic device translates the image H4 by m2 relative to the image H3 according to the result of the previous first superposition translation, and then performs the first superposition translation on the image H4 relative to the image H3 to obtain a second offset vector, wherein the size of the second offset vector is 1 and the direction is downward as shown in fig. 9.

Then, based on the position coincidence between the image H1 and the image H2, the electronic device translates the image H2 by m1 in the direction of the second offset vector and translates the image H2 by m1 × m2 in the direction of the first offset vector, and then translates the image H2 by the first overlay with respect to the image H1 to obtain a third offset vector, according to the result of the previous first overlay translation, as shown in fig. 9, the size of the third offset vector is 1, and the direction is right.

304. And combining the preset vectors corresponding to the first superposition translation for multiple times according to one or more preset sampling multiples to obtain the alignment vector.

In this embodiment of the application, after performing the first overlay translation on the second image with respect to the first image, for the offset vector obtained by performing the first overlay translation on the fourth image with respect to the third image in each size, the electronic device may multiply the offset vector by a product of a preset sampling multiple smaller than the size, and then add the vector obtained by the multiplication to the offset vector obtained by performing the first overlay translation on the second image with respect to the first image, so as to obtain the alignment vector. The preset vector corresponding to the first superposition translation is an offset vector obtained by the first superposition translation.

For example, as shown in fig. 9, after the first overlay translation of the second image with respect to the first image, i.e., the first overlay translation of the image H2 with respect to the image H1, a third offset vector is obtained, and the electronic device may calculate according to the following formula, obtaining an alignment vector: the alignment vector is m1 × m2 × first offset vector + m1 × second offset vector + third offset vector.

305. And acquiring a historical alignment vector of a previous frame of image in the current image stream.

306. Determining a common reference direction of the historical alignment vector and the alignment vector, and calculating an alignment component of the alignment vector in the common reference direction.

307. And controlling the second image to perform a second overlay translation relative to the first image according to the alignment component so as to align the second image with the first image.

Referring to fig. 10, fig. 10 is a fourth flowchart illustrating an image alignment method according to an embodiment of the present disclosure, where the image alignment method includes the following steps:

401. a first image and a second image of the same shooting scene are acquired.

402. And carrying out binarization processing on the first image to obtain a first binarized image, and carrying out binarization processing on the second image to obtain a second binarized image.

Wherein, only have black pixel and white pixel in the binary image, the grey value of pixel only has two kinds of situations in the binary image promptly: 1 or 0.

In some embodiments, when the first image is subjected to binarization processing to obtain the first binarized image, the electronic device may obtain a gray level median value of a pixel point in the first image; setting the gray of the pixel point with the gray value larger than the gray median value in the first image as one, and setting the gray of the pixel point with the gray value smaller than or equal to the gray median value in the first image as zero to obtain a first binary image. Similarly, the electronic device may obtain the second binary image according to the method described above.

In some embodiments, when the first image is subjected to binarization processing to obtain the first binarized image, the electronic device may obtain a mean value of gray levels of pixel points in the first image; and setting the gray scale of the pixel points with the gray scale value larger than the gray scale mean value in the first image as one, and setting the gray scale of the pixel points with the gray scale value smaller than or equal to the gray scale mean value in the first image as zero to obtain a first binary image. Similarly, the electronic device may obtain the second binary image according to the method described above. Compared with a binarization processing method based on a gray mean value, the binarization processing method based on the gray mean value can save the calculation amount and improve the processing speed.

403. And performing first superposition translation on the second binarized image relative to the first binarized image based on a plurality of preset vectors to determine an alignment vector.

When the second binary image is subjected to first superposition translation relative to the first binary image based on a plurality of preset vectors to determine the alignment vector, the following procedures are executed:

c. and carrying out one-time or continuous multiple down-sampling processing on the first binary image according to one or more preset sampling multiples to obtain a fifth image with one or more sizes, and carrying out one-time or continuous multiple down-sampling processing on the second binary image to obtain a sixth image with one or more sizes.

d. And performing first superposition translation on a sixth image and a fifth image of a next size based on a first superposition translation result of the sixth image and the fifth image of the previous size in the order from small to large until an alignment vector of the second image relative to the first image is obtained, wherein when performing the first superposition translation on the sixth image and the fifth image of each size, performing multiple superposition translations based on a plurality of preset vectors to obtain the result.

e. And combining the preset vectors corresponding to the first superposition translation for multiple times according to one or more preset sampling multiples to obtain the alignment vector.

It should be noted that the above flow is only different from the flow in the above embodiment in terms of implementation objects, and thus the specific implementation process of the above flow may refer to the above embodiment, and is not described herein again.

404. And acquiring a historical alignment vector of a previous frame of image in the current image stream.

405. Determining a common reference direction of the historical alignment vector and the alignment vector, and calculating an alignment component of the alignment vector in the common reference direction.

406. And controlling the second image to perform a second overlay translation relative to the first image according to the alignment component so as to align the second image with the first image.

Referring to fig. 11, fig. 11 is a schematic structural diagram of an image alignment apparatus according to an embodiment of the present disclosure. The image alignment apparatus 500 may include: a first obtaining module 501, a first processing module 502, a second obtaining module 503, a first determining module 504, and a second processing module 505.

A first obtaining module 501, configured to obtain a first image and a second image of the same shooting scene;

a first processing module 502 for performing a first overlay translation of the second image with respect to the first image based on a plurality of preset vectors to determine an alignment vector;

a second obtaining module 503, configured to obtain a historical alignment vector of a previous frame of image in the current image stream;

a first determining module 504, configured to determine a common reference direction of the historical alignment vector and the alignment vector, and calculate an alignment component of the alignment vector in the common reference direction;

a second processing module 505, configured to control the second image to perform a second overlay translation with respect to the first image according to the alignment component, so as to align the second image with the first image.

In some embodiments, after acquiring the first image and the second image of the same shooting scene, the image alignment apparatus 500 may further include:

the third processing module is used for carrying out binarization processing on the first image to obtain a first binarized image and carrying out binarization processing on the second image to obtain a second binarized image;

when performing a first overlay translation of the second image with respect to the first image based on a plurality of preset vectors to determine an alignment vector, the first processing module 502 is further configured to:

performing a first overlay translation of the second binarized image relative to the first binarized image based on a plurality of preset vectors to determine an alignment vector.

In some embodiments, when the first binary image is subjected to the binarization processing to obtain the first binary image, the third processing module may be configured to:

acquiring a gray average value of pixel points in the first image;

setting the gray level of the pixel point with the gray value larger than the gray average value in the first image as one, and setting the gray level of the pixel point with the gray value smaller than or equal to the gray average value in the first image as zero to obtain a first binary image.

In some embodiments, when performing the first overlay translation of the second image relative to the first image based on a plurality of preset vectors to determine the alignment vector, the first processing module 502 may be configured to:

performing multiple overlapping translation on the second image relative to the first image according to a plurality of preset vectors respectively;

and after each superposition translation, calculating a correlation value or a non-correlation value of the second image and the first image, and taking a preset vector corresponding to the maximum correlation value or the minimum non-correlation value as an alignment vector.

In some embodiments, when calculating the correlation or non-correlation of the second image with the first image, the first processing module 502 may be configured to:

carrying out XOR operation on the overlapped area of the second image after being overlapped and translated and the first image;

and calculating a correlation value or a non-correlation value of the overlapped and translated second image and the first image according to the result of the exclusive-or operation.

according to one or more preset sampling multiples, carrying out one-time or continuous multiple down-sampling processing on the first image to obtain a third image with one or more sizes, and carrying out one-time or continuous multiple down-sampling processing on the second image to obtain a fourth image with one or more sizes;

according to the order from small to large in size, performing first superposition translation on a fourth image and a third image of a next size based on a first superposition translation result of the fourth image and the third image of the previous size until an alignment vector of the second image relative to the first image is obtained, wherein when the first superposition translation is performed on the fourth image and the third image of each size, multiple times of superposition translation are performed based on a plurality of preset vectors to obtain the result;

and combining preset vectors corresponding to the first superposition translation for multiple times according to one or more preset sampling multiples to obtain the alignment vector.

In some embodiments, after determining the historical alignment vector and the common reference direction of the alignment vector, the image alignment apparatus 500 may further include:

a second determining module for determining the number of said common reference directions;

a fourth processing module, configured to, when there are multiple common reference directions, control the second image to perform a second overlay translation with respect to the first image according to the alignment vector, so as to align the second image with the first image;

when there is one of the common reference directions, the first determining module 504 performs the calculating of the alignment component of the alignment vector in the common reference direction.

In some embodiments, after obtaining the second image aligned with the first image, the image alignment apparatus 500 may further include:

and the fifth processing module is used for carrying out synthesis processing on the aligned first image and the aligned second image to obtain a synthetic image with a high dynamic range.

In some embodiments, when the current shooting mode is a video recording mode and the historical alignment vector of the previous frame of image in the current image stream is acquired, the second acquiring module 503 may be configured to:

acquiring a historical alignment vector of a previous video frame in a current video stream;

after obtaining the composite image with high dynamic range, the image alignment apparatus 500 may further include:

and the sixth processing module is used for carrying out video coding processing on the synthetic image to obtain a video corresponding to the shooting scene.

As can be seen from the above, in the image alignment apparatus provided in the embodiment of the present application, the first obtaining module 501 obtains the first image and the second image of the same shooting scene, the first processing module 502 performs the first overlay translation on the second image relative to the first image based on a plurality of preset vectors, to determine an alignment vector, a second obtaining module 503 obtains a historical alignment vector of a previous frame image in the current image stream, a first determining module 504 determines a common reference direction of the historical alignment vector and the alignment vector, calculates an alignment component of the alignment vector in the common reference direction, a second processing module 505 controls the second image to move relative to the first image in a superposition manner according to the alignment component so as to perform a second alignment process, and obtains a second image aligned with the first image, the overlapping moving directions of every two adjacent second alignment treatments of the image streams are ensured to be close or the same, and the image streams are prevented from shaking.

Fig. 12 shows a first structural schematic diagram of an electronic device according to an embodiment of the present application. The electronic device 600 includes a processor 601, a memory 602, and a camera assembly 603. The processor 601 is electrically connected to the memory 602 and the camera module 603.

The processor 601 is a control center of the electronic device 600, connects various parts of the whole electronic device using various interfaces and lines, performs various functions of the electronic device 600 by running or loading a computer program stored in the memory 602, and calls data stored in the memory 602, and processes the data, thereby performing overall monitoring of the electronic device 600.

The memory 602 may be used to store software programs and modules, and the processor 601 executes various functional applications and data processing by running the computer programs and modules stored in the memory 602. The memory 602 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, a computer program required for at least one function, and the like; the storage data area may store data created according to use of the electronic device, and the like. Further, the memory 602 may include high speed random access memory, and may also include non-volatile memory, such as at least one magnetic disk storage device, flash memory device, or other volatile solid state storage device. Accordingly, the memory 602 may also include a memory controller to provide the processor 601 with access to the memory 602.

The camera assembly 603 may include Image Processing circuitry, which may be implemented using hardware and/or software components, and may include various Processing units that define an Image Signal Processing (Image Signal Processing) pipeline. The image processing circuit may include at least: a camera, an Image Signal Processor (ISP Processor), control logic, an Image memory, and a display. Wherein the camera may comprise at least one or more lenses and an image sensor. The image sensor may include an array of color filters (e.g., Bayer filters). The image sensor may acquire light intensity and wavelength information captured with each imaging pixel of the image sensor and provide a set of raw image data that may be processed by an image signal processor.

The image signal processor may process the raw image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the image signal processor may perform one or more image processing operations on the raw image data, gathering statistical information about the image data. Wherein the image processing operations may be performed with the same or different bit depth precision. The raw image data can be stored in an image memory after being processed by an image signal processor. The image signal processor may also receive image data from an image memory.

The image Memory may be part of a Memory device, a storage device, or a separate dedicated Memory within the electronic device, and may include a DMA (Direct Memory Access) feature.

When image data is received from the image memory, the image signal processor may perform one or more image processing operations, such as temporal filtering. The processed image data may be sent to an image memory for additional processing before being displayed. The image signal processor may also receive processed data from the image memory and perform image data processing on the processed data in the raw domain and in the RGB and YCbCr color spaces. The processed image data may be output to a display for viewing by a user and/or further processed by a Graphics Processing Unit (GPU). Further, the output of the image signal processor may also be sent to an image memory, and the display may read image data from the image memory. In one embodiment, the image memory may be configured to implement one or more frame buffers.

The statistical data determined by the image signal processor may be sent to the control logic. For example, the statistical data may include statistical information of the image sensor such as auto exposure, auto white balance, auto focus, flicker detection, black level compensation, lens shading correction, and the like.

The control logic may include a processor and/or microcontroller that executes one or more routines (e.g., firmware). One or more routines may determine camera control parameters and ISP control parameters based on the received statistics. For example, the control parameters of the camera may include camera flash control parameters, control parameters of the lens (e.g., focal length for focusing or zooming), or a combination of these parameters. The ISP control parameters may include gain levels and color correction matrices for automatic white balance and color adjustment (e.g., during RGB processing), etc.

Referring to fig. 13, fig. 13 is a schematic structural diagram of an image processing circuit according to an embodiment of the present disclosure. As shown in fig. 13, for convenience of explanation, only aspects of the image processing technique related to the embodiment of the present invention are shown.

For example, the image processing circuitry may include: camera, image signal processor, control logic ware, image memory, display. The camera may include one or more lenses and an image sensor, among others. In some embodiments, the camera may be either a tele camera or a wide camera.

And the first image collected by the camera is transmitted to an image signal processor for processing. After the image signal processor processes the first image, statistical data of the first image (e.g., brightness of the image, contrast value of the image, color of the image, etc.) may be sent to the control logic. The control logic device can determine the control parameters of the camera according to the statistical data, so that the camera can carry out operations such as automatic focusing and automatic exposure according to the control parameters. The first image can be stored in the image memory after being processed by the image signal processor. The image signal processor may also read the image stored in the image memory for processing. In addition, the first image can be directly sent to the display for displaying after being processed by the image signal processor. The display may also read the image in the image memory for display.

In the embodiment of the present application, the processor 601 in the electronic device 600 loads instructions corresponding to one or more processes of the computer program into the memory 602 according to the following steps, and the processor 601 runs the computer program stored in the memory 602, thereby implementing various functions as follows:

acquiring a first image and a second image of the same shooting scene;

Referring to fig. 14, fig. 14 is a second schematic structural diagram of an electronic device according to an embodiment of the disclosure. In some embodiments, the electronic device 700 may further include: a display 704 and a power supply 705. The display 704 and the power source 705 are electrically connected to the processor 701 respectively.

The display 704 may be used to display information entered by or provided to the user as well as various graphical user interfaces that may be composed of graphics, text, icons, video, and any combination thereof.

The power supply 705 may be used to power various components of the electronic device 700. In some embodiments, the power supply 705 may be logically coupled to the processor 701 through a power management system, such that functions to manage charging, discharging, and power consumption management are performed through the power management system.

In this embodiment, the processor 701 in the electronic device 700 loads instructions corresponding to one or more processes of the computer program into the memory 702 according to the following steps, and the processor 701 executes the computer program stored in the memory 702, thereby implementing various functions as follows:

acquiring a first image and a second image of the same shooting scene;

In some embodiments, after acquiring the first image and the second image of the same shooting scene, the processor 701 may further perform:

carrying out binarization processing on the first image to obtain a first binarized image, and carrying out binarization processing on the second image to obtain a second binarized image;

the processor 701 may perform a first overlay translation of the second image with respect to the first image based on a plurality of preset vectors to determine an alignment vector:

In some embodiments, when performing binarization processing on the first image to obtain a first binarized image, the processor 701 may perform:

acquiring a gray average value of pixel points in the first image;

In some embodiments, when performing a first overlay translation of the second image with respect to the first image based on a plurality of preset vectors to determine an alignment vector, the processor 701 may perform:

In some embodiments, when calculating the correlation or non-correlation value of the second image with the first image, the processor 701 may perform:

In some embodiments, after determining the historical alignment vector and the common reference direction of the alignment vector, the processor 701 may further perform:

determining a number of the common reference directions;

performing said calculating an alignment component of said alignment vector in said common reference direction when there is one of said common reference directions;

and when the common reference direction is multiple, controlling the second image to perform second superposition translation relative to the first image according to the alignment vector so as to align the second image with the first image.

In some embodiments, after obtaining the second image aligned with the first image, the processor 701 may further perform:

and synthesizing the aligned first image and the second image to obtain a synthesized image with a high dynamic range.

In some embodiments, the current shooting mode is a video recording mode; in obtaining the historical alignment vector of the previous frame of image in the current image stream, the processor 701 may perform:

after obtaining a composite image with a high dynamic range, the processor 701 may further perform:

and carrying out video coding processing on the synthesized image to obtain a video corresponding to the shooting scene.

As can be seen from the above, after the alignment vector is determined, the electronic device according to the embodiment of the present application, instead of performing the second overlay movement on the second image according to the alignment vector, obtains the historical alignment vector of the previous frame of image in the current image stream, determines the common reference direction between the historical alignment vector and the alignment vector, and then performs the second overlay movement on the second image according to the alignment component of the alignment vector in the common reference direction, so that the overlay movement directions of the second alignment processing every two adjacent times of the image stream are ensured to be close to or the same, and the image stream is prevented from shaking.

An embodiment of the present application further provides a storage medium, where the storage medium stores a computer program, and when the computer program runs on a computer, the computer is caused to execute the image alignment method in any one of the above embodiments, such as: acquiring a first image and a second image of the same shooting scene; performing a first overlay translation of the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector; acquiring a historical alignment vector of a previous frame of image in a current image stream; determining a common reference direction of the historical alignment vector and the alignment vector, and calculating an alignment component of the alignment vector in the common reference direction; and controlling the second image to perform second superposition translation relative to the first image according to the alignment component so as to align the second image with the first image.

In the embodiment of the present application, the storage medium may be a magnetic disk, an optical disk, a Read Only Memory (ROM), a Random Access Memory (RAM), or the like.

In the foregoing embodiments, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

It should be noted that, for the image alignment method of the embodiment of the present application, it can be understood by a person skilled in the art that all or part of the process of implementing the image alignment method of the embodiment of the present application can be completed by controlling the relevant hardware through a computer program, where the computer program can be stored in a computer-readable storage medium, such as a memory of an electronic device, and executed by at least one processor in the electronic device, and during the execution process, the process of the embodiment of the image alignment method can be included. The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, etc.

In the image alignment apparatus according to the embodiment of the present application, each functional module may be integrated into one processing chip, or each module may exist alone physically, or two or more modules are integrated into one module. The integrated module can be realized in a hardware mode, and can also be realized in a software functional module mode. The integrated module, if implemented as a software functional module and sold or used as a stand-alone product, may also be stored in a computer readable storage medium such as a read-only memory, a magnetic or optical disk, or the like.

The foregoing detailed description is directed to an image alignment method, an image alignment apparatus, a storage medium, and an electronic device provided in the embodiments of the present application, and specific examples are applied in the present application to explain the principles and implementations of the present application, and the descriptions of the foregoing embodiments are only used to help understand the method and the core ideas of the present application; meanwhile, for those skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. An image alignment method, comprising:

acquiring a first image and a second image of the same shooting scene;

2. The image alignment method according to claim 1, wherein after acquiring the first image and the second image of the same shooting scene, further comprising:

the first overlay translation of the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector, comprising:

3. The image alignment method according to claim 2, wherein the binarizing the first image to obtain a first binarized image comprises:

acquiring a gray average value of pixel points in the first image;

4. The image alignment method of claim 1, wherein the first superimposed translation of the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector comprises:

5. The image alignment method of claim 4, wherein the calculating a correlation or non-correlation value of the second image with the first image comprises:

6. The image alignment method of claim 1, wherein said first superimposed translation of the second image relative to the first image based on a plurality of preset vectors to determine an alignment vector comprises:

7. The image alignment method of claim 1, wherein after determining the common reference direction of the historical alignment vector and the alignment vector, further comprising:

determining a number of the common reference directions;

8. The image alignment method according to any one of claims 1 to 7, further comprising, after obtaining the second image aligned with the first image:

9. The image alignment method according to claim 8, wherein the current shooting mode is a video recording mode; the obtaining of the historical alignment vector of the previous frame of image in the current image stream includes:

after the obtaining of the composite image with high dynamic range, the method further includes:

10. An image alignment apparatus, comprising:

11. A storage medium having stored thereon a computer program, characterized in that, when the computer program is run on a computer, it causes the computer to execute the image alignment method according to any one of claims 1 to 9.

12. An electronic device comprising a processor, a memory, said memory having a computer program, wherein said processor is adapted to perform the image alignment method of any of claims 1 to 9 by invoking said computer program.