WO2019242757A1

WO2019242757A1 - Imaging shake compensation method and device, and storage medium

Info

Publication number: WO2019242757A1
Application number: PCT/CN2019/092346
Authority: WO
Inventors: 吴永辉; 仇国润
Original assignee: 中兴通讯股份有限公司
Priority date: 2018-06-22
Filing date: 2019-06-21
Publication date: 2019-12-26
Also published as: CN110636203A

Abstract

Provided in the present disclosure are an imaging shake compensation method and device, and a computer readable storage medium. The method comprises: detecting whether a shake amount of a mobile terminal at the moment that the shutter responds is greater than a preset threshold; in response to a detection result indicating that the shake amount is greater than the preset threshold, detecting a direction in which a maximum shake amount of the mobile terminal at the moment that the shutter responds is detected; selecting a corresponding intervalled output mode for a pixel array of an imaging sensor according to the direction in which the maximum shake amount of the mobile terminal is detected so as to acquire a first original image and a second original image at the moment that the shutter responds; and compositing the first original image and the second original image to generate a target image.

Description

Method, device and storage medium for image shake compensation

Technical field

The present disclosure relates to, but is not limited to, the field of image processing technology.

Background technique

The camera function of mobile terminals is more and more concerned by consumers, and the quality of the camera function has become an important indicator for measuring the performance of mobile terminals. The sharpness of the captured image can be used to evaluate the quality of the camera function of the mobile terminal.

The image sharpness mainly depends on the performance of the imaging module itself and the optimization ability of the later image processing. In practical applications, in many scenes, such as scenes with low light, scenes in motion, scenes that capture moving objects, etc., when shooting, it is prone to imaging problems such as screen shake and smear. In order to solve the image smear caused by shake when shooting, optical image stabilization technology is usually used. Optical image stabilization needs to be implemented by adding hardware components, and the cost is relatively high.

Summary of the Invention

According to an embodiment of the present disclosure, a method for image shake compensation is provided, including: detecting whether a shake amount of a mobile terminal at a shutter response time is greater than a preset threshold; and in response to the shake amount being greater than the preset threshold, detecting a The maximum jitter direction of the mobile terminal at the shutter response time; according to the maximum jitter direction of the mobile terminal, selecting a corresponding interval output method for the pixel array of the image sensor to obtain the first frame of the original image at the shutter response time And the second frame of the original image; and performing image synthesis on the first frame of the original image and the second frame of the original image to generate a target image

According to an embodiment of the present disclosure, an apparatus for image shake compensation is further provided, which includes a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the processor performs Disclosed method for image shake compensation.

According to an embodiment of the present disclosure, there is also provided a computer-readable storage medium on which one or more computer programs are stored. When the one or more computer programs are executed by one or more processors, the one or more Processors execute a method of image shake compensation according to the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of a method for image shake compensation according to an embodiment of the present disclosure.

FIG. 2 is a structural block diagram of a mobile terminal according to an embodiment of the present disclosure.

FIG. 3 is a flowchart of step S110 of the method for image shake compensation shown in FIG. 1.

FIG. 4 shows a flowchart of step S120 of the method for image shake compensation shown in FIG. 1.

FIG. 5 shows a flowchart of step S130 of the method for image shake compensation shown in FIG. 1.

FIG. 6 shows a flowchart of step S140 of the method for image shake compensation shown in FIG. 1.

FIG. 7 is a structural block diagram of an apparatus for image shake compensation according to an embodiment of the present disclosure.

The realization, functional characteristics and advantages of the purpose of the present disclosure will be further described with reference to the embodiments and the drawings.

detailed description

It should be understood that the specific embodiments described herein are only used to explain the present disclosure and are not intended to limit the present disclosure.

As shown in FIG. 1, a method for image shake compensation according to an embodiment of the present disclosure includes steps S110 to S140:

In step S110, it is detected whether the amount of shake of the mobile terminal at the shutter response time is greater than a preset threshold.

In step S120, in response to the amount of shake being greater than a preset threshold, detecting a maximum shake direction of the mobile terminal at the shutter response time.

In step S130, according to the maximum jitter direction of the mobile terminal, a corresponding interval output method is selected for the pixel array of the image sensor to obtain the first frame original image and the second frame original image at the shutter response time.

In step S140, image synthesis is performed on the first frame original image and the second frame original image to generate a target image.

As shown in FIG. 2, the mobile terminal 100 includes a camera 110, an image sensor 120, a gyroscope 130, an image processor 140, and a control unit 150.

The camera 110 is used to capture images during shooting.

The image sensor 120 may be a Complementary Metal Oxide Semiconductor (CMOS) sensor, or a Charge Coupled Device (CCD) sensor. The image sensor 120 may include a plurality of rows and a plurality of columns of pixels arranged in an array.

The gyroscope 130 is used to detect changes in the azimuth and angle of the mobile terminal 100 in real time. The gyroscope 130 can detect in the directions of the three axes of X, Y, and Z, and record the value at time T and the value at time T + 1 in each axis direction. Time T is the previous time when the shutter is clicked to take a picture (ie, the shutter response time), and time T + 1 is the time when the shutter responds.

The image processor 140 may process and synthesize multiple frames of images, and convert the processed and synthesized images in the original data format to obtain images in the target data format, and process the images in the target data format again (for example, automatic white balance). , Auto exposure, auto focus, etc.) to get the target image.

The control unit 150 may control the interval output mode of the pixel array of the image sensor 120. For example, the gyroscope 130 measures the horizontal change amount (or horizontal shake amount) and vertical change amount (or vertical shake amount) from time T to time T + 1. When the horizontal change amount is greater than the vertical change amount, control The unit 150 may control the image sensor 120 to operate in a column interval output mode; when the horizontal variation is less than the vertical variation, the control unit 150 may control the image sensor 120 to operate in a row interval output mode.

The preset threshold in the above step S120 may be determined according to the accuracy of the image shake compensation. The preset threshold is used to determine whether a shake that occurs when a user uses the mobile terminal 100 for shooting reaches a degree that affects the quality of the picture. If the amount of shake of the mobile terminal 100 is less than or equal to a preset threshold at the shutter response time, it indicates that the shake that occurs is not sufficient to affect the quality of the mobile terminal 100's photographing and does not need to compensate for this image shake; if the mobile terminal 100 is at the shutter response time If the amount of jitter is greater than a preset threshold, it indicates that the occurrence of the jitter will affect the quality of the mobile terminal 100 taking pictures, and image jitter compensation is required.

As shown in FIG. 3, step S110 shown in FIG. 1 may include steps S111 to S113.

In step S111, the first orientation information of the mobile terminal at the shutter response time and the second orientation information of the mobile terminal at the moment immediately before the shutter response are measured by the gyroscope.

In step S112, the amount of position change between the first orientation information and the second orientation information is calculated to obtain a shake amount of the mobile terminal at the shutter response time.

In step S113, it is determined whether the amount of shake of the mobile terminal at the shutter response time is greater than a preset threshold.

Referring to FIG. 2, the shake amount of the mobile terminal at the shutter response time can be measured by the gyroscope 130 and output to the control unit 150. The control unit 150 may determine whether the amount of shake of the mobile terminal at the shutter response time is greater than a preset threshold.

If the amount of shake of the mobile terminal at the shutter response time is greater than a preset threshold, it indicates that the occurrence of shake will affect the quality of the mobile terminal's photographing. It is necessary to further detect the maximum shake direction of the mobile terminal at the shutter response time, and according to the maximum shake direction of the mobile terminal To choose the appropriate image shake compensation method.

Although the gyroscope 130 detects the azimuth change of the mobile terminal 100 in the directions of the three axes of X, Y, and Z, in actual shooting, the shake is usually sent in the X-axis direction (horizontal direction) or the Y-axis direction (vertical direction). on. It should be recognized that the “horizontal direction” and the “vertical direction” described herein are directions that refer to the pixel array of the image sensor 120, that is, the horizontal direction is a direction parallel to the row direction of the pixel array of the image sensor 120, and vertical The straight direction is a direction parallel to the column direction of the pixel array of the image sensor 120.

As shown in FIG. 4, step S120 shown in FIG. 1 may include steps S121 to S122.

In step S121, in response to the amount of shake being greater than a preset threshold, obtaining a horizontal shake amount and a vertical shake amount of the mobile terminal at the shutter response time.

In step S122, the horizontal shake amount and the vertical shake amount are compared. In response to the horizontal shake amount being greater than the vertical shake amount, it is determined that the maximum jitter direction of the mobile terminal is the horizontal direction, and in response to the vertical shake amount being greater than the horizontal shake amount, the movement is determined. The maximum jitter direction of the terminal is the vertical direction.

When the control unit 150 determines that the amount of shake of the mobile terminal 100 at the shutter response time is greater than a preset threshold, the control unit 150 further obtains the amount of shake of the mobile terminal 100 in the X-axis direction when the shutter responds through the gyro 130 (i.e. , Horizontal jitter amount) and the amount of jitter in the Y-axis direction (that is, the amount of vertical jitter), that is, the amount of displacement of the mobile terminal 100 in the X-axis direction from time T to time T + 1, and the mobile terminal 100 The amount of displacement in the Y-axis direction from time T to time T + 1. If the amount of jitter in the X-axis direction is greater than the amount of jitter in the Y-axis direction, it is determined that the maximum jitter direction of the mobile terminal 100 is a horizontal direction, and if the amount of jitter in the Y-axis direction is greater than the amount of jitter in the X-axis direction , It is determined that the maximum jitter direction of the mobile terminal 100 is a vertical direction.

After detecting the maximum shake direction of the mobile terminal 100 at the shutter response time through the above step S120, a corresponding interval output method may be selected for the pixel array of the image sensor 120 according to the maximum shake direction of the mobile terminal 100.

As shown in FIG. 5, step S130 shown in FIG. 1 may include steps S131 to S132.

In step S131, in response to determining that the maximum jitter direction of the mobile terminal is a horizontal direction, a column interval output method is selected for the pixel array of the image sensor.

In step S132, in response to determining that the maximum jitter direction of the mobile terminal is a vertical direction, a line interval output mode is selected for the pixel array of the image sensor.

If it is determined that the maximum jitter direction of the mobile terminal 100 is a horizontal direction, that is, a direction parallel to the row direction of the pixel array of the image sensor 120, the number D of column intervals may be selected according to the numerical interval where the horizontal jitter amount is located, where D is a positive integer, And the adjacent D column pixels are used as the processing unit.

If it is determined that the maximum jitter direction of the mobile terminal 100 is a vertical direction, that is, a direction parallel to the column direction of the pixel array of the image sensor 120, the number of row intervals d can be selected according to the numerical interval where the vertical jitter amount is located, where d is positive An integer, and the adjacent d-row pixels are used as a processing unit.

In the column interval output mode, each processing unit in the pixel array of the image sensor 120 consisting of pixels from the (N-1) * D + 1th column to the N * Dth column is selected as the first frame original image, where , N is a positive and odd number starting from 1, and each processing unit composed of pixels from the (M-1) * D + 1th column to the M * Dth column in the pixel array of the image sensor 120 is selected as the second frame The original image, where M is a positive even number starting from 2.

When the amount of horizontal jitter is greater than the amount of vertical jitter, the maximum jitter direction of the mobile terminal 100 is the horizontal direction, and the image sensor 120 works in a column interval output mode, that is, when the number D of column intervals is 1, the first frame is original. The image is composed of pixels in columns 1, 3, 5, 7 ... and the second frame of the original image is composed of pixels in columns 2, 4, 6, 8 ... According to the embodiment of the present disclosure, the value of the horizontal jitter amount may be divided into multiple numerical intervals, and the corresponding number of column intervals D is selected according to the numerical interval where the horizontal jitter amount is located. When the horizontal jitter amount is large, the image sensor 120 may The output is in the form of 2 columns, that is, when the number of column intervals D is 2, the first frame of the original image is composed of pixels in the 1, 2, 5, 6, 9, 10, ... columns, and the second frame of the original image Consists of pixels in the 3rd, 4th, 7th, 8th, 11th, 12th ... columns. The larger the value of the horizontal jitter amount, the larger the number D of column intervals.

In the line interval output mode, each processing unit in the pixel array of the image sensor 120 consisting of pixels from the (n-1) * d + 1th row to the pixels in the n * dth row is selected as the first frame original image, where , N is a positive and odd number starting from 1, and each processing unit composed of pixels from the (m-1) * d + 1th row to the m * dth column in the pixel array of the image sensor 120 is selected as the second frame The original image, where m is a positive even number starting from 2.

When the amount of horizontal jitter is less than the amount of vertical jitter, the maximum jitter direction of the mobile terminal 100 is the vertical direction, and the image sensor 120 works in a line interval output mode, that is, in the case where the number of line intervals d is 1, the first frame The original image is composed of pixels in rows 1, 3, 5, 7, ..., and the second frame of original image is composed of pixels in rows 2, 4, 6, 8, .... According to the embodiment of the present disclosure, the value of the vertical shake amount can be divided into multiple numerical intervals, and the corresponding number of line intervals d is selected according to the numerical interval where the vertical shake amount is located. When the vertical shake amount is large, the image The sensor 120 can output in two-line intervals, that is, when the number d of line intervals is two, the first frame of the original image is composed of pixels of the first, second, fifth, sixth, nineth, tenth, ... lines, and the second The original frame image consists of the 3rd, 4th, 7th, 8th, 11th, 12th ... pixels. The larger the value of the vertical jitter amount, the larger the number of line intervals d.

After the image processor 140 acquires the first frame original image and the second frame original image at the shutter response time, the first frame original image and the second frame original image are processed.

As shown in FIG. 6, step S140 shown in FIG. 1 may include steps S141 to S142.

In step S141, image synthesis is performed on the first frame original image and the second frame original image by the image processor to obtain an image in a raw data format.

In step S142, the image in the original data format is converted into the image in the target data format, and the image in the target data format is subjected to image processing to generate a target image.

As shown in FIG. 7, an image shake compensation apparatus 20 according to an embodiment of the present disclosure includes a memory 21 and a processor 22. A computer program is stored in the memory 21. The memory 21 and the processor 22 are connected through a data bus 23. When the computer program stored on the memory 21 is executed by the processor 22, the processor 22 may execute a method of image shake compensation according to various embodiments of the present disclosure.

An embodiment of the present disclosure also provides a computer-readable storage medium on which one or more computer programs are stored. When the one or more computer programs are executed by one or more processors, the one or more processes are processed. The processor performs a method of image shake compensation according to various embodiments of the present disclosure.

According to the image shake compensation method of the embodiment of the present disclosure, in a pixel array of an image sensor, an original image is divided into a first frame original image and a second frame original image by interlacing or interlacing, that is, the first frame original image is composed. There is a gap between all the row pixels or column pixels of. And there is also a gap between all the row pixels or column pixels that make up the second frame of the original image. This can avoid the occurrence of jitter due to the occurrence of adjacent row pixels or adjacent column pixels. Smear caused by overlapping. Then, only the first frame of the original image and the second frame of the original image are synthesized to produce a clear and complete target image. According to the technical solution of the present disclosure, it is possible to effectively solve the image smear phenomenon caused by dithering when dithering occurs without adding additional hardware.

The embodiments of the present disclosure have been described above with reference to the accompanying drawings, but the present disclosure is not limited to the specific implementations described above, and the specific implementations described above are only schematic and not restrictive. Those of ordinary skill in the art at Under the inspiration of the present disclosure, many forms can be made without departing from the scope of the present disclosure and the scope of protection of the claims, and these all fall into the protection of the present disclosure.

Claims

A method for image shake compensation includes:

Detecting whether the amount of shake of the mobile terminal at a shutter response time is greater than a preset threshold;

In response to the amount of jitter being greater than the preset threshold, detecting a maximum jitter direction of the mobile terminal at the shutter response time;

Selecting a corresponding interval output method for the pixel array of the image sensor according to the maximum jitter direction of the mobile terminal to obtain a first frame original image and a second frame original image at the shutter response time; and

Image synthesis is performed on the first frame original image and the second frame original image to generate a target image.
The method for image shake compensation according to claim 1, wherein the step of detecting whether the amount of shake of the mobile terminal at a shutter response time is greater than a preset threshold comprises:

Measuring, by a gyroscope, first orientation information of the mobile terminal at the shutter response time and second orientation information of the mobile terminal at a moment before the shutter response;

Calculating a position change amount between the first position information and the second position information to obtain a shake amount of the mobile terminal at the shutter response time; and

Determining whether the amount of shake of the mobile terminal at the shutter response time is greater than the preset threshold.
The method for image shake compensation according to claim 1, wherein in response to the amount of shake being greater than the preset threshold, the step of detecting a maximum shake direction of the mobile terminal at the shutter response time comprises:

Acquiring a horizontal shake amount and a vertical shake amount of the mobile terminal at the shutter response time;

Comparing the amount of horizontal jitter with the amount of vertical jitter;

In response to the horizontal jitter amount being greater than the vertical jitter amount, determining that a maximum jitter direction of the mobile terminal is a horizontal direction; and

In response to the vertical jitter amount being greater than the horizontal jitter amount, it is determined that the maximum jitter direction of the mobile terminal is a vertical direction.
The method for image shake compensation according to claim 3, wherein according to a maximum shake direction of the mobile terminal, a corresponding interval output mode is selected for a pixel array of an image sensor to obtain a first frame at the shutter response time The steps for the original image and the second frame of original image include:

In response to determining that the maximum jitter direction of the mobile terminal is a horizontal direction, a column interval output method is selected for a pixel array of the image sensor.
The method for image shake compensation according to claim 4, wherein in response to determining that the maximum shake direction of the mobile terminal is a horizontal direction, the step of selecting a column interval output mode for a pixel array of the image sensor comprises:

Selecting the number D of column intervals according to a numerical interval where the horizontal jitter amount is located, where D is a positive integer, and using adjacent pixels in the D column as a processing unit;

Selecting each processing unit composed of pixels from the (N-1) * D + 1th column to the N * Dth column in the pixel array of the image sensor as the first frame of the original image, where N is from 1 Positive odd number of;

Selecting each processing unit composed of pixels from the (M-1) * D + 1th column to the M * Dth column in the pixel array of the image sensor as the second frame original image, where M is from 2 Positive even number.
The method for image shake compensation according to claim 3, wherein according to a maximum shake direction of the mobile terminal, a corresponding interval output mode is selected for a pixel array of an image sensor to obtain a first frame at the shutter response time The steps for the original image and the second frame of original image include:

In response to determining that the maximum jitter direction of the mobile terminal is a vertical direction, a line interval output mode is selected for a pixel array of the image sensor.
The method for image shake compensation according to claim 6, wherein in response to judging that a maximum shake direction of the mobile terminal is a vertical direction, the step of selecting a line interval output mode for a pixel array of the image sensor comprises:

Selecting the number of row intervals d according to a numerical interval in which the vertical jitter amount is located, where d is a positive integer, and using adjacent d row pixels as a processing unit;

Selecting each processing unit consisting of pixels from the (n-1) * d + 1th row to the pixels from the n * dth row in the pixel array of the image sensor as the first frame of the original image, where n is from 1 Positive odd number of;

Selecting each processing unit consisting of pixels from the (m-1) * d + 1th row to the m * dth row of the pixel array of the image sensor as the second frame of the original image, where m is from 2 Positive even number.
The method for compensating image shake according to claim 1, wherein the step of generating a target image by performing image synthesis on the first frame original image and the second frame original image comprises:

Performing image synthesis on the first frame original image and the second frame original image by an image processor to obtain an image in a raw data format; and

Converting the image in the original data format into an image in the target data format, and performing image processing on the image in the target data format to generate the target image.
An image shake compensation device includes a memory and a processor, and a computer program is stored on the memory. When the computer program is executed by the processor, the processor executes any one of claims 1 to 8. The method for image shake compensation.
A computer-readable storage medium having stored thereon one or more computer programs, said one or more computer programs being executed by one or more processors, said one or more processors executing as claimed in claims 1 to The method of image shake compensation according to any one of 8.