CN110636203A - Image jitter compensation method, device and storage medium - Google Patents
Image jitter compensation method, device and storage medium Download PDFInfo
- Publication number
- CN110636203A CN110636203A CN201810648471.1A CN201810648471A CN110636203A CN 110636203 A CN110636203 A CN 110636203A CN 201810648471 A CN201810648471 A CN 201810648471A CN 110636203 A CN110636203 A CN 110636203A
- Authority
- CN
- China
- Prior art keywords
- mobile terminal
- image
- axis
- shaking
- amount
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
Abstract
The invention discloses a method and a device for compensating image jitter and a storage medium, belonging to the technical field of image processing. The method comprises the following steps: detecting whether the shaking amount of the mobile terminal at the shutter response moment is greater than a preset threshold value or not; if the shake amount is larger than a preset threshold value, further detecting the maximum shake direction of the mobile terminal at the shutter response moment; selecting a corresponding interval output mode for a pixel array of an image sensor according to the maximum shaking direction of the mobile terminal to acquire two frames of original images before and after the shutter response time; and carrying out image synthesis on the original images of the front frame and the back frame to generate a final target image. According to the technical scheme, extra hardware is not required to be added, and the image smear phenomenon caused by shaking during shooting can be effectively avoided when the mobile terminal shakes during shooting.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method and an apparatus for compensating image jitter, and a storage medium.
Background
As the photographing function of the mobile terminal is more and more paid attention and favored by consumers, the quality of the photographing function becomes an important index for measuring the performance of the mobile terminal. The quality of the photographing function of the camera is mainly measured according to the definition of the photographed image, and the definition is very important when the quality of one image is good or bad. The image definition mainly depends on the optimization of the performance of the module body and the later image processing, and certainly, a photographer is ensured to take a stable picture. However, in real life, a photographer/user cannot shoot a stable image (without shaking the image), and imaging problems such as image shaking and smear are likely to occur in many scenes, such as dark scenes, sports (buses and the like), and capturing children and the like. In order to solve image smear caused by shake during shooting, an optical anti-shake technology is generally adopted at present, optical anti-shake is realized by adding hardware components, and the input cost is higher.
Disclosure of Invention
The invention mainly aims to provide a method, a device and a storage medium for image shake compensation, aiming at effectively solving the image smear phenomenon caused by shake during shooting without adding extra hardware when the shake occurs during shooting of a mobile terminal.
To achieve the above object, an embodiment of the present invention provides a method for compensating image shake, including the steps of: detecting whether the shaking amount of the mobile terminal at the shutter response moment is greater than a preset threshold value or not; if the jitter amount is larger than the preset threshold value, further detecting the maximum jitter direction of the mobile terminal at the shutter response time; selecting a corresponding interval output mode for a pixel array of an image sensor according to the maximum shaking direction of the mobile terminal so as to obtain two frames of original images before and after the shutter response time; synthesizing the front and back frames of original images to generate a final target image
To achieve the above object, an embodiment of the present invention further provides an apparatus for image shake compensation, which includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program implements the steps of the foregoing method when executed by the processor.
To achieve the above object, the present invention provides a storage medium for a computer-readable storage, the storage medium storing one or more programs, the one or more programs being executable by one or more processors to implement the steps of the aforementioned method.
The invention provides a method, a device and a storage medium for compensating image shake, which detect whether the shake quantity of a mobile terminal at the shutter response time is larger than a preset threshold value, and further detect the maximum shake direction of the mobile terminal at the shutter response time when the shake quantity is larger than the preset threshold value. And selecting a corresponding interval output mode for the pixel array of the image sensor according to the maximum shaking direction of the mobile terminal so as to acquire two frames of original images before and after the shutter response time. And finally, carrying out image synthesis on the original images of the front frame and the back frame to generate a final target image. In this way, each row of pixels arranged at intervals or each column of pixels arranged at intervals in the pixel array of the image sensor form the original image of the previous frame, that is, intervals exist among all the row pixels or the column pixels forming the original image of the previous frame, and therefore, the phenomenon of smear caused by overlapping of adjacent row pixels or adjacent column pixels when shooting shakes can be avoided. All the remaining row pixels or all the remaining column pixels in the pixel array of the image sensor form a next frame of original image, namely, intervals also exist among all the row pixels or the column pixels forming the next frame of original image, and the phenomenon of smear caused by overlapping of adjacent row pixels or adjacent column pixels when shooting shakes can also be avoided. And finally, only image synthesis is needed to be carried out on the original images of the front frame and the back frame, and a clear and complete target image can be generated. Therefore, according to the technical scheme, additional hardware is not required to be added, and the image smear phenomenon caused by shaking in shooting can be effectively solved when the mobile terminal shakes in shooting.
Drawings
Fig. 1 is a flowchart of a method for compensating image jitter according to an embodiment of the present invention.
Fig. 2 is a block diagram of the mobile terminal of the present invention.
Fig. 3 is a first schematic diagram of a pixel array of an image sensor according to the present invention.
Fig. 4 is a schematic diagram of a pixel array of the image sensor of the invention.
Fig. 5 is a detailed flowchart of the method step S110 of image shake compensation shown in fig. 1.
Fig. 6 is a detailed flowchart of the method step S120 of image shake compensation shown in fig. 1.
Fig. 7 is a detailed flowchart of the method step S130 of image shake compensation shown in fig. 1.
Fig. 8 is a schematic diagram of a column interval output mode of a pixel array of an image sensor according to the present invention.
FIG. 9 is a schematic diagram of a row spacing output of a pixel array of an image sensor according to the present invention
Fig. 10 is a detailed flowchart of the method step S140 of image shake compensation shown in fig. 1.
Fig. 11 is a block diagram of an apparatus for compensating image shake according to a second embodiment of the present invention.
The implementation, functional features and advantages of the objects of the present invention will be further explained with reference to the accompanying drawings.
Detailed Description
It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
In the following description, suffixes such as "module", "part", or "unit" used to denote elements are used only for facilitating the explanation of the present invention, and have no peculiar meaning in itself. Thus, "module", "component" or "unit" may be used mixedly.
Example one
As shown in fig. 1, the present embodiment provides a method of image shake compensation, the method including the steps of:
step S110: and detecting whether the shaking amount of the mobile terminal at the shutter response moment is larger than a preset threshold value.
Specifically, the method for image shake compensation of the present invention is mainly implemented based on the mobile terminal 100 as shown in fig. 2, and the mobile terminal 100 mainly includes a camera 110, an image sensor 120, a gyroscope 130, an ISP image processor 140 and a control unit 150.
The camera 110 is mainly used for acquiring images during shooting.
The image sensor 120 may be a CMOS (ComplementarY Metal OXide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor, the number of pixels of the image sensor 120 may be 500, 800, 1000, 1200, 1300, 1600, etc., and the pixel array of the image sensor 100 may be 4: 3 or 16: 9 general arrangement, as shown in fig. 3, taking 1300 ten thousand pixels as an example, the pixel array is composed of single pixels of three primary colors of red, green and blue, and each pixel can be operated independently and driven by a control circuit under each pixel. Coordinates are piXel (0,0) to piXel (4096,3072), pixels are arranged in columns and columns, pixels are arranged in rows and rows, as shown in fig. 4, four squares are unit pixels, each unit piXel (unit piXel) is composed of two green pixels, one red piXel and one blue piXel, coordinates are unit piXel (0,0) to unit piXel (2048,1536), a conventional exposure mode of the CMOS sensor is performed in a row-by-row mode, and the exposure time of any one piXel (piXel) is the same, that is, the exposure starting point and the exposure time of each piXel (piXel) on the same row are the same, but the EV value of AE (automatic exposure) can be controlled by software in combination with a hardware circuit. The CMOS sensor is independent of each unit pixel control circuit, and the working mode of the conventional sensor is changed from progressive scanning to interlaced scanning by controlling the timing of each line of pixels AE (automatic exposure).
The gYroscope (gYroscope)120 is on the mobile terminal 100, and mainly functions to detect the variation of the orientation and angle of the mobile terminal 100 in real time, and the gYroscope 130 detects from three axes of X, Y and Z, and records the time value of each axial T and the time value of T +1, unit: rad/s, time T is defined as the previous time when the shutter is clicked to photograph after the camera APP is opened (i.e. shutter response), i.e. the time of acquiring the previous frame of original image mentioned later, and time T +1 is the time of shutter response, i.e. the time of acquiring the next frame of original image mentioned later.
The ISP (Image Signal Processing) Image processor 130 mainly processes and synthesizes a plurality of frames of images in the Image sensor 120 to obtain an nth frame Image and an N +1 th frame Image at the time of T +1, a camera real-time preview window is an N frame Image, next, the RAW data format of the Image sensor 120 is converted into JPEG or other formats, next, the Image of the Image sensor 120 is processed by 3A (3A: auto white balance AWB, auto exposure AE, auto focus AF), and finally, a clear, normally color-restored, normally exposed Image is output.
The control unit 150 mainly controls an interval output manner of the pixel array of the image sensor 120, and the image sensor 120 operates in an interval operation for each column of pixels when the X-axis variation > the Y-axis variation, depending on the X-axis and Y-axis variations measured by the gyroscope 130. When the X-axis variation is less than the Y-axis variation, the image sensor works in a mode of working at intervals of each row of pixels.
Based on the mobile terminal 100 shown in fig. 2, the method further introduces a concept of a preset threshold, where the preset threshold may be determined according to the accuracy of actually performing image shake compensation, and the preset threshold is used to determine whether the current shake occurring when the user uses the mobile terminal 100 to perform shooting has reached a degree that affects the quality of shooting, that is, if the shake amount of the mobile terminal at the shutter response time is less than or equal to the preset threshold, it indicates that the current shake occurring is not enough to affect the quality of shooting by the mobile terminal, the image sensor 120 may adopt a full-pixel output mode without performing subsequent steps of the method for image shake compensation, and if the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold, it indicates that the current shake occurring may affect the quality of shooting by the mobile terminal, and subsequent steps of the method for image shake compensation are required, the image smear phenomenon caused by shaking during shooting is effectively solved. Therefore, the step of detecting whether the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold is particularly important, as shown in fig. 5, the overall process specifically includes:
step S111: and measuring first orientation information of the mobile terminal at the shutter response moment and second orientation information of the mobile terminal at the moment before the shutter response moment through a gyroscope.
Step S112: and comparing the position variation between the first azimuth information and the second azimuth information to acquire the shake quantity of the mobile terminal at the shutter response time.
Step S113: and judging whether the shaking amount of the mobile terminal at the shutter response moment is larger than a preset threshold value.
Through the above method process, the shake amount of the mobile terminal at the shutter response time may be measured by the gyroscope 130 and output to the control unit 40, so as to determine whether the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold through the control unit 150, and when the shake amount is greater than the preset threshold, the following subsequent steps may be performed.
Step S120: and if the shaking amount is larger than the preset threshold, further detecting the maximum shaking direction of the mobile terminal at the shutter response time.
Specifically, based on the above description, if the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold, it is indicated that the currently occurring shake may affect the quality of photographing of the mobile terminal, and a maximum shake direction of the mobile terminal at the shutter response time needs to be further detected, so as to select an appropriate technical means for image shake compensation according to the maximum shake direction of the mobile terminal.
Since the gYroscope (gYroscope)120 detects the orientation variation of the mobile terminal 100 in real time from the three axes X, Y, and Z, and most of the Y axis or X axis shakes during the actual shooting, the present invention compares only the shake amount of the X axis with the shake amount of the Y axis to detect the maximum shake direction of the mobile terminal at the shutter response time. As shown in fig. 6, the overall process specifically includes:
step S121: and if the shake amount is larger than the preset threshold, further acquiring the X-axis shake amount and the Y-axis shake amount of the mobile terminal at the shutter response time.
Step S122: comparing the X-axis shaking amount with the Y-axis shaking amount, if the X-axis shaking amount is larger than the Y-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the X-axis direction, and if the Y-axis shaking amount is larger than the X-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the Y-axis direction.
In this way, when the control unit 150 determines that the amount of shake of the mobile terminal at the shutter response time is greater than the preset threshold, the control unit 150 further acquires the X-axis and Y-axis shaking amounts of the mobile terminal at the shutter response time through the gyro 110, namely, the displacement of the mobile terminal occurring in the X-axis direction between the time T and the time T +1 is obtained, and the displacement of the mobile terminal between the T time and the T +1 time in the Y-axis direction, if the X-axis shaking amount is larger than the Y-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the X-axis direction, if the Y-axis shaking amount is greater than the X-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the Y-axis direction, here, the X-axis direction is a direction parallel to the row direction of the pixel array of the image sensor 100, and the Y-axis direction is a direction parallel to the column direction of the pixel array of the image sensor 100.
Step S130: and selecting a corresponding interval output mode for the pixel array of the image sensor according to the maximum shaking direction of the mobile terminal so as to acquire two frames of original images before and after the shutter response time.
Specifically, when the maximum shaking direction of the mobile terminal at the shutter response time is detected in step S120, a corresponding interval output mode may be selected for the pixel array of the image sensor according to the maximum shaking direction of the mobile terminal, so as to obtain two frames of original images before and after the shutter response time. As shown in fig. 7, the overall process specifically includes:
step S131: and if the maximum shaking direction of the mobile terminal is judged to be the X-axis direction, selecting a row interval output mode for the pixel array of the image sensor to acquire two frames of original images before and after the shutter response time.
Specifically, if the maximum jitter direction of the mobile terminal is determined to be the X-axis direction, the number D of column intervals is selected according to the numerical interval in which the magnitude of the X-axis jitter amount falls. And selecting each processing unit of a previous frame original image composed of ND-D +1, ND-D +2, … …, ND-1 and ND column pixels in a pixel array of the image sensor by taking adjacent D column pixels as a processing unit, and selecting each processing unit of a next frame original image composed of MD-D +1, MD-D +2, … …, MD-1 and MD column pixels in the pixel array of the image sensor to acquire two frames of original images before and after the shutter response moment, wherein D is a natural number which is not zero, N is a positive odd number, and M is a positive even number.
That is, when the X-axis jitter amount is greater than the Y-axis jitter amount, it is determined that the maximum jitter direction of the mobile terminal is the X-axis direction, a column interval output mode is selected for the pixel array of the image sensor 120, and the operation mode of the image sensor 120 is that each column of pixels operates at intervals; that is, starting from the origin pixel of the image sensor 120, as shown in fig. 8, the previous frame original image is composed of the 1 st, 3 rd, 5 th, and 7 … 2047 th works of the pixel column, and the next frame original image is composed of the 2 nd, 4 th, 6 th, and 8 … 2048 th works of the pixel column. Furthermore, a plurality of numerical value intervals can be divided from possible values of the X-axis shaking amount, and the number D of the corresponding column intervals is selected according to the numerical value interval in which the X-axis shaking amount falls. That is, when the X-axis jitter amount changes greatly, the output pattern of the image sensor 120 is output every two or more columns, for example: every 2 columns are output, every adjacent 2 columns of pixels are a processing unit, the original image of the previous frame is formed by the 1 st, 2 nd, 5 th, 6 th, … th, 2045 th and 2046 th column works of pixel column, the original image of the next frame is formed by the 3 rd, 4 th, 7 th, 8 th, … th, 2047 th and 2048 th column works of pixel column, and the larger the value interval in which the X-axis shaking amount falls, the larger the number D of corresponding selected column intervals is.
Step S132: and if the maximum shaking direction of the mobile terminal is judged to be the Y-axis direction, selecting a line interval output mode for the pixel array of the image sensor to acquire two frames of original images before and after the shutter response time.
Specifically, if the maximum jitter direction of the mobile terminal is determined to be the Y-axis direction, the number d of line intervals is selected according to the numerical interval in which the Y-axis jitter amount falls. And selecting nd-d +1, nd-d +2, … …, nd-1 and nd column pixels in a pixel array of the image sensor as a processing unit to form each processing unit of a previous frame of original image, selecting md-d +1, md-d +2, … …, md-1 and md column pixels in the pixel array of the image sensor to form each processing unit of a next frame of original image, so as to obtain two frames of original images before and after the shutter response moment, wherein d is a natural number which is not zero, n is a positive odd number, and m is a positive even number.
That is, when the X-axis shake variation amount < the Y-axis shake variation amount, it is determined that the maximum shake direction of the mobile terminal is the Y-axis direction, a line interval output method is selected for the pixel array of the image sensor 120, the image sensor 120 operates in a line-by-line pixel interval operation, that is, from the origin pixel of the image sensor 120, as shown in fig. 9, the previous frame original image is composed of the 1 st, 3 rd, 5 th, and 7 … 1535 th column operations of the pixel row, and the next frame original image is composed of the 2 nd, 4 th, 6 th, and 8 … 1536 th column operations of the pixel column. Furthermore, a plurality of numerical value intervals can be divided from possible values of the Y-axis jitter amount, and the number d of the corresponding line intervals is selected according to the numerical value interval in which the Y-axis jitter amount falls. That is, when the Y-axis shake amount changes greatly, the output pattern of the image sensor 120 is output every two or more lines, for example: every 2 lines are output, every adjacent 2 lines of pixels are a processing unit, a former frame image is composed of 1 st, 2 nd, 5 th, 6 th, … th, 1533 th and 1534 th column works of a pixel row, a latter frame image is composed of 3 rd, 4 th, 7 th, 8 th, … th, 1535 th and 1536 th column works of the pixel row, and the larger the value interval in which the Y-axis jitter amount falls, the larger the number d of corresponding selection line intervals.
Step S140: and carrying out image synthesis on the front and rear frames of original images to generate a final target image.
Specifically, after the ISP image processor 140 acquires the two frames of original images of the image sensor 120 before and after the shutter response time, as shown in fig. 10, the two frames of original images are processed as follows:
step S141: and carrying out image synthesis on the original images of the front frame and the back frame by using an ISP image processor to obtain an image in an RAW data format.
Step S142: the RAW data format image is converted into a JPEG format image, and image processing procedures including auto white balance processing, auto exposure processing, and auto focus processing are performed again to generate a final target image.
Thus, the ISP image processor 140 can finally output a clear, normally color-restored, normally exposed image.
Example two
As shown in fig. 11, a second embodiment of the present invention provides an apparatus 20 for image shake compensation, where the apparatus 20 includes a memory 21, a processor 22, a program stored in the memory and running on the processor, and a data bus 23 for implementing connection communication between the processor 21 and the memory 22, and the program, when executed by the processor, implements the following specific steps as shown in fig. 1:
step S110: and detecting whether the shaking amount of the mobile terminal at the shutter response moment is larger than a preset threshold value.
Specifically, the method for image shake compensation of the present invention is mainly implemented based on the mobile terminal 100 as shown in fig. 2, and the mobile terminal 100 mainly includes a camera 110, an image sensor 120, a gyroscope 130, an ISP image processor 140 and a control unit 150.
The camera 110 is mainly used for acquiring images during shooting.
The image sensor 120 may be a CMOS (ComplementarY Metal OXide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor, the number of pixels of the image sensor 120 may be 500, 800, 1000, 1200, 1300, 1600, etc., and the pixel array of the image sensor 100 may be 4: 3 or 16: 9 general arrangement, as shown in fig. 3, taking 1300 ten thousand pixels as an example, the pixel array is composed of single pixels of three primary colors of red, green and blue, and each pixel can be operated independently and driven by a control circuit under each pixel. Coordinates are piXel (0,0) to piXel (4096,3072), pixels are arranged in columns and columns, pixels are arranged in rows and rows, as shown in fig. 4, four squares are unit pixels, each unit piXel (unit piXel) is composed of two green pixels, one red piXel and one blue piXel, coordinates are unit piXel (0,0) to unit piXel (2048,1536), a conventional exposure mode of the CMOS sensor is performed in a row-by-row mode, and the exposure time of any one piXel (piXel) is the same, that is, the exposure starting point and the exposure time of each piXel (piXel) on the same row are the same, but the EV value of AE (automatic exposure) can be controlled by software in combination with a hardware circuit. The CMOS sensor is independent of each unit pixel control circuit, and the working mode of the conventional sensor is changed from progressive scanning to interlaced scanning by controlling the timing of each line of pixels AE (automatic exposure).
The gYroscope (gYroscope)120 is on the mobile terminal 100, and mainly functions to detect the variation of the orientation and angle of the mobile terminal 100 in real time, and the gYroscope 130 detects from three axes of X, Y and Z, and records the time value of each axial T and the time value of T +1, unit: rad/s, time T is defined as the previous time when the shutter is clicked to photograph after the camera APP is opened (i.e. shutter response), i.e. the time of acquiring the previous frame of original image mentioned later, and time T +1 is the time of shutter response, i.e. the time of acquiring the next frame of original image mentioned later.
The ISP (Image Signal Processing) Image processor 130 mainly processes and synthesizes a plurality of frames of images in the Image sensor 120 to obtain an nth frame Image and an N +1 th frame Image at the time of T +1, a camera real-time preview window is an N frame Image, next, the RAW data format of the Image sensor 120 is converted into JPEG or other formats, next, the Image of the Image sensor 120 is processed by 3A (3A: auto white balance AWB, auto exposure AE, auto focus AF), and finally, a clear, normally color-restored, normally exposed Image is output.
The control unit 150 mainly controls an interval output manner of the pixel array of the image sensor 120, and the image sensor 120 operates in an interval operation for each column of pixels when the X-axis variation > the Y-axis variation, depending on the X-axis and Y-axis variations measured by the gyroscope 130. When the X-axis variation is less than the Y-axis variation, the image sensor works in a mode of working at intervals of each row of pixels.
Based on the mobile terminal 100 shown in fig. 2, the method further introduces a concept of a preset threshold, where the preset threshold may be determined according to the accuracy of actually performing image shake compensation, and the preset threshold is used to determine whether the current shake occurring when the user uses the mobile terminal 100 to perform shooting has reached a degree that affects the quality of shooting, that is, if the shake amount of the mobile terminal at the shutter response time is less than or equal to the preset threshold, it indicates that the current shake occurring is not enough to affect the quality of shooting by the mobile terminal, the image sensor 120 may adopt a full-pixel output mode without performing subsequent steps of the method for image shake compensation, and if the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold, it indicates that the current shake occurring may affect the quality of shooting by the mobile terminal, and subsequent steps of the method for image shake compensation are required, the image smear phenomenon caused by shaking during shooting is effectively solved. Therefore, the step of detecting whether the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold is particularly important, as shown in fig. 5, the overall process specifically includes:
step S111: and measuring first orientation information of the mobile terminal at the shutter response moment and second orientation information of the mobile terminal at the moment before the shutter response moment through a gyroscope.
Step S112: and comparing the position variation between the first azimuth information and the second azimuth information to acquire the shake quantity of the mobile terminal at the shutter response time.
Step S113: and judging whether the shaking amount of the mobile terminal at the shutter response moment is larger than a preset threshold value.
Through the above method process, the shake amount of the mobile terminal at the shutter response time may be measured by the gyroscope 130 and output to the control unit 40, so as to determine whether the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold through the control unit 150, and when the shake amount is greater than the preset threshold, the following subsequent steps may be performed.
Step S120: and if the shaking amount is larger than the preset threshold, further detecting the maximum shaking direction of the mobile terminal at the shutter response time.
Specifically, based on the above description, if the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold, it is indicated that the currently occurring shake may affect the quality of photographing of the mobile terminal, and a maximum shake direction of the mobile terminal at the shutter response time needs to be further detected, so as to select an appropriate technical means for image shake compensation according to the maximum shake direction of the mobile terminal.
Since the gYroscope (gYroscope)120 detects the orientation variation of the mobile terminal 100 in real time from the three axes X, Y, and Z, and most of the Y axis or X axis shakes during the actual shooting, the present invention compares only the shake amount of the X axis with the shake amount of the Y axis to detect the maximum shake direction of the mobile terminal at the shutter response time. As shown in fig. 6, the overall process specifically includes:
step S121: and if the shake amount is larger than the preset threshold, further acquiring the X-axis shake amount and the Y-axis shake amount of the mobile terminal at the shutter response time.
Step S122: comparing the X-axis shaking amount with the Y-axis shaking amount, if the X-axis shaking amount is larger than the Y-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the X-axis direction, and if the Y-axis shaking amount is larger than the X-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the Y-axis direction.
In this way, when the control unit 150 determines that the amount of shake of the mobile terminal at the shutter response time is greater than the preset threshold, the control unit 150 further acquires the X-axis and Y-axis shaking amounts of the mobile terminal at the shutter response time through the gyro 110, namely, the displacement of the mobile terminal occurring in the X-axis direction between the time T and the time T +1 is obtained, and the displacement of the mobile terminal between the T time and the T +1 time in the Y-axis direction, if the X-axis shaking amount is larger than the Y-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the X-axis direction, if the Y-axis shaking amount is greater than the X-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the Y-axis direction, here, the X-axis direction is a direction parallel to the row direction of the pixel array of the image sensor 100, and the Y-axis direction is a direction parallel to the column direction of the pixel array of the image sensor 100.
Step S130: and selecting a corresponding interval output mode for the pixel array of the image sensor according to the maximum shaking direction of the mobile terminal so as to acquire two frames of original images before and after the shutter response time.
Specifically, when the maximum shaking direction of the mobile terminal at the shutter response time is detected in step S120, a corresponding interval output mode may be selected for the pixel array of the image sensor according to the maximum shaking direction of the mobile terminal, so as to obtain two frames of original images before and after the shutter response time. As shown in fig. 7, the overall process specifically includes:
step S131: and if the maximum shaking direction of the mobile terminal is judged to be the X-axis direction, selecting a row interval output mode for the pixel array of the image sensor to acquire two frames of original images before and after the shutter response time.
Specifically, if the maximum jitter direction of the mobile terminal is determined to be the X-axis direction, the number D of column intervals is selected according to the numerical interval in which the magnitude of the X-axis jitter amount falls. And selecting each processing unit of a previous frame original image composed of ND-D +1, ND-D +2, … …, ND-1 and ND column pixels in a pixel array of the image sensor by taking adjacent D column pixels as a processing unit, and selecting each processing unit of a next frame original image composed of MD-D +1, MD-D +2, … …, MD-1 and MD column pixels in the pixel array of the image sensor to acquire two frames of original images before and after the shutter response moment, wherein D is a natural number which is not zero, N is a positive odd number, and M is a positive even number.
That is, when the X-axis jitter amount is greater than the Y-axis jitter amount, it is determined that the maximum jitter direction of the mobile terminal is the X-axis direction, a column interval output mode is selected for the pixel array of the image sensor 120, and the operation mode of the image sensor 120 is that each column of pixels operates at intervals; that is, starting from the origin pixel of the image sensor 120, as shown in fig. 8, the previous frame original image is composed of the 1 st, 3 rd, 5 th, and 7 … 2047 th works of the pixel column, and the next frame original image is composed of the 2 nd, 4 th, 6 th, and 8 … 2048 th works of the pixel column. Furthermore, a plurality of numerical value intervals can be divided from possible values of the X-axis shaking amount, and the number D of the corresponding column intervals is selected according to the numerical value interval in which the X-axis shaking amount falls. That is, when the X-axis jitter amount changes greatly, the output pattern of the image sensor 120 is output every two or more columns, for example: every 2 columns are output, every adjacent 2 columns of pixels are a processing unit, the original image of the previous frame is formed by the 1 st, 2 nd, 5 th, 6 th, … th, 2045 th and 2046 th column works of pixel column, the original image of the next frame is formed by the 3 rd, 4 th, 7 th, 8 th, … th, 2047 th and 2048 th column works of pixel column, and the larger the value interval in which the X-axis shaking amount falls, the larger the number D of corresponding selected column intervals is.
Step S132: and if the maximum shaking direction of the mobile terminal is judged to be the Y-axis direction, selecting a line interval output mode for the pixel array of the image sensor to acquire two frames of original images before and after the shutter response time.
Specifically, if the maximum jitter direction of the mobile terminal is determined to be the Y-axis direction, the number d of line intervals is selected according to the numerical interval in which the Y-axis jitter amount falls. And selecting nd-d +1, nd-d +2, … …, nd-1 and nd column pixels in a pixel array of the image sensor as a processing unit to form each processing unit of a previous frame of original image, selecting md-d +1, md-d +2, … …, md-1 and md column pixels in the pixel array of the image sensor to form each processing unit of a next frame of original image, so as to obtain two frames of original images before and after the shutter response moment, wherein d is a natural number which is not zero, n is a positive odd number, and m is a positive even number.
That is, when the X-axis shake variation amount < the Y-axis shake variation amount, it is determined that the maximum shake direction of the mobile terminal is the Y-axis direction, a line interval output method is selected for the pixel array of the image sensor 120, the image sensor 120 operates in a line-by-line pixel interval operation, that is, from the origin pixel of the image sensor 120, as shown in fig. 9, the previous frame original image is composed of the 1 st, 3 rd, 5 th, and 7 … 1535 th column operations of the pixel row, and the next frame original image is composed of the 2 nd, 4 th, 6 th, and 8 … 1536 th column operations of the pixel column. Furthermore, a plurality of numerical value intervals can be divided from possible values of the Y-axis jitter amount, and the number d of the corresponding line intervals is selected according to the numerical value interval in which the Y-axis jitter amount falls. That is, when the Y-axis shake amount changes greatly, the output pattern of the image sensor 120 is output every two or more lines, for example: every 2 lines are output, every adjacent 2 lines of pixels are a processing unit, a former frame image is composed of 1 st, 2 nd, 5 th, 6 th, … th, 1533 th and 1534 th column works of a pixel row, a latter frame image is composed of 3 rd, 4 th, 7 th, 8 th, … th, 1535 th and 1536 th column works of the pixel row, and the larger the value interval in which the Y-axis jitter amount falls, the larger the number d of corresponding selection line intervals.
Step S140: and carrying out image synthesis on the front and rear frames of original images to generate a final target image.
Specifically, after the ISP image processor 140 acquires the two frames of original images of the image sensor 120 before and after the shutter response time, as shown in fig. 10, the two frames of original images are processed as follows:
step S141: and carrying out image synthesis on the original images of the front frame and the back frame by using an ISP image processor to obtain an image in an RAW data format.
Step S142: the RAW data format image is converted into a JPEG format image, and image processing procedures including auto white balance processing, auto exposure processing, and auto focus processing are performed again to generate a final target image.
Thus, the ISP image processor 140 can finally output a clear, normally color-restored, normally exposed image.
EXAMPLE III
A third embodiment of the present invention provides a computer-readable storage medium, where one or more programs are stored, and the one or more programs are executable by one or more processors to implement the following specific steps as shown in fig. 1:
step S110: and detecting whether the shaking amount of the mobile terminal at the shutter response moment is larger than a preset threshold value.
Specifically, the method for image shake compensation of the present invention is mainly implemented based on the mobile terminal 100 as shown in fig. 2, and the mobile terminal 100 mainly includes a camera 110, an image sensor 120, a gyroscope 130, an ISP image processor 140 and a control unit 150.
The camera 110 is mainly used for acquiring images during shooting.
The image sensor 120 may be a CMOS (ComplementarY Metal OXide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor, the number of pixels of the image sensor 120 may be 500, 800, 1000, 1200, 1300, 1600, etc., and the pixel array of the image sensor 100 may be 4: 3 or 16: 9 general arrangement, as shown in fig. 3, taking 1300 ten thousand pixels as an example, the pixel array is composed of single pixels of three primary colors of red, green and blue, and each pixel can be operated independently and driven by a control circuit under each pixel. Coordinates are piXel (0,0) to piXel (4096,3072), pixels are arranged in columns and columns, pixels are arranged in rows and rows, as shown in fig. 4, four squares are unit pixels, each unit piXel (unit piXel) is composed of two green pixels, one red piXel and one blue piXel, coordinates are unit piXel (0,0) to unit piXel (2048,1536), a conventional exposure mode of the CMOS sensor is performed in a row-by-row mode, and the exposure time of any one piXel (piXel) is the same, that is, the exposure starting point and the exposure time of each piXel (piXel) on the same row are the same, but the EV value of AE (automatic exposure) can be controlled by software in combination with a hardware circuit. The CMOS sensor is independent of each unit pixel control circuit, and the working mode of the conventional sensor is changed from progressive scanning to interlaced scanning by controlling the timing of each line of pixels AE (automatic exposure).
The gYroscope (gYroscope)120 is on the mobile terminal 100, and mainly functions to detect the variation of the orientation and angle of the mobile terminal 100 in real time, and the gYroscope 130 detects from three axes of X, Y and Z, and records the time value of each axial T and the time value of T +1, unit: rad/s, time T is defined as the previous time when the shutter is clicked to photograph after the camera APP is opened (i.e. shutter response), i.e. the time of acquiring the previous frame of original image mentioned later, and time T +1 is the time of shutter response, i.e. the time of acquiring the next frame of original image mentioned later.
The ISP (Image Signal Processing) Image processor 130 mainly processes and synthesizes a plurality of frames of images in the Image sensor 120 to obtain an nth frame Image and an N +1 th frame Image at the time of T +1, a camera real-time preview window is an N frame Image, next, the RAW data format of the Image sensor 120 is converted into JPEG or other formats, next, the Image of the Image sensor 120 is processed by 3A (3A: auto white balance AWB, auto exposure AE, auto focus AF), and finally, a clear, normally color-restored, normally exposed Image is output.
The control unit 150 mainly controls an interval output manner of the pixel array of the image sensor 120, and the image sensor 120 operates in an interval operation for each column of pixels when the X-axis variation > the Y-axis variation, depending on the X-axis and Y-axis variations measured by the gyroscope 130. When the X-axis variation is less than the Y-axis variation, the image sensor works in a mode of working at intervals of each row of pixels.
Based on the mobile terminal 100 shown in fig. 2, the method further introduces a concept of a preset threshold, where the preset threshold may be determined according to the accuracy of actually performing image shake compensation, and the preset threshold is used to determine whether the current shake occurring when the user uses the mobile terminal 100 to perform shooting has reached a degree that affects the quality of shooting, that is, if the shake amount of the mobile terminal at the shutter response time is less than or equal to the preset threshold, it indicates that the current shake occurring is not enough to affect the quality of shooting by the mobile terminal, the image sensor 120 may adopt a full-pixel output mode without performing subsequent steps of the method for image shake compensation, and if the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold, it indicates that the current shake occurring may affect the quality of shooting by the mobile terminal, and subsequent steps of the method for image shake compensation are required, the image smear phenomenon caused by shaking during shooting is effectively solved. Therefore, the step of detecting whether the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold is particularly important, as shown in fig. 5, the overall process specifically includes:
step S111: and measuring first orientation information of the mobile terminal at the shutter response moment and second orientation information of the mobile terminal at the moment before the shutter response moment through a gyroscope.
Step S112: and comparing the position variation between the first azimuth information and the second azimuth information to acquire the shake quantity of the mobile terminal at the shutter response time.
Step S113: and judging whether the shaking amount of the mobile terminal at the shutter response moment is larger than a preset threshold value.
Through the above method process, the shake amount of the mobile terminal at the shutter response time may be measured by the gyroscope 130 and output to the control unit 40, so as to determine whether the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold through the control unit 150, and when the shake amount is greater than the preset threshold, the following subsequent steps may be performed.
Step S120: and if the shaking amount is larger than the preset threshold, further detecting the maximum shaking direction of the mobile terminal at the shutter response time.
Specifically, based on the above description, if the shake amount of the mobile terminal at the shutter response time is greater than the preset threshold, it is indicated that the currently occurring shake may affect the quality of photographing of the mobile terminal, and a maximum shake direction of the mobile terminal at the shutter response time needs to be further detected, so as to select an appropriate technical means for image shake compensation according to the maximum shake direction of the mobile terminal.
Since the gYroscope (gYroscope)120 detects the orientation variation of the mobile terminal 100 in real time from the three axes X, Y, and Z, and most of the Y axis or X axis shakes during the actual shooting, the present invention compares only the shake amount of the X axis with the shake amount of the Y axis to detect the maximum shake direction of the mobile terminal at the shutter response time. As shown in fig. 6, the overall process specifically includes:
step S121: and if the shake amount is larger than the preset threshold, further acquiring the X-axis shake amount and the Y-axis shake amount of the mobile terminal at the shutter response time.
Step S122: comparing the X-axis shaking amount with the Y-axis shaking amount, if the X-axis shaking amount is larger than the Y-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the X-axis direction, and if the Y-axis shaking amount is larger than the X-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the Y-axis direction.
In this way, when the control unit 150 determines that the amount of shake of the mobile terminal at the shutter response time is greater than the preset threshold, the control unit 150 further acquires the X-axis and Y-axis shaking amounts of the mobile terminal at the shutter response time through the gyro 110, namely, the displacement of the mobile terminal occurring in the X-axis direction between the time T and the time T +1 is obtained, and the displacement of the mobile terminal between the T time and the T +1 time in the Y-axis direction, if the X-axis shaking amount is larger than the Y-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the X-axis direction, if the Y-axis shaking amount is greater than the X-axis shaking amount, determining that the maximum shaking direction of the mobile terminal is the Y-axis direction, here, the X-axis direction is a direction parallel to the row direction of the pixel array of the image sensor 100, and the Y-axis direction is a direction parallel to the column direction of the pixel array of the image sensor 100.
Step S130: and selecting a corresponding interval output mode for the pixel array of the image sensor according to the maximum shaking direction of the mobile terminal so as to acquire two frames of original images before and after the shutter response time.
Specifically, when the maximum shaking direction of the mobile terminal at the shutter response time is detected in step S120, a corresponding interval output mode may be selected for the pixel array of the image sensor according to the maximum shaking direction of the mobile terminal, so as to obtain two frames of original images before and after the shutter response time. As shown in fig. 7, the overall process specifically includes:
step S131: and if the maximum shaking direction of the mobile terminal is judged to be the X-axis direction, selecting a row interval output mode for the pixel array of the image sensor to acquire two frames of original images before and after the shutter response time.
Specifically, if the maximum jitter direction of the mobile terminal is determined to be the X-axis direction, the number D of column intervals is selected according to the numerical interval in which the magnitude of the X-axis jitter amount falls. And selecting each processing unit of a previous frame original image composed of ND-D +1, ND-D +2, … …, ND-1 and ND column pixels in a pixel array of the image sensor by taking adjacent D column pixels as a processing unit, and selecting each processing unit of a next frame original image composed of MD-D +1, MD-D +2, … …, MD-1 and MD column pixels in the pixel array of the image sensor to acquire two frames of original images before and after the shutter response moment, wherein D is a natural number which is not zero, N is a positive odd number, and M is a positive even number.
That is, when the X-axis jitter amount is greater than the Y-axis jitter amount, it is determined that the maximum jitter direction of the mobile terminal is the X-axis direction, a column interval output mode is selected for the pixel array of the image sensor 120, and the operation mode of the image sensor 120 is that each column of pixels operates at intervals; that is, starting from the origin pixel of the image sensor 120, as shown in fig. 8, the previous frame original image is composed of the 1 st, 3 rd, 5 th, and 7 … 2047 th works of the pixel column, and the next frame original image is composed of the 2 nd, 4 th, 6 th, and 8 … 2048 th works of the pixel column. Furthermore, a plurality of numerical value intervals can be divided from possible values of the X-axis shaking amount, and the number D of the corresponding column intervals is selected according to the numerical value interval in which the X-axis shaking amount falls. That is, when the X-axis jitter amount changes greatly, the output pattern of the image sensor 120 is output every two or more columns, for example: every 2 columns are output, every adjacent 2 columns of pixels are a processing unit, the original image of the previous frame is formed by the 1 st, 2 nd, 5 th, 6 th, … th, 2045 th and 2046 th column works of pixel column, the original image of the next frame is formed by the 3 rd, 4 th, 7 th, 8 th, … th, 2047 th and 2048 th column works of pixel column, and the larger the value interval in which the X-axis shaking amount falls, the larger the number D of corresponding selected column intervals is.
Step S132: and if the maximum shaking direction of the mobile terminal is judged to be the Y-axis direction, selecting a line interval output mode for the pixel array of the image sensor to acquire two frames of original images before and after the shutter response time.
Specifically, if the maximum jitter direction of the mobile terminal is determined to be the Y-axis direction, the number d of line intervals is selected according to the numerical interval in which the Y-axis jitter amount falls. And selecting nd-d +1, nd-d +2, … …, nd-1 and nd column pixels in a pixel array of the image sensor as a processing unit to form each processing unit of a previous frame of original image, selecting md-d +1, md-d +2, … …, md-1 and md column pixels in the pixel array of the image sensor to form each processing unit of a next frame of original image, so as to obtain two frames of original images before and after the shutter response moment, wherein d is a natural number which is not zero, n is a positive odd number, and m is a positive even number.
That is, when the X-axis shake variation amount < the Y-axis shake variation amount, it is determined that the maximum shake direction of the mobile terminal is the Y-axis direction, a line interval output method is selected for the pixel array of the image sensor 120, the image sensor 120 operates in a line-by-line pixel interval operation, that is, from the origin pixel of the image sensor 120, as shown in fig. 9, the previous frame original image is composed of the 1 st, 3 rd, 5 th, and 7 … 1535 th column operations of the pixel row, and the next frame original image is composed of the 2 nd, 4 th, 6 th, and 8 … 1536 th column operations of the pixel column. Furthermore, a plurality of numerical value intervals can be divided from possible values of the Y-axis jitter amount, and the number d of the corresponding line intervals is selected according to the numerical value interval in which the Y-axis jitter amount falls. That is, when the Y-axis shake amount changes greatly, the output pattern of the image sensor 120 is output every two or more lines, for example: every 2 lines are output, every adjacent 2 lines of pixels are a processing unit, a former frame image is composed of 1 st, 2 nd, 5 th, 6 th, … th, 1533 th and 1534 th column works of a pixel row, a latter frame image is composed of 3 rd, 4 th, 7 th, 8 th, … th, 1535 th and 1536 th column works of the pixel row, and the larger the value interval in which the Y-axis jitter amount falls, the larger the number d of corresponding selection line intervals.
Step S140: and carrying out image synthesis on the front and rear frames of original images to generate a final target image.
Specifically, after the ISP image processor 140 acquires the two frames of original images of the image sensor 120 before and after the shutter response time, as shown in fig. 10, the two frames of original images are processed as follows:
step S141: and carrying out image synthesis on the original images of the front frame and the back frame by using an ISP image processor to obtain an image in an RAW data format.
Step S142: the RAW data format image is converted into a JPEG format image, and image processing procedures including auto white balance processing, auto exposure processing, and auto focus processing are performed again to generate a final target image.
Thus, the ISP image processor 140 can finally output a clear, normally color-restored, normally exposed image.
The method, the device and the storage medium for compensating image jitter provided by the embodiment of the invention detect whether the jitter amount of a mobile terminal at the shutter response time is greater than a preset threshold value, and further detect the maximum jitter direction of the mobile terminal at the shutter response time when the jitter amount is greater than the preset threshold value. And selecting a corresponding interval output mode for the pixel array of the image sensor according to the maximum shaking direction of the mobile terminal so as to acquire two frames of original images before and after the shutter response time. And finally, carrying out image synthesis on the original images of the front frame and the back frame to generate a final target image. In this way, each row of pixels arranged at intervals or each column of pixels arranged at intervals in the pixel array of the image sensor form the original image of the previous frame, that is, intervals exist among all the row pixels or the column pixels forming the original image of the previous frame, and therefore, the phenomenon of smear caused by overlapping of adjacent row pixels or adjacent column pixels when shooting shakes can be avoided. All the remaining row pixels or all the remaining column pixels in the pixel array of the image sensor form a next frame of original image, namely, intervals also exist among all the row pixels or the column pixels forming the next frame of original image, and the phenomenon of smear caused by overlapping of adjacent row pixels or adjacent column pixels when shooting shakes can also be avoided. And finally, only image synthesis is needed to be carried out on the original images of the front frame and the back frame, and a clear and complete target image can be generated. Therefore, according to the technical scheme, additional hardware is not required to be added, and the image smear phenomenon caused by shaking in shooting can be effectively solved when the mobile terminal shakes in shooting.
The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.
While the present invention has been described with reference to the embodiments shown in the drawings, the present invention is not limited to the embodiments, which are illustrative and not restrictive, and it will be apparent to those skilled in the art that various changes and modifications can be made therein without departing from the spirit and scope of the invention as defined in the appended claims.
Claims (10)
1. A method of image shake compensation, the method comprising the steps of:
detecting whether the shaking amount of the mobile terminal at the shutter response moment is greater than a preset threshold value or not;
if the jitter amount is larger than the preset threshold value, further detecting the maximum jitter direction of the mobile terminal at the shutter response time;
selecting a corresponding interval output mode for a pixel array of an image sensor according to the maximum shaking direction of the mobile terminal so as to obtain two frames of original images before and after the shutter response time;
and carrying out image synthesis on the front and rear frames of original images to generate a final target image.
2. The method of claim 1, wherein the step of detecting whether the amount of shake of the mobile terminal at the shutter response time is greater than a preset threshold specifically comprises:
measuring first azimuth information of the mobile terminal at a shutter response moment and second azimuth information of the mobile terminal at a shutter response previous moment through a gyroscope;
comparing the position variation between the first azimuth information and the second azimuth information to acquire the shake amount of the mobile terminal at the shutter response moment;
and judging whether the shaking amount of the mobile terminal at the shutter response moment is larger than a preset threshold value.
3. The method according to claim 1, wherein if the shake amount is greater than the preset threshold, the step of further detecting a maximum shake direction of the mobile terminal at a shutter response time specifically comprises:
if the jitter amount is larger than the preset threshold value, further acquiring an X-axis jitter amount and a Y-axis jitter amount of the mobile terminal at the shutter response time;
and comparing the X-axis shaking amount with the Y-axis shaking amount, if the X-axis shaking amount is larger than the Y-axis shaking amount, judging that the maximum shaking direction of the mobile terminal is the X-axis direction, and if the Y-axis shaking amount is larger than the X-axis shaking amount, judging that the maximum shaking direction of the mobile terminal is the Y-axis direction.
4. The method of claim 3, wherein the step of selecting a corresponding interval output mode for the pixel array of the image sensor according to the maximum shaking direction of the mobile terminal to obtain two frames of original images before and after the shutter response time specifically comprises:
and if the maximum shaking direction of the mobile terminal is judged to be the X-axis direction, selecting a row interval output mode for the pixel array of the image sensor to acquire two frames of original images before and after the shutter response time.
5. The method of claim 4, wherein if the maximum shaking direction of the mobile terminal is determined to be the X-axis direction, the step of selecting a column interval output mode for the image sensor pixel array to obtain two frames of original images before and after the shutter response time specifically comprises:
if the maximum shaking direction of the mobile terminal is judged to be the X-axis direction, selecting the number D of the column intervals according to the numerical value interval in which the X-axis shaking amount falls;
and selecting each processing unit of a previous frame original image composed of ND-D +1, ND-D +2, … …, ND-1 and ND column pixels in a pixel array of the image sensor by taking adjacent D column pixels as a processing unit, and selecting each processing unit of a next frame original image composed of MD-D +1, MD-D +2, … …, MD-1 and MD column pixels in the pixel array of the image sensor to acquire two frames of original images before and after the shutter response moment, wherein D is a natural number which is not zero, N is a positive odd number, and M is a positive even number.
6. The method of claim 3, wherein the step of selecting a corresponding interval output mode for the pixel array of the image sensor according to the maximum shaking direction of the mobile terminal to obtain two frames of original images before and after the shutter response time further comprises:
and if the maximum shaking direction of the mobile terminal is judged to be the Y-axis direction, selecting a row interval output mode for the pixel array of the image sensor to acquire two frames of original images before and after the shutter response time.
7. The method of claim 6, wherein if the maximum shaking direction of the mobile terminal is determined to be the Y-axis direction, the step of selecting a line interval output mode for the pixel array of the image sensor to obtain two frames of original images before and after the shutter response time specifically comprises:
if the maximum jitter direction of the mobile terminal is judged to be the Y-axis direction, selecting the number d of the line intervals according to the numerical value interval in which the Y-axis jitter amount falls;
and selecting nd-d +1, nd-d +2, … …, nd-1 and nd column pixels in a pixel array of the image sensor as a processing unit to form each processing unit of a previous frame of original image, selecting md-d +1, md-d +2, … …, md-1 and md column pixels in the pixel array of the image sensor to form each processing unit of a next frame of original image, so as to obtain two frames of original images before and after the shutter response moment, wherein d is a natural number which is not zero, n is a positive odd number, and m is a positive even number.
8. The method for image shake compensation according to claim 1, wherein the step of performing image synthesis on the two previous and next frames of original images to generate the final target image specifically comprises:
carrying out image synthesis on the front and rear frames of original images through an ISP image processor to obtain an RAW data format image;
the RAW data format image is converted into a JPEG format image, and image processing procedures including auto white balance processing, auto exposure processing, and auto focus processing are performed again to generate a final target image.
9. An apparatus for image shake compensation, comprising a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for enabling a connection communication between the processor and the memory, the program, when executed by the processor, implementing the steps of the method for image shake compensation according to any one of claims 1-8.
10. A storage medium for computer readable storage, characterized in that the storage medium stores one or more programs, which are executable by one or more processors to implement the steps of the method of image shake compensation according to any one of claims 1 to 8.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810648471.1A CN110636203A (en) | 2018-06-22 | 2018-06-22 | Image jitter compensation method, device and storage medium |
| PCT/CN2019/092346 WO2019242757A1 (en) | 2018-06-22 | 2019-06-21 | Imaging shake compensation method and device, and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201810648471.1A CN110636203A (en) | 2018-06-22 | 2018-06-22 | Image jitter compensation method, device and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110636203A true CN110636203A (en) | 2019-12-31 |
Family
ID=68966729
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201810648471.1A Withdrawn CN110636203A (en) | 2018-06-22 | 2018-06-22 | Image jitter compensation method, device and storage medium |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN110636203A (en) |
| WO (1) | WO2019242757A1 (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101345824A (en) * | 2007-07-09 | 2009-01-14 | 三星电子株式会社 | Method and apparatus for compensating hand-trembling of camera |
| CN102457675A (en) * | 2010-10-27 | 2012-05-16 | 展讯通信（上海）有限公司 | Image shooting anti-shaking manner for handheld camera equipment |
| US20130088610A1 (en) * | 2011-10-07 | 2013-04-11 | Samsung Electronics Co., Ltd | Photographing apparatus, motion estimating apparatus, image compensating method, motion estimating method, and computer-readable recording medium |
| CN103384309A (en) * | 2012-05-02 | 2013-11-06 | 佳能株式会社 | Image capturing apparatus and control method |
| CN103685918A (en) * | 2012-09-11 | 2014-03-26 | 索尼公司 | Processing apparatus, processing method, and program |
| CN103763483A (en) * | 2014-01-23 | 2014-04-30 | 贝壳网际（北京）安全技术有限公司 | Method and device for shaking resistance in photo taking process of mobile terminal and mobile terminal |
-
2018
- 2018-06-22 CN CN201810648471.1A patent/CN110636203A/en not_active Withdrawn
-
2019
- 2019-06-21 WO PCT/CN2019/092346 patent/WO2019242757A1/en active Application Filing
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101345824A (en) * | 2007-07-09 | 2009-01-14 | 三星电子株式会社 | Method and apparatus for compensating hand-trembling of camera |
| CN102457675A (en) * | 2010-10-27 | 2012-05-16 | 展讯通信（上海）有限公司 | Image shooting anti-shaking manner for handheld camera equipment |
| US20130088610A1 (en) * | 2011-10-07 | 2013-04-11 | Samsung Electronics Co., Ltd | Photographing apparatus, motion estimating apparatus, image compensating method, motion estimating method, and computer-readable recording medium |
| CN103384309A (en) * | 2012-05-02 | 2013-11-06 | 佳能株式会社 | Image capturing apparatus and control method |
| CN103685918A (en) * | 2012-09-11 | 2014-03-26 | 索尼公司 | Processing apparatus, processing method, and program |
| CN103763483A (en) * | 2014-01-23 | 2014-04-30 | 贝壳网际（北京）安全技术有限公司 | Method and device for shaking resistance in photo taking process of mobile terminal and mobile terminal |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2019242757A1 (en) | 2019-12-26 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US9473698B2 (en) | Imaging device and imaging method | |
| CN102870404B (en) | Imaging device and dark current correction method of same | |
| CN110213494B (en) | Photographing method and device, electronic equipment and computer readable storage medium | |
| EP1784008A2 (en) | Image-pickup device, image-pickup method, and program | |
| US8593547B2 (en) | Image processing apparatus, image capturing apparatus, and image processing method | |
| KR101120966B1 (en) | A hand jitter reduction system for cameras | |
| CN110475067B (en) | Image processing method and device, electronic equipment and computer readable storage medium | |
| CN107509044B (en) | Image synthesis method, image synthesis device, computer-readable storage medium and computer equipment | |
| CN111246100B (en) | Anti-shake parameter calibration method and device and electronic equipment | |
| CN109559352B (en) | Camera calibration method, device, electronic equipment and computer-readable storage medium | |
| JP5499853B2 (en) | Electronic camera | |
| CN109697737B (en) | Camera calibration method, device, electronic equipment and computer-readable storage medium | |
| US7864860B2 (en) | Image pickup apparatus and motion vector deciding method | |
| US20100245590A1 (en) | Camera sensor system self-calibration | |
| JP2006148550A (en) | Image processor and imaging device | |
| JP2006023339A (en) | Imaging apparatus | |
| JP2007318644A (en) | Imaging device, control method thereof, and its control method | |
| JP2014131190A (en) | Image pick-up apparatus, control method thereof, and control program | |
| CN110636203A (en) | Image jitter compensation method, device and storage medium | |
| US11399145B2 (en) | Image processing device, imaging apparatus, image processing method, and program | |
| US8373766B2 (en) | Image shooting device and image shooting method | |
| JP4705146B2 (en) | Imaging apparatus and imaging method | |
| JP4597887B2 (en) | Imaging device | |
| US11025884B2 (en) | Image capturing apparatus, control method thereof, and storage medium | |
| CN102480590B (en) | Electronic device, image photographing device and method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20191231 |
|
| WW01 | Invention patent application withdrawn after publication |