KR20220115494A - Image Signal Processor, Electronic Device and Image Stablization Method - Google Patents

Abstract

Provided is an electronic device for performing image stabilization. The electronic device of the present invention comprises: a camera module generating and outputting gyro data and frame data for inputted video; and an image signal processor, wherein the image signal processor includes: a motion vector module for calculating motion vector information; a gyro-based motion estimation unit for extracting camera rotation information on the camera module from the gyro data; a motion vector (MV)-based motion estimation unit for extracting frame rotation information from the frame data; an OIS 2D translation information estimation unit for estimating optical image stabilizer (OIS) 2D translation information, based on the motion vector information, the camera rotation information, and the frame rotation information; and a camera path optimizer for filtering out low-frequency components from the OIS 2D translation information and calculating stabilized camera motion information of the camera module by using the filtered OIS 2D translation information and the camera rotation information.

Description

Image Signal Processor, Electronic Device and Image Stablization Method

The present invention relates to image stabilization.

If a problem such as a user's hand shake occurs while the electronic device captures an image, the image may be acquired while being distorted. In order to compensate for the user's hand shake problem, the electronic device may perform image correction. Image correction may include, for example, Optical Image Stablization (OIS) correction or Digital Image Stablization (DIS or Electrical Image Stablization, EIS) correction.

OIS correction may refer to a correction in which the image stabilizer moves a lens or an image sensor in a direction that compensates for the shake of the electronic device when the electronic device shakes while the camera module acquires an image. There are two types of OIS correction: a shift method that moves in a plane and a tilt method that rotates. In addition, each OIS correction may include, for example, a lens moving method (a method of moving a lens) or a sensor moving method (a method of moving a sensor). That is, OIS correction may represent two-dimensional image movement (2D translation).

DIS or EIS correction may refer to an operation in which the electronic device corrects an image based on at least one of image motion and gyro data. The image movement may refer to information indicating the movement of an object (eg, a feature point of an imaged subject) within the frame image. The electronic device may determine the image motion by comparing frame images acquired through the camera module. The gyro data may mean information corresponding to the shaking of the electronic device. The gyro data may be acquired through a motion sensor that detects shaking of the electronic device. That is, the gyro data may represent 3D rotation.

In many cases, the electronic device actually shakes in three dimensions, but the camera module senses only gyro data, frame data, and motion vector information. That is, in the case of an electronic device that does not provide OIS 2D translation sensor information, only camera module rotation information and frame rotation information processed from information sensed by the camera module (gyro data, frame data, and motion vector information) are used. Therefore, digital image stabilization should be performed.

When digital image stabilization is performed using only camera module rotation information and frame rotation information in an electronic device using a shift OIS, OIS 2D translation information is reflected when an object is imaged Since stabilization is performed without the need for stabilization, wobbling may appear in the final processed image.

Therefore, the problem to be solved by the present invention is a device capable of performing digital image stabilization by estimating OIS two-dimensional translation information even in an electronic device that does not provide OIS two-dimensional translation information, reflecting the movement of the electronic device more accurately and to provide a method.

The problem to be solved by the present invention is to solve the problem caused by hand shake by optimizing the rotation information of the electronic device to appropriately perform digital image stabilization even in an electronic device that does not provide OIS two-dimensional translation information. It is to provide an apparatus and method that can accurately compensate.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

An electronic device according to some embodiments includes a camera module that generates and outputs gyro data for an input image and frame data, and an image signal processor, wherein the image signal processor includes a motion vector module that calculates motion vector information, and the gyro A gyro-based motion estimation unit for extracting camera rotation information of the camera module from data, an MV (Motion Vector)-based motion estimation unit for extracting frame rotation information from the frame data, the motion vector information, the camera rotation information and the frame An OIS 2D translation information estimator for estimating Optical Image Stablizer (OIS) 2D translation information based on rotation information, filtering a low frequency component from the OIS 2D translation information, and filtering the OIS 2D translation information and the camera and a camera path optimizer that calculates the stabilized camera motion information of the camera module by using the rotation information.

An electronic device according to some embodiments includes a camera module and an image signal processor that output gyro data including at least two gyro sensing values and frame data including at least two frame images, wherein the image signal processor includes the gyro Calculate frame rotation information and camera rotation information based on data, the frame data and motion vector information, estimate OIS 2D translation information based on the frame rotation information and the camera rotation information, and estimate the OIS 2D transform The stabilized camera motion information of the camera module is estimated based on the translation information, and final motion information calculated based on the stabilized camera motion information, the estimated OIS 2D translation information, and the camera rotation information is compensated.

An image stabilization method of an electronic device according to some embodiments includes sensing a frame image and obtaining gyro data, frame data, and a motion vector according to shaking of the electronic device, and extracting camera rotation information of the electronic device from the gyro data and extracting frame rotation information from the motion vector and the frame data, estimating raw Optical Image Stablization (OIS) 2D translation information based on the camera rotation information and the frame rotation information, the raw OIS 2D calculating filtered OIS 2D translation information by removing a bias component from the translation information, and performing digital image stabilization on the sensed frame image based on the filtered OIS 2D translation information.

1 is a block diagram illustrating a camera module according to some embodiments.
2 is a diagram illustrating an OIS correction operation according to some embodiments.
3 is a block diagram illustrating an image signal processor in accordance with some embodiments.
4 is a flowchart illustrating an image stabilization method according to some embodiments.
5 is a conceptual diagram illustrating an image stabilization method according to some embodiments.
6 is a graph illustrating an X-axis frequency component of motion information for a final image according to some embodiments.
7 is a graph showing a state in which the low frequency component is removed in FIG. 6 and only the X-axis high frequency component remains.
8 is a graph illustrating a Y-axis frequency component of motion information for a final image, according to some embodiments.
9 is a graph showing a state in which the low-frequency component is removed in FIG. 8 and only the Y-axis high-frequency component remains.
10 is a graph for explaining a boundary condition of a final image, according to some embodiments.
11 is a diagram for explaining a changed input pixel position range of a crop point after OIS compensation is applied according to a boundary condition, according to some embodiments.
12 is a block diagram of an electronic device in a network environment according to some embodiments.

1 is a block diagram illustrating an electronic device according to some exemplary embodiments.

Referring to FIG. 1 , an electronic device includes a camera module 100 and an application processor (AP) 200 .

The camera module 100 may include a lens assembly 101 , a flash 102 , an image sensor 110 , an image stabilizer 120 , and an image signal processor 201 . The AP 200 may include an image signal processor 201 and a memory 202 (eg, a buffer memory).

The lens assembly 101 may collect light emitted from a subject, which is an image capturing object. The lens assembly 101 may include one or more lenses. According to an embodiment, the camera module 100 may include a plurality of lens assemblies 101 . In this case, the camera module 100 may be, for example, a dual camera, a 360 degree camera, or a spherical camera. The plurality of lens assemblies 101 may have the same lens properties (eg, angle of view, focal length, autofocus, f number, or optical zoom), or at least one lens assembly may have at least one lens assembly and at least another lens assembly. It can have one different lens property. The lens assembly 101 may include, for example, a wide-angle lens or a telephoto lens.

The flash 102 may emit a light source used to enhance light emitted from a subject. The flash 102 may include one or more light emitting diodes (eg, a red-green-blue (RGB) LED, a white LED, an infrared LED, or an ultraviolet LED), or a xenon lamp.

According to an embodiment, the image sensor 110 may acquire an image corresponding to the subject by converting light transmitted from the subject through the lens assembly 101 into an electrical signal. According to an embodiment, the image sensor 110 may include, for example, one image sensor selected from among image sensors having different properties, such as an RGB sensor, a black and white (BW) sensor, an IR sensor, or a UV sensor, the same It may include a plurality of image sensors having properties, or a plurality of image sensors having different properties. Each image sensor included in the image sensor 110 may be implemented as, for example, a charged coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor.

According to an embodiment, the image stabilizer 120 responds to the movement of the camera module 100 or the electronic device 1101 including the same, and reduces a negative effect (eg, image shake) on the photographed image by the movement. At least one lens or image sensor 110 included in the lens assembly 101 may be moved in a specific direction or controlled (eg, read-out timing is adjusted, etc.) in order to compensate at least in part. According to an embodiment, the image stabilizer 120 may be implemented as, for example, an optical image stabilizer, and a gyro sensor (not shown) or an acceleration sensor (not shown) disposed inside or outside the camera module 100 . ) can be used to detect the movement.

According to an embodiment, the image signal processor 201 performs image processing (eg, depth map generation, 3D modeling, panorama generation, Feature point extraction, image synthesis, or image compensation (eg, noise reduction, resolution adjustment, brightness adjustment, blurring, sharpening, or softening) may be performed. Alternatively, an image signal processor ( The 201 may perform control (eg, exposure time control, readout timing control, etc.) on at least one (eg, image sensor 110) among components included in the camera module 100. Image The final image processed by the signal processor 201 may be stored back in the memory 202 for further processing or transferred to an external component of the camera module 100. According to an embodiment, the image signal processor 201 ) may be configured as at least a part of an application processor (not shown) as shown, or may be configured as a separate processor operating independently of the processor according to another embodiment. The final images processed by 201 ) may be displayed as it is by the processor or through a display device (not shown, refer to FIG. 12 ) after additional image processing.

According to an embodiment, the memory 1130 may temporarily store at least a portion of an image acquired through the image sensor 110 for a next image processing operation. For example, when image acquisition is delayed according to the shutter or a plurality of images are acquired at high speed, the acquired raw image (eg, high-resolution image) is stored in the memory 201, A corresponding copy image (eg, a lower resolution image) may be previewed through the display device. Thereafter, when a specified condition is satisfied (eg, a user input or a system command), at least a portion of the original image stored in the memory 202 may be acquired and processed by, for example, the image signal processor 201 . According to an embodiment, the memory 202 may be configured as at least a part of the memory 202 or as a separate memory operated independently of the memory 202 .

According to an embodiment, the electronic device (not shown, see FIG. 12 ) may include two or more camera modules 100 each having different properties or functions. In this case, for example, at least one camera module 100 may be a wide-angle camera or a front camera, and at least one other camera module may be a telephoto camera or a rear camera.

2 is a diagram illustrating an OIS correction operation according to some embodiments.

Referring to FIG. 2 , an electronic device (not shown, see FIG. 12 ) may include a camera module 100 . For example, the electronic device may further include a gyro sensor that detects shaking of the electronic device itself (Body) inside or outside the camera module 100 . For another example, the electronic device may further include an optical image stabilizer 120 that controls the movement of at least one of the lens 310 , the image sensor 330 , and the module 320 .

According to an embodiment, the camera module 100 may include a lens 310 (eg, the lens assembly 101 of FIG. 1 ) and an image sensor 330 (eg, the image sensor 110 of FIG. 1 ). have. The lens 310 may collect light reflected from the subject and transmit the light to the image sensor 330 . The image sensor 330 may acquire an image by converting light transmitted from the lens 310 into an electrical signal.

According to an embodiment, if the electronic device shakes while the electronic device acquires an image, the electronic device (or the image stabilizer 120 ) may primarily perform optical image stabilization correction to compensate for the shake of the electronic device. . For example, the image stabilizer 120 detects the shaking of the camera module 100 through a sensor (eg, a gyro sensor or an acceleration sensor) built into the camera module 100, and in response to the detected shaking, the camera module ( 100) may be controlled to physically move at least one of the components.

For example, referring to reference numeral 301 of FIG. 2 , the image stabilizer 120 may perform OIS correction in a lens shift method in which the lens 310 moves. For another example, referring to reference numeral 302 of FIG. 2 , the image stabilizer 120 is a module 320 including a lens 310 and an image sensor 330 to move OIS correction in a module tilt manner. can be performed. For another example, although not shown in FIG. 2 , the image stabilizer 120 may perform OIS correction in a sensor shift method in which the image sensor 330 moves. Hereinafter, in the case of OIS correction in a shift method in some embodiments, an image stabilization method will be described focusing on, for example, a camera module operating in a lens shift method or a sensor shift method.

According to an embodiment, the electronic device may detect the movement of the electronic device using a motion sensor (eg, a gyro sensor or an acceleration sensor) separate from a sensor used for OIS correction. In the following description, information corresponding to the motion of the electronic device may be referred to as gyro data, 3D rotation motion information, or camera rotation information (C_Path). In addition, in this document, information corresponding to the movement of the lens 310 and the image sensor 330 by OIS correction may be referred to as OIS movement data, 2D movement information, or OIS 2D translation information (T_Path). . According to an embodiment, the camera rotation information (C_Path) or the OIS 2D translation information (T_Path) may be expressed as an angle change over time or a relative position change of the module 320 over time. In this case, the change in the angle or the change in the relative position of the module 320 over time may be the cumulative sum of the translation difference values for each frame of the module 320 .

The image movement is a result of simultaneously reflecting the camera rotation movement (C_Path) and the OIS 2D translation information (T_Path), but the electronic device does not provide a sensor value for sensing the movement of the module 320 of FIG. 2 , unless a frame image movement is provided. information cannot be obtained.

If the electronic device performs DIS correction on the frame image without considering the OIS 2D translation information T_Path, the image may be overcorrected as much as the OIS correction is performed. Accordingly, according to various embodiments, the electronic device estimates the OIS 2D translation information T_Path based on the camera rotation information C_Path and the frame rotation information FR_Path, and uses the estimated OIS 2D translation information T_Path. DIS correction can be performed more accurately by reflecting the stabilized camera rotation information into motion compensation.

In the following description, the movement of the camera module itself (Body) is referred to as camera rotation information (C_Path), and movement information by lens shift or sensor shift is referred to as OIS 2D translation information (T_Path), and the camera module 320 ), the frame data and motion information based on the motion vector are referred to as frame rotation information (FR_Path).

3 is a block diagram illustrating an image signal processor in accordance with some embodiments.

Referring to FIG. 3 , the camera module 100 may include a frame angular velocity sensor 151 and a gyro sensor 153 according to some embodiments.

The frame angular velocity sensor 151 outputs frame data (Frame info). Specifically, the frame angular velocity sensor 151 may output instantaneous angular velocity information of the frame image whenever a frame time stamp is taken at a preset period. The motion vector module 210 may output position change information of feature points in the frame whenever a frame stamp is taken. The image sensor 110 generates a frame image, and the frame data generated from the frame angular velocity sensor 151 includes a frame time stamp and location information of a feature point in the frame image.

The gyro sensor 153 outputs gyro data (Gyro info). Specifically, the gyro sensor 153 may output instantaneous angular velocity information of the camera module itself, that is, the body, whenever a gyro time stamp is taken at a preset period. That is, the gyro data includes a gyro time stamp and location information (eg, three-dimensional coordinates) of the camera body.

According to some embodiments, the ISP 201 includes a motion vector module 210 , a motion vector (MV) base motion estimation unit 220 , an OIS 2D translation information estimation unit 230 , a gyro-based motion estimation unit 240 , It may include a camera path optimizer 250 , a motion compensation calculation unit 260 , and a geometric distortion compensation unit 270 .

The image signal processor 201 receives frame rotation information (FR_Path) and camera rotation information (C_Path) from frame data,/or motion data, and gyro data, and extracts OIS 2D translation information (T_Path, T_Path), and OIS 2D The camera rotation information is optimized by reflecting the translation information (T_Path), and a digitally stabilized final image can be output through the optimized body rotation information and the OIS 2D translation information (T_Path).

In the raw image, each pixel within one frame image (for example, the first frame generated from the first frame time stamp) according to the movement of the rolling shutter is different depending on the position of the camera body for each line or the rotation angle of the frame. It is sensed at a point in time, and location information is changed with respect to a feature point of the same subject. The MV-based motion estimator 220 may extract a change between the pixel position information, that is, rolling shutter information. This is called intra-frame rotation information (hereinafter, Intra_FR_Path). In addition, the final camera rotation information (Final C Rpath info.) in each frame is extracted by accumulating inter-frame rotation information (Inter_FR_Path) of the camera.

According to some embodiments, the motion vector module 210 receives the frame image and the frame data and extracts motion vector information (MV) of the frame image, that is, motion information of the frame. The motion vector information (MV) may be motion information extracted from a frame image. According to some embodiments, the MV-based motion estimator 220 extracts frame rotation information (FR_Path) based on frame data (Frame info) and/or motion vector information (MV). Frame rotation information (FR_Path) is inter-frame rotation information (Inter-frame rotation info.; Inter-FR_Path), intra-frame rotation information (Intra-frame rotation info.; Intra-FR_Path) and inter-frame rotation information is accumulated It may include final frame rotation information (Final FR_Path) obtained by summing.

Specifically, the MV-based motion estimation unit 220 generates a first frame time stamp generated from a first frame time stamp based on frame data and motion vector information (MV) received from the frame angular velocity sensor 151 and the motion vector module 210 . An angular change between the 'first rotation information' of the frame and the 'second rotation information' of the second frame generated from the second frame time stamp is extracted. In this case, the extracted information is referred to as inter-frame rotation information (Inter-Frame Rotation Path, hereinafter Inter_FR_Path). The final frame rotation information (Final FR_Path) in each frame is extracted by accumulating inter-frame rotation information of frames.

The gyro-based motion estimation unit 240 extracts camera rotation information based on the gyro data. Specifically, the gyro-based motion estimation unit 230 compares the gyro data, that is, the 'first rotation information' generated from the first gyro time stamp with the 'second rotation information' generated from the second gyro stamp, and The position change in the 3D world coordinate system between the 1st gyro time stamp and the 2nd gyro time stamp (including movement amount, movement direction, and rotation angular velocity of each of the x, y, and z axes, for example) can be extracted as camera rotation information (C_Path). can The camera rotation information (C_Path) is also inter-frame camera rotation information (Inter-frame Camera Rotation Info.; hereafter referred to as Inter-C_Path) and intra-frame camera rotation information (Intra-frame Camera Rotation Info.; hereafter referred to as Intra-C_Path), the final Includes camera rotation information (Final C_Path). The final camera rotation information (Final C_Path) may be calculated by accumulating the inter-frame camera rotation information (Inter_C_Path).

The OIS 2D translation information estimator 230 extracts OIS two-dimensional translation information (T_Path), which is information related to a lens shift or a sensor shift, based on the frame rotation information (FR_Path) and the camera rotation information (C_Path). The OIS 2D translation information T_Path may be information that the lens assembly 310 in the camera module moves on the same plane (Plane Shift) or information that the image sensor 110 moves on the same plane.

The OIS 2D translation information (T_Path) can be obtained by multiplying, for example, a difference value (eg, an angle value) between the camera rotation information (C_Path) and the frame rotation information (FR_Path) by the focal length of the image sensor or lens. . In this case, both the inter frame OIS 2D translation information (hereinafter, Inter_T_Path) and the intra frame OIS 2D translation information (Intra_T_Path) are obtained in the same manner. (Specifically, Inter_T_Path = Inter_C_Path - Inter_FR_Path, and it can be obtained as Intra_T_Path = Intra_C_Path - Intra_FR_Path.) And the final OIS 2D translation information (Final T_Path) is extracted by accumulating the OIS 2D translation inter-frame information (Inter_T_Path). .

The OIS 2D translation intra-frame information (Intra_T_Path) obtained here is used together with the intra-frame camera rotation information (Intra_C_Path) for image rolling shutter correction (for correcting pixel movement according to the shutter movement).

A detailed operation will be described in detail with reference to FIGS. 5 to 11 .

The camera path optimizer 250 refers to the OIS 2D translation information (T_Path) and the camera rotation information (C_Path) to calculate the stabilized camera movement information (S), which is a path that the camera module 100 itself moves. The camera path optimizer 250 optimizes the camera rotation information (C_Path) by referring to the extracted OIS 2D translation path (T_Path), the extracted intra OIS 2D translation information (Intra_T_Path), and the intra camera rotation information (Intra_C_Path). carry out This optimizing operation also includes distortion correction by the rolling shutter.

The rolling shutter refers to, according to some embodiments, when the image sensor is a CMOS sensor, the image signal processor 201 performs a line from the uppermost line to the lowermost line in the pixel array while the image sensor acquires a frame-by-frame raw image. It is possible to read out the light reflected in order up to . An operation in which the image sensor reads out light in line units may be referred to as a rolling shutter operation.

That is, in the rolling shutter method, since the time point at which the light reflected from the subject enters the image sensor 201 is different for each line, image distortion may occur even within one frame due to the time difference of the light. Because distortion caused by rolling shutter operation is caused by both camera rotation (C_Path) and OIS 2D translation (T_Path), the intra camera rotation information (Intra_C_Path) and intra OIS 2D translation extracted from S10 and S30 By using the information (Intra_T_Path), it is possible to correct the image distorted by the rolling shutter operation.

The motion compensation calculator 260 includes intra-frame camera rotation information (Intra_C_Path), intra-frame OIS 2D translation information (Intra_T_Path), final OIS 2D translation information (Final T_Path), final camera rotation information (Final C_Path), and A motion compensation amount for the input frame image is determined based on the stabilized camera motion information (S). That is, the motion compensation calculator 260 calculates a rotation correction amount corresponding to the intra-frame camera rotation information (Intra_C_Path) due to the rolling shutter operation in the stabilized camera motion information S, and the intra-frame OIS 2D translation information (Intra_T_Path). ), the final camera rotation information (Final C Path) and the final OIS 2D translation information (Final T_path) are reflected to determine the correction amount at each individual grid point of the frame image output from the image sensor 110 .

On the other hand, when the stabilized camera motion information S is applied to the frame image, the input pixel position of the pixel position of the corrected frame image is calculated to exist within the changed boundary margin range. A detailed description will be given with reference to FIGS. 10 to 11 .

The distortion compensation calculation unit 270 receives the correction amount of the individual grid calculated by the motion compensation calculation unit 260 as an input and applies it to the entire frame image output from the camera module 100 (eg, binning or interpolation). (Interpolation)) to perform digital image stabilization.

4 is a flowchart illustrating an image stabilization method according to some embodiments.

Referring to FIG. 4 , the image signal processor 201 performs the camera rotation information ( C_Path) and frame rotation information (FR_Path) are estimated.

Specifically, the image signal processor 201 receives the first gyro data (Gyro angle_t1, Gyro timestamp_t1) and the second gyro data (Gyro angle_t2, Gyro timestamp_t2), the motion information of the camera module 100, and rotation between the t1 and t2 time points. It is possible to estimate the final camera rotation information (Final C_Path) by calculating the angle change amount and accumulating it (S10). The estimated camera rotation information (C_Path) includes intra-frame camera rotation information (Intra_C_Path), inter-frame camera rotation information (Inter_C_Path), and final camera rotation information (Final C_Path). The camera rotation information (C_Path) may be referred to as three-dimensional (eg, x, y, z-axis) position change information based on the timestamp of the gyro sensor 153 , that is, rotational displacement of the camera module 100 . have. For example, the final camera rotation information (Final C_Path) may be calculated by accumulating the inter-frame camera rotation information (Inter_C_Path).

The image signal processor 201 receives the first frame data (Frame_t1, Frame timestamp_t1) at the time t1 and the second frame data (Frame_t2, Frame timestamp_t2) at the time t2 through the frame angular velocity sensor 151 and the motion vector module 210 , it is possible to estimate feature point motion information of the object between time t1 and time t2, that is, frame rotation information FR_Path (S20). In this case, the frame rotation information (FR_Path) includes intra-frame rotation information (Intra_FR_Path), inter-frame rotation information (Inter_FR_Path), and final frame rotation information (Final FR_Path). The final frame rotation information (Final FR_Path) may be calculated by accumulating the inter-frame rotation information (Inter_FR_Path).

The image signal processor 201 extracts OIS 2D translation information (T_Path) from the camera rotation information (C_Path) and the frame rotation information (FR_Path) (S30). Specifically, the OIS 2D translation information (T_Path) is estimated by multiplying the angle change value obtained by subtracting the frame rotation information from the camera rotation information by the converted focal length of the image sensor 110 in the camera module 100 . can The OIS 2D translation information (T_Path) may include inter-frame 2D translation information (Inter_T_Path), intra-frame 2D translation information (Intra_T_Path), and final OIS 2D translation information (Final T_Path). For example, inter-frame 2D translation information (Inter_T_Path) may be estimated based on inter-frame camera rotation information (Inter_C_Path) and inter-frame rotation information (Inter_FR_Path), and intra-frame 2D translation information (Intra_T_Path) may be estimated based on intra-frame camera rotation information (Intra_C_Path) and intra-frame 2D translation information (Intra_T_Path). That is, the final OIS 2D translation information (Final T_Path) may be calculated by accumulating the inter-frame 2D translation information (Inter_T_Path).

The image signal processor 201 removes bias values mixed in the estimated OIS 2D translation information T_Path to increase estimation accuracy ( S40 ). The bias value is an offset value possessed by the frame angular velocity sensor 151 or the gyro sensor 153. When the final OIS 2D translation information is estimated by integrating the inter-frame translation information (Inter_T_Path), the offset value is a low frequency component. may appear Since the final OIS 2D translation information (Final T_Path) cannot have a value larger than the image size of the image, it is necessary to consider that an offset value is included in the OIS 2D final translation path (Final T_Path) for more accurate estimation.

The image signal processor 201 , that is, the OIS 2D translation information estimator 230 applies a high-frequency filter to the final OIS 2D translation path (Final T_Path) estimated according to some embodiments to remove the low-frequency component of OIS 2D. The translation path (filtered_T_Path) is used in subsequent steps (S50 and S60). In this case, the high frequency filter may filter based on an arbitrary threshold frequency, and the threshold frequency serving as a filtering criterion may be tuned. That is, the final OIS 2D translation path (Final T_Path) may remove the low frequency component while leaving only the high frequency component based on the tuning threshold frequency. In an embodiment, the high-frequency filter may be implemented as a 1-Pole IIR filter.

The camera path optimizer 250 uses the filtered OIS 2D translation information (filtered_T_Path), the intra OIS 2D translation information (Intra_T_path), and the camera rotation information (C_Path) to the actual movement of the camera module 100 , that is, stabilized Estimate camera motion information (S). The camera rotation information (C_Path) used in this case may be intra prime camera rotation information (Intra_C_Path) and final camera rotation information (Final_C_Path).

The image signal processor 201 includes the stabilized camera motion information (S), filtered OIS 2D translation information (filtered T_Path) and intra-frame camera rotation information (Intra_C_Path) and intra-frame camera translation information (Intra_T_Path), Based on the camera rotation path, motion compensation of the input image may be performed. On the other hand, when the image signal processor 201 applies a preset boundary margin of the original input image (Raw Image) after motion compensation, the camera module 100 moves or OIS 2D translation into a range greater than or equal to the preset boundary margin. If it is moved, some images of a portion not filled in the boundary margin may not be displayed properly (eg, the corresponding portion may be displayed as a blank screen (eg, black)). In order to prevent this phenomenon, the image signal processor 201 obtains the stabilized camera motion information S within a range that satisfies the adjusted boundary margin range S50. The boundary margin adjustment will be described later with reference to FIGS. 10 and 11 .

Thereafter, the image signal processor 201 applies final motion compensation information that satisfies the boundary margin to the frame image to complete digital image stabilization (S60).

5 is a conceptual diagram illustrating an image stabilization method according to some embodiments.

를 시간에 대해 적분하면 거리

로 환산될 수 있다(즉,

)The camera rotation information (C_Path), the frame rotation information (FR_Path), and the OIS 2D translation information (T_Path) will be described in more detail. Angular velocity measured between the first time point t1 and the second time point t2

Integrating over time gives the distance

can be converted to (i.e.,

)

를 추출하고, MV베이스 모션 추정부(220)는 프레임 각속도 센서(151)의 출력값(Frame data, Frame timestamp)나 모션 벡터로부터 프레임 로테이션 정보(FR_Path)

를 추출한다. Referring to FIG. 5 , the gyro-based motion estimation unit 230 receives camera rotation information (C_Path) from an output value (Gyro data, Gyro timestamp) of the gyro sensor 153 .

, and the MV base motion estimation unit 220 receives frame rotation information (FR_Path) from the output value (Frame data, Frame timestamp) or motion vector of the frame angular velocity sensor 151 .

to extract

, A)로 산출될 수 있다. 프레임 로테이션 정보(FR_Path)는 센싱된 이미지의 이전 프레임과 현재 프레임 간의 로테이션 정보를 활용한다. 즉프레임 센서의 타임 스탬프 정보를 기초로 동일한 객체(예를 들어 특징점)의 t1 시점(Frame timestamp_t1)에서의 회전 정보와 t2 시점(Frame timestamp_t2)의 회전 정보를 비교하여 프레임 환산 회전 움직임

로 산출될 수 있다. 이때 t1 시점과 t2 시점은 기설정된 주기로 발행되는 타임 스탬프의 서로 다른 시점이다. The camera rotation information (C_Path) is rotation information of the camera module 100 itself, based on the timestamp information of the gyro sensor 153, rotation information of the time t1 (Gyro timestamp_t1) and rotation information of the time t2 (Gyro timestamp_t2). Compare camera rotation information (

, A) can be calculated. The frame rotation information FR_Path utilizes rotation information between the previous frame and the current frame of the sensed image. That is, based on the time stamp information of the frame sensor, the rotation information at time t1 (Frame timestamp_t1) of the same object (for example, feature point) is compared with the rotation information at time t2 (Frame timestamp_t2), and frame-converted rotation movement

can be calculated as In this case, time t1 and time t2 are different time points of time stamps issued at a preset period.

At this time, each of the camera rotation information (C_Path) and the frame rotation information (FR_Path) includes motion information within the same frame (Intra_C_Path, Inter_FR_Path), motion information between different frames (Inter_C_Path, Inter_FR_Path), and inter-frame motion. may include final rotation information (Final C_Path, Final_FR_Path) obtained by accumulating . According to some embodiments, the frame angular velocity sensor time stamp (Frame timestamp) and the gyro sensor time stamp (Gyro timestamp) may have the same period, different periods or a period of a predetermined multiple.

)와 최종 OIS 2D 트랜스레이션 정보(Final T_Path,

)가 조합된 최종 결과 위치(

+

)를 각속도만 고려한 회전 움직임으로 변환했을 때 추정되는 프레임의 회전 움직임(B)을 의미한다. 실제로 카메라 모듈(100)에서의 프레임 이미지의 움직임은 최종 카메라 로테이션 정보(Final C_Path,

, A)에 따른 움직임 후, 최종 OIS 2D 트랜스레이션 정보(Final T_Path,

)에 따라 평행이동하여, B' 위치로 이동한 것이다. 그러나, 이미지시그널 프로세서(201)에 최종 OIS 2D 트랜스레이션 정보(Final T_Path,

)가 센서로부터 제공되지 않으면, 현재 프레임 로테이션 정보는 단순히 모션 벡터(MV)의 회전 움직임(

)으로만 고려된 위치 B(position)에 기초한 것이다.The final frame rotation information (Final FR_Path) is the final camera rotation information (Final C_Path,

) and final OIS 2D translation information (Final T_Path,

) combined final result position (

+

) means the rotational movement (B) of the frame estimated when converted into rotational movement considering only the angular velocity. In fact, the movement of the frame image in the camera module 100 is the final camera rotation information (Final C_Path,

, after the movement according to A), the final OIS 2D translation information (Final T_Path,

) and moved to position B'. However, the final OIS 2D translation information (Final T_Path,

) is not provided from the sensor, the current frame rotation information is simply the rotational motion of the motion vector (MV) (

) is based on the position B (position) considered only.

)를 기초로 보정한다. 만약 OIS 2D 트랜스레이션 정보(

, B')의 활용 없이 프레임 로테이션 정보(

)만을 기초로 이미지 안정화를 수행하면, 출력된 프레임 이미지에 워블링이 나타날 수 있다. 여기서 안정화하려는 카메라 로테이션 정보(C_Path)는 카메라 인터-프레임 로테이션 정보(Inter_C_Path)를 누적합하여 산출한 <수학식 1>의 최종 카메라 로테이션 회전 경로(

)를 의미한다.The final image output from the image signal processor 201 is stabilized camera rotation information (

) is corrected based on If OIS 2D translation information (

, B') without using frame rotation information (

), wobbling may appear in the output frame image if image stabilization is performed based on only. Here, the camera rotation information (C_Path) to be stabilized is the final camera rotation rotation path (Equation 1) calculated by accumulating the camera inter-frame rotation information (Inter_C_Path)

) means

, T_Path )를 추정함으로써, B가 아닌 B'의 위치를 산출할 수 있다. 또한 이미지 시그널 프로세서(201)는 추정된 OIS 2D 트랜스레이션 정보(T_Path)를 안정화된 카메라 로테이션 정보(Optimized C_Path)의 산출에 이용하여, 디지털 이미지 안정화를 보다 정확하게 수행할 수 있다. Accordingly, even when the OIS 2D translation information (T_Path) is not separately provided, the image signal processor 201 performs the OIS 2D translation information (

, T_Path ), it is possible to calculate the location of B' instead of B. In addition, the image signal processor 201 may use the estimated OIS 2D translation information (T_Path) to calculate the stabilized camera rotation information (Optimized C_Path) to more accurately perform digital image stabilization.

<Equation 1>

)에서 프레임 로테이션 정보(

)을 차감한 각도값

에 비례한다. 이 때 수학식 1의 의 ,

는 인터-프레임 카메라 로테이션 정보(Inter C_Path)을 누적합하여 산출한 최종 카메라 로테이션 정보(Final C_Path)이고,

은 프레임의 인터-프레임 로테이션 정보(Inter FR_Path)을 누적합하여 산출한 최종 프레임 로테이션 정보(Final FR_Path)를 의미한다. Specifically, with reference to Equation 1, the OIS 2D translation information (T_Path, T in Equation 1) is the camera rotation information (

) in the frame rotation information (

) minus the angle value

proportional to In this case, in Equation 1,

is the final camera rotation information (Final C_Path) calculated by accumulating inter-frame camera rotation information (Inter C_Path),

denotes final frame rotation information (Final FR_Path) calculated by accumulating inter-frame rotation information (Inter FR_Path) of frames.

는 프레임 각속도 센서(151)에서 추정한 프레임 환산 회전의 인트라-프레임 x축 회전값 및 y축 회전값을 적용하고,

는 카메라 각속도에서 추정한 카메라 회전의 인트라-프레임 x축 회전값 및 y축 회전값을 적용하여, OIS 2D 트랜스레이션의 인트라-프레임 x축 회전값 및 y축 회전값을 산출할 수 있다. Also in Equation 1

applies the intra-frame x-axis rotation value and the y-axis rotation value of the frame conversion rotation estimated by the frame angular velocity sensor 151,

may calculate the intra-frame x-axis rotation value and y-axis rotation value of the OIS 2D translation by applying the intra-frame x-axis rotation value and the y-axis rotation value of the camera rotation estimated from the camera angular velocity.

)는 카메라 로테이션 정보(

)에서 프레임 로테이션 정보(

)을 차감한 각도값

에 렌즈 어셈블리(101)의 환산 초점 거리(

)를 곱한 값으로 구할 수 있다. OIS 2D 트랜스레이션 정보(T_Path)의 y축 움직임 패스(

)는 카메라 로테이션 정보(

)에서 프레임 로테이션 정보(

)을 차감한 각도값

에 렌즈 어셈블리(101)의 환산 초점 거리(

)를 곱한 값으로 구할 수 있다. 한편, 몇몇 실시예에 따라 OIS 움직임 이미지의 좌표는 좌측 상부 모서리를 기준좌표(0,0)으로 할 경우 반대방향으로 움직인 것으로 보아 산출된 값에 '-'부호를 붙일 수도 있으나, 기준 좌표의 위치에 따라 '+'부호를 붙일 수도 있다.The x-axis movement path of OIS 2D translation information (T_Path) (

) is the camera rotation information (

) in the frame rotation information (

) minus the angle value

The reduced focal length of the lens assembly 101 (

) can be obtained by multiplying The y-axis movement path of OIS 2D translation information (T_Path) (

) is the camera rotation information (

) in the frame rotation information (

) minus the angle value

The reduced focal length of the lens assembly 101 (

) can be obtained by multiplying On the other hand, according to some embodiments, the coordinates of the OIS motion image may be assigned a '-' sign to the calculated value as it is considered to have moved in the opposite direction when the upper left corner is the reference coordinate (0,0). Depending on the position, a '+' sign may be attached.

According to some embodiments, the 2D translation information estimator 220 may estimate the final OIS 2D translation information (Final T_Path) as described above.

6 is a graph illustrating an X-axis frequency component of OIS 2D translation information, according to some embodiments.

The image signal processor 201 may apply a high-frequency filter to the final OIS 2D translation information (Final T_Path). The final OIS 2D translation information shown in FIG. 6 is an OIS translation path obtained by accumulating inter-frame OIS 2D translation values.

The OIS 2D translation information (T_Path), that is, the moving path cannot actually deviate from the size of the image received from the image sensor 110 . However, the OIS 2D translation information T_Path includes not only gyro data used for OIS motion estimation, but also a bias value (ie, an offset value) generated during frame data generation. If the final OIS 2D translation information (Final T_Path) is calculated without removing the bias value and used for digital image stabilization, the final output stabilized image exceeds the boundary margin of the sensed frame image size, and some regions appear. it may not be Therefore, in order to remove a bias value when calculating the final OIS 2D translation information (Final T_Path), the image signal processor 201 may apply a high-frequency filter to the final OIS 2D translation information (Final T_Path).

The threshold frequency of the high-frequency filter is tunable. In the input image data, a high frequency band greater than the threshold frequency is passed based on the threshold frequency, and a low frequency band smaller than the threshold frequency is filtered. If only the bias component is removed and the high frequency component is left with almost no loss, the tuning threshold frequency must be set appropriately. For convenience of explanation, the OIS 2D translation information including the bias component is called raw OIS 2D translation information (raw T_Path), and the OIS 2D translation information from which the bias component is removed is the filtered OIS 2D translation. It is called information (filtered T_Path).

According to some embodiments, the high-frequency filter may use a 1-pole IIR filter. As a result, as shown in FIG. 6 , it is possible to extract a low frequency component from the raw OIS 2D translation information.

7 is a graph illustrating an X-axis final high-frequency component of motion information for an image, according to some embodiments. 8 is a graph illustrating a Y-axis component in which a low-frequency component and a high-frequency component coexist in FIG. 7 . 9 is a graph illustrating a state in which a low-frequency component is removed from the Y-axis movement of FIG. 8 and only a Y-axis high-frequency component remains, according to some embodiments.

For example, as shown in FIG. 7 , the x-axis raw OIS 2D translation information (Raw T_Path) may include a low frequency component. By filtering the low frequency component, as shown in FIG. 8 , a filtered OIS 2D translation path (filtered T_Path) of the x-axis including only the high frequency component can be obtained.

)를 얻을 수 있다.For example, as shown in FIG. 8 , the y-axis raw OIS 2D translation information (Raw T_Path) may include a low frequency component. When the low frequency component is filtered, the filtered OIS 2D translation information (filtered T_Path,

) can be obtained.

Accordingly, the filtered OIS 2D translation information (filtered T_Path) of the x-axis and the y-axis may include OIS shift information with improved accuracy by removing a bias component.

) 및 필터링된 OIS 2D 트랜스레이션 정보(filtered T_Path,

)에 기초하여 디지털 이미지 안정화를 수행할 수 있다. 몇몇 실시예에 따라 카메라 패스 옵티마이저(240)는 필터링된 OIS 2D 트랜스레이션 정보(filtered T_Path,

)를 반영하여, 카메라 로테이션 정보(C_Path)를 옵티마이징할 수 있다.Referring back to FIGS. 2 and 5 , the image signal processor 201 includes the camera rotation information (C_Path,

) and filtered OIS 2D translation information (filtered T_Path,

) based on digital image stabilization. According to some embodiments, the camera path optimizer 240 is filtered OIS 2D translation information (filtered T_Path,

), the camera rotation information (C_Path) may be optimized.

) 와 OIS 2D 트랜스레이션 정보(T_Path,

)가 반영된 결과로 촬상된다. 따라서 디지털 이미지 안정화는 카메라 로테이션 정보(C_Path) 및 필터링된 2D 트랜스레이션 정보(filtered T_Path,

)를 반영할 때, 변경된 바운더리 마진 범위(도 11 및 도 12)를 만족하도록, 안정화된 카메라 로테이션 정보(Optimized C_Path,

)를 도출해야 한다. The image signal processor 201 should perform digital image stabilization in consideration of motion information based on 3D spatial coordinates on a frame image captured with 2D image coordinates. The input frame image is camera rotation information (C_Path,

) and OIS 2D translation information (T_Path,

) is captured as the reflected result. Therefore, digital image stabilization includes camera rotation information (C_Path) and filtered 2D translation information (filtered T_Path,

), stabilized camera rotation information (Optimized C_Path,

) should be derived.

), 카메라 인트라-프레임 로테이션 정보(

), OIS 2D 트랜스레이션 정보(filtered T_Path,

)와 인트라-프레임 OIS 트랜스레이션 정보(Inter_T_Path)의 조합이므로, 수학식 2와 같이 산출될 수 있다. Specifically, for image stabilization, the movement of the image (input frame image) before image stabilization is

), camera intra-frame rotation information (

), OIS 2D translation information (filtered T_Path,

) and intra-frame OIS translation information (Inter_T_Path), it can be calculated as in Equation (2).

<Equation 2>

는 3차원 월드 좌표계의 한 지점 P에 카메라 모듈(100)의 카메라 로테이션 정보(

)을 적용하고 그 결과를 카메라 모듈(100)의 내부 프로젝션 매트릭스(K)를 곱하여 픽셀 좌표계로 변환한 후, 필터링된 2D 트랜스레이션 정보(filtered T_Path,

)를 적용한 결과로 나타난다. 이 때 인트라-프레임 카메라 로테이션 정보(Intra C_Path)와 인트라-프레임 2D 트랜스레이션 정보(Intra_T_Path)를 반영하여 롤링 셔터에 의한 왜곡을 보정한다. That is, pixel coordinates in the input frame image

is camera rotation information of the camera module 100 at a point P in the three-dimensional world coordinate system (

) and multiply the result by the internal projection matrix (K) of the camera module 100 to convert it into a pixel coordinate system, and then filter 2D translation information (filtered T_Path,

) as a result of applying At this time, the distortion caused by the rolling shutter is corrected by reflecting the intra-frame camera rotation information (Intra C_Path) and the intra-frame 2D translation information (Intra_T_Path).

)는 안정화의 대상으로 인터-프레임 카메라 로테이션 정보(Inter C_Path)를 누적합하여 산출한 최종 카메라 로테이션 경로(Final C_Path)를 로테이션 매트릭스로 나타낸 값이다. Camera rotation information (

) is a rotation matrix representing the final camera rotation path (Final C_Path) calculated by accumulating inter-frame camera rotation information (Inter C_Path) as an object of stabilization.

)는 몇몇 실시예에 따라 x축 쉬프트값(Tx)과 y축 쉬프트값(Ty)가 모두 고려된, 3x3 매트릭스일 수 있고, 카메라 내부 프로젝션 매트릭스는 수학식 3과 같을 수 있다.At this time, the filtered OIS 2D translation information (filtered T_Path,

) may be a 3x3 matrix in which both the x-axis shift value (Tx) and the y-axis shift value (Ty) are considered, and the camera internal projection matrix may be as Equation (3).

<Equation 3>

Equation 3 represents a transformation relationship matrix K between a point P(x,y,z) of the world coordinate system and a two-dimensional pixel coordinate point formed after this point is projected by a camera sensor, that is, a projection matrix. f is the converted focal length, and (W, H) is the number of horizontal and vertical pixels of the input image frame. For convenience of calculation, if the image pixel coordinate center is set to (W/2, H/2), Equation 3 is changed to Equation 4 below.

<Equation 4>

)는 3차원 월드 좌표계의 한 지점 P에 카메라 모듈의 안정화된 카메라 로테이션 정보(

)를 반영한 뒤 여기에 카메라 내부 프로젝션 매트릭스를 적용하여 산출할 수 있다.The final image after digital image stabilization may be calculated as in Equation 5. The pixel coordinates of the stabilized image (

) is the stabilized camera rotation information of the camera module at a point P in the 3D world coordinate system (

) can be reflected and calculated by applying the camera's internal projection matrix to it.

<Equation 5>

)에 대한 입력 프레임 이미지의 좌표(

)는 백워드 워핑(backward warping) 방식으로 월드 좌표계(3차원 좌표계)로 변환한 후(

), 안정화 경로와 로우 경로의 차이 즉 카메라 보정 회전값 (

,

)를 반영하고 뒤이어 OIS 2D 트랜스레이션 정보를 반영하여 산출될 수 있다. 이 때 인트라-프레임 카메라 로테이션(Intra C_Path)과 인트라-프레임 2D 트랜스레이션 정보(Intra_T_Path)를 반영하여 롤링 셔터 효과를 보정한다.As in Equation 7, the final frame image coordinates (

) to the coordinates of the input frame image (

) after converting to the world coordinate system (three-dimensional coordinate system) in a backward warping method (

), the difference between the stabilization path and the raw path, i.e. the camera compensation rotation value (

,

) and subsequently OIS 2D translation information can be reflected and calculated. At this time, the rolling shutter effect is corrected by reflecting the intra-frame camera rotation (Intra C_Path) and the intra-frame 2D translation information (Intra_T_Path).

)은 입력된 프레임 이미지(

)에 반영되는 투영 변환 매트릭스(

)를 산출할 수 있다. Equation 6 is Equation 4 and Equation 5 applied to Equation 2. That is, the final frame image (

) is the input frame image (

) reflected in the projection transformation matrix (

) can be calculated.

<Equation 6>

(Optimized C_Path)를 추정한다.According to some embodiments, the camera path optimizer 240 provides stabilized camera rotation information.

(Optimized C_Path) is estimated.

10 is a graph illustrating an input pixel position range of a crop point of a final image before OIS compensation is applied, according to some embodiments. FIG. 11 is a diagram for explaining that an input pixel position range for rotational motion is translated by an OIS compensation amount after OIS compensation is applied, according to some embodiments.

When the OIS optical correction is applied, the image signal processor adjusts the range of the boundary margin for the correction of the camera rotation movement C_Path to the image captured by the image sensor 110 by the OIS optical correction amount. Referring to FIG. 10 , the image actually captured before the OIS optical correction is applied is region B, but the processed image is cropped and only region A is displayed. That is, the actually captured image (B) is cropped by Xmargin on both sides of the first direction from the inside of the boundary on the basis of the center coordinate (0,0) of the image, and both sides of the second direction are cropped by Ymargin from the inside of the boundary, respectively. final output. At this time, the area between the boundary of the output area A and the boundary of the actually captured image area B is referred to as a boundary margin.

=

=0인 경우, 카메라 모듈(100)에서 촬상된 입력 픽셀들은 B 영역에 있고, 이미지 안정화 후에 크롭되어 출력되는 대상은 A 영역의 이미지이다. 즉, 실제로 촬상되는 B 영역은 너비 W, 높이 H로 촬상되고, 이미지 안정화 후 출력되는 A 영역은 B 영역 좌우로 각각 Xmargin 및 상하로 각각 Ymargin만큼 영상 중심부(0,0) 안쪽으로 크롭된다. 출력되는 A 영역의 꼭지점 좌표는 수학식 7과 같이 산출될 수 있다.Specifically, referring to FIG. 10 and Equation 7, when there is no compensation for OIS 2D translation, that is, in Equation 7

=

When =0, input pixels captured by the camera module 100 are in region B, and the cropped and output object after image stabilization is an image of region A. That is, the area B actually imaged is imaged with a width W and a height H, and the area A output after image stabilization is cropped to the inside of the image center (0,0) by Xmargins to the left and right of the B area, and Ymargins up and down, respectively. The output coordinates of the vertices of the region A may be calculated as in Equation (7).

<Equation 7>

However, the center coordinate of the captured image is shifted from (0,0) to (Tx, Ty) due to motion compensation based on camera rotation or OIS 2D translation. That is, the coordinates of the vertices of the region B of FIG. 10 are moved together with the coordinates of the vertices of the region B′ of FIG. 11 .

In this case, when the position of the input pixel of the cropped region A is within the motion-corrected region B′ (moved within the boundary margin), the output image may be normally output without a cut portion. However, if the input pixel of the cropped area A exceeds the area B' (exceeds the boundary margin), the output image appears as black or the like. Therefore, when the input image is shifted by optical correction (B→B'), the boundary margin of the crop region A (right of Table 1) must also be changed.

) 도 12에서 설명한 바와 같이 움직임 보정 전의 바운더리 마진 조건(표 1의 왼쪽)은, Tx 및 Ty 정보가 포함된 움직임 보정 후의 바운더리 마진(표 1의 오른쪽)과 같이 변경한다.On the other hand, the correctable range of camera rotation is (

) As described in FIG. 12 , the boundary margin condition before motion compensation (left of Table 1) is changed to the same as the boundary margin after motion compensation including Tx and Ty information (right of Table 1).

<Table 1>

는 카메라 회전에 의한 카메라 로테이션 정보(C_Path), 렌즈 또는 센서 쉬프트에 의한 OIS 2D 트랜스레이션 정보(T_Path)를 반영하여 구한 안정화된 카메라 로테이션 정보

와 카메라 로테이션 정보

의 차이값으로 수학식 8과 같은 변형 메트릭스, 예를 들면 로드리게스 회전 매트릭스(Rodrigues' rotation matrix) 로 근사해서 나타낼 수 있다. For example, the image warping angle for rotation stabilization (

is stabilized camera rotation information obtained by reflecting camera rotation information (C_Path) by camera rotation and OIS 2D translation information (T_Path) by lens or sensor shift

and camera rotation information

As a difference value of , it can be approximated by a transformation matrix such as Equation 8, for example, a Rodrigues' rotation matrix.

<Equation 8>

According to some embodiments, the motion compensation calculator 260 may generate a deformation matrix (eg, Equation 8) and apply the deformation matrix to the input frame image.

According to some embodiments, the geometry distortion compensator 270 receives the grid change value of each pixel to which motion compensation is applied from the motion compensation calculation unit 260, and performs digital processing such as binning or interpolation to reflect the entire input frame image. and output the final frame image.

Accordingly, the electronic device including the camera module 100 to which the lens or image sensor of the present invention is shifted obtains camera rotation information and frame rotation information from the sensors, and even if there is no separate OIS motion information, the motion vector module value Alternatively, image stabilization may be performed by calculating OIS 2D translation information from sensing output values of the sensor. Accordingly, when both camera and frame rotation information are obtained from the sensor, optical and digital image stabilization is performed to output a stabilized image without wobbling even in a low-light environment or a background lacking repeating patterns or feature points occupies most of the screen. There are advantages that can be

12 is a block diagram of an electronic device in a network environment according to some embodiments.

Referring to FIG. 12 , in the network environment 1000 , the electronic device 1101 communicates with the electronic device 1102 through the first network 198 (eg, short-range wireless communication) or the second network 1199 ( For example, it may communicate with the electronic device 1104 or the server 1108 through long-distance wireless communication. According to an embodiment, the electronic device 1101 may communicate with the electronic device 1104 through the server 1108 . According to an embodiment, the electronic device 1101 includes a processor 1120 , a memory 1130 , an input device 1150 , a sound output device 1155 , a display device 1160 , an audio module 1170 , and a sensor module ( 1176), interface 1177, haptic module 1179, camera module 100, power management module 1188, battery 1189, communication module 1190, subscriber identification module 1196, and antenna module 1197 ) may be included. In some embodiments, at least one of the components (eg, the display device 1160 or the camera module 100 ) may be omitted from the electronic device 1101 , or other components may be added to the electronic device 1101 . In some embodiments, some components may be integrated and implemented, such as when a sensor module 1176 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) is embedded in the display device 1160 (eg, a display). can

According to an embodiment, the processor 1120 drives, for example, software (eg, a program 1140 ) to drive at least one other component (eg, hardware) of the electronic device 1101 connected to the processor 1120 . or a software component), and may perform various data processing and operations. The processor 1120 loads and processes commands or data received from other components (eg, the sensor module 1176 or the communication module 1190 ) into the volatile memory 1132 , and stores the result data in the non-volatile memory 1134 . can be stored in According to an embodiment, the processor 1120 is operated independently of the main processor 1121 (eg, central processing unit or application processor), and additionally or alternatively, uses less power than the main processor 1121, or Alternatively, the auxiliary processor 1123 (eg, a graphic processing unit, an image signal processor, a sensor hub processor, or a communication processor) specialized for a specified function may be included. The auxiliary processor 1123 may be operated separately from or embedded in the main processor 1121 .

In this case, the auxiliary processor 1123 may, for example, act on behalf of the main processor 1121 while the main processor 1121 is in an inactive (eg, sleep) state, or the main processor 1121 may be active (eg, in an active (eg) state). : While in the application execution) state, together with the main processor 1121, at least one of the components of the electronic device 1101 (eg, the display device 1160, the sensor module 1176, or the communication module ( 1190)) and at least some of the related functions or states. According to an embodiment, the coprocessor 1123 (eg, an image signal processor or a communication processor) is implemented as some component of another functionally related component (eg, the camera module 100 or the communication module 1190). can be The memory 1130 includes various data used by at least one component (eg, the processor 1120 or the sensor module 1176 ) of the electronic device 1101 , for example, software (eg, a program 1140 ). ) and input data or output data for commands related thereto. The memory 1130 may include a volatile memory 1132 or a non-volatile memory 1134 .

The program 1140 is software stored in the memory 1130 , and may include, for example, an operating system 1142 , middleware 1144 , or an application 1146 .

According to an embodiment, the input device 1150 receives a command or data to be used in a component (eg, the processor 1120 ) of the electronic device 1101 from the outside (eg, a user) of the electronic device 1101 . The device may include, for example, a microphone, a mouse, or a keyboard.

According to an embodiment, the sound output device 1155 is a device for outputting a sound signal to the outside of the electronic device 1101, for example, a speaker used for general purposes such as multimedia playback or recording playback, and a phone reception. It may include a dedicated receiver. According to an embodiment, the receiver may be formed integrally with or separately from the speaker.

According to an embodiment, the display device 1160 is a device for visually providing information to a user of the electronic device 1101 , for example, a display, a hologram device, or a projector and a control circuit for controlling the corresponding device. may include. According to an embodiment, the display device 1160 may include a touch circuitry or a pressure sensor capable of measuring the intensity of the pressure applied to the touch.

According to an embodiment, the audio module 1170 may interactively convert a sound and an electrical signal. According to an embodiment, the audio module 1170 acquires a sound through the input device 1150 , or an external electronic device (eg, a sound output device 1155 ) connected to the electronic device 1101 by wire or wirelessly. Sound may be output through the electronic device 1102 (eg, a speaker or headphones).

According to an embodiment, an electric signal or data value corresponding to an internal operating state (eg, power or temperature) of the electronic device 1101 or an external environmental state may be generated. The sensor module 1176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, a temperature sensor, a humidity sensor, Alternatively, it may include an illuminance sensor.

According to an embodiment, the interface 1177 may support a designated protocol that can be connected to an external electronic device (eg, the electronic device 1102 ) by wire or wirelessly. According to an embodiment, the interface 1177 may include a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

According to an embodiment, the connection terminal 1178 is a connector that can physically connect the electronic device 1101 and an external electronic device (eg, the electronic device 1102 ), for example, an HDMI connector, a USB connector, or an SD card. It may include a card connector, or an audio connector (eg, a headphone connector).

According to an embodiment, the haptic module 1179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can recognize through tactile or kinesthetic sense. The haptic module 1179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

According to an embodiment, the camera module 100 may capture still images and moving images. According to an embodiment, the camera module 100 may be configured and operated as described with reference to FIGS. 1 to 11 .

According to an embodiment, the power management module 1188 is a module for managing power supplied to the electronic device 1101 , and may be configured as, for example, at least a part of a power management integrated circuit (PMIC).

According to an embodiment, the battery 1189 is a device for supplying power to at least one component of the electronic device 1101, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell. may include

According to an embodiment, the communication module 1190 establishes a wired or wireless communication channel between the electronic device 1101 and an external electronic device (eg, the electronic device 1102 , the electronic device 1104 , or the server 1108 ). , and may support performing communication through an established communication channel. The communication module 1190 may include one or more communication processors that support wired communication or wireless communication, which are operated independently of the processor 1120 (eg, an application processor). According to an embodiment, the communication module 1190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 1194 (eg, : LAN (local area network) communication module, or power line communication module), and using a corresponding communication module among them, the first network 1198 (eg, Bluetooth, WiFi direct, or infrared data association (IrDA)) communication network) or the second network 1199 (eg, a cellular network, the Internet, or a telecommunication network such as a computer network (eg, LAN or WAN)). The above-described various types of communication modules 1190 may be implemented as a single chip or as separate chips.

According to an embodiment, the wireless communication module 1192 may distinguish and authenticate the electronic device 1101 in the communication network using user information stored in the subscriber identification module 1196 .

According to an embodiment, the antenna module 1197 may include one or more antennas for externally transmitting or receiving a signal or power. According to an embodiment, the communication module 1190 (eg, the wireless communication module 1192 ) may transmit a signal to or receive a signal from the external electronic device through an antenna suitable for a communication method.

Some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input/output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) to signal (eg commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 1101 and the external electronic device 1104 through the server 1108 connected to the second network 1199 . Each of the electronic devices 1102 and 1104 may be the same or a different type of device as the electronic device 1101 . According to an embodiment, all or some of the operations performed by the electronic device 1101 may be performed by one or a plurality of external electronic devices. According to an embodiment, when the electronic device 1101 is to perform a function or service automatically or upon request, the electronic device 1101 performs the function or service by itself instead of or in addition to it. At least some related functions may be requested from the external electronic device. Upon receiving the request, the external electronic device may execute the requested function or additional function, and may transmit the result to the electronic device 1101 . The electronic device 1101 may provide the requested function or service by processing the received result as it is or additionally. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those of ordinary skill in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: camera module
200: application processor
201: image signal processor
202: memory

Claims (20)

a camera module that generates and outputs gyro data and frame data for an input image; and
an image signal processor;
The image signal processor
a motion vector module for calculating motion vector information;
a gyro-based motion estimation unit for extracting camera rotation information of the camera module from the gyro data;
a motion vector (MV) based motion estimation unit for extracting frame rotation information from the frame data;
an OIS 2D translation information estimator for estimating Optical Image Stablizer (OIS) 2D translation information based on the motion vector information, the camera rotation information, and the frame rotation information;
Filtering the low-frequency component from the OIS 2D translation information,
and a camera path optimizer configured to calculate stabilized camera motion information of the camera module by using the filtered OIS 2D translation information and the camera rotation information. The method of claim 1, wherein the OIS 2D translation information is
An electronic device for calculating raw OIS 2D translation information by reflecting a converted focal length to a value obtained by subtracting the frame rotation information from the camera rotation information. The method of claim 2, wherein the filtering
An electronic device that removes a low frequency component less than a tuning threshold frequency from the raw OIS 2D translation information. The method of claim 1, wherein the camera path optimizer is
and extracting final motion information based on the stabilized camera motion information, the filtered OIS 2D translation information, and the camera rotation information. The electronic device of claim 4 , wherein the final motion information is adjusted according to a boundary margin of the input image. The electronic device of claim 2 , wherein the camera rotation information includes a final camera rotation path calculated by accumulating intra-frame camera rotation information, inter-frame camera rotation information, and the inter-frame camera rotation information. The electronic device of claim 2 , wherein the frame rotation information includes final frame rotation information calculated by accumulating intra-frame rotation information, inter-frame rotation information, and the inter-frame rotation information. 5. The method of claim 4,
Each of the frame images of the input image by reflecting the rotation correction amount corresponding to the intra-frame camera rotation information, the intra-frame OIS 2D translation information, the camera rotation information and the OIS 2D translation information in the stabilized camera motion information The electronic device further comprising a motion compensation calculation unit for calculating a correction amount at each grid point. The electronic device of claim 8 , further comprising a distortion compensation calculator configured to perform digital image stabilization by applying the correction amount of the motion compensation calculator to the entire frame image of the input image. a camera module for outputting gyro data including at least two gyro sensing values and frame data including at least two frame images; and
an image signal processor;
The image signal processor
calculating frame rotation information and camera rotation information based on the gyro data, the frame data, and the motion vector information;
estimating OIS 2D translation information based on the frame rotation information and the camera rotation information;
estimating the stabilized camera motion information of the camera module based on the estimated OIS 2D translation information,
Compensating for final motion information calculated based on the stabilized camera motion information, the estimated OIS 2D translation information, and the camera rotation information. The electronic device of claim 10 , wherein the estimated OIS 2D translation information is calculated by applying a converted focal length to a value obtained by subtracting the frame rotation information from the camera rotation information. 12. The method of claim 11, wherein the image signal processor is
Filtering from the estimated OIS 2D translation information based on a tuning threshold frequency to estimate the stabilized camera motion information of the camera module, the electronic device. The method of claim 10, wherein the calculated final motion information is
The electronic device of claim 1, wherein the stabilized camera motion information, the estimated OIS 2D translation information, and the motion information according to the camera rotation information are adjusted according to a boundary margin of an input image. The electronic device of claim 10 , wherein the camera rotation information includes final camera rotation information obtained by accumulating intra-frame camera rotation information, inter-frame camera rotation information, and the inter-frame camera rotation information. The electronic device of claim 14 , wherein the frame rotation information includes intra-frame rotation information, inter-frame rotation information, and final frame rotation information obtained by accumulating the inter-frame rotation information. The method of claim 10, wherein the stabilized camera motion information
An electronic device that applies camera rotation information to pixel coordinates of an input image, and applies an internal projection matrix of the camera module and the OIS 2D translation information. A method for stabilizing an image of an electronic device, the method comprising:
sensing a frame image and obtaining gyro data, frame data, and motion vectors according to shaking of the electronic device;
extracting camera rotation information of the electronic device from the gyro data, and extracting frame rotation information from the motion vector and the frame data;
estimating raw Optical Image Stablization (OIS) 2D translation information based on the camera rotation information and the frame rotation information;
calculating filtered OIS 2D translation information by removing a bias component from the raw OIS 2D translation information; and
and performing digital image stabilization on the sensed frame image based on the filtered OIS 2D translation information. The method of claim 17, wherein the raw OIS 2D translation information is
An image stabilization method of an electronic device, which is a value estimated corresponding to a value obtained by subtracting the frame rotation information from the camera rotation information. 19. The method of claim 18, wherein the frame rotation information
Image stabilization of an electronic device including inter-frame rotation information extracted between at least two frames, intra-frame rotation information extracted from one frame, and final frame rotation information obtained by accumulating the inter-frame rotation information Way. 20. The method of claim 19, wherein the digital image stabilization comprises:
estimating stabilized camera motion information of the electronic device based on the filtered OIS 2D translation information and the camera rotation information;
calculating final motion information based on the stabilized camera motion information, the filtered OIS 2D translation information, and the camera rotation information; and
Compensating for motion according to the calculated final motion information to the sensed frame image, the image stabilization method of an electronic device.

Images