KR100705929B1 - Compensation Apparatus for motion in the mobile terminal - Google Patents

Abstract

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a shake motion compensating device, in particular, capable of performing image correction by shake of a terminal.

According to an aspect of the present invention, there is provided a shake motion correcting apparatus, including: an image sensor configured to detect and output image data of a subject; A shake correction processor configured to analyze the detected image data, calculate a shake motion vector, and output a shake correction image having a predetermined size from the calculated motion vector; A digital zoom processor configured to enlarge or reduce the shake corrected image at a predetermined ratio required by the output data by using a weight obtained by a biliner method; And an image format processing unit converting the data processed by the digital zoom processing unit into a format required by the output data and outputting the converted data.

Terminal, camera, shake, zoom processing unit

Description

Compensation Apparatus for motion in the mobile terminal

1 is a block diagram showing an anti-shake device in a conventional portable terminal.

2 is a diagram illustrating input and output images of conventional image stabilization and digital zoom.

3 is a block diagram showing a shake motion compensation device in a mobile terminal according to an embodiment of the present invention.

4 is a diagram illustrating an input / output image grid for explaining a biliner interpolation method according to an exemplary embodiment of the present invention.

FIG. 5 is a detailed configuration diagram of the digital zoom processing unit of FIG. 3. FIG.

FIG. 6 is a detailed view of an interpolation coefficient generator of FIG. 5. FIG.

FIG. 7 is a detailed configuration diagram of the biliner interpolation part of FIG. 5. FIG.

FIG. 8 is a diagram illustrating read / write of the line buffer memory of FIG. 5; FIG.

<Explanation of symbols for the main parts of the drawings>

101 ... Lens 102 ... Image Sensor

110 Image sensor interface 112 Image processing unit

113.Stabilizer 114 Digital zoom processor

115.Image format processing unit 120 ... Shake interpolation unit

121 Line buffer memory 122 Interpolation coefficient generator

123 .... Coordinate generator 124 ... Line buffer memory controller

125 Interpolator 126 First-in-first-out memory

127 First-in-first-out memory controller 130 Clock signal and timing control signal generator

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a shake motion compensation device, in particular, capable of performing image correction by camera shake.

Today's portable terminals offer not only a simple telephone function, but also a variety of additional functions. In particular, composite portable terminals (hereinafter referred to as portable terminals) such as portable phones, smart phones, PDAs, and the like, which are equipped with a camera having a combination of a telephone function and other additional functions, are being developed. Among them, a mobile terminal, which is a combination of a camera and a mobile phone, can take a picture, store it, play it when necessary, use it as a wallpaper, and send it to another device using an email or a phone number.

Accordingly, portable terminals having cameras, such as camera phones, have become popular in the existing digital camera market, and individuals are entering an era of one-person cameras with one camera. Cameras embedded in such portable terminals are being developed with the trend of high pixels and high functions.

The camera module mounted on the portable terminal is composed of an image sensor unit, an image processing unit, and a controller made of a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS). The image sensor unit converts light into an electrical signal and outputs the image sensor, and the image processing and control unit processes an output image of the image sensor unit and controls a motor.

Such an image processing and control unit is called an image signal processor (ISP) or a camera control processor (ISP), and may be implemented as a single chip or an independent chip in combination with an image sensor.

Recently, a camera shake correction function is required for an image processor chip (ISP chip) for a camera module in order to prevent blurring or blurring of an image due to camera shake caused by an external vibration of a camera during still image or video recording. .

Image stabilization methods include optical image stabilization (OIS) and digital image stabilization (DIS).

The optical image stabilization method is a method of detecting the movement of an optically moving image, and since a separate device such as a gyro sensor is required mechanically, it is not suitable for a portable terminal such as a PDA in terms of price and size. On the other hand, the digital image stabilization method does not require an additional optical device and is simply applied to a digital signal processor (DSP), which is easier to apply than a mobile terminal, and is used for image stabilization in a mobile terminal.

1 is a block diagram showing a shake motion compensation device in a conventional portable terminal.

Referring to FIG. 1, a motion estimator unit 11 calculates a motion vector of a continuous image frame input from an image sensor, and then a motion compensation unit 12. In the motion predictor 11, the image is shifted in the opposite direction in the same size as the motion vector and the direction is changed to perform image stabilization.

In this case, when the motion is generated and the correction is performed, the corrected image has a smaller resolution than the input image, so that a digital zoom processor 13 for enlarging or reducing the corrected image in accordance with the output data resolution is required. When the data input from the motion compensator 12 passes through the digital zoom processor 13, an image in which image stabilization is corrected at a resolution of a desired size is finally output.

FIG. 2 shows an image stabilizer B that is shifted by a motion vector when the subject moves from A to B due to camera shake, and is enlarged to an output image size and displayed as a zoomed image C. FIG.

In the captured input image 20, the A image 21 is a general image to which image stabilization is not applied, and the corrected image B 22 is a image stabilization image detected by a motion vector, and the output image C 22 ′. Is a zoomed image in which the image stabilizer 22 is enlarged to an output image size.

To implement the digital zoom function, an interpolation method should be used. Among the most widely used interpolation methods, the nearest neighbor interpolation method, mean value interpolation method, and bilinear method are used. There are interpolation method, bicubic interpolation method, and B-Spline interpolation method. Among them, the nearest neighbor interpolation method and the average value interpolation method are easy to implement because of the small amount of calculation, but it is easy to obtain image quality that is far from the sharpness of the image. . On the other hand, cubic interpolation and B-spline interpolation are excellent in terms of image quality, but complicated calculations require a lot of hardware resources and are not suitable for portable terminals such as mobile phones and PDA phones, which place importance on size, power and cost.

On the other hand, bilinear interpolation can be an interpolation method that satisfactorily meets these two problems. It is also difficult to implement hardware that processes images input from the image sensor in real time.

The present invention provides a shake motion compensation device for correcting shake of an input image using a bilinear interpolation method.

In the mobile terminal according to the present invention for achieving the above object, the shaking motion of the dressing suit,

In a mobile terminal,

An image sensor for detecting and outputting image data of the subject;

A shake correction processor configured to analyze the detected image data, calculate a shake motion vector, and output a shake correction image having a predetermined size from the calculated motion vector;

A digital zoom processor configured to enlarge or reduce the shake corrected image at a predetermined ratio required by the output data by using a weight obtained by a biliner method;

And an image format processing unit converting the data processed by the digital zoom processing unit into a format required by the output data and outputting the converted data.

Preferably, the digital zoom processor obtains a weight by a 2 * 2 biliner method using two lines adjacent to an input pixel and two adjacent pixels included in each line.

Preferably, the weight is each value inversely proportional to the distance from an adjacent input pixel for each output pixel.

Preferably, the zoom factor in the row direction of the input image data is obtained by the number of pixels in the row direction of the output image relative to the number of pixels in the row direction of the input image, and the zoom factor in the column direction of the input image data. Is obtained by the number of pixels in the row direction of the output image compared to the number of pixels in the row direction of the input image.

Preferably, the digital zoom processor includes a line buffer memory having a plurality of line memories and temporarily storing each line data; A coordinate generator for accumulating the coordinates of adjacent pixels, respectively, and generating integer and real part data calculated from a zoom factor using the accumulated coordinate information; A line buffer memory controller for controlling read / write of the line memory using the coordinate information of the integer part of the coordinate generation unit as an address value; An interpolation coefficient generator for generating an interpolation coefficient using the real part of the coordinate generator; An interpolation unit for multiplying the interpolation coefficients generated by the interpolation coefficient generator and multiplying the multiplied values by the line data output from the line buffer memory to output interpolation data; A first-in first-out memory for sequentially storing and outputting output data of the interpolation unit; Each of the units includes a clock signal and a timing control signal generator for generating a clock signal and a timing control signal.

Preferably, the coordinate generator includes an X and Y coordinate accumulator for accumulating X and Y coordinates, an X and Y coordinate integer unit for separating and outputting integer data from the accumulated X and Y coordinate values, and the accumulated And an X, Y coordinate real part for separating real data from the X and Y coordinate values and outputting the real data to the interpolation coefficient generator.

Preferably, the interpolation coefficient generator is further configured to calculate an interpolation coefficient for compensating distances in a horizontal direction and a column direction based on a distance between adjacent pixels input using real data of X coordinates and real data of Y coordinates. It is characterized in that it occurs.

Preferably, the interpolation coefficient generator and the interpolator are used in a 2 * 2 biliner method.

Preferably, the interpolation section includes a plurality of registers for storing adjacent pixels output from the line memory; A plurality of multipliers for multiplying and outputting an output pixel value of each register and an interpolation coefficient of the interpolation coefficient generator; And an adder for adding the output data of the multiplier and outputting interpolation data having a ratio required by the output data.

Preferably, the line buffer memory stores the first two line data of the image frame in the first and second line memories, and stores the first two line data of the image frame in the third memory when data corresponding to the next line is input. And sequentially outputting data stored in the line memory.

Hereinafter, with reference to the accompanying drawings as follows.

3 is a block diagram illustrating an apparatus for compensating for shaking motion in a mobile terminal according to an exemplary embodiment of the present invention.

Referring to FIG. 3, an image sensor 102 which detects light reflected through the camera lens 101 as an electrical signal, converts the light into digital image data, and outputs the digital image data; And an image signal processor 103 for processing the image data output from the image sensor 102, and adjusting and outputting the shake correction and the size of the shake correction image data by a biliner interpolation method.

The image signal processor 103 may include an interface unit 111 for interfacing with the image sensor 102, an image processor 112 for processing image data input through the interface unit 111 to improve image quality, and A shake correction processor 113 for calculating a shake motion vector after analyzing the image data input from the image processor 112 and outputting a shake correction image having a predetermined size based on the calculated motion vector, and the shake correction processor The digital zoom processor 114 adjusts and outputs the output image of 113 by a biliner interpolation method at a predetermined ratio required by the output data, and converts the image data processed by the digital zoom processor 114 into a constant image format. It is characterized in that it comprises an image format processing unit 115 for converting according to the timing to output the final data.

The shake correction device in the mobile terminal according to the present invention configured as described above will be described with reference to the accompanying drawings.

As shown in FIG. 3, light reflected and input through the camera lens 101 is detected as an electrical signal by the image sensor 102, and the electrical signal is converted into digital image data and output.

Here, the image sensor 102 includes a sensor such as a CCD or CMOS to obtain an image image of the photographed subject, and the electrical signal processing portion and the portion converting into a digital signal are integrally formed or independent. It may be configured as.

The image signal processor 110 is applied to a portable terminal to configure a camera module together with the image sensor 102 and the lens 101. The image signal processing unit 110 is implemented as one image signal processing (ISP) chip to perform digital image stabilization (DIS) shake correction.

The image signal processing unit 110 processes and shakes image data input from the image sensor 102, and outputs the image size through adjustment such as enlargement or reduction by a biliner method on the size of the corrected image data. Done. Here, the image signal processor 110 is implemented as an image signal processing chip of the camera module, and includes a motion vector compensation and a shake correction function.

The image signal processor 102 includes an image sensor interface 111, an image processor 112, a shake correction processor 113, a digital zoom processor 114, and an image format processor 115. The image sensor interface 111 is configured to interface with an image sensor or a camera, and receives image data.

The image processor 112 may output an image by improving the quality by interpolating the image data input through the image sensor interface 111. Also, the image processor 112 may include a color space conversion function and a white balance function.

The shake correction processor 113 receives the image data processed by the image processor 112, analyzes the image data, calculates a motion vector of the shake (eg, hand shake), and based on the calculated motion vector. To output a shake correction image of a certain size. That is, the shake correction is performed by shifting the input image in the same direction as the calculated motion vector and in the opposite direction.

In this case, since the shake corrected image has an image smaller than the input image, the shake corrected image data is output to the digital zoom processor 114 to match the resolution of the output data.

The digital zoom processing unit 114 receives the shake corrected image data and adjusts the size by performing enlargement or reduction at a ratio required by the output data by a biliner interpolation method. The image format processor 115 converts the image data processed by the digital zoom processor 114 according to an output format and outputs final data according to timing.

The present invention includes a shake correction function in the digital zoom processor 114. Specifically, the camera shake correction method in the digital zoom processing unit 114 is as follows. The digital zoom processor 114 uses a bilinear interpolation method. For example, 2 * 2 biliner interpolation method. That is, interpolation is performed using two pixels adjacent to two lines on both sides with respect to the input pixel. This biliner interpolation method is also possible by n * n (n = 2 or more).

In the biliner interpolation method, as illustrated in FIG. 4, square pixels located at each corner of the input grid represent input image data of the digital zoom processor 114, and S1, S2, S3, Represented by S4 respectively. Pixels marked with circular dots at each corner of the output grid represent output image data of the digital zoom processor 114.

This biliner interpolation method calculates the weight that is inversely proportional to the distance from the adjacent input pixel for each output pixel, and multiplies each weight by the input pixel to obtain the sum-of-product (SOP). It is a principle. That is, the pixel PN value of the output image of FIG. 4 may be obtained as in Equation 1.

PN = S1 × (1-dx) × (1-dy) + S2 × dx × (1-dy) + S3 × (1-dx) × dy + S4 × dx × dy

In Equation 1, S1, S2, S3, and S4 mean four pixel values on the input grid adjacent to the output image pixel PN, and when the intervals S1-S2 between pixels of the input grid are 1, Each weight value is dx and dy when the distances away from S1 in the x- and y-axis directions are dx and dy, respectively, as shown in Equation 1 (1-dx) × (1-dy), dx × (1-dy), ( 1-dx) x dy and dx x dy.

The distances dx and dy are determined from a zoom factor in the row direction and the column direction of the image as in Equation 2.

dx = Row _i / Row _o

dy = Col _i / Col _o

In Equation 2, Row _i is the number of pixels in the row direction of the input image, Row _o is the number of pixels in the row direction of the output image, Col _i is the number of pixels in the column direction of the input image, Col _i is It means the number of pixels in the column direction of the output image, respectively. That is, the zoom factor is determined using the number of pixels in the row and column direction of the input image and the number of pixels in the row / column direction of the output image.

5 is a detailed block diagram of a digital zoom processor according to an exemplary embodiment of the present invention.

As shown in FIG. 5, the digital zoom processor includes a shake interpolation unit 120 and a clock signal and timing control signal generator 130. The shake interpolation unit 120 includes a line buffer memory 121, The interpolation coefficient generator 122, the XY coordinate generator 123, the line buffer memory controller 124, the interpolator 125, the first-in first-out memory 126, and the first-in first-out memory controller 127.

The line buffer memory 121 is composed of four 1-port 1-line memories 121a to 121d, and serves as a buffer for temporarily storing or outputting data through a read or write operation. Here, the 1-port 1-line memory operates by reading line data once and writing another time by one address data and a control signal. In other words, it is not possible to read / write at the same time.

The line buffer memory controller 124 controls the read / write of the line buffer memory 121.

The interpolation coefficient generator 122 generates a biliner interpolation coefficient, and calculates an interpolation coefficient required to perform interpolation in the interpolator 125. That is, a 2 * 2 biliner interpolation coefficient is generated.

The coordinate generator 123 accumulates dx and dy calculated by the zoom factor using the X coordinate accumulator 123a and the Y coordinate accumulator 123e, respectively. Here, the coordinate generator 123 operates by horizontal and vertical synchronization signals and clock signals input from the clock signal and the timing generator 130, and the accumulators 123a and 123b are sized by X and Y. Accumulate.

The values accumulated by the accumulators 123a and 123b are divided into integer parts 123b and 123e and real parts 123c and 123f, respectively, and the X coordinate integer part 123a and the Y coordinate integer part 123e are X and Y coordinates. After extracting the integer data of each of the output to the line buffer memory controller 124, the line buffer memory controller 124 is used to generate an address value for reading / writing the line buffer memory 121 using the integer data do.

The X coordinate real part 123c and the Y coordinate real part 123f extract the real data accumulated by the accumulators 123a and 123b and output the extracted real data to the interpolation coefficient generator 122 and the interpolation coefficient generator 122. Is used to calculate the interpolation coefficient by the real data.

That is, the line buffer memory controller 124 generates an address value by the X and Y coordinate integer units 123b and 123e of the coordinate generation unit 123 and transmits the address value to each line memory 121a to 121d of the line buffer memory 121. Input data is stored and output. The interpolation coefficient generator 122 generates an interpolation coefficient by using real data input by the X and Y coordinate real parts 123c and 123f of the coordinate generator 123.

The interpolator 125 interpolates each pixel of the line data output from the line buffer memory 121 by a zoom factor by the interpolation coefficient generator 122 and then outputs the interpolated coefficients. That is, interpolation is performed by a 2 * 2 biliner interpolation method. This is interpolated using pixels located on both sides of the input pixel, that is, all four pixels.

The first-in-first-out (FIFO) memory 125 is used to match the pixel input / output clock speeds of the input data and the output data, and the first-in-first-out memory controller 127 controls data read / write of the first-in first-out memory 127. .

The clock signal and timing control signal generator 130 receives an input horizontal synchronization signal Hsync_i, an input vertical synchronization signal Vsync_i, and an input pixel clock signal Clock_i, and outputs an output horizontal synchronization signal Hsync_o and an output vertical synchronization. A signal Vsync_o and an output pixel clock Clock_o are generated, and a clock and timing control signal necessary for each block described above are also generated.

6 is a diagram illustrating input and output data of the interpolation coefficient generator according to the present invention, wherein the input data are real data dx of X coordinates and real data dy of Y coordinates calculated by the XY coordinate generator 123, and output data is interpolated. An interpolation coefficient value required by the executive unit 125, which is equal to the weight value for four adjacent input pixels in Equation 1. That is, the weight value is a value inversely proportional to the distance from an adjacent input pixel with respect to each output pixel, and according to the direction and position, (1-dx) × (1-dy), dx × (1-dy), (1- dx) x dy, dx x dy.

7 is a detailed configuration diagram of the interpolation unit according to the present invention. The interpolator is a 2x2 Bilinear interpolator.

Referring to the operation thereof, the first to fourth registers (REG # 1 to REG # 4) (131, 132, 135, and 136) are registers for storing four adjacent pixels received from the line memories 121a to 121d. (133, 134, 137, 138) and the adder 139 is an arithmetic operator for calculating the sum of the products (SOP) by applying the corresponding weight to four adjacent pixel data. Here, the multipliers and multipliers of the multipliers 133, 134, 137 and 138 are values calculated from the 2x2 bilinear interpolation coefficient generator 122 and the output values of the registers. Is calculated.

That is, the first multiplier 133 multiplies the 1 pixel value of the first line data output from the first register 131 by the weight (dx × (1-dy)) output from the interpolation coefficient generator 122. Will print. The second multiplier 134 calculates the 2-pixel value of the first line data output from the second register 132 and the weight ((1-dx) × (1-dy)) output from the interpolation coefficient generator 122. Will be multiplied. The third multiplier 137 multiplies the 1 pixel value of the second line data output from the third register 135 by the weight (dx × dy) output from the interpolation coefficient generator 122. The fourth multiplier 138 multiplies two pixel values of the second line data output from the fourth register 136 by the weight ((1-dx) × dy) output from the interpolation coefficient generator 122. .

The adder 139 adds the outputs of the multipliers 133, 134, 137, and 138 to output interpolation data corresponding to the output image size.

8 is a diagram illustrating a write / write state of the line buffer memory of FIG. 5 according to the present invention.

As shown in the drawing, the line buffer memory 121 writes or reads data into four 1-port 1-line memories 121a to 121d, and a line unit of data inputted to a corresponding memory in which data is read / written. It is put together.

That is, when the input of the image frame is started and data is input, the first two lines # 1 and 2 are sequentially stored in the memory # 1 and the memory # 2. Then, when the data corresponding to the third line is input, the input data starts to be stored in the memory # 3, and at the same time, the previous two line data stored in the memory # 1 and the memory # 2 are output to the interpolator 125. To start. In addition, when the data corresponding to the fourth line is input, the input data starts to be stored in the memory # 4 and at the same time, the previous two line data stored in the memory # 2 and the memory # 3 begin to be output to the interpolator 125. In this manner, read / write operations of the respective memories are selectively performed sequentially, and the bilinear interpolator 125 performs interpolation using four adjacent pixels.

So far, the present invention has been described with reference to the preferred embodiments, and those skilled in the art to which the present invention pertains to the detailed description of the present invention and other forms of embodiments within the essential technical scope of the present invention. Could be implemented. Here, the essential technical scope of the present invention is shown in the claims, and all differences within the equivalent range will be construed as being included in the present invention.

According to the shake motion correcting apparatus in a portable terminal according to the present invention, a shake correction function can be embedded in an image signal processing chip by hardware implementation of the shake motion correcting apparatus for processing an image signal of a camera module of the portable terminal. .

It also has the effect of processing image data in real time, using less hardware resources such as memory capacity and logic gate counts.

In addition, there is an effect that can be applied when the digital zoom function itself is applied to the portable terminal as well as the zoom function for the shake correction.

Claims (10)

In a mobile terminal, An image sensor for detecting and outputting image data of the subject; A shake correction processor configured to analyze the detected image data, calculate a shake motion vector, and output a shake correction image having a predetermined size from the calculated motion vector; A digital zoom processor configured to enlarge or reduce the shake corrected image at a predetermined ratio required by the output data by using a weight obtained by a biliner method; And an image format processing unit for converting the data processed by the digital zoom processing unit into a format required by the output data and outputting the converted image format. The method of claim 1, Wherein the digital zoom processor obtains a weight by a 2 * 2 biliner method using two lines adjacent to an input pixel and two adjacent pixels included in each line. The method of claim 2, And the weights are respective values that are inversely proportional to a distance from an adjacent input pixel with respect to each output pixel. The method of claim 3, The zoom factor in the row direction of the detected image data is obtained by the number of pixels in the row direction of the output image compared to the number of pixels in the row direction of the input image. And the zoom factor in the column direction of the detected image data is determined by the number of pixels in the row direction of the output image relative to the number of pixels in the row direction of the input image. The method of claim 1, The digital zoom processor includes a line buffer memory having a plurality of line memories and temporarily storing each line data; A coordinate generator for accumulating the coordinates of adjacent pixels, respectively, and generating integer and real part data calculated from a zoom factor using the accumulated coordinate information; A line buffer memory controller for controlling read / write of the line memory using the coordinate information of the integer part of the coordinate generation unit as an address value; An interpolation coefficient generator for generating an interpolation coefficient using the real part of the coordinate generator; An interpolation unit for multiplying the interpolation coefficients generated by the interpolation coefficient generator and multiplying the multiplied values by the line data output from the line buffer memory to output interpolation data; A first-in first-out memory for sequentially storing and outputting output data of the interpolation unit; And a clock signal and a timing control signal generator for generating a clock signal and a timing control signal in each unit. The method of claim 5, The coordinate generator includes an X and Y coordinate accumulator for accumulating X and Y coordinates, an X and Y coordinate integer unit for separating and outputting integer data from the accumulated X and Y coordinate values, and the accumulated X and Y coordinates. And an X, Y coordinate real part for separating real data from coordinate values and outputting the real data to an interpolation coefficient generator. The method of claim 6, The interpolation coefficient generator generates an interpolation coefficient for compensating the distances apart in the lateral direction and the column direction based on the distance between adjacent pixels input using the real data of the X coordinate and the real data of the Y coordinate. An apparatus for compensating for shaking motion in a portable terminal, characterized in that. The method of claim 5, The interpolation coefficient generating unit and the interpolation unit is a shake motion compensation device for a mobile terminal, characterized in that using a 2 * 2 biliner method. The method of claim 5,  The interpolation section includes a plurality of registers for storing adjacent pixels output from the line memory; A plurality of multipliers for multiplying and outputting an output pixel value of each register and an interpolation coefficient of the interpolation coefficient generator; And an adder for adding the output data of the multiplier and outputting interpolation data having a ratio required by the output data. The method of claim 5, The line buffer memory stores the first two line data of the image frame in the first and second line memories, and stores data in the third memory when the data corresponding to the next line is input to the stored first and second line memories. An apparatus for compensating for shaking motion in a portable terminal, characterized by sequentially performing an operation of outputting stored data.

Images