JP4499686B2 - Image processing apparatus and method, recording medium, and program - Google Patents

Description

  The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly, to an image processing apparatus and method, a recording medium, and a program that reduce the effects of camera shake when imaging with a portable imaging apparatus.

  Portable video cameras are becoming commonly used. As an individual imaging device used in such a video camera, for example, a charge transfer type individual imaging device represented by a CCD (Charge Coupled Device) sensor or an X-type represented by a CMOS (Complementary Metal Oxide Semiconductor) sensor. There is a Y-address type individual imaging device.

  CMOS sensors consume less power than CCD sensors, are driven by a single low voltage, and can be easily integrated with peripheral circuits. Therefore, CMOS sensors are considered for use in video processing devices such as video cameras. ing.

  However, it has been difficult to record high-quality moving images and still images using a CMOS sensor as an image sensor of an image processing apparatus such as a video camera. One of the reasons is that distortion occurs in an image captured by camera shake. In the case of a CCD sensor that has already been used as an image sensor of an image processing apparatus, the amount of camera shake correction required to execute processing for reducing the effects of camera shake was obtained within one field or one frame. A single value calculated based on camera shake information is used. This is because the exposure period of all the pixels is equal, and image distortion does not occur, so that camera shake can be corrected with a single value.

  In contrast to a CCD sensor, a CMOS sensor captures and processes an image of a subject by the following mechanism, and thus the image is distorted by camera shake. The distortion is considered to occur as follows.

  A charge transfer type solid-state imaging device such as a CCD sensor can read out pixel data by exposing all pixels at the same time, but an XY address type solid-state imaging device such as a CMOS sensor The data is sequentially read in units of one pixel or one line. Even in a solid-state imaging device that sequentially reads out in units of one pixel, the difference in readout time between pixels in one line is negligibly small compared to the difference in readout time between lines. Therefore, in the following discussion, only the time difference between lines is considered as shown in FIG.

  For example, as shown in FIG. 1, a description will be given by taking as an example a CMOS sensor in which the readout cycle of all pixels is 1/60 seconds and one image is composed of lines 1 to N. Referring to FIG. 2, in the exposure period of line 1, the start time is time t1, and the end time is time t2. The end time of the exposure period for line 2 is time t3, and the end time of the exposure period for line 3 is time t4.

  The time difference between time t2 and time t3 is time difference Δt, and the time difference between time t3 and time t4 is time difference Δt. That is, in the exposure period, a time difference is generated for each line by the time difference Δt. Accordingly, in this case, a time difference close to 1/60 second occurs in the exposure period between the top line 1 and the bottom line N of the screen. As described above, when one image is read out and camera shake occurs in a situation where the exposure period is shifted between the top line 1 and the bottom line N, the time difference of the exposure period for each line. As a result, there is a problem that the image of the object that is the subject is deformed.

  With reference to FIG. 3 and FIG. 4, the problem that the captured image of the subject is deformed will be described. FIG. 3 and FIG. 4 each illustrate an example in which a stationary square is imaged as a subject at the center of the image. The center diagram in FIG. 3 shows a state in which a normal image with no camera shake and no distortion is captured when the subject is imaged by the image processing apparatus. It is a figure which shows being imaged.

  The right side of FIG. 3 is a diagram illustrating that when an object is imaged, an image of the distorted object is captured because the image processing apparatus has a camera shake in the right direction. Similarly, the diagram on the left side of FIG. 3 is a diagram illustrating that when the subject is imaged, an image of the distorted subject is captured because the image processing apparatus has leftward camera shake.

  The center diagram in FIG. 4 shows a state in which a normal image without distortion is captured when a subject is imaged by the image processing apparatus, and as a result, a normal image without distortion is captured. It is a figure which shows being imaged. The upper diagram in FIG. 4 is a diagram illustrating that when the subject is imaged, an image of the subject that is expanded in the vertical direction is captured because the image processing apparatus has an upward camera shake. The lower diagram in FIG. 4 is a diagram illustrating that when the subject is imaged, an image of the subject contracted in the vertical direction is captured because the image processing apparatus has a camera shake in the lower direction. .

  In this way, in the CMOS sensor, a shift in exposure period (shift in imaging timing) occurs for each line. Therefore, with a single value calculated based on camera shake information obtained in one field or one frame, When camera shake correction is performed, the effect of camera shake cannot be completely taken, and as a result, an image that can be provided to the user may be a distorted image.

In order to correct such image distortion due to camera shake, in order to correct horizontal camera shake, the readout position is changed for each line, and in order to correct vertical camera shake, the position of the readout line is set in the vertical direction. There has been proposed an image processing apparatus that changes to the above. (For example, see Patent Document 1)
JP 2001-358999 A

  However, in the method proposed in Patent Document 1, it is assumed that camera shake information is acquired for each line constituting the image. However, acquiring camera shake information for each line detects camera shake. There was a problem that it was difficult because of the sampling frequency of the sensor

  Further, in Patent Document 1 described above, there is a problem that camera shake correction can be performed only in pixel units. Such a problem similarly occurs in the conventional image device.

  In the conventional image processing apparatus, since it is necessary to temporarily store image data of a captured image, a memory having a capacity capable of storing at least one screen is required. Therefore, there is a problem that the memory capacity cannot be reduced.

  The present invention has been made in view of such a situation, and generates a camera shake correction amount for each line from camera shake information obtained during one field or one frame so that camera shake can be corrected with high accuracy. To. Further, by determining a coefficient for interpolating pixels from camera shake information and generating pixel data for interpolation, camera shake correction can be performed with higher accuracy. Furthermore, it is possible to reduce the capacity of a memory for storing image data by performing camera shake detection in a short time.

An image processing apparatus according to an aspect of the present invention is an image processing apparatus that processes a subject image captured by an imaging unit that captures an image using an XY address type solid-state imaging device, and the subject captured by the imaging unit. Storage means for storing an image, output means for outputting vibration detection information relating to vibration applied to the image processing apparatus, and vibration detection information output by the output means within each exposure period for each line of the subject image Conversion means for converting vibration detection information located at the center of each exposure period and vibration detection information before and after the exposure period into vibration amounts indicating the magnitude of vibration applied to the image processing apparatus, Calculating means for calculating a correction amount for reducing the influence of the vibration on the subject image based on the vibration amount; Control is performed so that image data corresponding to the number of pixels of the subject image necessary for correction for reducing reverberation is stored in the storage unit, and stored in the storage unit based on a plurality of different correction amounts. Control of correcting the subject image and outputting the corrected subject image is performed.

An image processing method according to an aspect of the present invention includes an imaging unit that captures a subject image with an XY address type solid-state imaging device, and a storage unit that stores the subject image captured in the processing of the imaging step. In the image processing method of the image processing apparatus, an output step of outputting vibration detection information related to vibration applied to the image processing apparatus, and output in the process of the output step within each exposure period for each line of the subject image Of the vibration detection information, a conversion for converting the vibration detection information located at the center of each exposure period and the vibration detection information before and after the exposure information into a vibration amount indicating the magnitude of vibration applied to the image processing apparatus. And a calculation step for calculating a correction amount for reducing the influence of the vibration on the subject image based on the vibration amount, Control is performed so that image data corresponding to the number of pixels of the subject image necessary for correction to reduce the influence of the vibration on the subject image is stored, and the storage unit is based on a plurality of different correction amounts. The subject image stored in is corrected to output a corrected subject image.

A recording medium program according to an aspect of the present invention includes an imaging unit that captures a subject image with an XY address type solid-state imaging device, and a storage unit that stores the subject image captured in the processing of the imaging step. A computer program for controlling the image processing apparatus, the output step for outputting vibration detection information relating to vibration applied to the image processing apparatus; and the output step within each exposure period for each line of the subject image Among the vibration detection information output in the processing, the vibration detection information centered in time in each exposure period and the vibration detection information before and after the vibration detection information indicate the magnitude of the vibration applied to the image processing apparatus. And a calculation step for calculating a correction amount for reducing the influence of the vibration on the subject image based on the vibration amount. And controlling the storage means to store image data for the number of pixels of the subject image necessary for correction to reduce the influence of the vibration on the subject image, and a plurality of different data On the basis of the correction amount, a program for correcting the subject image stored in the storage unit and outputting the corrected subject image is recorded.

A program according to one aspect of the present invention includes an image processing unit that captures an image of a subject using an XY address type solid-state imaging device, and a storage unit that stores the subject image captured in the processing of the imaging step. A computer program for controlling the apparatus, wherein an output step for outputting vibration detection information relating to vibration applied to the image processing apparatus, and output in the processing of the output step within each exposure period for each line of the subject image Among the detected vibration detection information, the vibration detection information located at the center in time of each exposure period and the vibration detection information before and after that are converted into a vibration amount indicating the magnitude of vibration applied to the image processing apparatus. And a calculating step for calculating a correction amount for reducing the influence of the vibration on the subject image based on the vibration amount. The storage unit is controlled so that image data corresponding to the number of pixels of the subject image necessary for correction to reduce the influence on the subject image due to the vibration is stored, and a plurality of different correction amounts are set. On the basis of the above, a program for correcting the subject image stored in the storage unit and outputting the corrected subject image is performed.

In the image processing apparatus and method, and the program according to one aspect of the present invention, a captured subject image is stored, vibration detection information regarding vibration applied to the image processing device is output , and each exposure for each line of the subject image is performed. Among the vibration detection information output by the output means within the period, the vibration detection information located at the center in time of each exposure period and the vibration detection information before and after the vibration detection information indicate the magnitude of vibration applied to the image processing apparatus. Based on the vibration amount, a correction amount for reducing the influence on the subject image due to the vibration is calculated, and the object image necessary for the correction for reducing the influence on the subject image due to the vibration is calculated. Image data for the number of pixels is stored, and the subject image is corrected based on a plurality of different correction amounts.

  According to the present invention, it is possible to correct camera shake applied during imaging.

  According to the present invention, camera shake can be corrected more accurately.

  According to the present invention, camera shake of 1 pixel or less can be corrected.

  According to the present invention, the capacity of a memory for storing image data necessary for camera shake correction can be reduced.

  According to the present invention, even when the exposure time is shortened, the capacity of a memory for storing image data necessary for camera shake correction can be reduced.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 5 is a diagram showing a configuration of an embodiment of an image processing apparatus to which the present invention is applied. The image sensor 11 of the image processing apparatus 10 is configured by, for example, an XY address type solid-state image sensor (CMOS or the like). Data of the subject image captured by the image sensor 11 is supplied to an AFE (Analog Front End) 12.

  The AFE 12 converts the supplied image data into image data of a digital signal and supplies it to the signal processing unit 13. The signal processing unit 13 calculates a luminance signal and a color difference signal from the supplied image data, and supplies the calculated signals to the image interpolation unit 14.

  The image data supplied to the image interpolation unit 14 is image data of the subject imaged by the image sensor 11, but all data of the image captured by the image sensor 11 is supplied to the image interpolation unit 14. However, only the data read according to the timing from the TG (Timing Generator) 15 is supplied to the image interpolation unit 14.

  The data supplied to the image interpolation unit 14 is once supplied to the memory 17 under the control of the memory controller 16. Conversely, the data stored in the memory 17 is read out according to an instruction from the memory controller 16 and supplied to the image interpolation unit 14. The image interpolation unit 14 performs an interpolation process for correcting camera shake (details will be described later) on the supplied data, and outputs the data to a recording medium (not shown) for recording or recording. Output to a non-display and so on.

  Here, the user shake is described as an example of vibration applied to the image processing apparatus 10, but the present invention can be applied to vibration other than camera shake.

  The image interpolation unit 14, the TG 15, and the memory controller 16 perform control based on the camera shake correction amount calculated by the camera shake correction amount calculation unit 19 according to the camera shake amount detected by the camera shake detection unit 18.

  The camera shake detection unit 18 uses, for example, a method using a sensor such as an angular velocity sensor or a sensorless camera shake detection method using image processing or the like, and detects camera shake applied to the image processing apparatus 10 during shooting. For example, when the camera shake detection unit 18 is configured by an angular velocity sensor, the angular velocity sensor supplies the image stabilization amount calculation unit 19 with angular velocity data added to the pitching direction and the yawing direction.

  Based on the detected camera shake amount, the camera shake correction amount calculation unit 19 calculates data of the camera shake correction amount for correcting the motion accompanying the camera shake. That is, the camera shake correction amount calculation unit 19 calculates, for each line, a correction amount that indicates how many pixels are moved by using the supplied data to reduce the influence of the added camera shake.

  Here, for each line, as described with reference to FIGS. 1 and 2, in an individual image sensor such as a CMOS, image data is sequentially read out in units of one line. Therefore, a camera shake correction amount is calculated for each line.

  Although it is necessary to calculate the camera shake correction amount for each line, the timing at which the camera shake amount data is acquired depends on the sampling frequency of the camera shake detection unit 18, and the camera shake correction amount is not necessarily calculated for each line. The amount of camera shake data is not always supplied.

  When the sampling frequency of the camera shake detection unit 18 is different from the frequency of reading out data for each line of the image captured by the image sensor 11, in order to obtain the camera shake correction amount for each line, the sampling interval of the camera shake detection unit 18 is determined. It is necessary to interpolate the camera shake correction amount data when the calculation based on the camera shake correction amount data in Step 1, ie, the camera shake amount data from the camera shake detection unit 18 cannot be performed, by some method.

  In addition, when the sampling frequency of the camera shake detection unit 18 is the same as the frequency of reading data for each line of the image captured by the image sensor 11, but the phase is different, in order to obtain the camera shake correction amount for each line, It is necessary to interpolate the data of the camera shake correction amount between the samplings of the camera shake detection unit 18.

  Interpolation of camera shake correction amount data will be described, but here, camera shake detection (timing at which camera shake amount data is supplied from the camera shake detection unit 18 to the camera shake correction amount calculation unit 19) reads 100 lines. A case of three points in between will be described as an example. As shown in FIG. 6, here, for convenience of explanation, it is assumed that the amount of camera shake from the camera shake detection unit 18 can be acquired for the line 1 ′, the line 50 ′, and the line 100 ′. This line 1 'is a reference line for explanation, and will be described as one line in the lines constituting the captured image.

  In the state shown in FIG. 6, the amount of camera shake when reading line 1 ′ is P1 pixel, the amount of camera shake when reading line 50 ′ is P2 pixel, and the amount of camera shake when reading line 100 ′ is , P3 pixel.

  The camera shake amount is a value obtained by converting the information output from the camera shake detection unit 18 and input to the camera shake correction amount calculation unit 19 into a pixel unit value. Note that the camera shake amount in pixel units after conversion by the camera shake correction amount calculation unit 19 is in pixel units including the decimal part, and correction in an amount of 1 pixel or less is possible in the subsequent processing.

  The camera shake amounts P1 to P3 are differences based on the camera shake amount acquired at a predetermined timing. The predetermined timing is immediately after the start of image capturing. In the state shown in FIG. 6, from the reference, right hand shake occurs at line 1 ′, right hand shake occurs until line 50 ′, and left hand shake occurs until line 100 ′. Is the situation.

  For example, when a camera shake amount obtained as shown in FIG. 6 is captured in the image processing apparatus 10 when a stationary square is imaged as a subject, the image processing apparatus 10 does not perform correction as it is. The image as shown in (1) is taken (provided to the user). Such a distorted image is corrected, and camera shake correction is performed so that the quadrilateral is finally provided to the user as a quadrilateral image.

  In the present embodiment, an example in which an XY address type individual image pickup device such as a CMOS is used as the image pickup device 11 (FIG. 5) is described as an example. When the individual image sensor is used as the image sensor 11, the system of the present invention is applied by substituting a single value for the camera shake for each line in the screen as a method for correcting the image blur due to the camera shake. Is possible (in one image, this is done by using the same value).

  In order to correct the camera shake amount for each line, it is necessary to acquire the camera shake amount for each line. However, referring to FIG. 6 again, in the example shown in FIG. 6, only the amount of camera shake at the time of the line 1 ′, the line 50 ′, and the line 100 ′ is acquired. The reason why only such discrete data can be acquired is that it depends on the timing (sampling frequency) at which the camera shake detector 18 detects camera shake. In such a case, since the camera shake amounts relating to the lines 2 ′ to 49 ′ and the lines 51 ′ to 99 ′ are not acquired, it is necessary to perform processing by interpolating the camera shake amount for each line.

  Interpolation processing (camera shake correction amount calculation processing) performed by the camera shake correction amount calculation unit 19 will be described with reference to the flowchart of FIG. In step S <b> 11, the camera shake correction amount calculation unit 19 acquires camera shake amount data (camera shake information for calculating the camera shake amount) detected from the camera shake detection unit 18.

  As described above, the camera shake correction amount calculation unit 19 acquires three camera shake amounts between the lines 1 'to 100'. From these three camera shake amounts, camera shake amounts relating to the lines 2 'to 99' (excluding the line 50 ') are calculated in step S12. The calculation method will be described with reference to FIGS. 9A to 9C. As shown in FIG. 9A, for example, the camera shake amount relating to the line 1 ′ is also used for the lines 2 ′ to 40 ′, the camera shake amount relating to the line 50 ′ is also used for the lines 41 ′ to 60 ′, and the hand shake amount relating to the line 100 ′. Interpolation may be performed by a method in which the amount of camera shake relating to one line is applied to the preceding and following lines, such as using the amount of camera shake for the lines 61 ′ to 99 ′.

  Further, as shown in FIG. 9B, calculation may be performed using a linear function. In the example shown in FIG. 9B, camera shake amounts for lines 2 ′ to 49 ′ are calculated from a linear function having two camera shake amounts for line 1 ′ and line 50 ′, respectively. The camera shake amounts for the lines 51 ′ to 99 ′ are respectively calculated from a linear function having two camera shake amounts for 100 ′.

  Furthermore, as shown in FIG. 9C, it may be calculated using another function that is not a linear function. The example shown in FIG. 9C is basically the same as the example shown in FIG. 9B, except that other functions that are not linear functions are used. Usually, camera shake is not considered to change in the same direction at the same speed and with the same magnitude. Therefore, if interpolation is performed using a linear function that changes linearly, an error may occur. Considering that interpolation (correction) is performed more appropriately and accurately, as shown in FIG. 9C. It is better to use a curve-changing function.

  Which method in FIGS. 9A to 9C is used to calculate the correction amount depends on the processing capability of the camera shake correction amount calculation unit 19. When the camera shake correction amount calculation unit 19 is configured by a microcomputer or the like, the correction amount is calculated (interpolated) by one of the methods shown in FIGS. 9A to 9C in accordance with the processing capability of the microcomputer. What should I do?

  Specifically, when the processing capability of the camera shake correction amount calculation unit 19 is low, the correction amount is calculated (interpolated) by the method shown in FIG. When the processing capability is relatively high, the correction amount may be calculated (interpolated) by the method shown in FIG.

  As described above, when the camera shake amount for each line is obtained from the discrete camera shake amount, the camera shake correction amount is calculated in step S13. The camera shake correction amount is a pixel unit value indicating how much correction should be performed in order to reduce the amount of camera shake (a state in which the effect of camera shake is reduced).

  For example, as shown in FIG. 6, when the amount of camera shake relating to the line 1 ′ is P1 pixel, the absolute value is different in order to change the state where the camera shake amount is P1 pixel into a state without camera shake. In this case, it is sufficient to calculate the amount of different sign, that is, in this case, −P1 pixel as the correction amount.

  10A to 10C are graphs showing the camera shake correction amount calculated by the camera shake correction amount calculating unit 19. FIG. 10A shows the camera shake correction amount calculated when the camera shake correction amount calculation unit 19 interpolates the camera shake amount by the method described with reference to FIG. 9A. FIG. 10B shows the camera shake correction amount calculated when the camera shake correction amount calculation unit 19 interpolates the camera shake amount by the method described with reference to FIG. 9B. FIG. 10C shows the camera shake correction amount calculated when the camera shake correction amount calculation unit 19 interpolates the camera shake amount by the method described with reference to FIG. 9C.

  As described above, when the camera shake correction amount is calculated, the reading position is determined in step S14 (FIG. 6). When the readout position is determined, if there are two pixels at that position, a process of interpolating a new pixel is performed in step S15. The reading position determination process in step S14 and the pixel interpolation in step S15 will be described. For convenience of explanation, the processing in step S14 and the processing in step S15 will be described together below, but the processing up to step S14 is performed by the camera shake correction amount calculation unit 19 (FIG. 5), and the processing in step S15. Is performed by the image interpolation unit 14.

  The left diagram in FIG. 11 is a graph showing the horizontal camera shake correction amount calculated by the camera shake correction amount calculation unit 19, and the right diagram in FIG. 11 is added when a rectangular object is imaged. FIG. 11 is a diagram illustrating an example of an image A that is captured when camera shake is not corrected, and the center diagram of FIG. 11 is a diagram when a portion a of the image illustrated in the right diagram is enlarged. .

  The part a in the image A includes 10 lines 20 'to 29'. Among them, the pixel b to be read is located on the line 23 '. The calculated camera shake correction amount is calculated for each line. Based on the correction amount for each line, that is, based on data indicating how many pixels the reading position is shifted, the reading start position of each line is determined. It is determined.

  When the read start positions are connected by a line, a read start line b as shown in the center diagram of FIG. 11 is created. The read start line b shown in the center diagram of FIG. 11 is linearly illustrated, but is not necessarily linear, and a predetermined line has a read start position shifted from other lines. Sometimes it is. The read start line b is a line shown for explanation, and is not a line generated as a result of processing by the camera shake correction amount calculation unit 19 or a line necessary for executing the processing.

  As shown in FIG. 12, when reading is started from the reading start line b, as shown in the lower side of FIG. 12, the image of the part a that constitutes a part of one side of the square becomes an image without distortion. You can see that it has been corrected.

  In this embodiment, in order to enable correction of one pixel or less, when attention is paid to either the horizontal direction or the vertical direction, when the pixel b to be read is positioned on one pixel, 2 It may be when it is located on one pixel. First, a case where attention is paid to the horizontal direction will be described. The example illustrated in FIG. 13A illustrates a case where the pixel b is located on one pixel. In the case as shown in FIG. 13A, the pixel data of the pixel G3 is read out as the pixel data of the pixel b.

  The example illustrated in FIG. 13B illustrates a case where the pixel b is located on two pixels. In the example illustrated in FIG. 13B, the pixel b is located across the pixel G2 and the pixel G3. In such a case, data of the pixel b is created from a plurality of adjacent pixels, and the created data is read as pixel data of the pixel b.

  A case where the data of the pixel b is generated from a plurality of adjacent pixels will be described with reference to the example illustrated in FIG. 13B. First, when the pixel b spans the pixel G2 and the pixel G3, the pixel data of the pixel b is generated using only the pixel data of the pixel G2 and the pixel G3. In such a case, pixel data of the pixel b is generated by multiplying the ratio of the pixel b straddling the pixel G2 and the pixel G3.

For example, assuming that the pixel data of the pixel G2 is data D2, the pixel data of the pixel G3 is data D3, and the ratio of the pixel b to the pixel G2 is Kb (0 <Kb <1), it is calculated based on the following equation: .
Pixel data of pixel b = D2 × Kb + D3 × (1−Kb)

  The pixel data of the pixel b to be read may be generated by simply multiplying the data of the pixel on which the pixel b is applied and the ratio of the pixel b, or the pixel on which the pixel b is not applied. The pixel data of the pixel b may be generated including this data.

For example, in the example illustrated in FIG. 13B, pixel data of the pixel b is generated using the pixels G1 to G4. In this case, the pixel data (referred to as D1 to D4) of the pixels G1 to G4 are multiplied by coefficients (referred to as K1 to K4) that are uniquely derived from the ratio of the pixel b applied to the pixel G2. , Add the multiplied values. That is, it is calculated based on the following formula.
Pixel b data = D1 × K1 + D2 × K2 + D3 × K3 + D4 × K4

  These coefficients K1 to K4 are the ratio of the pixel b applied to the pixel G2 (that is, when the camera shake correction amount includes a pixel below the decimal point, the numerical value after the decimal point (for example, the camera shake correction amount is 1.5 pixels). When it is calculated, it may be stored in the image interpolation unit 14 or the camera shake correction amount calculation unit 19 as a table associated with 0.5)), and the coefficient may be read from the stored table.

  Thus, it is possible to correct camera shake more accurately by generating pixel data of the pixel b from a plurality of pixels. Even when the pixel b is positioned on one pixel, the pixel data may be generated as described above from a plurality of adjacent pixel data.

  In this way, a process of interpolating (generating) the readout position in the horizontal direction and the readout pixel data as necessary is performed. Next, a process of interpolating (generating) the readout position in the vertical direction and the readout pixel as necessary will be described. The process in the vertical direction is basically the same as the process in the horizontal direction described above.

  The diagram on the left side of FIG. 14 is a graph of the camera shake correction amount in the vertical direction calculated by the camera shake correction amount calculation unit 19. The right side of FIG. 14 is a diagram illustrating an example of an image A that is captured when the influence of camera shake applied in the upward direction is not corrected when a rectangular subject is imaged. It shows the state that has been. The middle diagram in FIG. 14 is a diagram when the portion a in the image shown in the right diagram is enlarged.

  The part a in the image A includes seven lines 20 'to 26'. Among them, the pixel b to be read is located on the line 23 '. In the diagram shown in the center of FIG. 14, a code is assigned to a line (hereinafter referred to as an original line as appropriate) that should be positioned when an image is taken without camera shake. A gap is provided for illustration. In the example illustrated in FIG. 14, the original line and the captured image line do not match, and thus an image exists in the gap between the lines.

  In the above-described horizontal camera shake correction, correction is performed by changing the reading start position in one line. However, in the vertical camera shake correction, correction is performed by selecting the line itself. To be done. In other words, camera shake correction in the vertical direction is calculated as a correction amount based on how many pixels the imaged line is moved up or down to obtain an image imaged without the original camera shake. The reading is performed based on the correction amount.

  As in the case of horizontal correction, in vertical correction, a pixel b to be read is positioned on one pixel (line) as shown in FIG. 15A, and a plurality of pixels b as shown in FIG. 15B. It can be considered that it is located on a pixel (line). A method of generating pixel data of the pixel b when the pixel b is positioned on a plurality of pixels is calculated using a coefficient, as in the case of the horizontal correction described with reference to FIG. 13B. Therefore, the description thereof is omitted here.

  When such interpolation (correction) is performed, a line of the captured image is placed on the original line as shown in the left diagram of FIG. As a result, as shown in the diagram on the right side of FIG. 16, in this case, an image is captured so that the portion constituting one side of the quadrangle is located on the original line. In this way, even in the vertical direction, even if the image is shaken during imaging, it is possible to provide the user with an image in which the influence of shake is reduced.

  As described above, in the present embodiment, the correction in the horizontal direction and the vertical direction (steps S14 and S15 in FIG. 8) is performed.

  By the way, in the above-described embodiment, the camera shake amount is interpolated in step S12, and the camera shake correction amount is calculated in step S13. That is, after the discrete data acquired from the camera shake detection unit 18 is interpolated to generate the camera shake amount for each line, the correction amount for each line is calculated.

  However, the influence of camera shake may be corrected according to the flowchart shown in FIG. When the camera shake amount is acquired from the camera shake detection unit 18 in step S21, the camera shake correction amount calculation unit 19 calculates the camera shake correction amount with respect to the acquired camera shake amount in step S22. In step S23, camera shake correction amount interpolation (generation) processing is performed so that the camera shake correction amount is prepared for each line using the calculated camera shake correction amount.

  That is, in this case, a discrete camera shake correction amount is calculated from the discrete data acquired from the camera shake detection unit 18, and the camera shake correction amount is interpolated to prepare a camera shake correction amount for each line. As described above, even if the camera shake correction amount is interpolated after calculating the camera shake correction amount, the camera shake correction amount is calculated using the interpolated camera shake amount after interpolating the camera shake amount as described above. It is possible to obtain the same effect as when performing

  Since the process after step S24 is the same as the process after step S14 described above, the description thereof will be omitted. In addition, the processing in step S22 and step S23 is also a basic processing, for example, when calculating the camera shake correction amount, calculating using a linear function, calculating using a curvilinear function, etc. Since it can be performed in the same manner as described in describing the flowchart, the description thereof is omitted.

  As described above, the camera shake correction amount calculation unit 19 calculates the camera shake correction amount. As shown in FIG. 5, the camera shake correction amount is transmitted to the image interpolation unit 14, the TG 15, and the memory controller 16. Supplied. The pixel readout based on the camera shake correction amount performed by each unit will be described. As shown in FIG. 18A, here, a case where the predetermined position of the pixel G1 is set as the reading start position R0 will be described as an example.

  In such a case, first, reading of pixel data supplied from the image sensor 11 to the AFE 12 is controlled by the TG 15. The control of reading out pixel data by the TG 15 is performed in units of pixels in integer multiples. Therefore, the reading is started from the reading start position R1 as shown in FIG. 18B. In this case, since the read start position R1 is the head position of the pixel G1, only the pixel data after the pixel G1 is supplied to each part after the AFE 12, as shown in FIG. 18C.

  That is, rough camera shake correction is performed under the control of TG15.

  Next, the image interpolation unit 14 and the memory controller 16 perform correction below the decimal point that is less than one pixel. First, the pixel data shown in FIG. 18C is sequentially stored in the memory 17 under the control of the memory controller 16 via the image interpolation unit 14. The memory controller 16 reads out the pixel data stored in the memory 17 as necessary, and supplies it to the image interpolation unit 14.

  The reading by the memory controller 16 and the processing in the image interpolation unit 14 will be described. In this case, as shown in the upper part of FIG. 18D, the pixel data is read from the reading start position R2 in the middle of the pixel G1. Although reading is performed, it is not possible to read from the middle of the pixel, so pixel data such as pixel G1 ′, pixel G2 ′,... Is generated (interpolated) as shown in the lower part of FIG. Processing such as output is performed.

  This interpolation process has already been described with reference to FIGS. 13 and 15, but is performed by a process of multiplying the pixel data by a coefficient or the like. With respect to the horizontal direction and the vertical direction, processes such as pixel generation (pixel interpolation) are performed. Through such a process, a fine correction of camera shake is performed. In order to perform such processing, a memory 17 for storing pixel data is necessary, and a memory controller 16 for controlling writing to and reading from the memory 17 is necessary. The configuration is as shown in FIG.

  In the configuration example shown in FIG. 5, pixel data (image data) output from the signal processing unit 13 is stored in the memory 17 under the control of the memory controller 16 via the image interpolation unit 14. It is like that. With this configuration, the resolution of the image is converted before the processing such as pixel interpolation by the image interpolation unit 14 described above is performed (when the data from the signal processing unit 13 is supplied to the memory controller 16). Thus, it is possible to perform processing such as camera shake correction on the image whose resolution has been converted.

  That is, the image interpolation unit 14 can perform processing such as resolution conversion on input data, and can perform processing such as pixel interpolation on output data. As described above, the pixel data (image data) stored in the memory 17 and the stored data are made to pass through the image interpolation unit 14, so that the processing can be varied.

  In the above-described embodiment, the case where the image processing apparatus 10 includes the TG 15 has been described as an example. However, the TG 15 is not necessarily a necessary part for processing for camera shake correction. The present invention can be applied even when 10 does not include the TG 15.

  When the image processing apparatus 10 does not include the TG 15, the pixel interpolation (reading and generation) described above may be performed by the image interpolation unit 14 and the memory controller 16. When the read start position based on the camera shake correction amount calculated by the camera shake correction amount calculation unit 19 is the read start position R0 as shown in FIG. 18A, the rough correction performed by the TG 15 is not performed and the process shown in FIG. 18D is performed. Generation of such pixel data may be started from the pixel G1. In other words, rough correction and fine correction may be collectively performed by the image interpolation unit 14 and the memory controller 16.

  In the above-described embodiment, the capacity of the memory 17 is not described, but the capacity of the memory 17 may be a capacity that can store data for one image, for example. However, as will be described below, by processing the camera shake correction amount and the like, it is possible to perform processing even if the capacity is less than the capacity for storing data for one image. That is, even if the capacity of the memory 17 is small, it is possible to perform the above-described accurate camera shake correction.

  First, the case of performing horizontal camera shake correction will be described. As shown in FIG. 19, description will be given assuming that the camera shake detection unit 18 (FIG. 5) is an image processing apparatus 10 that detects camera shake information of 8 times in one field or one frame. If the exposure period is set as shown in FIG. 19, the camera shake information of line 1 can be calculated from the data from information a1 to information a6. Here, it is assumed that the setting is made as such.

  As described above, in the present embodiment, the camera shake correction amount for correcting the added camera shake is prepared by interpolating (calculating) using data related to a predetermined line. I tried to do it. When interpolating the camera shake amount or the camera shake correction amount, referring to FIG. 9B again, for example, if there is no camera shake information of two lines, line 1 ′ and line 50 ′, lines 2 ′ to The camera shake amount 49 and the camera shake correction amount cannot be acquired.

  However, as shown in FIG. 18, when information a1 to a6 is used as camera shake information of line 1 (for example, line 1 ′), in other words, it can be acquired within the exposure period of the line. When possible camera shake information (six in this case) is used, the next camera shake amount cannot be calculated until the next information a7 is acquired. In this case, the information a7 is acquired for the line i (for example, the line 50 ').

  Therefore, when the camera shake amount is calculated using the six pieces of camera shake information, the lines 2 to (i−1) (for example, the lines 2 ′ to 49 ′) between the lines 1 to i are represented by the line i. Interpolation processing cannot be performed until the information a7 is acquired. That is, in such camera shake detection, when a line at a predetermined position is read out, camera shake information at that line is not acquired, and the position where pixel data is read out cannot be determined by camera shake correction. For this reason, it is necessary to store the image data in the memory 17 once, calculate the camera shake correction amount, and then perform reading (and processing such as interpolation).

  However, here, the camera shake amount of the line 1 is calculated from the information a2 to a5. That is, out of camera shake information that can be acquired during the exposure period, information on both ends is excluded, and the amount of camera shake is calculated from four pieces of information located in the center.

  As described above, when the camera shake amount is calculated using the information on the central portion (information centered in time and information before and after the information), the camera shake related to the line i at the end of the exposure period of the line 1. Information a3 to a6 required when calculating the amount has already been detected (acquired). Therefore, since the amount of camera shake related to line i can be calculated at the time of line 1, the amount of camera shake related to line 2 to line (i-1) is calculated for each line from the amount of camera shake of line 1 and the amount of camera shake of line i. In addition, it can be obtained (calculated) at the time of line 1.

  Therefore, it is possible to calculate the camera shake amount and the camera shake correction amount without temporarily storing the image data in the memory 17 in order to calculate the camera shake amount, and thus storing the image data in the memory 17. Even without this, camera shake correction can be performed in the horizontal direction.

  If the camera shake amount of the line 1 is calculated using the information a3 and the information a4 of the central portion, that is, if the camera shake amount of the line 1 is calculated using only two pieces of information of the central portion, When the camera shake amount of line 1 is calculated (when the exposure period of the first line is completed), it is possible to calculate the camera shake amount of line i and further line j.

  In this case, the camera shake amount for one line is generated from a small amount of information, but the camera shake amount of other lines (for example, line i and line j for line 1) is also used. That is, it is possible to calculate a camera shake amount with high accuracy by performing processing such as correcting the camera shake amount of one line using the camera shake amounts of a plurality of lines. It is also possible to calculate a camera shake correction amount with higher accuracy.

  Next, a case where camera shake correction in the vertical direction is performed will be described. The correction of camera shake in the vertical direction is basically the same as that in the case of the camera shake correction in the horizontal direction described above, and therefore, differences will be described. A description will be given with reference to FIG. 19 again. Also in the vertical direction, as in the case of the horizontal direction described above, if the camera shake amount of the first line (line 1) is calculated, for example, by calculating from the information of information a2 to a5, the first line At the end of the exposure period, information a3 to a6 necessary for calculating the amount of camera shake of line i has already been detected (acquired).

  For this reason, as in the case of the horizontal direction described above, the image processing apparatus 10 does not need to include the memory 17 having a capacity for storing image data for one screen. The applied camera shake can be corrected with high accuracy.

  However, unlike the horizontal camera shake correction, in the vertical camera shake correction, as described with reference to FIGS. 14 and 15, it is necessary to perform processing across a plurality of lines. The image processing apparatus 10 needs to include a memory 17 having a capacity sufficient to store pixel (image) data corresponding to the number of pixels (for example, 10 lines) necessary for pixel interpolation.

  As described above, the image processing apparatus 10 needs to include the memory 17, but the capacity thereof may be equal to or less than the capacity capable of storing one image, and the capacity of the memory 17 can be reduced. Therefore, the cost of the memory 17 itself can be reduced by using a capacity other than the capacity required for storing image data for camera shake correction in the memory 17 for other processing or by reducing the capacity. It becomes possible.

  As described above, when the camera shake detection unit 18 (FIG. 5) calculates the camera shake amount (camera shake correction amount) with information acquired in a short period of time, the capacity of the memory 17 can be reduced. In the above-described embodiment, for the sake of simplicity, it is assumed that camera shake information is detected eight times within one field or one frame. However, in applying the present invention, the number of times of camera shake detection is not limited. The present invention can be applied any number of times.

  By the way, in the imaging process of the imaging apparatus using the XY address type solid-state imaging device, the image information of each pixel is read out immediately after the exposure period ends. As described above, when the method capable of reducing the capacity of the memory 17 is applied, a problem may occur. The problem is that when the exposure period becomes shorter (the shutter speed becomes faster, in other words, the camera shake detection interval> half of the exposure time is satisfied), the pixel information is read out at the next time. There is a case where the amount of camera shake is not detected.

  The problem will be described with reference to FIG. In region 1, the exposure center of each line exists between information a4 and information a5 related to camera shake detection. At this time, the amount of camera shake needs to be interpolated from the information a4 and information a5, but at the time when each line in the region 1 is read, the camera shake detection of the information a5 is not yet performed. . Therefore, in region 1, the amount of camera shake cannot be obtained by interpolation.

  In region 2, the read time of each line exceeds the detection time of information a5. Since the exposure center of each line exists between information a4 and information a5 related to camera shake detection, and the camera shake of information a4 and information a5 is detected, the amount of camera shake of each line is the amount of information a4 and information a5. It can be obtained by interpolation from camera shake information.

  In the area 3, the exposure center of each line exceeds the detection time of the information a5 and exists between the camera shake detection information a5 and the information a6. At this time, the camera shake amount needs to be interpolated from the information a5 and the information a6, but the camera shake detection of the information a6 has not been performed yet at the time of reading each line in the area 3. Therefore, in region 3, the amount of camera shake cannot be obtained by interpolation.

  As described above, when the condition of (camera shake detection interval)> (half of the exposure time) is satisfied, there may be a region where the amount of camera shake cannot be interpolated.

  Although it is possible to shorten the camera shake detection interval and avoid the above-mentioned problems by increasing the sampling frequency of camera shake detection, the burden on the system increases due to an increase in calculation amount and communication amount.

  Therefore, the amount of camera shake is predicted as a method of reducing the system burden and solving the above-described problems. The camera shake amount at the next time is predicted from past camera shake information. For example, as shown in FIG. 21, in order to calculate the predicted camera shake amount F1, the past camera shake amounts P1 and P2 are used, the straight line 1 is obtained linearly from the camera shake amounts P1 and P2, and the straight line 1 is extended. The amount of camera shake F1, which is the upper point, is calculated.

  This method is merely an example, and there is no limit to the number of past camera shake data used for prediction, the prediction algorithm, and the like.

In the above-described embodiment, when the camera shake amount (camera shake correction amount) interpolation process in the vicinity of the upper end or the lower end of the screen is executed, the camera shake data in the preceding and succeeding fields or frames is used for the interpolation. You may make it perform the calculation of. In such a case, it is possible to calculate a correction amount or the like for correcting camera shake with higher accuracy, and thus it is possible to correct camera shake with higher accuracy.

  In the embodiment described above, the description of the captured image is not particularly given, but the captured image may be either a still image or a moving image. That is, the present invention can be applied to a camera that captures a still image, and can also be applied to a camera that captures a moving image.

  As described above, according to the present invention, it is possible to obtain data for correcting camera shake for each line constituting an image from data relating to a small amount of detected camera shake by interpolation. In addition, in order to correct a camera shake amount of 1 pixel or less, pixel data is interpolated, thereby causing a camera shake in an image processing apparatus such as a video camera using an XY address type solid-state image pickup device such as a CMOS. Can be corrected with accuracy of 1 pixel unit or less for each line.

  Furthermore, according to the present invention, it is possible to reduce the capacity of the memory 17 for storing the image data of the captured image. Of course, the present invention can be applied not only to an XY address type solid-state image pickup device such as a CMOS but also to a video camera using a charge transfer type individual image pickup device such as a CCD.

  The above-described processing for correcting camera shake can be executed by hardware having each function, but can also be executed by software. When a series of processing is executed by software, various functions can be executed by installing a computer in which the programs that make up the software are installed in dedicated hardware, or by installing various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

  In order to describe the recording medium, a personal computer that handles the recording medium will be briefly described. FIG. 22 is a diagram illustrating an internal configuration example of a general-purpose personal computer. A CPU (Central Processing Unit) 201 of the personal computer executes various processes in accordance with programs stored in a ROM (Read Only Memory) 202. A RAM (Random Access Memory) 203 appropriately stores data and programs necessary for the CPU 201 to execute various processes. The input / output interface 205 is connected to an input unit 206 including a keyboard and a mouse, and outputs a signal input to the input unit 206 to the CPU 201. The input / output interface 205 is also connected to an output unit 207 including a display and a speaker.

  The input / output interface 205 is also connected to a storage unit 208 composed of a hard disk or the like, and a communication unit 209 that exchanges data with other devices via a network such as the Internet. The drive 210 is used when data is read from or written to a recording medium such as the magnetic disk 231, the optical disk 232, the magneto-optical disk 233, and the semiconductor memory 234.

  As shown in FIG. 22, the recording medium is distributed to provide a program to the user separately from the personal computer, and a magnetic disk 231 (including a flexible disk) on which the program is recorded, an optical disk 232 (CD- Consists of package media including ROM (Compact Disc-Read Only Memory), DVD (including Digital Versatile Disc), magneto-optical disk 233 (including MD (Mini-Disc) (registered trademark)), or semiconductor memory 234 In addition, it is configured by a hard disk including a ROM 202 storing a program and a storage unit 208 provided to a user in a state of being pre-installed in a computer.

  In this specification, the steps for describing the program provided by the medium are performed in parallel or individually in accordance with the described order, as well as the processing performed in time series, not necessarily in time series. The process to be executed is also included.

It is a figure for demonstrating reading of the image data from an XY address type image pick-up element. It is a figure for demonstrating the shift | offset | difference of the read for every line. It is a figure for demonstrating that the image of a to-be-photographed object is distorted by camera shake. It is a figure for demonstrating that the image of a to-be-photographed object is distorted by camera shake. It is a figure which shows the structure of one Embodiment of the image processing apparatus to which this invention is applied. It is a figure for demonstrating the amount of camera shake. It is a figure for demonstrating that the image of a to-be-photographed object is distorted by camera shake. It is a flowchart for demonstrating operation | movement of an image processing apparatus. It is a figure for demonstrating the interpolation of the amount of camera shake. It is a figure for demonstrating the camera-shake correction amount. It is a figure for demonstrating a reading position. It is a figure for demonstrating reading. It is a figure for demonstrating the interpolation of a pixel. It is a figure for demonstrating reading. It is a figure for demonstrating the interpolation of a pixel. It is a figure for demonstrating the image by the interpolated pixel. It is a flowchart for demonstrating other operation | movement of an image processing apparatus. It is a figure for demonstrating reading of a pixel. It is a figure for demonstrating the information required for calculation of the amount of camera shake. It is a figure for demonstrating the information required for calculation of the amount of camera shake. It is a figure for demonstrating prediction of the amount of camera shake. It is a figure explaining a medium.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Image processing apparatus, 11 Image pick-up element, 12 AFE, 13 Signal processing part, 14 Image interpolation part, 15 TG, 16 Memory controller, 17 Memory, 18 Shake detection part, 19 Shake correction amount calculation part

Claims (4)

In an image processing apparatus for processing a subject image picked up by an image pickup means for picking up an image by an XY address type solid-state image pickup device,
Storage means for storing the subject image captured by the imaging means;
Output means for outputting vibration detection information relating to vibration applied to the image processing device;
Of the vibration detection information output by the output means within each exposure period for each line of the subject image, the vibration detection information positioned at the center in time of each exposure period and the vibration detection information before and after the vibration detection information, Conversion means for converting into a vibration amount indicating a magnitude of vibration applied to the image processing apparatus;
Calculating means for calculating a correction amount for reducing the influence of the vibration on the subject image based on the vibration amount;
Control is performed so that image data corresponding to the number of pixels of the subject image necessary for correction to reduce the influence of the vibration on the subject image is stored in the storage unit, and based on a plurality of different correction amounts. An image processing apparatus that performs control to correct the subject image stored in the storage unit and output the corrected subject image. An imaging means for imaging a subject image by an XY address type solid-state imaging device;
In an image processing method of an image processing apparatus, comprising: storage means for storing the subject image captured in the processing of the imaging step;
An output step of outputting vibration detection information relating to vibration applied to the image processing device;
Of the vibration detection information output in the processing of the output step within each exposure period for each line of the subject image, vibration detection information located at the center in time of each exposure period and vibration detection information before and after the vibration detection information. A conversion step for converting into a vibration amount indicating a magnitude of vibration applied to the image processing device;
Calculating a correction amount for reducing the influence of the vibration on the subject image based on the vibration amount; and
The storage unit is controlled so as to store image data for the number of pixels of the subject image necessary for correction to reduce the influence of the vibration on the subject image, and based on a plurality of different correction amounts. An image processing method comprising: correcting the subject image stored in the storage unit and outputting the corrected subject image. An imaging means for imaging a subject image by an XY address type solid-state imaging device;
A computer program for controlling an image processing apparatus comprising: storage means for storing the subject image captured in the processing of the imaging step;
An output step of outputting vibration detection information relating to vibration applied to the image processing device;
Of the vibration detection information output in the processing of the output step within each exposure period for each line of the subject image, vibration detection information located at the center in time of each exposure period and vibration detection information before and after the vibration detection information. A conversion step for converting into a vibration amount indicating a magnitude of vibration applied to the image processing device;
Calculating a correction amount for reducing the influence of the vibration on the subject image based on the vibration amount; and
The storage unit is controlled so as to store image data for the number of pixels of the subject image necessary for correction to reduce the influence of the vibration on the subject image, and based on a plurality of different correction amounts. And a control unit that corrects the subject image stored in the storage unit and outputs the corrected subject image. An imaging means for imaging a subject image by an XY address type solid-state imaging device;
A computer program for controlling an image processing apparatus comprising: storage means for storing the subject image captured by the imaging means;
An output step of outputting vibration detection information relating to vibration applied to the image processing device;
Of the vibration detection information output in the processing of the output step within each exposure period for each line of the subject image, vibration detection information located at the center in time of each exposure period and vibration detection information before and after the vibration detection information. A conversion step for converting into a vibration amount indicating a magnitude of vibration applied to the image processing device;
Calculating a correction amount for reducing the influence of the vibration on the subject image based on the vibration amount; and
The storage unit is controlled so as to store image data for the number of pixels of the subject image necessary for correction to reduce the influence of the vibration on the subject image, and based on a plurality of different correction amounts. A program for correcting the subject image stored in the storage means and outputting the corrected subject image.

Images