JP4449777B2 - Imaging apparatus, image processing apparatus and method, recording medium, and program - Google Patents

Description

The present invention relates to an imaging apparatus, an image processing apparatus and method, a recording medium, and a program, and more particularly to an imaging apparatus, an image processing apparatus and method, a recording medium, and an imaging apparatus that reduce the influence of vibration when imaging with a portable imaging apparatus. Regarding the program.

  Portable video cameras are becoming commonly used. As an individual imaging device used in such a video camera, for example, a charge transfer type individual imaging device represented by a CCD (Charge Coupled Device) sensor or an X-type represented by a CMOS (Complementary Metal Oxide Semiconductor) sensor. There is a Y-address type individual imaging device.

  A CMOS sensor consumes less power than a CCD sensor, is driven with a single low voltage, and can be easily integrated with a peripheral circuit, and thus may be used in an image processing apparatus such as a video camera.

  However, it has been difficult to record high-quality moving images and still images using a CMOS sensor as an image sensor of an image processing apparatus such as a video camera. One reason for this is that distortion occurs in an image picked up by hand shake. In the case of a CCD sensor that is already used as an image sensor of an image processing apparatus, the amount of correction required for executing processing for reducing the influence of camera shake is the camera shake obtained in one field or one frame. A single value calculated based on the information is used. This is because the exposure period of all the pixels is the same, and image distortion does not occur, so that camera shake can be corrected with a single value.

  In contrast to a CCD sensor, a CMOS sensor captures and processes an image of a subject by the following mechanism, and thus the image is distorted by camera shake. The distortion is considered to occur as follows.

  A charge transfer type solid-state imaging device such as a CCD sensor can read out pixel data by exposing all pixels at the same time, but an XY address type solid-state imaging device such as a CMOS sensor The data is sequentially read in units of one pixel or one line. When pixel data is sequentially read out in units of one line, for example, when one image is composed of 1 to N lines and it takes t seconds to read out one line, it takes N × t seconds to read out the data of one image. Time is needed.

  In other words, the Nth line is read after about N × t seconds have elapsed since the first line was read. Due to this time difference, when vibrations such as camera shake are applied when an image is captured (when data for each line is read out), the image is shifted between the first line and the Nth line. In some cases, the position where the image exists is displaced, and the captured image is distorted.

In order to reduce such distortion of an image, it has been proposed to perform correction so as to cancel the vibration when a vibration such as a camera shake is applied. (For example, see Patent Document 1)
JP 2004-266322 A

  According to the method disclosed in Patent Document 1, it is possible to reduce the influence on the image due to vibration such as camera shake. However, the following may occur during long exposure.

  FIG. 1 is a diagram for explaining vibration applied at the time of photographing with an XY address type solid element and a correction amount for canceling the vibration. First, with reference to FIG. 1, the influence on the image by different exposure times will be described. The curve in FIG. 1 is a Sine wave, which means that vibration indicating the characteristics of the Sine wave has been applied.

  In the description of FIG. 1, it is assumed that the vibration indicating the characteristics of the Sine wave is vibration applied in the vertical direction with respect to a device that captures an image. In other words, when vibration of a predetermined magnitude is continuously applied in the vertical direction and the displacement of the vibration is expressed with time, the explanation will be continued with an example of a Sine wave as shown in FIG. .

  In the upper diagram of FIG. 1 (the diagram in which the Sine wave is illustrated), the numerical values on the vertical axis indicate the numbers of lines constituting one image, and the horizontal axis indicates time. Here, it is assumed that the number of lines constituting one screen is 9, the line read out first in time is set as line 8, and the line read out last is set as line 0. The other lines are also described as line 1 and the like. In the XY address type solid-state element, reading is performed for each line (or for each pixel), so the timing of starting reading of each line is different. Therefore, as shown in FIG. 1, when the reading of one screen is illustrated for each line over time, a portion from which data for one screen is read is a parallelogram.

  The lower diagram of FIG. 1 is a diagram obtained by extracting the vibration applied when the line 8 or the line 0 is read and the values used for the correction process from the upper diagram of FIG. The left diagram in FIG. 1 is a diagram at the time of one-field exposure, and the right diagram is a diagram at the time of four-field exposure. In FIG. 1, 1 field exposure is appropriately described as short exposure, and 4 field exposure is appropriately described as long exposure.

  Referring to FIG. 1, at the time of one-field exposure, line 8 is read from time t1 to time t2, and line 0 is read from time t3 to time t4. An upward vibration is applied between time t1 and time t2 when the line 8 is read. Similarly, upward vibration is applied between time t3 and time t4 when line 0 is read.

  The correction performed to reduce the influence of vibration is performed by changing the data reading start position on the line in consideration of the direction and force of the applied vibration. The reading start position uses the direction and magnitude of vibration acquired at the center time of exposure time (hereinafter referred to as correction amount determination point as appropriate), and how much correction is applied to add vibration. This is determined by calculating a correction amount indicating whether an image equivalent to an unexisting image can be obtained.

  For example, the line 8 is read during the period from time t1 to time t2 (exposed from time t1 to time t2), but the time (time t1 + time t2) / 2 within the period (determination of correction amount is determined). Based on the direction (in this case, upward) and magnitude (value obtained by an acceleration sensor, etc.) of the vibration applied to the point), a correction amount is calculated, and a reading start position is determined.

  Similarly, the reading start position of line 0 is also determined based on the direction and magnitude of vibration applied at the time of (time t3 + time t4) / 2. By performing such processing for each of the lines 8 to 0, the image shifted due to vibration is corrected, and an image similar to that when no vibration is applied is generated.

  In the case of one-field exposure (short-time exposure), the correction amount decision point is located approximately at the center with respect to the blur width in the exposure period, and there are few errors (noise and calculation errors) in the exposure period. It is done. As a result, it is considered that good correction can be performed.

  In the case of 4-field exposure, the same correction as in the case of 1-field exposure is performed. However, in the case of long-time exposure such as 4-field exposure, the correction amount determination point is not only located at a position deviated from the center with respect to the blur width in the exposure period, but also an error in the exposure period. Is also considered large.

Referring to the diagram on the right side of FIG. 1, during the four-field exposure, the line 8 is exposed in the period from the time t5 to the time t7. The vibration applied during the period is initially upward. Although there is, it has changed downward from the middle. The line 0 is also read during a period from time t6 to time t8, and the vibration applied during the period is initially upward, but changes from midway to downward.

  Since the correction amount determination point when the line 8 is read is when the upward vibration is applied, the correction performed when the upward vibration is applied is performed. However, actually, the downward vibration is also applied during the period in which the line 8 is read, and the correction when the downward vibration is originally applied to the downward vibration is performed. Need to be

  Similarly, the correction amount determination point within the period in which the line 0 is read is when the downward vibration is applied, but the upward vibration is also applied during the period in which the line 0 is read. .

  In this way, if the direction of vibration applied within the period during which one line is read out, only correction that can be performed only in one direction before or after the change can be performed, and as a result, appropriate correction can be performed. It is not considered.

  Further, with reference to FIG. 2, the influence of the exposure time on the image in the applied vibration and the correction for canceling the vibration will be described. As in FIG. 1, FIG. 2 shows the relationship between the applied sine wave of vibration and the acquisition time of the value used for correction, and the lower figure shows the portion related to line 8 and line 0 from the upper figure. It is the extracted figure.

  Both the left and right views of FIG. 2 show a four-field exposure. The left diagram in FIG. 2 is the same as the right diagram in FIG.

  Referring to the diagram on the right side of FIG. 2, the line 8 is read during a period from time t9 to time t11, and the line 0 is read during a period from time t10 to time t12. In the period in which the line 8 is read, upward vibration is applied. Even during the period in which line 0 is read out, downward vibrations are also applied in the latter half, but judging from the overall ratio, only upward vibrations are being applied.

  Since both the correction amount determination points for line 8 and line 0 exist when an upward vibration is applied, the correction performed when the upward vibration is applied is performed.

  As described above, even when the exposure is performed for a long time, the vibration applied when one predetermined line is read may be in the same direction. In such a case, unlike the case described above (described with reference to the right side of FIG. 1), the direction of vibration applied during the readout period and the vibration of the portion where the correction amount determination point exists. Since there is no section in the period in which correction is performed in a direction different from the direction that should be corrected as in the case described above, only the direction of correction is considered. It is thought that appropriate correction can be performed.

  When a moving image is captured, the state shown on the left side of FIG. 2 and the state shown on the right side may occur alternately. The video image | photographed in such a state is demonstrated with reference to FIG. In FIG. 3, an image A is a subject to be photographed, and in this case, a triangle. Further, the description will be continued assuming that a horizontal vibration is applied when the image A is taken. In FIG. 3, the horizontal arrows and the dotted line shown at the image 2 are given for the sake of explanation, and are not a part of the photographed image.

  Image 1 is a video (one image in the video) taken at time T1, and is corrected in a state as shown on the right side of FIG. 2 (that is, in a direction different from the direction that should be corrected within the readout period). In the state where there is no section that would cause the following to occur (hereinafter referred to as state 1).

  Image 2 is an image taken at time T2, and there is a state as shown on the left side of FIG. 2 (that is, a section in which correction is performed in a direction different from the direction to be originally corrected in the readout period) In such a state, hereinafter referred to as state 2).

  Image 3 is an image taken at time T3 and is an image taken in state 1. Image 4 is an image taken at time T4 and is an image taken in state 2.

  Images 1 and 3 are images taken in state 1, and the triangle of image A that is the subject of the image is blurred due to lateral vibration, and there are multiple lower right vertices. It has been done. That is, the image is taken as if several triangles are overlapped, and the image is blurred. The right angles that make up the triangle are taken with the right angles maintained.

  The image 2 and the image 4 are images taken in the state 2, and the triangle of the image A to be photographed is shaken due to the vibration in the horizontal direction, and is the same as the images 1 and 3. The image is blurred. Further, in the image 2 and the image 4, the right angle constituting the triangle is not maintained as a right angle but is taken in an obtuse angle.

  In summary, the image captured in the state 1 has a large vibration accumulated due to the long-time exposure, and therefore blur (blur) exists. In addition to bleeding, the image taken in state 2 also has distortion. That is, as shown in FIG. 3, in the image 2 and the image 4, there is an inclination as indicated by a dotted line that does not originally exist in the image A.

  When images such as image 1, image 2, image 3, and image 4 are taken continuously, vibrations such as “bleed”, “bleed + distortion”, “bleed”, “bleed + distortion” The image that has been adversely affected is repeated. In this way, if there is an image with distortion and an image without distortion, it is an extreme expression, but the video in which the triangle that is the subject in this case becomes larger or smaller is provided to the user. become. Such a video is not preferable for the user.

  In image 2 and image 4, not only bleeding but also distortion occurs in a state where there is a section in which correction is performed in a direction different from the direction to be originally corrected within a period in which data is read from the line. It is considered that the cause was that the image was taken in (State 2). That is, it is considered that one of the causes of distortion is caused by processing for correcting the influence of vibration.

  Such a problem exists as a problem that occurs at the time of long exposure, and how to remove it or not to provide an unobtrusive image due to distortion to the user was a problem.

  The present invention has been made in view of such a situation, and an object of the present invention is to reduce an adverse effect given to a photographed image due to vibrations such as camera shake even during long exposure. It is another object of the present invention to more appropriately correct for vibration during long exposure.

The imaging apparatus according to the present invention is based on an amount of vibration from an imaging unit that captures an image with an XY address type solid-state imaging device and a detection unit that detects the applied vibration. A first correction amount that is a correction amount in units of lines based on the vibration amount when the exposure time of the imaging unit is less than 1/60 second , When the exposure time is 1/60 second or more, the value is smaller than the first correction amount for at least a part of the lines, and the difference in the correction amount for each line is the first correction amount for each line. A calculation unit that calculates a second correction amount that is a correction amount in units of lines, which is a value smaller than a difference in correction amount, and a correction amount in units of lines that is calculated by the calculation unit. The correction for vibration applied to the And a correcting unit that performs the down units.

When the first correction amount of at least one line is used as a representative amount, the calculating means sets the second correction amount that is the same value as the first correction amount for the at least one line. For the lines other than the at least one line, the second correction amount corresponding to a value obtained by multiplying the first correction amount in each line by a predetermined coefficient less than 1 is calculated.

In the imaging apparatus of the present invention, an image is picked up by an XY address type solid-state imaging device, applied vibration is detected, and a correction amount in units of lines for the applied vibration is calculated based on the vibration amount. Is done. The calculation calculates a first correction amount that is a correction amount for each line based on the vibration amount when the exposure time is less than 1/60 seconds , and at least when the exposure time is 1/60 seconds or more. Correction for each line in which a value is smaller than the first correction amount and a difference in the correction amount for each line is smaller than a difference in the first correction amount for each line for some lines. This is done by calculating a second correction amount that is an amount. Further, based on the calculated correction amount, correction for vibration applied to the image is corrected on a line basis.

The image processing apparatus of the present invention is a means for calculating a correction amount for each line with respect to the applied vibration based on the vibration amount from the detecting means for detecting the applied vibration, and the exposure time is 1/60. When less than a second , a first correction amount that is a correction amount in units of lines is calculated based on the vibration amount, and when the exposure time is 1/60 second or more, at least for some lines, A second correction amount that is smaller than the first correction amount and that has a difference in correction amount for each line that is smaller than the difference in the first correction amount for each line. A calculating unit that calculates a correction amount; and a correcting unit that corrects the vibration applied to the image in units of lines based on the correction amount in units of lines calculated by the calculating unit.

The image processing method of the present invention is a step of calculating a correction amount in line units for the applied vibration based on the vibration amount from the detecting means for detecting the applied vibration, and the exposure time is 1/60. When less than a second , a first correction amount that is a correction amount in units of lines is calculated based on the vibration amount, and when the exposure time is 1/60 second or more, at least for some lines, A second correction amount that is smaller than the first correction amount and that has a difference in correction amount for each line that is smaller than the difference in the first correction amount for each line. A calculation step for calculating a correction amount; and a correction step for correcting the vibration applied to the image in units of lines based on the correction amount in units of lines calculated in the processing of the calculation step.

The recording medium of the present invention is a step of calculating a correction amount in line units for the applied vibration based on the vibration amount from the detecting means for detecting the applied vibration, and the exposure time is 1/60 second. when less than the time to calculate the first correction amount is a correction amount for each line based on the vibration amount, when the exposure time is more than time 1/60 seconds, for at least a part of the line, the A second correction that is a correction amount in units of lines, the value being smaller than the first correction amount, and the difference in the correction amount for each line being smaller than the difference in the first correction amount for each line. A computer comprising: a calculation step for calculating an amount; and a correction step for correcting, in line units, vibration applied to the image based on the correction amount in line units calculated in the processing of the calculation step. reading Noh program is recorded.

The program of the present invention is a step of calculating a correction amount in line units for the applied vibration based on the vibration amount from the detecting means for detecting the applied vibration, wherein the exposure time is 1/60 second. When the exposure time is less than 1/60 seconds , a first correction amount that is a correction amount in units of lines is calculated based on the vibration amount. A second correction amount that is a correction amount in units of lines, the value of which is smaller than the correction amount of 1, and the difference of the correction amount for each line is smaller than the difference of the first correction amount for each line. A computer comprising: a calculation step for calculating a correction value for the vibration applied to the image based on the correction amount for the line unit calculated in the processing of the calculation step; A possible program.

In the image processing apparatus and method, and the program of the present invention, the correction amount in units of lines for the applied vibration is calculated based on the vibration amount of the applied vibration. The calculation calculates a first correction amount that is a correction amount for each line based on the vibration amount when the exposure time is less than 1/60 seconds , and at least when the exposure time is 1/60 seconds or more. Correction for each line in which a value is smaller than the first correction amount and a difference in the correction amount for each line is smaller than a difference in the first correction amount for each line for some lines. This is done by calculating a second correction amount that is an amount. Further, based on the calculated correction amount, correction for vibration applied to the image is corrected on a line basis.

  According to the present invention, it is possible to reduce an adverse effect given to a photographed image due to vibration applied during photographing.

  According to the present invention, it is possible to prevent an image taken at the time of long exposure from becoming an annoying image due to an adverse effect caused by vibration applied at the time of shooting.

  Embodiments of the present invention will be described below with reference to the drawings.

[Configuration example of image processing apparatus]
FIG. 4 is a diagram showing a configuration of an embodiment of an image processing apparatus to which the present invention is applied. The image sensor 11 of the image processing apparatus 10 is configured by, for example, an XY address type solid-state image sensor (CMOS or the like). Data of the subject image captured by the image sensor 11 is supplied to an AFE (Analog Front End) 12.

  The AFE 12 converts the supplied image data into image data of a digital signal and supplies it to the signal processing unit 13. The signal processing unit 13 calculates a luminance signal and a color difference signal from the supplied image data, and supplies the calculated signals to the image interpolation unit 14.

  The image data supplied to the image interpolation unit 14 is image data of the subject imaged by the image sensor 11, but all data of the image captured by the image sensor 11 is supplied to the image interpolation unit 14. However, only the data read according to the timing from the TG (Timing Generator) 15 is supplied to the image interpolation unit 14.

  The data supplied to the image interpolation unit 14 is once supplied to the memory 17 under the control of the memory controller 16. Conversely, the data stored in the memory 17 is read out according to an instruction from the memory controller 16 and supplied to the image interpolation unit 14. The image interpolation unit 14 performs interpolation processing (details will be described later) for reducing adverse effects due to vibration on the supplied data, and outputs the data to a recording medium (not shown) for recording or recording. Output to a non-display etc. and display it. The vibration applied to the image processing apparatus 10 includes a user's hand shake.

  The image interpolation unit 14, the TG 15, and the memory controller 16 perform control based on the correction amount calculated by the correction amount calculation unit 19 according to the vibration amount detected by the vibration detection unit 18.

  The vibration detection unit 18 uses, for example, a method using a sensor such as an angular velocity sensor or a sensorless vibration detection method using image processing or the like, and detects vibration applied to the image processing apparatus 10 during photographing. For example, when the vibration detection unit 18 is configured by an angular velocity sensor, the angular velocity sensor supplies the correction amount calculation unit 19 with angular velocity data applied to the pitching direction and the yawing direction.

  Based on the detected vibration amount, the correction amount calculation unit 19 calculates correction amount data for correcting the movement caused by the vibration. When calculating the correction amount, the correction amount calculation unit 19 refers to information related to the exposure time acquired by the exposure time acquisition unit 20. The exposure time acquisition unit 20 acquires information related to the exposure time from a portion that controls a shutter (not shown).

  The correction amount calculation unit 19 calculates, for each line, a correction amount indicating how many pixels are moved by using the supplied data to reduce the influence of the applied vibration. Here, for each line, in an individual imaging device such as a CMOS, image data is sequentially read out in units of one line. Therefore, the correction amount is calculated for each line.

[Operation of Image Processing Device]
The operation of the image processing apparatus 10 will be described with reference to the flowcharts of FIGS. As described above, the image processing apparatus 10 executes processing for reducing the influence of the applied vibration.

  In step S <b> 11, the correction amount calculation unit 19 acquires vibration amount data detected from the vibration detection unit 18. For example, information as shown in FIG. 8 is detected by the vibration detection unit 18 and supplied to the correction amount calculation unit 19.

In the example illustrated in FIG. 8, a case where one screen is configured by eight lines by the vibration detection unit 18 and eight vibration amounts (hand shake amounts) are acquired for each line on the one screen is illustrated. Yes. That is, the vibration amount P8 is in the case of the line 8, the vibration amount P7 is in the case of the line 7, the vibration amount P6 is in the case of the line 6, the vibration amount P5 is in the case of the line 5, and the vibration amount P4 is in the case of the line 4. However, the vibration amount P3 is acquired for the line 3, the vibration amount P2 is acquired for the line 2, the vibration amount P1 is acquired for the line 1, and the vibration amount P0 is acquired for the line 0.

  Here, for convenience of explanation, it is assumed that one screen is composed of 9 lines, but actually, it is composed of 9 lines or more. Here, the vibration amount is acquired every 9 lines. However, in reality, there is a difference between the sampling frequency for acquiring the vibration amount and the line readout timing (the readout time depends on the exposure time, etc.). However, since there is a case where it does not coincide with a timing having a certain period such as a sampling frequency because it changes, the vibration amount cannot always be acquired at the center of the exposure time for each line.

  Therefore, the vibration amount acquired in a predetermined line is commonly used in several lines including the line. Alternatively, interpolation is performed using the obtained discrete vibration amount, and a vibration amount for each line is generated.

  Thus, when the vibration amount is acquired for each line, the correction amount is calculated in step S12. The correction amount is a pixel unit value indicating how much correction should be performed in order to set the vibration amount to a state without vibration (a state in which the influence of vibration is reduced).

  For example, as shown in FIG. 8, when the vibration amount related to the line 8 is P8 pixels, the absolute value having a different sign is used to change the vibration amount of the P8 pixel to a state without vibration. That is, in this case, -P8 pixel may be calculated as the correction amount. Similarly, for the lines 7 to 0 other than the line 8, the correction amount is calculated from the vibration amount for each line.

  Thus, when the correction amount is calculated, the exposure time is acquired in step S13 (FIG. 5). Information relating to the exposure time is acquired by the exposure time acquisition unit 20 and supplied to the correction amount calculation unit 19. In step S14, the correction amount calculation unit 19 determines whether or not the supplied exposure time is equal to or greater than the first threshold value (whether or not the condition that exposure time> first threshold value is satisfied).

  In the present embodiment, the exposure time is divided into short exposure and long exposure. The processing (correction) for reducing the influence on vibration is different between the short-time exposure and the long-time exposure. Therefore, in step S14, at that time (time when the process of step S14 is executed), a process for determining whether the exposure is a short exposure or a long exposure is executed.

  Here, the first threshold value used in the process of step S14 is a value provided to separate short-time exposure and long-time exposure. In step S14, if it is determined that the exposure time is smaller than the first threshold, in other words, if it is determined that the exposure time is short exposure, the process proceeds to step S15, and the processing at the time of short exposure is performed. Executed. On the other hand, if it is determined in step S14 that the exposure time is greater than the first threshold, in other words, if it is determined that the exposure time is long exposure, the process proceeds to step S16, and the exposure time for the long exposure time is determined. Processing is executed.

  The short exposure process executed in step S15 and the long exposure process executed in step S16 will be described. First, with reference to the flowchart of FIG. 6, the process at the time of short exposure performed in step S15 is demonstrated.

  In step S31, the reading start position is determined. The reading start position is determined based on the correction amount calculated in step S12. The correction amount is calculated for each line, but based on the correction amount for each line, that is, based on data indicating how many pixels the reading position is shifted, the reading start position of each line is determined. The

  When the read start position is determined, data is read from the determined read start position in step S32. In this way, the correction amount is calculated for each line, and the position on the line where data reading is started is determined based on the calculated correction amount, thereby correcting the influence of vibration for each line. Can do. Therefore, the image provided to the user side can be an image in which adverse effects due to vibration are reduced.

  It should be noted that the invention described in Japanese Patent Application Laid-Open No. 2004-266322 filed earlier by the present applicant can be applied to the above-described correction-related processing.

  Next, the long time exposure process executed in step S16 (FIG. 5) will be described with reference to the flowchart of FIG. In step S51, a representative amount is selected. Referring to FIG. 8, the vibration amount is acquired every nine lines, and the correction amount is calculated every nine lines. Of these, the correction amount for at least one line is selected as the representative amount.

  As will be described below, at the time of long exposure, the correction amount calculated for each line is not used for each line, but the correction amount for at least one line is used as a representative amount. That is, in the case shown in FIG. 8, nine correction amounts are calculated, but at least one of the nine correction amounts is used, and a correction process described later is executed.

  As the representative amount, for example, in one screen, the correction amount of the line for which the correction amount is calculated first (in FIG. 8, the correction amount calculated from the vibration amount P8 in the line 8 (hereinafter simply referred to as the correction amount). It is described as an amount P8, and other correction amounts are also described in the same manner) as a representative amount (in this case, the correction amount for one line is used).

  Alternatively, in one screen, the correction amount of the line for which the correction amount is calculated first (correction amount P8 in FIG. 8) and the correction amount of the line for which the correction amount is calculated last (in FIG. 8). May calculate an average value of the correction amount P0) and use the average value as a representative amount (in this case, the correction amounts of two lines are used).

  Alternatively, a correction amount acquired at a line located at the center of the image (a correction amount acquired at a position corresponding to the center of exposure of all frames. For example, in FIG. 8, it is calculated from the vibration amount P4 in the line 4. The correction amount P4) may be set as the representative amount.

  Regarding the method of selecting the representative amount, the method as described above can be considered, and any method may be used, or a method other than the method described above may be used.

  Here, the description will be continued by taking as an example the case where the correction amount calculated first is the representative amount. By using the correction amount calculated first as the representative amount, there are the following advantages. If the last correction amount is used as the representative amount (and the last correction amount is also used), the data of the last line must be acquired in order to determine the representative amount. . Therefore, in order to determine the representative amount, a memory having a capacity capable of storing data from the first line to data of the last line, that is, a memory having a capacity capable of storing data for one screen is required.

  However, by using the correction amount calculated first as the representative amount, at least in the process of determining the representative amount, there is no need for a memory having a capacity capable of storing data for one screen. Any memory having a capacity capable of storing data for one line may be used. That is, the memory capacity can be reduced.

  When the representative amount is determined in step S51, a coefficient is calculated in step S52. Here, a description of this coefficient will be added. For the correction amount other than the representative amount, a value obtained by multiplying the correction amount calculated from each line by a predetermined coefficient is used as a new correction amount, and is used as the correction amount for the line.

  In other words, in the processing at the time of short exposure, the correction amount is calculated for each line from the vibration amount acquired for each line, and the correction amount is used to perform correction. In this process, the correction amount is calculated for each line from the vibration amount acquired for each line, and the correction amount other than the representative amount is further multiplied by a coefficient, and the correction amount multiplied by the coefficient is used. And correction is performed.

  The coefficient is determined depending on the exposure time. This will be described with reference to FIG. FIG. 9 is a diagram showing the relationship between the exposure time and the coefficient. As described above, the first threshold value is a threshold value provided for identifying short-time exposure and long-time exposure in step S14 (FIG. 5). When the exposure time becomes equal to or greater than the first threshold value, a long-time exposure is processed, and a coefficient to be multiplied by the correction amount is set.

  When the exposure time is equal to or shorter than the first threshold value, that is, when the exposure is performed for a short time, 1 is set as the coefficient. Since the coefficient is 1, the value does not change even if it is multiplied in the calculation. Therefore, as described with reference to FIG. 6, processing such as calculating the coefficient and multiplying the correction amount is performed at the time of short exposure. Not done.

  When the exposure time is between the first threshold value and the second threshold value, in the example shown in FIG. 9, a coefficient is determined (calculated) in proportion to the exposure time in a linear function.

  When the exposure time exceeds the second threshold, the coefficient is set to 0. When the coefficient is 0, as a result, the correction amount is 0, which means that processing is performed. This means that correction for correction amounts other than the representative amount is not performed.

  The relationship between the exposure time and the coefficient shown in FIG. 9 is an example, and does not mean that the present invention is limited to the determination of such a coefficient. As another method of calculating the coefficient, the coefficient may be set to 0 when the exposure time exceeds the first threshold value. That is, in such a case, since the coefficient is 1 or 0 with the first threshold as a boundary, it is determined whether correction is performed. Alternatively, when the exposure time is equal to or greater than the first threshold, the coefficient is set to 1 (or 1 or less) for even lines, and the coefficient is set to 0 for odd lines. Also good.

  In addition, when the exposure time is longer than the second threshold, the coefficient is set to 0. However, the coefficient may be set to a value other than 0, for example, 0.5. In other words, when the exposure time is equal to or greater than the second threshold value, the correction may not be performed at all, but may be set such that a weaker correction is performed than in the normal (during short-time exposure).

  In this way, any method may be used for calculating the coefficient. However, in the case of long exposure, the correction amount calculated for each line is not used for correction, but the coefficient is multiplied. Is done. Here, as shown in FIG. 9, the description will be continued assuming that the coefficient is set (calculated) according to the exposure time.

  When the coefficient is calculated in step S52, the correction amount is calculated in step S53. The correction amount has already been calculated in step S12 (FIG. 5). In the process in step S52, a new correction amount is calculated by multiplying the correction amount already calculated by the set coefficient. Is done.

  As shown in FIG. 8, the process of step S <b> 53 will be described by taking as an example a case where nine correction amounts are calculated in one image. The correction amount P8 calculated from the line 8 is set as a representative amount, and the other correction amounts P7 to P0 calculated from the lines 7 to 0 are multiplied by the coefficient calculated in step S52. . A correction amount multiplied by the coefficient is set as a new correction amount.

  In the present embodiment, the correction amount set as the representative amount is used for correction as it is, and the correction amount other than the representative amount is multiplied by a coefficient and used for correction.

  When multiplying the coefficients, the already calculated correction amount itself may be multiplied, or the correction amount may be multiplied after being processed in some manner. For example, the difference between the correction amount of each line and the representative amount may be calculated, and the difference may be multiplied by a coefficient. In this case, the representative amount remains as it is, but the correction amount of the other lines is a value obtained by multiplying the difference from the representative amount by a coefficient.

  Also, the representative amount itself may be multiplied by a coefficient in the same manner as other correction amounts.

  In step S54, the reading start position is determined based on the set correction amount. Then, reading of data from the determined reading start position is executed in step S55.

  As described above, a value in the range of 0 to 1 is assigned as the coefficient. Therefore, the correction amount multiplied by the coefficient (correction amount calculated in step S53) is smaller than the correction amount before the coefficient is multiplied (correction amount calculated in step S12).

  That is, the correction amount calculated in step S12 is used for correction as it is during short-time exposure, but is converted into a small correction amount by multiplication by a coefficient and used for correction during long-time exposure. . Therefore, even if the same magnitude of vibration is applied, the amount of correction is small during long exposure compared to short exposure.

  However, in general, the amount of vibration is more easily accumulated during long exposure than during short exposure, and the correction amount tends to increase. In addition, in the period during which one line is being read, the direction of vibration is more likely to change during long exposure than during short exposure. As described above, the correction is performed in the opposite direction to the applied vibration, with a magnitude equal to the applied vibration as a correction amount, in order to make the applied vibration free from vibration. It is done by using. Thus, when the applied vibration is in one direction, the correction is performed according to that direction.

  Therefore, when vibration is applied in different directions in a section (one line) to be corrected, one of the directions is set as a direction to be corrected, and the other direction cannot be set as a correction target. For example, when an upward vibration and a downward vibration are applied within the readout period of one line, if the upward vibration is a correction target (when the vibration amount is acquired during the upward vibration) ) In that line, it is not possible to perform correction for the downward vibration at the same time.

  Due to such a state, there is a high possibility that correction is performed with a large correction amount in one direction of the applied vibration during long exposure. Such a correction may be a correction that promotes the influence of vibration in a direction in which the correction is not performed. For example, as described with reference to FIG. 3, distortion may occur in the image 2 or the like.

  In the present embodiment, the correction amount is multiplied by a coefficient so that the correction amount is smaller than that during short-time exposure during long exposure, and the correction amount multiplied by the coefficient (correction amount set to a small value). ) Is used to perform correction, so that distortion can be reduced.

  The fact that the correction amount is multiplied by a coefficient so that the correction amount is smaller than that during the short-time exposure will be further described with reference to the drawings. FIG. 10 shows the correction amount calculated when the camera shake amount as shown in FIG. 8 is acquired.

  Since the correction amount is an amount added in a direction to cancel the influence of the camera shake amount, if FIG. 10 corresponds to FIG. 8, the direction of the camera shake amount and the direction of the correction amount are opposite to each other. Therefore, the correction amount in FIG. 10 should be illustrated in the reverse direction. Further, since the direction of the correction amount for each line also depends on the direction of the amount of camera shake, the direction may be different for each line. However, here, for convenience of explanation, they are shown in the same direction. The same applies to the other drawings.

  In FIG. 10, the correction amount of the line 8 is the correction amount C8. This correction amount C8 is set as a representative amount. The exposure time when the correction amount as shown in FIG. 10 is calculated is when it is smaller than the first threshold. That is, the correction amount calculated at the time of short-time exposure is shown. In this case, the correction amounts C7 to C0 other than the representative amount are multiplied by “1” as a coefficient (or the process of multiplying the coefficient is omitted, and the calculated correction amount is used as it is). .

  FIG. 11A and FIG. 11B show the case where the correction amount similar to that in FIG. 10 is calculated, but the correction amount when the exposure time is larger than the first threshold, that is, when the long exposure time is calculated. Show. Because of the long exposure time, the correction amount once calculated (referred to as the first correction amount) is further multiplied by a coefficient, so that the correction amount actually used for correction (referred to as the second correction amount) is obtained. Calculated. In FIG. 11A and FIG. 11B, a white circle (a circle in which the circle is not filled) represents the first correction amount, and a black circle (a circle in which the circle is filled) represents the second correction amount.

  The second correction amount shown in FIG. 11A shows when 0.5 is multiplied as a coefficient, and the second correction amount shown in FIG. 11B shows when 0 is multiplied as a coefficient. In the figure, comparing the white circle representing the first correction amount and the black circle representing the second correction amount, it can be seen that the correction amount is smaller in the long exposure than in the short exposure. . That is, the second correction amount (black circle) used for correction at the time of long exposure is smaller than the first correction amount (white circle) used for correction at the time of short exposure. This can be understood by referring to FIGS. 11A and 11B.

  As shown in FIG. 11B, when “0” is multiplied as a coefficient, the first correction amount is multiplied by 0 to calculate the second correction amount. The calculated value is 0. Therefore, when it is determined that the exposure time is multiplied by 0 as a coefficient, processing such as calculating the first correction amount and the second correction amount may be omitted.

  For example, in FIG. 5, when the processes of step S12, step S13, and step S14 are interchanged and exposure time information is first acquired and it is determined that the exposure time is multiplied by 0 as a coefficient (that is, exposure time) If it is determined that is equal to or longer than the second exposure time), subsequent processing (processing for calculating the first correction amount and the second correction amount) may be omitted. However, in this case, since it is necessary to calculate the representative amount, only the representative amount needs to be calculated.

  Since this can be applied when multiplying by “0” as a coefficient, the order of processing in the flowchart shown in FIG. 5 is appropriately changed so that the correction can be performed efficiently. It is a change within the scope of application.

  FIG. 12 is a diagram for explaining how to multiply other coefficients. FIG. 12 also shows the correction amounts C8 to C0 for each line, as in FIG. The method of multiplying coefficients in FIG. 12 is a method of multiplying the difference from the representative amount by a coefficient. That is, for example, the correction amount C7 of the line 7 is relative to the difference between the representative amount C8 (correction amount C8) (first correction amount for the line 8) and the correction amount C7 (first correction amount for the line 7). A correction amount (second correction amount) obtained by multiplying the coefficient by the coefficient is used as the correction amount used for the actual correction in the line 7.

  FIG. 12 shows that the coefficient is multiplied by “1” because of the correction amount at the time of short-time exposure. That is, the calculated correction amount (first correction amount) is used as it is.

  On the other hand, FIG. 13A shows the correction amount when “0.5” is multiplied as the coefficient, and FIG. 13B shows the correction amount when “0” is multiplied as the coefficient. 13A and 13B, as in FIGS. 11A and 11B, the white circle represents the first correction amount, and the black circle represents the second correction amount.

  FIG. 13B shows the second correction amount when 0 is multiplied as a coefficient, but this 0 is multiplied for the difference from the representative amount, and the difference from the representative amount. No matter how large, if it is multiplied by 0 as a result, its value becomes 0. Therefore, when “0” is multiplied as a coefficient, the correction amount (second correction amount) in all lines is the same value as the representative amount (the difference from the representative amount is 0), and the correction amount is used. Will be corrected.

  Therefore, in such a case (when 0 is multiplied as a coefficient), the processing related to the calculation of the correction amount may be omitted when the representative amount is calculated.

  FIG. 14 is a diagram conceptually showing an image before correction and an image after correction when correction is performed with the correction amount shown in FIG. A portion illustrated by a dotted line represents an image before correction, and a portion illustrated by a solid line represents an image after correction. In FIG. 14, the image before correction and the image after correction are also shifted in the vertical direction, but are only shifted for the sake of explanation, and the correction amount in FIG. It does not mean that it will be corrected.

  Referring to FIG. 14, the read start position is shifted for each line by the correction amount for each line. By performing such correction, it is assumed that the actual image has reduced distortion and bleeding.

  Similarly, FIG. 15A corresponds to FIG. 13A, and FIG. 15B corresponds to FIG. 15B. The diagram shown in FIG. 15A shows images before and after correction when the coefficient is multiplied by 0.5. To compare the corrected image shown in FIG. 14 (image when the coefficient is “1”) and the corrected image shown in FIG. 15A (image when the coefficient is “0.5”), It can be seen that the corrected image shown in FIG. 15A has a smaller reading start position shift.

  Referring to FIG. 15B, it can be seen that when the coefficient is 0, an image obtained by shifting the uncorrected image by the representative amount is substantially the corrected image.

  By doing so, it can be understood that, at the time of long-time exposure, the influence due to the amount of shake (vibration) applied can be corrected at least roughly. That is, as described above, it can be seen that even if the blur cannot be corrected, at least the distortion can be corrected. Therefore, the adverse effects due to vibration can be reduced even during long exposure.

  FIG. 16 is a diagram for explaining how to multiply other coefficients. FIG. 16 also shows correction amounts C8 to C0 for each line, as in FIG. The method of multiplying coefficients in FIG. 16 is a method of multiplying the representative amount by a coefficient. In other words, this is a method of calculating the second correction amount by multiplying all the calculated first correction amounts by the coefficients.

  Since the correction amount shown in FIG. 16 is obtained by multiplying the first correction amount by “1” as a coefficient, the value does not change, and the first correction amount = the second correction amount. Therefore, in such a case (in the case of short-time exposure), as described above, processing such as coefficient setting and multiplication may be omitted.

  FIG. 17A and FIG. 17B show correction amounts when “0.5” and “0” are multiplied as coefficients, respectively. Referring to FIG. 17A, the representative amount is also multiplied by “0.5”. Also, referring to FIG. 17B, the representative example is also multiplied by “0”.

  FIG. 18 shows an image before correction and an image after correction when correction is performed with the correction amount shown in FIG. 19A shows an image before correction and an image after correction when correction is performed with the correction amount shown in FIG. 17A, and FIG. 19B shows an image before correction and an image after correction when correction is performed with the correction amount shown in FIG. 17B. . Even when the coefficient including the representative amount is multiplied, as in the case described above, the adverse effect due to vibration can be reduced even during long exposure.

  By executing the processing as in the present embodiment, as shown in FIG. 20, even when the image A is photographed under a situation where a lateral vibration is applied during long exposure. To prevent distortion from occurring in image 1 ′, image 2 ′, image 3 ′, and image 4 ′ captured at time T1 ′, time T2 ′, time T3 ′, and time T4 ′, respectively. Is possible.

  In the above-described embodiment, during long exposure, a representative amount is selected from a plurality of correction amounts, and a correction amount other than the representative amount is multiplied by a coefficient. Instead of selecting the representative amount, all correction amounts may be multiplied by a coefficient (as described with reference to FIG. 16 and the like). Further, the correction amount may be selected from among a plurality of correction amounts, for example, a coefficient may be multiplied only for the correction amount of an odd number of lines.

  In the above-described embodiment, correction is performed with a correction amount multiplied by a coefficient during long exposure, but instead of a coefficient multiplied by the correction amount, a vibration amount (correction amount) within the exposure period is used. ) Is calculated, a correction amount is obtained from the average value, and the correction amount may be used to perform correction. That is, during long exposure, an average value of vibration amounts applied while one line is read may be calculated, a correction amount may be calculated from the average value, and correction may be performed using the correction amount.

  Or you may make it use the average of the vibration amount for a longer period, for example, the whole exposure period (1 field). When the average of the vibration amount for the entire exposure period is used, it can be realized by providing a memory having a capacity capable of storing such data.

  When the average value is calculated, the image processing apparatus 10 (FIG. 4) may be configured to include a filter in order to obtain the average value. A filter that performs processing such as LPF (Low Pass Filter), interpolation, and fitting (fitting: approximation) can be applied.

[About recording media]
The series of processes described above, for example, processes related to correction can be executed by hardware having the respective functions, but can also be executed by software. When a series of processing is executed by software, various functions can be executed by installing a computer in which the programs that make up the software are installed in dedicated hardware, or by installing various programs. For example, it is installed from a recording medium in a general-purpose personal computer or the like.

  FIG. 21 is a diagram illustrating an internal configuration example of a general-purpose personal computer. A CPU (Central Processing Unit) 41 of the personal computer executes various processes according to a program stored in a ROM (Read Only Memory) 42. A RAM (Random Access Memory) 43 appropriately stores data and programs necessary for the CPU 41 to execute various processes. The input / output interface 45 is connected to an input unit 46 including a keyboard and a mouse, and outputs a signal input to the input unit 46 to the CPU 41. The input / output interface 45 is also connected to an output unit 47 including a display, a speaker, and the like.

  The input / output interface 45 is also connected to a storage unit 48 configured by a hard disk and a communication unit 49 that exchanges data with other devices via a network such as the Internet. The drive 50 is used when data is read from or written to a recording medium such as the magnetic disk 61, the optical disk 62, the magneto-optical disk 63, and the semiconductor memory 64.

  As shown in FIG. 21, the recording medium is distributed to provide a program to the user separately from the personal computer, and includes a magnetic disk 61 (including a flexible disk) on which the program is recorded, an optical disk 62 (CD- Consists of package media including ROM (Compact Disc-Read Only Memory), DVD (including Digital Versatile Disc), magneto-optical disk 63 (including MD (Mini-Disc) (registered trademark)), or semiconductor memory 64 In addition, it is configured by a hard disk including a ROM 42 storing a program and a storage unit 48 provided to the user in a state of being pre-installed in a computer.

  In this specification, the steps for describing the program provided by the medium are performed in parallel or individually in accordance with the described order, as well as the processing performed in time series, not necessarily in time series. The process to be executed is also included.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

It is a figure explaining the relationship between exposure time and a vibration amount. It is a figure explaining the relationship between exposure time and a vibration amount. It is a figure for demonstrating distortion. It is a figure which shows the structure of one Embodiment of the image processing apparatus to which this invention is applied. It is a flowchart for demonstrating operation | movement of an image processing apparatus. It is a flowchart for demonstrating the process at the time of short time exposure. It is a flowchart for demonstrating the process at the time of long exposure. It is a figure for demonstrating the amount of vibrations. It is a figure for demonstrating a threshold value. It is a figure for demonstrating the corrected amount and a coefficient. It is a figure for demonstrating the corrected amount and a coefficient. It is a figure for demonstrating the corrected amount and a coefficient. It is a figure for demonstrating the corrected amount and a coefficient. It is a figure for demonstrating the image before correction | amendment, and the image after correction | amendment. It is a figure for demonstrating the image before correction | amendment, and the image after correction | amendment. It is a figure for demonstrating the corrected amount and a coefficient. It is a figure for demonstrating the corrected amount and a coefficient. It is a figure for demonstrating the image before correction | amendment, and the image after correction | amendment. It is a figure for demonstrating the image before correction | amendment, and the image after correction | amendment. It is a figure for demonstrating distortion. It is a figure explaining a medium.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 Image processing apparatus, 11 Image sensor, 12 AFE, 13 Signal processing part, 14 Image interpolation part, 15 TG, 16 Memory controller, 17 Memory, 18 Vibration detection part, 19 Correction amount calculation part, 20 Exposure time acquisition part

Claims (6)

An image pickup means for picking up an image with an XY address type solid-state image pickup device;
A means for calculating a correction amount in line units for the applied vibration based on the vibration amount from the detecting means for detecting the applied vibration, wherein an exposure time of the imaging means is less than 1/60 second Sometimes, a first correction amount that is a correction amount in units of lines is calculated based on the vibration amount, and when the exposure time is 1/60 second or more, the first correction amount is applied to at least a part of the lines. A second correction amount, which is a correction amount for each line, is smaller than the correction amount, and the difference in the correction amount for each line is smaller than the difference in the first correction amount for each line. Calculating means for
An imaging apparatus comprising: correction means for performing correction for vibration applied to the image in units of lines based on the correction amount in units of lines calculated by the calculation unit. When the first correction amount of at least one line is used as a representative amount, the calculating means sets the second correction amount that is the same value as the first correction amount for the at least one line. To calculate
The second correction amount corresponding to a value obtained by multiplying the first correction amount in each line by a predetermined coefficient less than 1 is calculated for lines other than the at least one line. Imaging device. Based on the amount of vibration from the detecting means for detecting the applied vibration, the means for calculating a correction amount in units of line with respect to the applied vibration, the vibration when the exposure time is less than 1/60 seconds A first correction amount that is a correction amount in units of lines is calculated based on the amount, and when the exposure time is 1/60 second or more, a value is calculated from the first correction amount for at least some lines. Calculating means for calculating a second correction amount, which is a correction amount in units of lines, in which the difference in correction amount for each line is smaller than the difference in the first correction amount for each line. ,
An image processing apparatus comprising: correction means for performing correction for vibration applied to the image for each line based on the correction amount for each line calculated by the calculation means. A step of calculating a correction amount in line units for the applied vibration based on the vibration amount from the detecting means for detecting the applied vibration, and the vibration when the exposure time is less than 1/60 second A first correction amount that is a correction amount in units of lines is calculated based on the amount, and when the exposure time is 1/60 second or more, a value is calculated from the first correction amount for at least some lines. And calculating a second correction amount that is a correction amount in units of lines, in which the difference in correction amount for each line is smaller than the difference in the first correction amount for each line; ,
An image processing method comprising: a correction step of performing correction for vibration applied to the image in units of lines based on the correction amount in units of lines calculated in the processing of the calculation step. A step of calculating a correction amount in line units for the applied vibration based on the vibration amount from the detecting means for detecting the applied vibration, and the vibration when the exposure time is less than 1/60 second A first correction amount that is a correction amount in units of lines is calculated based on the amount, and when the exposure time is 1/60 second or more, a value is calculated from the first correction amount for at least some lines. And calculating a second correction amount that is a correction amount in units of lines, in which the difference in correction amount for each line is smaller than the difference in the first correction amount for each line; ,
A computer-readable program is recorded that includes: a correction step for correcting the vibration applied to the image in units of lines based on the correction amount in units of lines calculated in the processing of the calculation step. Recording medium. A step of calculating a correction amount in line units for the applied vibration based on the vibration amount from the detecting means for detecting the applied vibration, and the vibration when the exposure time is less than 1/60 second A first correction amount that is a correction amount in units of lines is calculated based on the amount, and when the exposure time is 1/60 second or more, a value is calculated from the first correction amount for at least some lines. And calculating a second correction amount that is a correction amount in units of lines, in which the difference in correction amount for each line is smaller than the difference in the first correction amount for each line; ,
A computer-readable program comprising: a correction step for correcting line-by-line with respect to vibration applied to the image based on the correction amount for the line unit calculated in the calculation step.

Images