JP2015126436A - Image processor, image processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To satisfactorily reduce uncomfortable fluctuations of images due to camera shakes.SOLUTION: An image fluctuation amount of an input image is acquired at every points of time. An image fluctuation amount at every points of time, which is obtained by accumulating image fluctuation amount at every points of time, is subjected to a stabilization filtering processing to acquire a correction amount at every points of time on the basis of the accumulated image fluctuation amount before and after the stabilization filtering processing. The image fluctuation at every points of time of the input image is corrected by using the correction amount at every points of time to output the corrected image. A filter parameter of a stabilization filter at every points of time is set according to the reliability of a piece of information relevant to the state a camera for obtaining an input image and/or the fluctuation amount of the image acquired at every points of time.

Description

  The present technology relates to an image processing apparatus, an image processing method, and a program, and more particularly to an image processing apparatus, an image processing method, and a program that correct an image fluctuation amount due to camera shake or the like.

  Vibration or unnatural movement of the camera as represented by camera shake causes unpleasant image shaking in the moving image. Conventionally, as a technique for reducing this unpleasant image shake, an optical image stabilization method, an electronic image stabilization method, and the like have been proposed and adopted. For example, in the electronic image stabilization method, an unpleasant image shake is reduced by cutting out and deforming a part of the captured and developed image.

  The amount of variation between images includes movements intended by the user, such as panning and tilting, in addition to those not intended by the user, such as camera shake. Therefore, in order to perform a desired correction, it is necessary to calculate the correction amount by processing the variation amount. Conventionally, a technique for performing correction based on the amount of change between images has been proposed (see, for example, Patent Document 1 and Patent Document 2).

JP 2011-029735 A Special table 2008-541626

  An object of the present technology is to satisfactorily reduce unpleasant image shaking due to camera shake or the like.

The concept of this technology is
An image fluctuation amount acquisition unit for acquiring an image fluctuation amount at each time of the input image;
Stabilization filter processing is performed on the accumulated image fluctuation amount at each time obtained by integrating the acquired image fluctuation amounts at each time, and each time based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
An image correction unit that corrects image fluctuation at each time of the input image using the acquired correction amount at each time and outputs a corrected image,
The correction amount acquisition unit
The image processing apparatus sets a filter parameter of the stabilization filter at each time according to shooting information of the camera that obtains the input image.

  In the present technology, the image fluctuation amount at each time of the input image is acquired by the image fluctuation amount acquisition unit. Here, the image fluctuation amount at each time is a fluctuation amount between adjacent images, that is, a fluctuation amount between images. The amount of change between images is obtained by, for example, pattern matching.

  The correction amount acquisition unit accumulates the acquired image fluctuation amount at each time to obtain the accumulated image fluctuation amount at each time, and the accumulated filter fluctuation amount at each time is subjected to a stabilization filter process. Further, the correction amount acquisition unit acquires the correction amount at each time based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. Then, the image correction unit corrects the image fluctuation at each time of the input image using the acquired correction amount at each time.

  In the correction amount acquisition unit, the filter parameter of the stabilization filter at each time is set according to the shooting information of the camera. Thereby, when the estimation result of the fluctuation amount is low in reliability and easily erroneous, the correction can be weakened, and erroneous correction due to erroneous detection can be prevented. For example, the shooting information of the camera is the state information of the camera that obtains the input image and / or the reliability of the acquired image fluctuation amount at each time.

  In the present technology, for example, the correction amount acquisition unit obtains a modulation value from the state information of the camera that obtains the input image, and adjusts the predetermined filter parameter with the modulation value, or adjusts the predetermined filter parameter with reliability. Then, the filter parameter of the stabilization filter may be set. In this case, for example, the camera status information includes the output signal of the gyro sensor of the camera, the filter parameter is the number of taps, and the correction amount acquisition unit determines whether the camera moves based on the output signal of the gyro sensor. The smaller the modulation value, the smaller the modulation value may be obtained.

  In the present technology, for example, the correction amount acquisition unit obtains a modulation value from the state information of the camera that obtains the input image, and adjusts the predetermined correction margin with the modulation value, or adjusts the predetermined correction margin with reliability. Then, the filter parameter of the stabilization filter may be set so that the correction amount does not exceed the modulated correction margin. As a result, it is possible to ensure that the correction amount falls within the correction margin, and it is possible to prevent failure. In this case, for example, the camera state information includes camera shutter speed information, the filter parameter is the number of taps, and the correction amount acquisition unit obtains a smaller modulation value as the shutter speed becomes slower. May be.

  In the present technology, for example, the filter parameter is the number of taps, and the correction amount acquisition unit obtains a first modulation value that modulates the number of taps of the stabilization filter from the state information of the camera that obtains the input image. A second modulation value for modulating the correction margin is obtained, the predetermined number of taps is modulated with the first modulation value to obtain the first number of taps, the predetermined margin is modulated with the second modulation value, and the correction amount Finds the maximum number of taps that does not exceed this modulated correction margin as the second tap number, and sets the smaller of the first tap number and the second tap number as the tap number of the stabilization filter. May be.

  In the present technology, for example, the correction amount acquisition unit performs a smoothing process on the filter parameters of the stabilization filter set at each time, and the smoothing process is performed on the accumulated image fluctuation amount at each time. A stabilization filtering process may be performed based on the filter parameters. In this case, for example, the filter parameter may be the number of taps, and the smoothing process to the filter parameter may be a process of taking an envelope on the side with the smaller number of taps. As a result, the time change of the stabilization filter itself is reduced, and it is possible to adjust the correction effect without feeling uncomfortable.

  Further, in the present technology, for example, the correction amount acquisition unit performs a smoothing process on the accumulated image fluctuation amount at each time after the stabilization filter process is performed, and at each time after the smoothing is performed. The correction amount at each time may be acquired based on the accumulated image variation amount and the accumulated image variation amount at each time before the stabilization filter processing is performed. In this case, for example, the smoothing process to the accumulated image fluctuation amount at each time after the stabilization filter process is performed falls between the accumulated image fluctuation amounts at each time before and after the stabilization filter process is performed. In this way, the process of taking an envelope may be performed. As a result, it is possible to remove fine fluctuations in the accumulated image fluctuation amount given by fluctuations in the filter parameters, and it is possible to obtain a more appropriate correction amount as the correction amount at each time.

  According to the present technology, it is possible to satisfactorily reduce unpleasant image shake due to camera shake or the like. Note that the effect described here is not necessarily limited, and any effect described in the present disclosure may be used.

1 is a block diagram illustrating a configuration example of an image processing apparatus as a first embodiment. It is a schematic diagram which shows the correction principle of an image variation according to an electronic image stabilization method. It is a figure for demonstrating the variation | change_quantity between the images acquired by the image variation | change_quantity acquisition part. It is a figure which shows an example of the image variation | change_quantity in each time, a camera path (cumulative image variation | change_quantity), and a stabilization camera path | pass. It is a figure for demonstrating correction | amendment of the image variation by an electronic image stabilization system. It is a block diagram which shows the structural example of the image processing apparatus as 2nd Embodiment. It is a block diagram which shows the structural example of the image processing apparatus as 3rd Embodiment. It is a block diagram which shows the structural example of the image processing apparatus as 4th Embodiment. It is a block diagram which shows the structural example of the image processing apparatus as 5th Embodiment. It is a block diagram which shows the structural example of the image processing apparatus as 6th Embodiment. It is a figure which shows roughly the analysis process of a camera state analysis part. It is a figure which shows an example of the smoothing process of a modulation value. It is a flowchart which shows an example of the production | generation process procedure of the filter parameter (tap number) in predetermined time. It is a figure which shows the relationship between the correction margin set beforehand and the correction margin narrowed reflecting the modulation value. It is a figure which shows the relationship between the tap number set beforehand, the tap number reduced reflecting the modulation value and analysis reliability, and the tap number which fits in the correction margin narrowed reflecting the modulation value and analysis reliability. It is a flowchart which shows another example of the production | generation procedure of the filter parameter (tap number) in predetermined time. It is a flowchart which shows another example of the production | generation procedure of the filter parameter (tap number) in predetermined time. It is a figure which shows an example of the smoothing process of a tap number. It is a figure which shows an example of the smoothing process of the accumulated image fluctuation amount after a filter. It is a flowchart which shows roughly the process in the image processing apparatus as 6th Embodiment. It is a block diagram which shows the structural example of the computer apparatus which processes by software.

Hereinafter, modes for carrying out the invention (hereinafter referred to as “embodiments”) will be described. The description will be given in the following order.
1. 1. First embodiment 2. Second embodiment 3. Third embodiment 4. Fourth embodiment 5. Fifth embodiment 6. Sixth embodiment Modified example

<1. First Embodiment>
[Configuration of image processing apparatus]
FIG. 1 shows a configuration example of an image processing apparatus 100 according to the first embodiment. The image processing apparatus 100 corrects image fluctuations at each time (each frame) using an electronic image stabilization method. FIG. 2 is a schematic diagram illustrating the principle of correcting image fluctuations according to the electronic image stabilization method. In this electronic image stabilization method, as is well known, a cut-out position is moved at each time from an imaged / developed image according to the amount of image fluctuation, a part is cut out, and deformation such as enlargement is performed. As a result, image fluctuation correction is performed.

  The image processing apparatus 100 includes an image fluctuation amount acquisition unit 101, a correction amount acquisition unit 102, an image correction unit 103, and a camera state analysis unit 108. The image fluctuation amount acquisition unit 101 acquires the image fluctuation amount at each time (each frame) of the input image. The input image is an image obtained by capturing and developing with a camera. Here, the development means various processes (gain adjustment process, special mode process, etc.) on the captured image.

  As shown in FIG. 3, the image fluctuation amount acquisition unit 101 acquires a fluctuation amount between adjacent images, that is, a fluctuation amount between images, as the image fluctuation amount at each time. In this case, the image fluctuation amount acquisition unit 101 acquires the fluctuation amount by, for example, pattern matching. For example, the image variation amount acquisition unit 101 calculates a projection matrix for fitting a pattern between two images, and analyzes the projection matrix to acquire an image variation amount.

  The correction amount acquisition unit 102 integrates the image fluctuation amount at each time acquired by the image fluctuation amount acquisition unit 101 to obtain the accumulated image fluctuation amount at each time. Further, the correction amount acquisition unit 102 performs a stabilization filter process on the accumulated image fluctuation amount at each time. Then, the correction amount acquisition unit 102 acquires the difference at each time as the correction amount based on the accumulated image fluctuation amount at each time before and after the stabilization filter process.

  FIG. 4A shows an example of the image fluctuation amount at each time. Note that the image fluctuation amount indicates the amount of movement of the pattern between images as described above. The factors of the image fluctuation amount include the movement of the frame frame accompanying the movement of the camera, the vibration caused by the camera shake or walking given to the camera during photographing, the vibration of the optical system of the camera itself, and the like. The image fluctuation amount at each time is arranged in time series as a relative value between images.

  FIG. 4B shows an example of the cumulative image fluctuation amount at each time. This accumulated image fluctuation amount is also referred to as a “camera pass”. This cumulative image fluctuation amount is an amount indicating how much the camera has moved or shaken from the cumulative start time to the current time, and is arranged in time series.

  A solid line P in FIG. 4C indicates a stabilized camera path (stabilized cumulative image fluctuation amount) obtained by performing the stabilization filter process on the cumulative image fluctuation amount at each time. Here, stabilization refers to unpleasant vibration (for example, high-frequency vibration) of the camera path (cumulative image fluctuation amount) in order to obtain “smooth video with reduced unnatural camera shake and camera shake”. Means the process to be removed. A filter used in the stabilization filter process is referred to as a stabilization filter. A result obtained by performing the filtering process using the stabilization filter on the camera path is referred to as a stabilization camera path.

  In this specification, for the sake of simplicity of explanation, correction processing is mainly described as horizontal and vertical translation components as components of image fluctuation amounts, but the components of image fluctuation amounts are limited to translation. The same processing can be performed for different image fluctuation amounts such as roll rotation, pitch rotation, yaw rotation, shear deformation, and expansion / contraction deformation.

  For the sake of simplicity, the stabilization filter mainly includes a simple “average value filter” having “number of taps” as a filter parameter (a filter that outputs an average value of numerical values for a given number of taps). ). However, the stabilization filter is not limited to this average value filter, and other filters having appropriate characteristics and parameters may be employed.

  The image correction unit 103 uses the correction amount at each time acquired by the correction amount acquisition unit 102 to correct the image variation at each time of the input image and outputs a corrected image. In this case, the image correction unit 103 corrects the image fluctuation by moving the cutout position at each time based on the correction amount from the image at each time of the input image, cutting out a part, and performing deformation such as enlargement. (See FIG. 2).

  The camera state analysis unit 108 converts a predetermined number of state information of the camera into two types of modulation values “modulation value for the number of taps” and “modulation value for the correction margin” and outputs the result. These modulation values are expressed in the form of weights from 0.0 to 1.0. The camera status information includes setting information set by the camera autonomously, setting information set by the user, and output information of various sensors loaded on the camera. Specifically, the shutter speed, gain, frame rate setting information, gyro sensor output signal, and the like.

  The “modulation value with respect to the number of taps” has an effect of adjusting the frequency characteristic of the stabilization effect according to the state of the camera. For example, by analyzing the magnitude of the shake from the output signal of the gyro sensor, the situation of the photographer's way of shooting (running or stopping) is determined, and the modulation value is determined. For example, when the motion is at a standstill with a small amount of motion, it is desirable that the image is strongly stabilized, so the value is close to 1.0. In situations where the vehicle is running, a small value is used to suppress overcorrection for large shaking. Value.

  The “modulation value for the correction margin” has an effect of limiting the correction amount at each time. For example, when the shutter speed becomes slow, the in-plane exposure blur becomes conspicuous. This exposure blur becomes conspicuous when a large correction amount occurs. Therefore, when the shutter speed is slow, the modulation value is set to a small value, and the correction amount is limited.

  As described above, in the correction of the image fluctuation by the electronic image stabilization method, as shown by a broken line frame in FIG. 5B, a part of the image before correction (see FIG. 5A) is cut out and enlarged. Then, a deformed and corrected image (corrected image) is generated. This is to prevent reference to a region that has not moved to the image before correction. This allowable range of the cutout area is referred to as “correction margin”.

  The correction amount acquisition unit 102 sets the number of taps of the stabilization filter based on the modulation value. That is, the correction amount acquisition unit 102 obtains a first tap number by modulating a predetermined tap number (fixed tap number) with “a modulation value for the tap number”. Further, the correction amount acquisition unit 102 modulates a predetermined correction margin (fixed correction margin) with the “modulation value with respect to the correction margin”, and sets the maximum number of taps whose correction amount does not exceed the modulated correction margin to the second value. The number of taps. Then, the correction amount acquisition unit 102 sets the smaller one of the first tap number and the second tap number as the tap number of the stabilization filter.

  The operation of the image processing apparatus 100 shown in FIG. 1 will be described. An input image obtained by being imaged and developed by the camera is supplied to the image fluctuation amount acquisition unit 101. The image fluctuation amount acquisition unit 101 acquires the image fluctuation amount at each time (each frame) of the input image (see FIG. 4A). This image fluctuation amount is supplied to the correction amount acquisition unit 102.

  The correction amount acquisition unit 102 is supplied with the image variation amount at each time (each frame) of the input image acquired by the image variation amount acquisition unit 101. In the correction amount acquisition unit 102, the image variation amount at each time is integrated to obtain the accumulated image variation amount (camera path) at each time (see FIG. 4B). Then, the correction amount acquisition unit 102 performs a stabilization filter process on the accumulated image fluctuation amount at each time to obtain a stabilized accumulated image fluctuation amount (stabilized camera path) at each time (FIG. 4). (See (c)).

  In the correction amount acquisition unit 102, the difference at each time is acquired as a correction amount based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. In this case, the correction amount acquisition unit 102 is supplied with two modulation values “a modulation value for the number of taps” and “a modulation value for the correction margin” obtained from the camera state analysis unit 108 based on the camera state information. Is done. Then, the correction amount acquisition unit 102 adjusts the tap number of the stabilization filter, adjusts the correction margin based on these modulation values, and sets the tap number of the stabilization filter based on the result. . Thereby, the frequency characteristic of the stabilization effect is adjusted according to the state of the camera, and the correction amount at each time is limited.

  The correction amount at each time acquired by the correction amount acquisition unit 102 is supplied to the image correction unit 103. The image correction unit 103 corrects the image variation at each time of the input image using the correction amount at each time, and outputs a corrected image. In this case, in the image correction unit 103, the cutout position is moved at each time based on the correction amount from the image at each time of the input image, a part of the cutout position is cut out, and a deformation such as enlargement is applied to change the image fluctuation. Correction is performed.

  As described above, in the image processing apparatus 100 shown in FIG. 1, the number of stabilization filter taps (filter parameters) at each time is set according to the state of the camera that obtains the input image. Therefore, when the estimation result of the image fluctuation amount acquired by the image fluctuation amount acquisition unit 101 is low in reliability and easily erroneous, the correction amount can be weakened, and erroneous correction due to erroneous detection can be prevented.

  Further, in the image processing apparatus 100 shown in FIG. 1, a first tap number obtained by modulating a predetermined tap number (fixed tap number) with a “modulation value for the tap number” and a predetermined correction margin (fixed). The smaller one of the second tap numbers, which is the maximum number of taps not exceeding the correction margin obtained by modulating the “correction margin for the correction margin” with the “modulation value for the correction margin”, is set as the tap number of the stabilization filter. is there. Therefore, it is possible to ensure that the correction amount is within the correction margin after modulation, and it is possible to prevent the occurrence of failure. In this case, the cut-off frequency can be brought to the lowest frequency.

<2. Second Embodiment>
[Configuration of image processing apparatus]
FIG. 6 shows an example of the configuration of an image processing apparatus 100A as the second embodiment. Similar to the image processing apparatus 100 shown in FIG. 1, the image processing apparatus 100A corrects image fluctuations at each time (each frame) by using an electronic image stabilization method. 6, parts corresponding to those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted as appropriate.

  The image processing apparatus 100A includes an image fluctuation amount acquisition unit 101A, a correction amount acquisition unit 102A, and an image correction unit 103. The image fluctuation amount acquisition unit 101A acquires the image fluctuation amount at each time (each frame) of the input image and obtains the reliability (analysis reliability) of the accumulated image fluctuation amount at each acquired time.

  The reliability is a weight representing the estimation accuracy of 0.0 to 1.0 when calculating the image change amount. For example, the image fluctuation amount acquisition unit 101A calculates a projection matrix for fitting a pattern between two images in the same manner as the image fluctuation amount acquisition unit 101 in the image processing apparatus 100 shown in FIG. To obtain the image fluctuation amount. The image fluctuation amount acquisition unit 101A further converts the fitting error into a weight to obtain a reliability.

This will be specifically described. Consider calculating a projection matrix from images A and B. First, matching information (for example, using block matching) is generated from the patterns of images A and B. Since it becomes a pair of coordinates of the image,
{(Xa_n, Ya_n) ⇔ (Xb_n, Yb_n) | n = 1, ...}
It becomes. From this pair information, a projection matrix can be obtained by a widely known method such as the least square method.

  Next, in each of n = 1,..., The result of converting the coordinates (Xa_n, Ya_n) on the image A into the coordinates on the image B using a projection matrix is defined as (X′b_n, Y′b_n). At this time, a value obtained by adding the distances between (Xb_n, Yb_n) and (X′b_n, Y′b_n) for all n can be used as an error. In the conversion to the weight, for example, the above error value is compared with a preset threshold value and converted to a weight of 0.0 to 1.0.

  The correction amount acquisition unit 102A integrates the image variation amount at each time acquired by the image variation amount acquisition unit 101A to obtain the accumulated image variation amount at each time. Further, the correction amount acquisition unit 102A performs a stabilization filter process on the accumulated image fluctuation amount at each time. Then, based on the accumulated image fluctuation amount at each time before and after the stabilization filter process, a difference at each time is acquired as a correction amount.

  Further, the correction amount acquisition unit 102A sets the number of taps of the stabilization filter based on the reliability obtained by the image variation amount acquisition unit 101A. That is, the correction amount acquisition unit 102A obtains the first tap number by adjusting the predetermined tap number (fixed tap number) with reliability. Further, the correction amount acquisition unit 102A adjusts a predetermined correction margin (fixed correction margin) with reliability, and sets the maximum number of taps whose correction amount does not exceed the adjusted correction margin as the second tap number. . Then, the correction amount acquisition unit 102A sets, for example, the smaller one of the first tap number and the second tap number as the tap number of the stabilization filter.

  The image correction unit 103 uses the correction amount at each time acquired by the correction amount acquisition unit 102A to correct the image variation at each time of the input image and outputs a corrected image. In this case, the image correction unit 103 corrects the image fluctuation by moving the cutout position at each time based on the correction amount from the image at each time of the input image, cutting out a part, and performing deformation such as enlargement. (See FIG. 2).

  The operation of the image processing apparatus 100A shown in FIG. 6 will be described. An input image obtained by being imaged and developed by the camera is supplied to the image fluctuation amount acquisition unit 101A. The image fluctuation amount acquisition unit 101A acquires the image fluctuation amount at each time (each frame) of the input image (see FIG. 4A) and the reliability of the acquired cumulative image fluctuation amount at each time. (Analysis reliability) is obtained.

  The correction amount acquisition unit 102A is supplied with the image fluctuation amount at each time (each frame) of the input image acquired by the image fluctuation amount acquisition unit 101A. In the correction amount acquisition unit 102A, the image fluctuation amount at each time is integrated to obtain the accumulated image fluctuation amount (camera path) at each time (see FIG. 4B). Then, in the correction amount acquisition unit 102A, the cumulative image fluctuation amount at each time is subjected to a stabilization filter process, and a stabilized cumulative image fluctuation amount (stabilized camera path) at each time is obtained (FIG. 4). (See (c)).

  In the correction amount acquisition unit 102A, the difference at each time is acquired as the correction amount based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. In this case, the reliability described above is also supplied to the correction amount acquisition unit 102A from the image variation amount acquisition unit 101A. Then, the correction amount acquisition unit 102A adjusts the tap number of the stabilization filter, adjusts the correction margin based on the reliability, and sets the tap number of the stabilization filter based on the result. Thereby, the frequency characteristic of the stabilization effect is adjusted according to the reliability, and the correction amount at each time is limited.

  The correction amount at each time acquired by the correction amount acquisition unit 102A is supplied to the image correction unit 103. The image correction unit 103 corrects the image variation at each time of the input image using the correction amount at each time, and outputs a corrected image. In this case, in the image correction unit 103, the cutout position is moved at each time based on the correction amount from the image at each time of the input image, a part of the cutout position is cut out, and a deformation such as enlargement is applied to change the image fluctuation. Correction is performed.

  As described above, in the image processing apparatus 100A shown in FIG. 6, the number of taps (filter parameter) of the stabilization filter at each time is set according to the reliability of the obtained image fluctuation amount at each time. It is. For this reason, when the estimation result of the image variation amount acquired by the image variation amount acquisition unit 101A is low in reliability and easily erroneous, the correction amount can be weakened, and erroneous correction due to erroneous detection can be prevented.

  In addition, in the image processing apparatus 100A shown in FIG. 6, the first tap number obtained by adjusting the predetermined tap number (fixed tap number) with reliability and the predetermined correction margin (fixed correction margin) are set. The smaller one of the second tap numbers, which is the maximum number of taps not exceeding the correction margin obtained by adjusting the reliability, is set as the tap number of the stabilization filter. Therefore, it can be ensured that the correction amount is within the adjusted correction margin, and the occurrence of failure can be prevented. In this case, the cut-off frequency can be brought to the lowest frequency.

<3. Third Embodiment>
[Configuration of image processing apparatus]
FIG. 7 shows a configuration example of an image processing device 100B as the third embodiment. Similar to the image processing apparatus 100 shown in FIG. 1, the image processing apparatus 100B corrects image fluctuations at each time (each frame) using an electronic image stabilization method. 7, parts corresponding to those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted as appropriate.

  The image processing apparatus 100B includes an image fluctuation amount acquisition unit 101, a correction amount acquisition unit 102B, and an image correction unit 103. The image fluctuation amount acquisition unit 101 acquires the image fluctuation amount at each time (each frame) of the input image. The correction amount acquisition unit 102B integrates the image fluctuation amount at each time acquired by the image fluctuation amount acquisition unit 101 to obtain the accumulated image fluctuation amount at each time.

  Further, the correction amount acquisition unit 102B performs a stabilization filter process on the accumulated image fluctuation amount at each time. Then, based on the accumulated image fluctuation amount at each time before and after the stabilization filter process, a difference at each time is acquired as a correction amount. Here, the correction amount acquisition unit 102B sets the maximum number of taps that does not exceed the predetermined correction margin (fixed correction margin) as the tap number (filter parameter) of the stabilization filter.

  The image correction unit 103 uses the correction amount at each time acquired by the correction amount acquisition unit 102B to correct the image variation at each time of the input image and outputs a corrected image. In this case, the image correction unit 103 corrects the image fluctuation by moving the cutout position at each time based on the correction amount from the image at each time of the input image, cutting out a part, and performing deformation such as enlargement. (See FIG. 2).

  The operation of the image processing apparatus 100B shown in FIG. 7 will be described. An input image obtained by being imaged and developed by the camera is supplied to the image fluctuation amount acquisition unit 101. The image fluctuation amount acquisition unit 101 acquires the image fluctuation amount at each time (each frame) of the input image (see FIG. 4A).

  The correction amount acquisition unit 102B is supplied with the image variation amount at each time (each frame) of the input image acquired by the image variation amount acquisition unit 101. In the correction amount acquisition unit 102B, the image fluctuation amount at each time is integrated to obtain the accumulated image fluctuation amount (camera path) at each time (see FIG. 4B). Then, the correction amount acquisition unit 102B performs a stabilization filter process on the accumulated image fluctuation amount at each time to obtain a stabilized accumulated image fluctuation amount (stabilized camera path) at each time (FIG. 4). (See (c)).

  In the correction amount acquisition unit 102B, the difference at each time is acquired as the correction amount based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. In this case, in the correction amount acquisition unit 102B, the correction amount is set to the maximum tap number that does not exceed a predetermined correction margin (fixed correction margin) as the tap number (filter parameter) of the stabilization filter.

  The correction amount at each time acquired by the correction amount acquisition unit 102B is supplied to the image correction unit 103. The image correction unit 103 corrects the image variation at each time of the input image using the correction amount at each time, and outputs a corrected image. In this case, in the image correction unit 103, the cutout position is moved at each time based on the correction amount from the image at each time of the input image, a part of the cutout position is cut out, and a deformation such as enlargement is applied to change the image fluctuation. Correction is performed.

  As described above, in the image processing apparatus 100B shown in FIG. 7, the number of taps (filter parameter) of the stabilization filter is set to the maximum number of taps whose correction amount does not exceed a predetermined correction margin (fixed correction margin). Is set. Therefore, it is possible to ensure that the correction amount falls within a predetermined correction margin, and it is possible to prevent the occurrence of failure. In this case, the cut-off frequency can be brought to the lowest frequency.

<4. Fourth Embodiment>
[Configuration of image processing apparatus]
FIG. 8 shows an example of the configuration of an image processing apparatus 100C as the fourth embodiment. Similar to the image processing apparatus 100 shown in FIG. 1, the image processing apparatus 100C corrects image fluctuations at each time (each frame) using an electronic image stabilization method. 8, parts corresponding to those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted as appropriate.

  The image processing apparatus 100 </ b> C includes an image fluctuation amount acquisition unit 101, a correction amount acquisition unit 102 </ b> C, an image correction unit 103, a filter generation unit 104, and a filter parameter smoothing unit 105. The image fluctuation amount acquisition unit 101 acquires the image fluctuation amount at each time (each frame) of the input image.

  The filter generation unit 104 generates the number of taps (filter parameter) at each time according to the feature of the image fluctuation amount (variation component) at each time acquired by the image fluctuation amount acquisition unit 101. The filter parameter smoothing unit 105 performs a smoothing process on the number of taps at each time generated by the filter generation unit 104 to obtain a smoothed tap number (filter parameter) at each time.

  The correction amount acquisition unit 102 </ b> C obtains the cumulative image fluctuation amount at each time by integrating the image fluctuation amounts at each time acquired by the image fluctuation amount acquisition unit 101. Then, the correction amount acquisition unit 102 </ b> C uses the number of taps (filter parameter) smoothed at each time obtained by the filter parameter smoothing unit 105 for the accumulated image fluctuation amount at each time to perform the stabilization filter process. Apply. Then, based on the accumulated image fluctuation amount at each time before and after the stabilization filter process, a difference at each time is acquired as a correction amount.

  The image correction unit 103 uses the correction amount at each time acquired by the correction amount acquisition unit 102C to correct the image variation at each time of the input image and outputs a corrected image. In this case, the image correction unit 103 corrects the image fluctuation by moving the cutout position at each time based on the correction amount from the image at each time of the input image, cutting out a part, and performing deformation such as enlargement. (See FIG. 2).

  The operation of the image processing apparatus 100C shown in FIG. 8 will be described. An input image obtained by being imaged and developed by the camera is supplied to the image fluctuation amount acquisition unit 101. The image fluctuation amount acquisition unit 101 acquires the image fluctuation amount at each time (each frame) of the input image (see FIG. 4A).

  The image fluctuation amount at each time is supplied to the filter generation unit 104. In the filter generation unit 104, the number of taps (filter parameters) at each time is generated according to the feature of the image fluctuation amount (variation component) at each time. Then, the number of taps at each time is smoothed by the filter parameter smoothing unit 105, and the smoothed tap number (filter parameter) at each time is obtained. For example, the smoothing process is an envelope process that is equal to or less than the number of input taps (see FIG. 18).

  Image correction amount at each time (each frame) of the input image acquired by the image variation amount acquisition unit 101 is supplied to the correction amount acquisition unit 102C. In the correction amount acquisition unit 102C, the image fluctuation amount at each time is integrated to obtain the accumulated image fluctuation amount (camera path) at each time (see FIG. 4B). Then, the correction amount acquisition unit 102C performs a stabilization filter process using the smoothed tap number (filter parameter) of each time obtained from the filter parameter smoothing unit 104 on the accumulated image fluctuation amount at each time. Thus, a stabilized cumulative image fluctuation amount (stabilized camera path) at each time is obtained (see FIG. 4C).

  In the correction amount acquisition unit 102C, the difference at each time is acquired as the correction amount based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. The correction amount at each time acquired by the correction amount acquisition unit 102 </ b> C is supplied to the image correction unit 103. The image correction unit 103 corrects the image variation at each time of the input image using the correction amount at each time, and outputs a corrected image. In this case, in the image correction unit 103, the cutout position is moved at each time based on the correction amount from the image at each time of the input image, a part of the cutout position is cut out, and a deformation such as enlargement is applied to change the image fluctuation. Correction is performed.

  As described above, in the image processing apparatus 100C shown in FIG. 8, the number of taps (filter parameter) at each time is smoothed, and the stabilization filter is used by using the number of taps at each time after the smoothing process. Processing is performed. Therefore, the time change of the filter itself is reduced, and correction without a sense of incongruity is possible.

<5. Fifth embodiment>
[Configuration of image processing apparatus]
FIG. 9 shows an example of the configuration of an image processing apparatus 100D as the fifth embodiment. Similar to the image processing apparatus 100 shown in FIG. 1, the image processing apparatus 100D corrects image fluctuations at each time (each frame) using an electronic image stabilization method. 9, parts corresponding to those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted as appropriate.

  The image processing apparatus 100D includes an image fluctuation amount acquisition unit 101, a correction amount acquisition unit 102D, an image correction unit 103, a filter processing unit 106, and a fluctuation amount smoothing unit 107. The image fluctuation amount acquisition unit 101 acquires the image fluctuation amount at each time (each frame) of the input image.

  The filter processing unit 106 integrates the image variation amount at each time acquired by the image variation amount acquisition unit 101 to obtain a cumulative image variation amount at each time. Then, the filter processing unit 106 performs a stabilization filter process on the accumulated image fluctuation amount at each time, and outputs the accumulated image fluctuation amount at each time after the filter processing. The fluctuation amount smoothing unit 107 performs a smoothing process on the accumulated image fluctuation amount at each time after the filter processing, and obtains a smoothed cumulative image fluctuation amount at each time after the filter processing. For example, this smoothing process is a process of taking an envelope so as to satisfy a constraint condition that it falls within the accumulated image fluctuation amount at each time before and after the stabilization filter process is performed (see FIG. 19).

  Based on the smoothed accumulated image fluctuation amount at each time after the filter processing and the image fluctuation amount at each time obtained by the image fluctuation amount obtaining unit 101, the correction amount acquisition unit 102D at each time. The difference is acquired as a correction amount. The image correction unit 103 uses the correction amount at each time acquired by the correction amount acquisition unit 102D to correct the image variation at each time of the input image and outputs a corrected image. In this case, the image correction unit 103 corrects the image fluctuation by moving the cutout position at each time based on the correction amount from the image at each time of the input image, cutting out a part, and performing deformation such as enlargement. (See FIG. 2).

  The operation of the image processing apparatus 100C shown in FIG. 9 will be described. An input image obtained by being imaged and developed by the camera is supplied to the image fluctuation amount acquisition unit 101. The image fluctuation amount acquisition unit 101 acquires the image fluctuation amount at each time (each frame) of the input image (see FIG. 4A).

  The image fluctuation amount at each time is supplied to the filter processing unit 106. In the filter processing unit 106, the image fluctuation amount at each time is integrated to obtain the accumulated image fluctuation amount (camera path) at each time (see FIG. 4B). Then, the filter processing unit 106 performs stabilization filter processing on the accumulated image fluctuation amount at each time to obtain a stabilized accumulated image fluctuation amount (stabilized camera path) at each time (FIG. 4 ( c)).

  The accumulated image fluctuation amount at each time after the filtering is supplied to the fluctuation amount smoothing unit 107. The fluctuation amount smoothing unit 107 performs smoothing processing on the accumulated image fluctuation amount at each time after the filter processing, and obtains a smoothed cumulative image fluctuation amount at each time after the filter processing. The smoothed cumulative image fluctuation amount at each time after the filter processing is supplied to the correction amount acquisition unit 102D. In the correction amount acquisition unit 102D, based on the smoothed accumulated image fluctuation amount at each time after the filter processing and the image fluctuation amount at each time acquired by the image fluctuation amount acquisition unit 101, the correction amount acquisition unit 102D at each time. The difference is acquired as a correction amount.

  The correction amount at each time acquired by the correction amount acquisition unit 102D is supplied to the image correction unit 103. The image correction unit 103 corrects the image variation at each time of the input image using the correction amount at each time, and outputs a corrected image. In this case, in the image correction unit 103, the cutout position is moved at each time based on the correction amount from the image at each time of the input image, a part of the cutout position is cut out, and a deformation such as enlargement is applied to change the image fluctuation. Correction is performed.

  As described above, in the image processing apparatus 100D shown in FIG. 9, the accumulated image fluctuation amount at each time subjected to the stabilization filter process is subjected to the smoothing process and used for calculating the correction amount at each time. Is. Therefore, it is possible to remove fine fluctuations in the accumulated image fluctuation amount given by fluctuations in the number of taps (filter parameters) of the stabilization filter, and it is possible to obtain a more appropriate correction amount as the correction amount at each time.

<6. Sixth Embodiment>
[Configuration of image processing apparatus]
FIG. 10 shows a configuration example of an image processing apparatus 200 as the sixth embodiment. Similar to the image processing apparatus 100 shown in FIG. 1, the image processing apparatus 200 corrects image fluctuations at each time (each frame) by an electronic image stabilization method. Note that “{***}” represents a set of values, and is a notation for handling a plurality of values collectively. Further, “*** _ i, *** _ n” indicates the value of the time of interest.

  The image processing apparatus 200 includes an imaging / developing unit 201, an image storage unit 202, an image change amount calculation unit 203, a fluctuation analysis unit 204, a cumulative fluctuation amount storage unit 205, and an analysis reliability storage unit 206. doing. The image processing apparatus 200 includes a camera state acquisition unit 207, a camera state analysis unit 208, a modulation value storage unit 209, a modulation value smoothing unit 210, and a filter generation unit 211. Further, the image processing apparatus 200 includes a filter parameter storage unit 212, a parameter smoothing unit 213, a filter calculation unit 214, a post-filtered cumulative variation storage unit 215, a post-filtered cumulative variation smoothing unit 216, a difference A calculation unit 217 and a correction unit 218 are included.

  The imaging / developing unit 201 images a subject and outputs an image. Here, the development means various processes (gain adjustment process, special mode process, etc.) on the captured image. The image storage unit 202 outputs an image (input image) output from the imaging / development unit 201 so that the image change amount calculation unit 203 and the correction unit 218 can refer to an image at a necessary time (frame) at a necessary timing. ).

  The image change amount calculation unit 203 inputs an image from the image storage unit 202, and similarly to the image variation amount acquisition unit 101A in the image processing apparatus 100A in FIG. 6, the projection matrix H_i for fitting a pattern between two images. And the fitting error is converted into a weight to obtain the analysis reliability c_i. The analysis reliability may be a value obtained by converting the area of the moving subject area in the screen into a weight.

  The fluctuation analysis unit 204 receives the projection matrix H_i calculated by the image change amount calculation unit 203, calculates the image fluctuation amount at each time, and continues to accumulate from a specific time, thereby accumulating the image fluctuation amount at each time. tx_acc_i and ty_acc_i are output. For example, the fluctuation analysis unit 204 analyzes the components of the projection matrix, calculates the translation amount, continues to accumulate inside, and outputs the accumulated image fluctuation amount at each time.

  The accumulated variation amount storage unit 205 holds the accumulated image variation amounts tx_acc_i and Ty_acc_i at each time output from the variation analysis unit 204 so that the accumulated image variation amount at a necessary time can be referred to at a timing required by another block. Memory. The analysis reliability storage unit 206 is a storage destination for buffering the analysis reliability c_i at each time output from the image fluctuation amount calculation unit 203.

  The camera state acquisition unit 207 acquires various numerical values indicating the camera state from the camera at the time of imaging, and outputs a numerical value group as a camera state value {p_k} _i. The state information of the camera includes a state set by the camera autonomously, a state set by the user, and output states of various sensors loaded on the camera. Specifically, the shutter speed, gain, frame rate setting state, gyro sensor output state, and the like.

  The camera state analysis unit 208 inputs the camera state value {p_k} _i acquired by the camera state acquisition unit 207, and similarly to the camera state analysis unit 108 in the image processing apparatus 100 of FIG. Two types of modulation values {w_k} _i are output: “modulation value for parameter)” and “modulation value for correction margin”. These modulation values are expressed in the form of weights from 0.0 to 1.0. Since the camera state value {p_k} _i is composed of a plurality of types of numerical values, the camera state analysis unit 208 analyzes each and finally converts them into two types of modulation values {w_k} _i.

  FIG. 11 schematically shows an analysis process of the camera state analysis unit 208. The camera state analysis unit 208 converts the camera state value into a weight compared with a preset threshold value, and finally integrates and outputs the result by multiplication. In the illustrated example, camera state values a, b,..., X are input.

  In the process of obtaining “modulation values for the number of taps (filter parameters)”, the camera state values a, b,..., X are converted into modulation weights wa, wb,. A modulation value Wtot is obtained by multiplication by a multiplier. On the other hand, in the process of obtaining the “modulation value for the correction margin”, the camera state values a, b,..., X are converted into modulation weights wa ′, wb ′,. Is multiplied by a multiplier to obtain a modulation value Wtot ′.

  Returning to FIG. 10, the modulation value storage unit 209 is a storage destination for buffering the modulation value {w_k} _i at each time (frame) obtained by the camera state analysis unit 208. The modulation value smoothing unit 210 obtains and smoothes the modulation values {{w_k} _m} at the corresponding time and a plurality of surrounding times from the modulation value storage unit 209 for each time, and smoothes the time variation. A modulation value {w_k} _i is obtained.

  FIG. 12 shows an example of the modulation value smoothing process. The illustrated example shows an envelope processing that envelopes the modulation value on the smaller side in order to avoid artifacts due to erroneous correction, as an example of the smoothing processing.

  Returning to FIG. 10, the filter generation unit 211 receives the accumulated image fluctuation amount {tx_acc_n}, {ty_acc_n}, the analysis reliability {c_n}, the smoothed modulation value {w_k} _i, and the filter parameter (number of taps) { a_n} _i is generated. The flowchart in FIG. 13 illustrates an example of a filter parameter (number of taps) generation processing procedure at a predetermined time. For the sake of simplicity, the case of generating the number of taps of the average value filter will be described.

  In step ST1, the filter generation unit 211 starts processing. Thereafter, in step ST2, the filter generation unit 211 calculates the number of taps according to the characteristics of the fluctuation component of the accumulated image fluctuation amount {tx_acc_n}, {Ty_acc_n}. In addition, in step ST3, the filter generation unit 211 calculates the maximum number of taps. At this time, the filter generation unit 211 multiplies a predetermined number of taps (a preset number of taps) by both the “modulation value for the number of taps” and the analysis reliability to reflect the maximum number of taps (modulation degree and analysis reliability). And the number of taps reduced). Thereby, the number of taps is adjusted by reflecting the modulation value and the analysis reliability. Note that it is also conceivable to obtain the maximum number of taps by multiplying a predetermined number of taps by either the “modulation value for the number of taps” or the analysis reliability.

  Further, the filter generation unit 211 calculates the maximum correction margin in step ST4. At this time, the filter generation unit 211 reflects the maximum correction margin (modulation value and analysis reliability) by multiplying a predetermined correction margin (preset correction margin) by the “modulation value for the correction margin” and the analysis reliability. Narrowed correction margin). Thereby, the correction margin is adjusted by reflecting the modulation value and the analysis reliability. It is also conceivable to obtain the maximum number of taps by multiplying a predetermined correction margin by either “modulation value for correction margin” or analysis reliability. FIG. 14 shows the relationship between the preset correction margin and the correction margin narrowed to reflect the modulation value.

  Next, in step ST5, the filter generation unit 211 calculates the number of correctable taps. In this case, the filter generation unit 211 performs a filtering process on the accumulated image fluctuation amount while increasing the number of taps in order from a minimum of 1, calculates a difference before and after the filtering process, and a difference value (corresponding to a correction amount). The number of taps is output immediately before the maximum correction margin calculated in step ST4 is exceeded or when the number of taps reaches the maximum number of taps calculated in step ST3.

  In this case, when the maximum tap number calculated in step ST3 is the first tap number and the tap number immediately before exceeding the maximum correction margin calculated in step ST4 is the second tap number, in step ST5, The smaller one of the first tap number and the second tap number is selected. As a result of calculating the number of taps in step ST5, it is possible to acquire the number of taps within a range that takes modulation into consideration and falls within the correction margin.

  FIG. 15 shows the relationship between the number of taps set in advance, the number of taps reduced by reflecting the modulation value and analysis reliability, and the number of taps that fits in the correction margin narrowed by reflecting the modulation value and analysis reliability. ing. In some cases, the number of taps reduced by reflecting the modulation value and analysis reliability is smaller than the number of taps that can be accommodated in the correction margin narrowed by reflecting the modulation value and analysis reliability.

  Next, in step ST6, the filter generation unit 211 selects the smaller of the number of taps calculated in step ST2 and the number of taps calculated in step ST5. And the filter production | generation part 211 complete | finishes a process in step ST7.

The number of taps (filter parameter) generated by the filter generation unit 211 is
(1) Consider the modulation obtained from the camera status (weaken if you are not good)
(2) Consider the reliability of the estimation result of the amount of variation between images (decrease if low reliability)
(3) Guarantee that it will fit within the correction margin ("If it is less than this tap number, it will not fail")
The result reflects.

  In addition, as shown in the flowchart of FIG. 16, the structure which deleted step ST2 and step ST6 in the flowchart of FIG. 13 is also considered. In addition, a configuration in which the processes related to step ST3 and step ST4 are omitted in the flowcharts of FIGS. For example, the flowchart of FIG. 17A shows a configuration in which the process according to step ST3 is omitted from the flowchart of FIG. Moreover, the flowchart of FIG.17 (b) has shown the structure which abbreviate | omitted the process which concerns on step ST4 in the flowchart of FIG.

  Returning to FIG. 10, the filter parameter storage unit 212 is a storage destination for buffering the number of taps (filter parameters) {a_n} _i at each time (frame) generated by the filter generation unit 211. The parameter smoothing unit 213 acquires the number of taps {{a_n} _m} at the corresponding time and a plurality of times around it from the filter parameter storage unit 212 and smoothes them, and the number of taps {a_smth_n} whose time variation is smooth Get _i.

  As described above, the maximum number of taps guaranteed to be within the correction margin is obtained by the filter generation unit 211. Therefore, the parameter smoothing unit 213 performs appropriate smoothing under a constraint condition that does not exceed the number of input taps. FIG. 18 illustrates an example of the tap number smoothing process. The example shown in the figure shows an envelope process that is equal to or less than the number of input taps as an example of the smoothing process.

  Returning to FIG. 10, the filter calculation unit 214 uses the stabilization filter (average value filter) that is configured by using the smoothed tap number {a_smth_n} _i for the accumulated image fluctuation amount {tx_acc_n} and {ty_acc_n}. To obtain post-filtering cumulative image fluctuation amounts tx_acc_filter_i and ty_acc_filter_i at each time (frame). The post-filtering cumulative variation storage unit 215 is a storage destination for buffering the cumulative image variation tx_acc_filter_i and ty_acc_filter_i at each time (frame) after filtering output from the filter calculation unit 214.

  The post-filter cumulative fluctuation amount smoothing unit 216 acquires and smoothes the post-filter cumulative fluctuation amount {tx_acc_filter_n}, {ty_acc_filter_n} from the post-filter cumulative fluctuation amount storage unit 215 at the corresponding time and a plurality of times around it. The post-filter cumulative image fluctuation amount tx_acc_filter_smth_i, ty_acc_filter_smth_i at each time (frame) in which the time fluctuation is smoothed is obtained.

  FIG. 19 shows an example of the smoothing process of the post-filter accumulated image fluctuation amount. The illustrated example shows, as an example of the smoothing process, a process of taking an envelope so as to satisfy a constraint condition that it falls within the accumulated image fluctuation amount at each time before and after the stabilization filter process. This restriction condition is a restriction condition for the correction amount output from the difference calculation unit 217 to fall within the correction margin.

  Returning to FIG. 10, the difference calculation unit 217 obtains the difference between the accumulated image fluctuation amount tx_acc_i, ty_acc_i and the smoothed filtered cumulative image fluctuation amount tx_acc_filter_smth_i, ty_acc_filter_smth_i at each time (frame). ) Correction amounts tx_stb_i and ty_stb_i are obtained.

  The correction unit 218 uses the correction amount tx_stb_i, ty_stb_i at each time (frame) obtained by the difference calculation unit 217 to correct the image fluctuation at each time of the input image and outputs a corrected image. In this case, the correction unit 218 corrects the image variation by moving the cutout position at each time based on the correction amount from the image at each time of the input image, cutting out a part, and performing deformation such as enlargement ( (See FIG. 2).

  The operation of the image processing apparatus 200 shown in FIG. 10 will be described. An image (input image) is supplied from the image storage unit 202 to the image change amount calculation unit 203. The image change amount calculation unit 203 calculates a projection matrix H_i for fitting a picture between two images at each time (frame), and converts the fitting error into a weight to obtain an analysis reliability c_i. can get.

  The projection matrix H_i at each time obtained by the image change amount calculation unit 203 is supplied to the fluctuation analysis unit 204. In the fluctuation analysis unit 204, the image fluctuation amount is calculated from the projection matrix H_i at each time, and accumulation is started from a specific time to obtain the accumulated image fluctuation amounts tx_acc_i and ty_acc_i at each time (FIG. 4A). , (B)). The accumulated image variation amounts tx_acc_i and ty_acc_i at each time are stored in the accumulated variation amount storage unit 205 so that the accumulated image variation amount at a required time can be referred to at a timing required by another block. Also, the analysis reliability c_i at each time obtained by the image change amount calculation unit 203 is held in the analysis reliability storage unit 206.

  The camera state acquisition unit 207 acquires various numerical values indicating the camera state from the camera at the time of shooting. The camera state value {p_k} _i as a numerical group output from the camera state acquisition unit 207 is supplied to the camera state analysis unit 208. The camera state analysis unit 208 analyzes the camera state value {p_k} _i to obtain two types of modulation values {w_k} _i, which are “a modulation value for the number of taps (filter parameter)” and “a modulation value for the correction margin”. It is done. These modulation values are expressed in the form of weights from 0.0 to 1.0.

  The modulation value {w_k} _i at each time obtained by the camera state analysis unit 208 is held in the modulation value storage unit 209. In the modulation value smoothing unit 210, the modulation value {w_k} _m of the corresponding time and a plurality of times around it is acquired from the modulation value storage unit 209 and smoothed for each time, and the time variation is smoothed. The value {w_k} _i is obtained.

  The filter generation unit 211 receives the accumulated image fluctuation amount {tx_acc_n}, {ty_acc_n}, analysis reliability {c_n}, and smoothed modulation value {w_k} _i. Based on these inputs, the filter generation unit 211 generates the tap number {a_n} _i, which is a filter parameter at each time. The number of taps {a_n} _i at each time is stored in the filter parameter storage unit 212.

  The parameter smoothing unit 213 obtains the number of taps {{a_n} _m} of the corresponding time and a plurality of surrounding times from the filter parameter storage unit 212 and smoothes the taps at each time at which time variation is smoothed. The number {a_smth_n} _i is obtained. Since the maximum number of taps guaranteed to be within the correction margin is obtained by the filter generation unit 211, the parameter smoothing unit 213 performs appropriate smoothing under the constraint that the input tap number is not exceeded. Is done.

  The tap number {a_smth_n} _i at each time obtained by the parameter smoothing unit 213 is supplied to the filter calculation unit 214. In the filter calculation unit 214, the stabilization filter processing is performed on the accumulated image fluctuation amount {tx_acc_n}, {ty_acc_n} by a stabilization filter (average value filter) configured using the smoothing tap number {a_smth_n} _i. The From the filter calculation unit 214, the filtered cumulative image fluctuation amount {tx_acc_filter}, {ty_acc_filter} at each time is obtained (see FIG. 4C) and held in the post-filtering cumulative fluctuation amount storage unit 215.

  The post-filter cumulative fluctuation amount smoothing unit 216 acquires the smoothed post-filter image fluctuation amounts {tx_acc_filter_n} and {ty_acc_filter_n} from the post-filter cumulative fluctuation amount storage unit 215 at the corresponding time and a plurality of times around it. Thus, the post-filter cumulative image fluctuation amount tx_acc_filter_smth_i and ty_acc_filter_smth_i at which the time fluctuation is smoothed are obtained.

  The difference calculation unit 217 obtains a correction amount tx_stb_i, ty_stb_i at each time by obtaining a difference between the accumulated image variation amount tx_acc_i, ty_acc_i and the smoothed filtered cumulative image variation amount tx_acc_filter_smth_i, ty_acc_filter_smth_i at each time. The correction amounts tx_stb_i and ty_stb_i are supplied to the correction unit 218.

  The correction unit 218 uses the correction amounts tx_stb_i and ty_stb_i at each time to correct the image variation at each time of the input image and obtain a corrected image. In this case, in the correction unit 218, the cutout position is moved at each time based on the correction amount from the image at each time of the input image, a part is cut out, and deformation such as enlargement is applied to correct the image fluctuation. Performed (see FIG. 2).

  The flowchart in FIG. 20 schematically shows processing in the image processing apparatus 200 shown in FIG. In step ST11, the image processing apparatus 200 starts processing. Thereafter, the image processing apparatus 200 inputs an image to the image change amount calculation unit 203 in step ST12. Next, in step ST13, the image processing apparatus 200 uses the image change amount calculation unit 203 to calculate a projection matrix H_i at each time. Next, in step ST14, the image processing apparatus 200 analyzes the projection matrix H_i at each time with the fluctuation analysis unit 204 to obtain an image fluctuation amount at each time, and starts accumulation from a specific time. Accumulated image fluctuation amounts tx_acc_i and ty_acc_i at each time are obtained.

  The image processing apparatus 200 performs the processing from step ST15 to step ST17 in parallel with the processing from step ST12 to step ST14. That is, in step ST15, the image processing apparatus 200 acquires various numerical values indicating the camera state from the camera at the time of shooting by the camera state acquisition unit 207, and acquires a camera state value {p_k} _i as a numerical group. To do.

  In step ST <b> 16, the image processing apparatus 200 analyzes the camera state value {p_k} _i using the camera state analysis unit 208, and calculates “modulation value for the number of taps (filter parameter)” and “modulation value for the correction margin”. A kind of modulation value {w_k} _i is obtained. In step ST <b> 17, the image processing apparatus 200 smoothes the modulation value {w_k} _i using the modulation value smoothing unit 210. The image processing apparatus 200 moves to the process of step ST18 after the process of step ST17 or after the process of step ST14.

  In step ST18, the image processing apparatus 200 uses the filter generation unit 211 to input the accumulated image fluctuation amount {tx_acc_n}, {ty_acc_n}, the analysis reliability {c_n}, and the smoothed modulation value {w_k} _i. The number of taps {a_n} _i, which is a filter parameter at each time, is generated (see FIG. 13). In step ST19, the image processing apparatus 200 uses the parameter smoothing unit 213 to smooth the tap number {a_n} _i at each time.

  Next, in step ST20, the image processing apparatus 200 uses the stabilization filter (average value filter) configured by using the smoothing tap number {a_smth_n} _i at each time in the filter calculation unit 214. Apply stabilization filter processing to {tx_acc_n} and {ty_acc_n}. In step ST <b> 21, the image processing apparatus 200 performs a smoothing process on the filtered cumulative image fluctuation amounts tx_acc_n_i and ty_acc_n_i at each time by the filtered cumulative fluctuation amount smoothing unit 216.

  Next, in step ST <b> 22, the image processing apparatus 200 performs a difference calculation at the difference calculation unit 217. In this case, the difference calculation unit 217 obtains the difference between the accumulated image fluctuation amount tx_acc_i, ty_acc_i and the smoothed filtered cumulative image fluctuation amount tx_acc_filter_smth_i, ty_acc_filter_smth_i at each time, and the correction amount tx_stb_i, ty_stb_i at each time is obtained. can get.

  Next, in step ST23, the image processing device 200 uses the correction amount tx_stb_i, ty_stb_i at each time to correct the image variation at each time of the input image, thereby obtaining a corrected image. Then, the image processing apparatus 200 outputs a corrected image in step ST24. After step ST24, the image processing apparatus 200 ends the process at step ST25.

  As described above, in the image processing apparatus 200 shown in FIG. 10, the number of stabilization filter taps (filter parameters) at each time is set according to the state of the camera that obtains the input image. Therefore, the correction amount is weakened when the projection matrix H_i calculated by the image change amount calculation unit 203 at each time, and accordingly, the accumulated image variation amounts tx_acc_i and ty_acc_i obtained at the time by the variation analysis unit 204 are unreliable and error-prone. It is possible to prevent erroneous correction due to erroneous detection.

  In addition, in the image processing apparatus 200 shown in FIG. 10, the number of taps (filter parameter) of the stabilization filter at each time is set according to the reliability of the projection matrix at each time obtained by the image change amount calculation unit 203. Is. Therefore, the correction amount is weakened when the projection matrix H_i calculated by the image change amount calculation unit 203 at each time, and accordingly, the accumulated image variation amounts tx_acc_i and ty_acc_i obtained at the time by the variation analysis unit 204 are unreliable and error-prone. It is possible to prevent erroneous correction due to erroneous detection.

  Further, in the image processing apparatus 200 shown in FIG. 10, the first tap number obtained by modulation with the “modulation value with respect to the tap number” and the correction margin obtained by modulation with the “modulation value with respect to the correction margin” are exceeded. The smaller one of the second tap numbers, which is the maximum number of taps, is set as the tap number of the stabilization filter. Therefore, it is possible to ensure that the correction amount is within the correction margin after modulation, and it is possible to prevent the occurrence of failure. In this case, the cut-off frequency can be brought to the lowest frequency.

  In addition, in the image processing apparatus 200 shown in FIG. 10, the number of taps (filter parameters) at each time is smoothed, and the stabilization filter processing is performed using the number of taps at each time after the smoothing process. Is. Therefore, the time change of the filter itself is reduced, and correction without a sense of incongruity is possible.

  Further, in the image processing apparatus 200 shown in FIG. 10, the smoothing process is performed on the accumulated image fluctuation amount at each time when the stabilization filter process is performed, and the correction amount calculation at each time is used. Therefore, it is possible to remove fine fluctuations in the accumulated image fluctuation amount given by fluctuations in the number of taps (filter parameters) of the stabilization filter, and it is possible to obtain a more appropriate correction amount as the correction amount at each time.

<7. Modification>
In the above-described embodiments, the camera information for obtaining the input image and the reliability of the image fluctuation amount at each time are listed as the shooting information of the camera. However, the present invention is not necessarily limited to these, and other information similar to these may be used alone or in combination.

  In the above-described embodiment, the example in which the modulation value obtained from the camera state and the reliability of the image fluctuation amount estimation are reflected in the filter parameter of the stabilization filter has been shown. That is, the stabilization effect adjustment process is reflected in the filter parameter. However, it is also conceivable to reflect this modulation value and reliability in the correction amount. In this case, when you want to weaken the stabilization effect, you can adjust the stabilization effect by reflecting the modulation amount and reliability in the correction amount so that the cutout part does not deviate greatly from the center. It is. When the estimation result of the fluctuation amount is low in reliability and easy to error, the correction can be weakened, and erroneous correction due to erroneous detection can be prevented.

  The image processing apparatus in each of the above embodiments can be realized by hardware or software. FIG. 21 shows a configuration example of a computer apparatus 300 that performs processing by software. The computer apparatus 300 includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a data input / output unit (data I / O) 304, and an image memory 305. .

  The ROM 302 stores a processing program for the CPU 301 and the like. The RAM 303 functions as a work area for the CPU 301. The CPU 301 reads out the processing program stored in the ROM 302 as necessary, transfers the read processing program to the RAM 303 and develops it, reads the developed processing program, and executes image processing.

  In this computer apparatus 300, an input image (image data) is input via the data I / O 304 and stored in the image memory 305. The input image stored in the image memory 305 is subjected to image processing for correcting image fluctuations by the CPU 301. The processed corrected image (image data) is output to the outside from the image memory 305 via the data I / O 304.

Moreover, this technique can also take the following structures.
(1) An image fluctuation amount acquisition unit that acquires an image fluctuation amount at each time of an input image;
Stabilization filter processing is performed on the accumulated image fluctuation amount at each time obtained by integrating the acquired image fluctuation amounts at each time, and each time based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
The correction amount acquisition unit
An image processing apparatus that sets a filter parameter of the stabilization filter at each time according to shooting information of a camera that obtains the input image.
(2) The correction amount acquisition unit
The image according to (1), wherein the filter parameter of the stabilization filter at each time is set according to the state information of the camera that obtains the input image and / or the reliability of the obtained image fluctuation amount at each time. Processing equipment.
(3) The image processing apparatus according to (1), further including: an image correction unit that corrects image fluctuation at each time of the input image using the acquired correction amount at each time and outputs a corrected image.
(4) The shooting information of the camera is the state information of the camera that obtains the input image or the reliability of the acquired image fluctuation amount at each time,
The correction amount acquisition unit
The modulation value is obtained from the state information of the camera that obtains the input image, and a predetermined filter parameter is adjusted by the modulation value, or the predetermined filter parameter is adjusted by the reliability, and the filter parameter of the stabilization filter is set. The image processing apparatus according to (1).
(5) The camera status information includes an output signal of the gyro sensor of the camera,
The filter parameter is the number of taps,
The correction amount acquisition unit
The image processing device according to (4), wherein the modulation value is determined to be smaller as the movement of the camera is smaller based on an output signal of the gyro sensor.
(6)
The shooting information of the camera is the status information of the camera that obtains the input image, or the reliability of the acquired image fluctuation amount at each time,
The correction amount acquisition unit
A modulation value is obtained from the state information of the camera that obtains the input image, and a predetermined correction margin is adjusted with the modulation value, or a predetermined correction margin is adjusted with the reliability, and the correction amount is adjusted with the adjusted correction margin. The image processing apparatus according to (1), wherein the filter parameter of the stabilization filter is set so that the cut-off becomes the lowest frequency within a range not exceeding.
(7) The camera state information includes shutter speed information of the camera,
The filter parameter is the number of taps,
The correction amount acquisition unit
The image processing apparatus according to (6), wherein the modulation value is determined to be smaller as the shutter speed is slower.
(8) The photographing information of the camera is state information of the camera that obtains the input image,
The filter parameter is the number of taps,
The correction amount acquisition unit
Obtaining a first modulation value for modulating the number of taps of the stabilization filter from the state information of the camera that obtains the input image, and obtaining a second modulation value for modulating the correction margin,
A predetermined number of taps is modulated with the first modulation value to obtain a first number of taps, a predetermined margin is modulated with the second modulation value, and the correction amount does not exceed the modulated correction margin. Find the maximum number of taps as the second number of taps,
The image processing apparatus according to (1), wherein a smaller one of the first tap number and the second tap number is set as the tap number of the stabilization filter.
(9) The correction amount acquisition unit
The filter parameter of the stabilization filter at each time set is smoothed, and the stabilization filter process is performed on the accumulated image fluctuation amount at each time using the filter parameter subjected to the smoothing process. To (8).
(10) The filter parameter is the number of taps,
The image processing apparatus according to (9), wherein the smoothing process to the filter parameter is a process of taking an envelope on the side with a smaller number of taps.
(11) The correction amount acquisition unit
A smoothing process is performed on the cumulative image fluctuation amount at each time after the stabilization filter processing is performed, and the cumulative image fluctuation amount at each time after the smoothing is performed and the stabilization filter processing is performed. The image processing apparatus according to any one of (1) to (10), wherein a correction amount at each time is acquired based on a cumulative image fluctuation amount at each time before the time.
(12) The smoothing process to the cumulative image fluctuation amount at each time after the stabilization filter processing is performed is between the cumulative image fluctuation amounts at each time before and after the stabilization filter processing is performed. The image processing apparatus according to (11), wherein the image processing apparatus is a process for taking an envelope.
(13)
Obtaining an image variation amount at each time of the input image;
Stabilization filter processing is performed on the accumulated image fluctuation amount at each time obtained by integrating the acquired image fluctuation amounts at each time, and each time based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. Obtaining a correction amount of time,
In the step of acquiring the correction amount,
An image processing method for setting a filter parameter of a stabilization filter at each time according to shooting information of a camera that obtains the input image.
(14)
Image fluctuation amount acquisition means for acquiring an image fluctuation amount at each time of the input image;
Stabilizing filter processing is performed on the accumulated image variation amount at each time obtained by accumulating the acquired image variation amount at each time, and each time based on the accumulated image variation amount at each time before and after the stabilization filter processing. Function as a correction amount acquisition means for acquiring a correction amount of time,
The correction amount acquisition means includes
A program for setting a filter parameter of the stabilization filter at each time according to shooting information of a camera that obtains the input image.
(15) An image fluctuation amount acquisition unit that acquires an image fluctuation amount at each time of the input image;
Stabilizing filter processing is performed on the accumulated image variation amount at each time obtained by accumulating the acquired image variation amount at each time, and each time based on the accumulated image variation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
An image correction unit that corrects image fluctuation at each time of the input image using the acquired correction amount at each time and outputs a corrected image,
The correction amount acquisition unit
An image processing apparatus that sets the filter parameter of the stabilization filter so that the cutoff frequency is the lowest in a range where the correction amount does not exceed a predetermined correction margin.
(16) An image fluctuation amount acquisition unit that acquires an image fluctuation amount at each time of the input image;
Stabilizing filter processing is performed on the accumulated image variation amount at each time obtained by accumulating the acquired image variation amount at each time, and each time based on the accumulated image variation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
An image correction unit that corrects image fluctuation at each time of the input image using the acquired correction amount at each time and outputs a corrected image,
The correction amount acquisition unit
A smoothing process is performed on the filter parameter of the stabilization filter at each time obtained based on the accumulated image fluctuation amount at each time, and the smoothing process is performed on the accumulated image fluctuation amount at each time. An image processing apparatus that performs stabilization filter processing.
(17) The filter parameter is the number of taps,
The image processing apparatus according to (16), wherein the smoothing process to the filter parameter is a process of taking an envelope on the side with a smaller number of taps.
(18) An image fluctuation amount acquisition unit that acquires an image fluctuation amount at each time of the input image;
Stabilizing filter processing is performed on the accumulated image variation amount at each time obtained by accumulating the acquired image variation amount at each time, and each time based on the accumulated image variation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
An image correction unit that corrects image fluctuation at each time of the input image using the acquired correction amount at each time and outputs a corrected image,
The correction amount acquisition unit
A smoothing process is performed on the cumulative image fluctuation amount at each time after the stabilization filter processing is performed, and the cumulative image fluctuation amount at each time after the smoothing is performed and the stabilization filter processing is performed. An image processing apparatus that acquires a correction amount at each time based on a cumulative image fluctuation amount at each time before the time.
(19) The smoothing process to the accumulated image variable at each time after the stabilization filter processing is performed is between the accumulated image fluctuation amounts at each time before and after the stabilization filter processing is performed. The image processing apparatus according to (18), wherein the image processing apparatus takes an envelope.
(20) An image fluctuation amount acquisition unit that acquires an image fluctuation amount at each time of the input image;
Stabilization filter processing is performed on the accumulated image fluctuation amount at each time obtained by integrating the acquired image fluctuation amounts at each time, and each time based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
An image correction unit that corrects image fluctuations at each time of the input image using the acquired correction amount at each time and outputs a corrected image; and
A correction amount adjustment unit that adjusts the correction amount at each time acquired by the correction amount acquisition unit according to the state information of the camera that obtains the input image and / or the reliability of the acquired image variation amount at each time. When,
An image processing apparatus comprising: an image correction unit that corrects image fluctuations at each time of the input image using the adjusted correction amount at each time and outputs a corrected image.

DESCRIPTION OF SYMBOLS 100,100A-100D ... Image processing apparatus 101, 101A ... Image fluctuation amount acquisition part 102, 102A-102D ... Correction amount acquisition part 103 ... Image correction part 104 ... Filter generation part 105. ··· Filter parameter smoothing unit 106 ··· Filter processing unit 107 ··· Fluctuation amount smoothing unit 108 ··· Camera state analysis unit 200 ··· Image processing device 201 ··· Imaging / image unit 202 ··· Image storage unit 203 ... Image change amount calculation unit 204 ... Fluctuation analysis unit 205 ... Cumulative variation amount storage unit 206 ... Analysis reliability storage unit 207 ... Camera state acquisition unit 208 ... Camera State analysis unit 209 ... Modulation value storage unit 210 ... Modulation value smoothing unit 211 ... Filter generation unit 212 ... Filter parameter storage unit 213 ... Para Meter smoothing unit 214 ... filter calculation unit 215 ... filtered accumulated fluctuation amount storage unit 216 ... filtered cumulative variation amount smoothing unit 217 ... difference calculation unit 218 ... correction unit 300 ...・ Computer device

Claims (20)

An image fluctuation amount acquisition unit for acquiring an image fluctuation amount at each time of the input image;
Stabilization filter processing is performed on the accumulated image fluctuation amount at each time obtained by integrating the acquired image fluctuation amounts at each time, and each time based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
The correction amount acquisition unit
An image processing apparatus that sets a filter parameter of the stabilization filter at each time according to shooting information of a camera that obtains the input image. The correction amount acquisition unit
The image processing according to claim 1, wherein a filter parameter of the stabilization filter at each time is set according to state information of a camera that obtains the input image and / or reliability of the obtained image fluctuation amount at each time. apparatus. The image processing apparatus according to claim 1, further comprising: an image correcting unit that corrects image fluctuation at each time of the input image using the acquired correction amount at each time and outputs a corrected image. The shooting information of the camera is the status information of the camera that obtains the input image, or the reliability of the acquired image fluctuation amount at each time,
The correction amount acquisition unit
The modulation value is obtained from the state information of the camera that obtains the input image, and a predetermined filter parameter is adjusted by the modulation value, or the predetermined filter parameter is adjusted by the reliability, and the filter parameter of the stabilization filter is set. The image processing apparatus according to claim 1. The camera state information includes the output signal of the gyro sensor of the camera,
The filter parameter is the number of taps,
The correction amount acquisition unit
The image processing apparatus according to claim 4, wherein the modulation value is determined to be smaller as the movement of the camera is smaller based on an output signal of the gyro sensor. The shooting information of the camera is the status information of the camera that obtains the input image, or the reliability of the acquired image fluctuation amount at each time,
The correction amount acquisition unit
A modulation value is obtained from the state information of the camera that obtains the input image, and a predetermined correction margin is adjusted with the modulation value, or a predetermined correction margin is adjusted with the reliability, and the correction amount is adjusted with the adjusted correction margin. The image processing apparatus according to claim 1, wherein the filter parameter of the stabilization filter is set so that the cut-off becomes the lowest frequency within a range not exceeding. The camera status information includes shutter speed information of the camera,
The filter parameter is the number of taps,
The correction amount acquisition unit
The image processing apparatus according to claim 6, wherein the modulation value is determined to be smaller as the shutter speed is slower. The shooting information of the camera is the status information of the camera that obtains the input image,
The filter parameter is the number of taps,
The correction amount acquisition unit
Obtaining a first modulation value for modulating the number of taps of the stabilization filter from the state information of the camera that obtains the input image, and obtaining a second modulation value for modulating the correction margin,
A predetermined number of taps is modulated with the first modulation value to obtain a first number of taps, a predetermined margin is modulated with the second modulation value, and the correction amount does not exceed the modulated correction margin. Find the maximum number of taps as the second number of taps,
The image processing apparatus according to claim 1, wherein a smaller one of the first tap number and the second tap number is set as the tap number of the stabilization filter. The correction amount acquisition unit
The smoothing process is performed on the filter parameter of the stabilization filter at each time set, and the stabilization filter process is performed on the accumulated image fluctuation amount at each time by the filter parameter subjected to the smoothing process. The image processing apparatus described. The filter parameter is the number of taps,
The image processing apparatus according to claim 9, wherein the smoothing process to the filter parameter is a process of taking an envelope on the side with a smaller number of taps. The correction amount acquisition unit
A smoothing process is performed on the cumulative image fluctuation amount at each time after the stabilization filter processing is performed, and the cumulative image fluctuation amount at each time after the smoothing is performed and the stabilization filter processing is performed. The image processing apparatus according to claim 1, wherein a correction amount at each time is acquired based on a cumulative image fluctuation amount at each time before the time. The smoothing process to the accumulated image fluctuation amount at each time after the stabilization filter processing is performed is enveloped so as to fall between the accumulated image fluctuation amounts at each time before and after the stabilization filter processing is performed. The image processing apparatus according to claim 11, wherein the image processing apparatus is a process of drawing a line. Obtaining an image variation amount at each time of the input image;
Stabilization filter processing is performed on the accumulated image fluctuation amount at each time obtained by integrating the acquired image fluctuation amounts at each time, and each time based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. Obtaining a correction amount of time,
In the step of acquiring the correction amount,
An image processing method for setting a filter parameter of a stabilization filter at each time according to shooting information of a camera that obtains the input image. Computer
Image fluctuation amount acquisition means for acquiring an image fluctuation amount at each time of the input image;
Stabilizing filter processing is performed on the accumulated image variation amount at each time obtained by accumulating the acquired image variation amount at each time, and each time based on the accumulated image variation amount at each time before and after the stabilization filter processing. Function as a correction amount acquisition means for acquiring a correction amount of time,
The correction amount acquisition means includes
A program for setting a filter parameter of the stabilization filter at each time according to shooting information of a camera that obtains the input image. An image fluctuation amount acquisition unit for acquiring an image fluctuation amount at each time of the input image;
Stabilizing filter processing is performed on the accumulated image variation amount at each time obtained by accumulating the acquired image variation amount at each time, and each time based on the accumulated image variation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
An image correction unit that corrects image fluctuation at each time of the input image using the acquired correction amount at each time and outputs a corrected image,
The correction amount acquisition unit
An image processing apparatus that sets the filter parameter of the stabilization filter so that the cutoff frequency is the lowest in a range where the correction amount does not exceed a predetermined correction margin. An image fluctuation amount acquisition unit for acquiring an image fluctuation amount at each time of the input image;
Stabilizing filter processing is performed on the accumulated image variation amount at each time obtained by accumulating the acquired image variation amount at each time, and each time based on the accumulated image variation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
An image correction unit that corrects image fluctuation at each time of the input image using the acquired correction amount at each time and outputs a corrected image,
The correction amount acquisition unit
A smoothing process is performed on the filter parameter of the stabilization filter at each time obtained based on the accumulated image fluctuation amount at each time, and the smoothing process is performed on the accumulated image fluctuation amount at each time. An image processing apparatus that performs stabilization filter processing. The filter parameter is the number of taps,
The image processing apparatus according to claim 16, wherein the smoothing process to the filter parameter is a process of taking an envelope on the side with a smaller number of taps. An image fluctuation amount acquisition unit for acquiring an image fluctuation amount at each time of the input image;
Stabilizing filter processing is performed on the accumulated image variation amount at each time obtained by accumulating the acquired image variation amount at each time, and each time based on the accumulated image variation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
An image correction unit that corrects image fluctuation at each time of the input image using the acquired correction amount at each time and outputs a corrected image,
The correction amount acquisition unit
A smoothing process is performed on the cumulative image fluctuation amount at each time after the stabilization filter processing is performed, and the cumulative image fluctuation amount at each time after the smoothing is performed and the stabilization filter processing is performed. An image processing apparatus that acquires a correction amount at each time based on a cumulative image fluctuation amount at each time before the time. The smoothing processing to the accumulated image variable at each time after the stabilization filter processing is performed is an envelope so that it falls between the accumulated image variation amounts at each time before and after the stabilization filter processing is performed. The image processing apparatus according to claim 18, wherein An image fluctuation amount acquisition unit for acquiring an image fluctuation amount at each time of the input image;
Stabilization filter processing is performed on the accumulated image fluctuation amount at each time obtained by integrating the acquired image fluctuation amounts at each time, and each time based on the accumulated image fluctuation amount at each time before and after the stabilization filter processing. A correction amount acquisition unit for acquiring a correction amount of time;
An image correction unit that corrects image fluctuations at each time of the input image using the acquired correction amount at each time and outputs a corrected image; and
A correction amount adjustment unit that adjusts the correction amount at each time acquired by the correction amount acquisition unit according to the state information of the camera that obtains the input image and / or the reliability of the acquired image variation amount at each time. When,
An image processing apparatus comprising: an image correction unit that corrects image fluctuations at each time of the input image using the adjusted correction amount at each time and outputs a corrected image.