JP2012169701A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress a multiple image and enable frame rate conversion processing to a dynamic image including an object moving at a non-uniform speed.SOLUTION: A shutter speed and information of an imaging frame rate and a display frame rate are acquired from photographic information of a dynamic image signal to acquire the required number of frame images for frame conversion. A motion vector of a subject in the plurality of frame images acquired is calculated, and a motion distance and a motion speed of the subject are calculated from the calculated motion vector for each frame image. Next, a first filter processing based on the shutter speed and the motion speed and a second filter processing based on the motion distance are applied to the plurality of frame images.

Description

  In particular, the present invention relates to an image processing apparatus, an image processing method, and a program suitable for use in converting a frame rate.

  Conventionally, when converting a high frame rate moving image to a low frame rate moving image, a single low frame rate moving image is averaged by using a plurality of images from the high frame rate moving image. There is a technology to convert to an image for use. Hereinafter, such processing is referred to as frame rate conversion processing. However, in an image shot at a shutter speed that is shorter than the frame rate of the moving image, the subject appears sharp, but when the moving image obtained by the frame rate conversion process is observed, it is perceived as a multiple image. appear. Therefore, in order to suppress such multiple images, blurring is generally performed.

  For example, Patent Document 1 discloses a conventional technique for adding blur to a captured moving image. In the technique described in Patent Document 1, a motion vector for each region is calculated for an input moving image processing target frame, and an evaluation value of a motion blur amount is calculated based on imaging information at the time of imaging such as a shutter speed. To do. Based on this calculated evaluation value, motion blur addition and reduction processing are performed. As a result, information that can be played back in each frame is played back discontinuously, and both the phenomenon that the smoothness of movement is lost (hereinafter referred to as jerkiness) and the phenomenon that the movement of the subject is greatly blurred (hereinafter referred to as blur) are suppressed. To be able to.

  In addition, a technique is disclosed in which deterioration due to jerkiness can be suppressed more naturally by obtaining a representative motion vector from a motion vector in a moving image and adding motion blur (see, for example, Patent Document 2).

JP 2010-4329 A JP 2010-15383 A

  However, both of the techniques described in Patent Document 1 and Patent Document 2 add motion blur considering a motion vector between frames, but do not consider a change in motion between a plurality of frames. For this reason, the following problems arise when applied to frame rate conversion processing in which a plurality of frame images are averaged to generate one frame.

  For example, in the case of a moving image in which a moving object moves at a uniform speed, the same blurred image arranged at equal intervals in a plurality of frames is synthesized, so the amount of blur is appropriately controlled according to only the motion vector. be able to. However, in the case of a moving image in which a moving object moves at a non-uniform speed, even if the direction of the motion vector from the previous frame is the same, the distance moved from a certain reference frame varies. For this reason, when a motion vector suddenly changes, isolated blurred images are synthesized, and there is a problem that a multiple image is seen when frame rate conversion processing is performed.

  In view of the above-described problems, an object of the present invention is to enable frame rate conversion processing by suppressing multiple images for a moving image including an object moving at a non-uniform speed.

  The image processing apparatus of the present invention includes an input unit that inputs a plurality of frame images from a moving image signal, an acquisition unit that acquires shutter speed and frame rate information from the shooting information of the moving image signal, and an input unit that inputs the information. Calculating a motion vector of the subject in the plurality of frame images, and calculating a motion distance of the subject for each frame image from the calculated motion vector; and a plurality of frame images input by the input unit. On the other hand, it has a filter means for applying a first filter process based on the shutter speed and frame rate acquired by the acquisition means and a second filter process based on the motion distance calculated by the calculation means. It is characterized by.

  According to the present invention, it is possible to generate a smooth moving image in which multiple images are suppressed when frame rate conversion is performed on an input moving image.

It is a block diagram which shows the schematic structural example of PC which concerns on embodiment. It is a block diagram which shows the detailed structural example of the image process part which concerns on embodiment. It is a figure which shows an example of the frame rate change at a high shutter speed. It is a figure which shows an example of the frame rate change at a low shutter speed. It is a figure which shows an example of the frame rate change which added the motion blur by the conventional method. It is a figure which shows an example of the frame rate change which added the motion blur by embodiment. It is a figure explaining the outline | summary which calculates the filter correction coefficient (alpha). It is a flowchart which shows an example of the procedure which detects a motion vector. It is a flowchart which shows an example of the procedure which determines a valid block. It is a flowchart which shows an example of the procedure which determines an effective motion vector. It is a figure explaining the movement distance for every frame image, and its average value. It is a figure explaining the outline | summary which calculates filter correction coefficient (beta). It is a flowchart which shows an example of the process sequence which applies a filter. It is a figure which shows an example of a blur addition filter.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram illustrating a schematic configuration example of a personal computer (PC) according to the present embodiment.
In FIG. 1, when a USB memory 101 storing a high frame rate moving image signal captured by a high-speed camera or the like is inserted into the PC 102, the PC 102 can take in these moving image signals. Specifically, all the frame images of the moving image signal are read into a memory (not shown) by the reading unit 104 of the PC 102. Furthermore, an imaging parameter including information on a shutter speed when capturing a moving image and a display parameter including information on a frame rate are read as imaging information. The image processing unit 105 performs a frame rate conversion process using these read moving image signals and parameters. The processed moving image signal is converted into a monitor signal by the output unit 106, and the converted monitor signal is displayed on the monitor 103 as a moving image.

FIG. 2 is a block diagram illustrating a detailed configuration example of the image processing unit 105 according to the present embodiment.
In FIG. 2, the moving image signal input by the reading unit 104 is input from the first input terminal 201 to the image processing unit 105. On the other hand, the imaging parameters are input from the second input terminal 202 to the image processing unit 105, and the display parameters are input from the third input terminal 203 to the image processing unit 105.

  The shutter speed acquisition unit 210 acquires shutter speed information from the imaging parameters input to the second input terminal 202, and the imaging frame rate acquisition unit 211 acquires imaging frame rate information from the input imaging parameters. . The display frame rate acquisition unit 212 acquires display frame rate information from the display parameters input to the third input terminal 203. The frame number calculation unit 213 inputs information on the acquired imaging frame rate and display frame rate, and calculates the number of frame images necessary for the frame rate conversion process. For example, when the imaging frame rate is 100 fps and the display frame rate is 20 fps, the number of frames required for the frame rate conversion process is at least five.

  The frame image acquisition unit 204 acquires n consecutive frame images necessary for the frame rate conversion processing calculated by the frame number calculation unit 213 from the moving image signal input to the first input terminal 201. The motion vector calculation unit 205 calculates a motion vector between frames from n frame images. The motion distance calculation unit 207 calculates a motion distance from the motion vector calculated by the motion vector calculation unit 205. On the other hand, the motion speed calculation unit 206 calculates the motion speed of the target object from the motion vector calculated by the motion vector calculation unit 205 and the imaging frame rate acquired by the imaging frame rate acquisition unit 211.

  The first filter selection unit 208 determines a filter correction coefficient α for adding motion blur based on the motion distance between adjacent frame images and the shutter speed. On the other hand, the second filter selection unit 209 determines a filter correction coefficient β for adding motion blur based on the motion distance between a plurality of frame images. Details of the filter correction coefficients α and β will be described later.

  The first filter application unit 214 uses the filter correction coefficient α determined by the first filter selection unit 208 to add motion blur to a plurality of frame images of the moving image signal. I do. Then, the second filter application unit 215 moves the frame image after the filter is applied by the first filter application unit 214 using the filter correction coefficient β determined by the second filter selection unit 209. A second filter process for adding blur is performed. The frame rate conversion processing unit 216 applies all the filters for adding motion blur, and then performs frame rate conversion processing on the moving image signal to generate a plurality of new frame images. Then, the new frame image generated by the frame rate conversion processing unit 216 is output from the output terminal 217 as an output moving image signal.

3 to 6 are conceptual diagrams for explaining the frame rate conversion processing.
3 to 6, the horizontal axis represents time, and one rectangular area represents one frame image. The horizontal axis of the rectangle is the horizontal position of one scanning line, and the vertical direction of the rectangle represents the luminance.

  In the example described below, the input display frame rate is InputRate = 100 fps, and the output display frame rate is OutputRate = 20 fps. Hereinafter, an example in which one frame image is generated from five input frame images when performing the frame rate conversion process using this condition will be described.

  FIG. 3A shows an example of an input moving image signal with a high shutter speed. The shutter speed is 1/1000 sec, and frame conversion processing is performed using frame images from the (N−2) th frame image to the (N + 2) th frame image. The impulse signal moves from left to right at a uniform speed from the (N−2) th frame image to the (N + 2) th frame image. In the example shown in FIG. Since it is an image, the impulse signal is sharp.

  FIG. 3B shows a time-series image of a moving image subjected to frame rate conversion. The result of average synthesis using the (N−2) th frame image and the (N + 2) th frame image is an image of five impulse signals. In this case, the subject appears sharp, and a moving image obtained by the frame rate conversion process is recognized as a multiple image.

  FIG. 4A shows an example of an input moving image signal with a low shutter speed. The shutter speed is 1/100 sec, and frame conversion processing is performed using the (N−2) th frame image to the (N + 2) th frame image. The impulse signal moves from left to right at a uniform speed from the (N−2) th frame image to the (N + 2) th frame image. In the example shown in FIG. 4A, a moving image shot at the open shutter speed. Since it is an image, the impulse signal appears to have motion blur.

  FIG. 4B shows a time-series image of a moving image subjected to frame rate conversion. The result of average synthesis using the (N−2) th frame image and the (N + 2) th frame image is an image of five impulse signals. In this case, the subject has motion blur, and when the moving image obtained by the frame rate conversion process is observed, it appears as a smooth motion.

  FIG. 5A shows an example of an input moving image signal with a high shutter speed. The shutter speed is 1/1000 sec, and frame conversion processing is performed using the (N−2) th frame image to the (N + 2) th frame image. The impulse signal moves from left to right at an unequal speed from the (N−2) th frame image to the (N + 2) th frame image. In the example shown in FIG. Since it is a moving image, the impulse signal is sharply reflected.

  FIG. 5B shows the result of the frame image obtained by performing the average synthesis process. In the example shown in FIG. 5B, the subject appears sharp, and when a moving image obtained by the frame rate conversion process is observed, it is recognized as a multiple image.

  FIG. 5 (c) shows a time-series image of a moving image that has been subjected to frame rate conversion by adding motion blur by a conventional method. In the conventional method, only the speed of the subject between adjacent frames is considered, so when motion blur is added, the amount of motion blur added differs due to the effect of the distance that the subject moves in space. . Due to the difference in the amount of motion blur addition, as shown in FIG. 5C, a luminance step occurs in the synthesized frame image. This level difference causes jerkiness when observed as a moving image, causing deterioration in image quality.

  FIG. 6A shows an example of an input moving image signal with a high shutter speed, which is the same as FIG. 5A. FIG. 6B shows the result of a frame image obtained by performing an average synthesis process after adding motion blur by a conventional method. Since the added amount of motion blur is non-uniform, if a frame rate conversion process is performed as it is and a moving image is observed, it is recognized as a multiple image.

  FIG. 6C shows a time-series image of a moving image subjected to frame rate conversion according to the present embodiment. In the present embodiment, the variance of motion vectors of a plurality of frames on the time axis is further taken, and the additional amount of motion blur is determined again based on the variance value. The detailed processing procedure will be described later. When the moving speed of the moving object varies along the time axis, the distance that the subject moves in the space differs, and the added amount of motion blur varies. Therefore, by re-correcting the step as shown in FIG. 6B, motion blur is added to the entire image as shown in FIG. 6C, and the motion is smooth when viewed as a moving image.

<Relationship between subject movement speed and optimum shutter speed>
FIG. 7A shows the relationship between the movement speed and the shutter speed.
As shown in FIG. 7A, there is an optimum shutter speed corresponding to the movement speed of the subject obtained from the motion vector. When shooting at this optimal shutter speed, moving objects appear sharp. Therefore, when shooting at a shutter speed faster than the optimum shutter speed, the moving object appears sharper and is more likely to deteriorate due to jerkiness.

  The horizontal axis in FIG. 7A indicates the motion speed obtained from the motion vector, and the vertical axis indicates the shutter speed at the time of shooting. As shown in FIG. 7A, there are a region that is sharply projected with an optimum shutter speed curve 701 as a boundary and a region where motion blur exists. When the optimal shutter speed curve 701 is approached from this motion blur area, the motion blur amount decreases, and the possibility that jerkiness will occur increases. Hereinafter, the possibility of the occurrence of jerkiness is referred to as jerkiness occurrence degree e.

  Here, the frame rate is f (frame / second), the motion speed is v (Pixel / frame), and the shutter speed is s (second). When shooting at a certain shutter speed, if the motion blur is 1 pixel or less as shown in the following formula (1), the image is sharply displayed.

  The optimum shutter speed curve 701 is a curve representing the relationship between the motion speed and the shutter speed when the motion blur is 1 pixel, and is calculated by the following equation (2).

  If the distance between the arbitrary point (v0, s0) in the vs coordinate system shown in FIG. 7A and the optimum shutter speed curve 701 is Distance, the jerkiness occurrence degree e at the arbitrary point (v0, s0) is Is calculated by the following equation (3).

  When the distance between the arbitrary point (v0, s0) and the optimum shutter speed curve 701 is zero, or when the arbitrary point (v0, s0) is in a sharp region, the jerkiness occurrence degree e is ∞.

  FIG. 7B is a diagram illustrating the relationship between the jerkiness degree e and the filter correction coefficient α. As shown in FIG. 7B, the first filter selection unit 208 can determine the value of the filter correction coefficient α based on the value of the jerkiness occurrence degree e, and controls the amount of motion blur addition using the filter correction coefficient α. can do. Note that the three-dimensional data of the movement speed, the shutter speed, and the filter correction coefficient α is held in advance in the image processing unit 105.

  The filter correction coefficient α used for the filter for adding motion blur is a positive decimal number from 0 to 1, and the amount of motion blur added increases as the value increases.

<Calculation of motion vector>
FIG. 8 is a flowchart illustrating an example of a processing procedure in which the motion vector calculation unit 205 calculates the relative movement amount between frames. In the present embodiment, an example will be described in which a motion vector is obtained for each block, and the motion amount of the entire screen is obtained as an affine parameter from the motion vector.
First, in step S801, a valid block is determined as preprocessing. This is a process of excluding blocks that may not find the correct motion vector. Details of this processing will be described later.

  Next, in step S802, the motion vector of the effective block is calculated. Here, a general block matching method will be described. In the block matching method, a sum of squares of differences or a sum of absolute differences between pixels in a block is used as a matching evaluation value. First, the evaluation value is obtained if the target block for which the motion vector is obtained is sequentially moved within the search range of the reference image. The position having the smallest evaluation value among all the evaluation values obtained in the search range is the position having the highest correlation with the symmetric block, and the movement amount is a motion vector.

  This method of obtaining the search range pixel by pixel is called full search. On the other hand, a method of obtaining a minimum evaluation value while thinning out the search range and then performing a detailed search for the vicinity thereof is called a step search. Step search is well known as a method for obtaining a motion vector at high speed. By the above method, in step S802, the motion vector of the effective block is calculated.

  Next, in step S803, it is determined whether or not motion vectors have been detected for all blocks. As a result of the determination, if there is still a block that has not been subjected to the detection process, the process returns to step S801. If a motion vector has been detected for all blocks, the process proceeds to the next step S804.

  In step S804, a valid motion vector is determined. This process is a process of excluding those determined to have incorrect calculation results from the obtained motion vectors. Details of this processing will be described later. In step S805, an affine parameter is detected from a valid motion vector.

  Here, details of the affine parameter detection processing will be described. If the center coordinate of the target block is (x, y) and the center coordinate of the block in the reference image has moved to (x ′, y ′) from the calculation result of the motion vector, these relationships are expressed by the following equations: It can be expressed as (4).

  Here, the 3 × 3 matrix shown in Expression (4) is an affine transformation matrix. When each element of the matrix is an affine parameter and a = 1, b = 0, d = 0, and e = 1, this conversion is parallel movement, c is the horizontal movement amount, and f is the vertical movement amount. It becomes. When a = cos θ, b = −sin θ, c = 0, d = sin θ, e = cos θ, and f = 0, this conversion is rotational movement at the rotation angle θ. The above equation (4) can be expressed as the following equation (5) in the form of a generalized matrix.

  Here, x and x ′ are 1 × 3 matrices, and A is a 3 × 3 affine matrix. When there are n effective motion vectors, the coordinate values of the target image can be represented by an n × 3 matrix as in the following equation (6).

  Similarly, the coordinate values after movement can also be represented by an n × 3 matrix as shown in Equation (6). Therefore, n motion vectors can be expressed as the following equation (7).

  Here, if the affine matrix A in Equation (7) is obtained, it becomes the amount of displacement of the entire screen. When the equation (4) is modified, the affine matrix A can be expressed as the following equation (8).

  Since the amount of motion can be expressed by affine transformation parameters in this way, in addition to shift blurring that occurs when the camera is held, it is possible to handle roll blurring in the in-plane direction and zoom blurring in the front-rear direction. Is possible.

  When trying to obtain a correlation between blocks by block matching, an image in the block needs to have some feature amount. In a block that is flat and contains almost only a DC component, a correct motion vector cannot be obtained. Conversely, if an edge is included in the horizontal direction or the vertical direction, matching becomes easy. Therefore, in step S801, such flat block is excluded.

FIG. 9 is a flowchart illustrating an example of a processing procedure for determining an effective block by the motion vector calculation unit 205 in step S801 of FIG. In FIG. 9, processing for one block will be described.
First, in step S901, the difference value between the maximum value and the minimum value is calculated for one horizontal line in the block. For example, when the block size is composed of 50 × 50 pixels, the maximum value and the minimum value are obtained from the 50 pixels in the horizontal direction in the block, and the difference value is calculated.

  Next, in step S902, it is determined whether or not the difference value for the number of horizontal lines has been calculated. If there are still horizontal lines for which the difference value has not been calculated, the process returns to step S901, and the difference values for all the horizontal lines are calculated. If so, the process proceeds to step S903. In step S903, the maximum difference value is obtained from the calculated number of difference values.

  Next, in step S904, the preset threshold value Tx is compared with the maximum difference value calculated in step S903. As a result of this comparison, if the maximum difference value is smaller than the threshold value Tx, it can be considered that the block does not have a feature amount in the horizontal direction. On the other hand, if the maximum difference value is greater than or equal to the threshold value Tx, it can be considered that the feature amount is in the horizontal direction, and thus the process proceeds to step S906 and the same, and the same verification is performed in the vertical direction.

  Next, in step S906, the difference value between the maximum value and the minimum value is calculated for one vertical line in the block. For example, when the block size is composed of 50 × 50 pixels, the maximum value and the minimum value are obtained from 50 pixels in the vertical direction in the block, and the difference value is calculated.

  Next, in step S907, it is determined whether or not the difference value for the number of vertical lines has been calculated. If there is still a vertical line for which the difference value has not been calculated, the process returns to step S906, and the difference values for all the vertical lines are calculated. If so, the process proceeds to step S908. In step S908, the maximum difference value is obtained from the calculated number of difference values.

  Next, in step S909, the preset threshold value Ty is compared with the maximum difference value calculated in step S908. As a result of this comparison, if the maximum difference value is smaller than the threshold value Ty, the block is regarded as a block having no feature amount in the vertical direction, and the process proceeds to step S905 to determine an invalid block. On the other hand, when the maximum difference value is equal to or greater than the threshold value Ty, since the block has characteristics in both the horizontal and vertical directions, it can be expected that accurate block matching is performed. Therefore, the process proceeds to step S910 to be an effective block. .

Next, a method for determining an effective motion vector will be described with reference to the flowchart of FIG. FIG. 10 is a flowchart illustrating an example of a processing procedure for determining an effective motion vector in step S804 of FIG. 8 by the motion vector calculation unit 205.
First, in step S1001, the motion vector detected in step S802 of FIG. 8 is input, and in step S1002, the occurrence frequency is calculated. Next, in step S1003, it is determined whether the occurrence frequency of all motion vectors has been calculated. If there is still a motion vector for which the occurrence frequency has not been calculated, the process returns to step S1001. On the other hand, when the occurrence frequencies of all the motion vectors are calculated, the motion vector having the maximum occurrence frequency is obtained in step S1004.

  Next, in step S1005, a motion vector is input again. In step S1006, it is determined whether this motion vector is a motion vector having the maximum occurrence frequency or a motion vector in the vicinity thereof. When the entire screen is only shifted, the motion vector of each block is almost the same as the motion vector with the highest occurrence frequency, and when there is roll motion, many motion vectors are in the vicinity of the motion vector with the highest occurrence frequency. Is considered to occur. Therefore, in the case of a motion vector included in these values, the process proceeds to step S1007 and is determined to be a valid motion vector.

  On the other hand, in the case of a motion vector deviating from these values, the process proceeds to step S1008, where it is determined as an invalid motion vector. Then, in step S1009, it is determined whether or not the processing for determining whether the motion vector is valid or invalid has been completed. If there is a motion vector that has not been determined yet, the process returns to step S1005. If determination has been completed for all the motion vectors, the process ends.

<Processing for uneven speed>
When performing the frame rate conversion processing, a plurality of frame images are averaged to generate one frame image. When a moving object moves at a non-uniform speed between a plurality of frame images, for a plurality of frame images to be used, the movement distance calculation unit 207 calculates a movement distance of the moving object based on a motion vector for each frame image. Then, the second filter selection unit 209 calculates the distance average Ave by the following equation (9).

  Here, N is the frame image number, and D [n] is the moving distance of the moving object between the nth frame image and the (n-1) th frame image. Based on the calculated plurality of motion distances D [N] and the distance average Ave, the second filter selection unit 209 calculates the standard deviation S from the motion distance of each frame image by the following equation (10). A filter correction coefficient β is calculated from the standard deviation S.

  FIG. 11 is a diagram illustrating a state in which a moving object is moving at an uneven speed between a plurality of frame images. FIG. 12 is a diagram illustrating the relationship between the standard deviation S and the filter correction coefficient β when a moving object is moving at an uneven speed between a plurality of frame images. In FIG. 12, the horizontal axis represents the standard deviation S of the movement distance, and the vertical axis represents the blur addition parameter (filter correction coefficient β). In this way, the second filter selection unit 209 can determine the filter correction coefficient β based on the standard deviation S. Then, blur addition processing of coefficients α and β is performed on all motion vectors calculated on the frame image.

<Bokeh addition filter selection>
FIG. 14 shows an example of a blur addition filter, which is a filter W with U rows and V columns. Here, the first filter applying unit 214 and the second filter applying unit 215, the image I _m after filtering is obtained by the following equation (11).

  Here, as described above, the filter correction coefficient α is determined by the shutter speed and the motion speed, and the filter correction coefficient β is determined by the motion distance and the standard deviation thereof in a plurality of frame images.

<Overall processing>
FIG. 13 is a flowchart illustrating an example of a processing procedure up to the frame rate conversion processing in the present embodiment.
First, in step S1301, the frame image acquisition unit 204 acquires continuous frame images. Next, in step S1302, the imaging frame rate acquisition unit 211 acquires imaging frame rate information from the input imaging parameters. As described above, the imaging frame rate is used when calculating the motion speed.

  In step S1303, the frame number calculation unit 213 calculates the number of consecutive frame images Num necessary for the frame rate conversion process. In step S1304, the shutter speed acquisition unit 210 acquires information on the value of the shutter speed at the time of shooting.

  In step S1305, the motion vector calculation unit 205 initializes an integer n for counting. In step S1306, the motion vector calculation unit 205 determines whether the integer n is smaller than the number Num. If the number n is smaller than the number Num, the process proceeds to the next step S1307. The process proceeds to S1312.

  Next, in step S1307, the motion vector calculation unit 205 calculates a motion vector between the nth frame image and the n + 1th frame image. In step S1308, the motion distance calculation unit 207 calculates a subject motion distance D [n] based on a motion vector between the nth frame image and the n + 1th frame image.

  Next, in step S1309, the motion speed calculation unit 206 calculates the motion speed from the shooting frame rate acquired in step S1302. Then, the first filter selection unit 208 determines the filter correction coefficient α as described with reference to FIG. 7 from the calculated movement speed and the shutter speed acquired in step S1304.

  In step S1310, the first filter application unit 214 applies a filter to the subject of the frame image using the determined filter correction coefficient α to add motion blur. In step S1311, the value of the integer n is incremented by 1, and the process returns to step S1306.

  On the other hand, in step S1312, the second filter selection unit 209 obtains the standard deviation S of the subject movement distance of each frame image from D [n] calculated by the number Num necessary for the frame rate conversion process. Next, in step S1313, the second filter selection unit 209 determines the filter correction coefficient β of each frame image from the standard deviation S calculated for each frame image. Then, the second filter application unit 215 applies a different low-pass filter to each frame image to add motion blur. When motion blur is added to all frame images, the process ends.

  As described above, according to the present embodiment, for a plurality of frame images, the filter correction coefficients α and β for adding motion blur are adjusted according to the motion speed, shutter speed, and motion distance of the imaging target between the frames. Can do. Accordingly, when performing frame rate conversion processing, a moving image with reduced jerkiness can be generated even if there is an object moving at a non-uniform speed.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

204 frame image acquisition unit 205 motion vector calculation unit 207 motion distance calculation unit 210 shutter speed acquisition unit 211 imaging frame rate acquisition unit 214 first filter application unit 215 second filter application unit

Claims (6)

Input means for inputting a plurality of frame images from a moving image signal;
Acquisition means for acquiring information on shutter speed and frame rate from the shooting information of the moving image signal;
Calculating means for calculating a motion vector of a subject in a plurality of frame images input by the input means, and calculating a motion distance of the subject for each frame image from the calculated motion vector;
A first filtering process based on a shutter speed and a frame rate acquired by the acquisition unit and a second distance based on a motion distance calculated by the calculation unit for a plurality of frame images input by the input unit. An image processing apparatus comprising: filter means for applying filter processing.   The image processing apparatus according to claim 1, wherein the filter unit further applies the second filter process based on a variance of the motion vector.   The image processing apparatus according to claim 1, wherein the filter unit applies the second filter process after applying the first filter process.   4. The method according to claim 1, further comprising: a frame number calculation unit that calculates the number of frame images input by the input unit based on the frame rate information acquired by the acquisition unit. The image processing apparatus described. An input step of inputting a plurality of frame images from the moving image signal;
An acquisition step of acquiring shutter speed and frame rate information from the shooting information of the moving image signal;
Calculating a motion vector of a subject in a plurality of frame images input in the input step, and calculating a motion distance of the subject for each frame image from the calculated motion vector;
A first filtering process based on the shutter speed and the frame rate acquired in the acquisition process, and a second filter based on the motion distance calculated in the calculation process for the plurality of frame images input in the input process. And an image processing method comprising: a filter process for applying the filter process. An input step of inputting a plurality of frame images from the moving image signal;
An acquisition step of acquiring shutter speed and frame rate information from the shooting information of the moving image signal;
Calculating a motion vector of a subject in a plurality of frame images input in the input step, and calculating a motion distance of the subject for each frame image from the calculated motion vector;
A first filtering process based on the shutter speed and the frame rate acquired in the acquisition process, and a second filter based on the motion distance calculated in the calculation process for the plurality of frame images input in the input process. A program that causes a computer to execute a filtering step to apply filtering processing.

Images