JP2013239791A - Image processing device, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide technology which can raise the frame rate of a moving image containing image distortion without collapsing the moving image.SOLUTION: An image processing device of the present invention is designed to interpolate between frames of an input image signal to raise the frame rate of the input image signal. The image processing device includes: distortion correction means which, when the input image signal is a distorted image signal, applies distortion correction processing to the input image signal to correct image distortion in it; detection means for detecting a motion vector from the distortion corrected image signal; distortion processing means for distorting the motion vector detected by the detection means according to the extent of image distortion in the input image signal; and generation means for generating from the input image signal and the motion vector distorted by the distortion processing means an interpolation frame for interpolating between the frames of the input image signal.

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program.

  When shooting with a digital camera, an image with intentional distortion may be shot using a fisheye lens or the like. Techniques relating to image processing for images having such intentional distortion are disclosed in Patent Documents 1 to 3, for example. Specifically, Patent Document 1 discloses a technique for extracting image feature amounts after correcting image distortion. Patent Document 2 discloses a technique for correcting camera shake of a fisheye image. Patent Document 3 discloses a technique for obtaining a motion vector that eliminates the influence of image distortion.

  In recent years, moving images have been shot using a digital single-lens reflex camera, and moving images using a fisheye lens have also been taken. A technique for improving the quality of moving images includes frame rate conversion (hereinafter referred to as FRC) that increases the number of frames of moving images. FRC is effective in improving the smoothness of moving images and suppressing flicker. Therefore, it is desirable to perform FRC on a moving image shot using a fisheye lens. In FRC, the number of frames is increased by generating interpolation frames between frames. Vector interpolation is often used as a method for generating an interpolation frame. In vector interpolation, a motion vector is obtained from frames before and after an interpolation frame generation position (time position). Then, one of the previous and subsequent frames is generated as an interpolated frame image at the midpoint of the start and end points of the motion vector.

  However, in a moving image having intentional distortion, the object does not travel straight along the motion vector, but moves in a distorted trajectory. For this reason, when conventional vector interpolation is used, an erroneous interpolation frame is generated (one of the previous and subsequent frames is generated as an interpolation frame image at an incorrect position), and the moving image is corrupted. . Note that the motion vector obtained by the technique disclosed in Patent Document 3 is a motion vector that eliminates the influence of image distortion, that is, a motion vector obtained from a moving image without distortion, and using such a motion vector, Even correct interpolation frames cannot be generated.

JP 2007-306199 A JP 2006-295626 A JP 2010-041419 A

  An object of the present invention is to provide a technique capable of increasing the frame rate of a moving image with image distortion without breaking the moving image.

The first aspect of the present invention is:
An image processing apparatus that interpolates between frames of an input image signal to increase a frame rate of the input image signal,
When the input image signal is a distorted image signal, distortion correction means for performing a distortion correction process for correcting image distortion on the input image signal;
Detecting means for detecting a motion vector from the image signal after the distortion correction processing;
Distortion processing means for distorting the motion vector detected by the detection means according to how the input image signal is distorted;
Generating means for generating an interpolated frame for interpolating between frames of the input image signal from the input image signal and the motion vector distorted by the distortion processing means;
An image processing apparatus comprising:

The second aspect of the present invention is:
An image processing apparatus that interpolates between frames of an input image signal to increase a frame rate of the input image signal,
When the input image signal is a distorted image signal, distortion correction means for performing a distortion correction process for correcting image distortion on the input image signal;
Detecting means for detecting a motion vector from the image signal after the distortion correction processing;
Generating means for generating an interpolated frame for interpolating between frames of the image signal after the distortion correction processing, from the image signal after the distortion correction processing and the motion vector detected by the detection means;
Distortion processing means for generating an interpolation frame for interpolating between the frames of the input image signal by distorting the interpolation frame generated by the generation means according to a method of distortion of the image of the input image signal;
An image processing apparatus comprising:

The third aspect of the present invention is:
An image processing apparatus that interpolates between frames of an input image signal to increase a frame rate of the input image signal,
First generating means for detecting a motion vector from the input image signal and generating an interpolation frame for interpolating between frames of the input image signal using the detected motion vector;
A second generation means for setting the frame of the input image signal as an interpolation frame for interpolating between frames of the input image signal;
When the input image signal is not a distorted image signal, an interpolation frame is generated by the first generation means, and when the input image signal is a distorted image signal, an interpolation frame is generated by the second generation means. Control means for controlling the processing of the first generation means and the processing of the second generation means,
An image processing apparatus comprising:

The fourth aspect of the present invention is:
An image processing method for interpolating between frames of an input image signal to increase a frame rate of the input image signal,
When the computer is a signal of a distorted image, the distortion correction step of performing a distortion correction process to correct the distortion of the image to the input image signal;
A detection step in which a computer detects a motion vector from the image signal after the distortion correction processing;
A distortion processing step in which the computer distorts the motion vector detected in the detection step according to a method of distortion of the image of the input image signal;
A generation step of generating an interpolated frame for interpolating between frames of the input image signal from the input image signal and the motion vector distorted in the distortion processing step;
An image processing method characterized by comprising:

According to a fifth aspect of the present invention,
An image processing method for interpolating between frames of an input image signal to increase a frame rate of the input image signal,
When the computer is a signal of a distorted image, the distortion correction step of performing a distortion correction process to correct the distortion of the image to the input image signal;
A detection step in which a computer detects a motion vector from the image signal after the distortion correction processing;
A computer that generates an interpolated frame for interpolating between frames of the image signal after the distortion correction processing, from the image signal after the distortion correction processing and the motion vector detected in the detection step;
A distortion processing step for generating an interpolation frame for interpolating between frames of the input image signal by distorting the interpolation frame generated in the generation step according to a method of distortion of the image of the input image signal;
An image processing method characterized by comprising:

The sixth aspect of the present invention is:
An image processing method for interpolating between frames of an input image signal to increase a frame rate of the input image signal,
A first generation step in which a computer detects a motion vector from the input image signal and generates an interpolated frame for interpolating between frames of the input image signal using the detected motion vector;
A second generation step in which the computer sets the frame of the input image signal as an interpolation frame for interpolating between frames of the input image signal;
An interpolation frame is generated in the first generation step when the input image signal is not a distorted image signal, and an interpolation frame is generated in the second generation step when the input image signal is a distorted image signal. A control step for controlling the processing of the first generation step and the processing of the second generation step, so that
An image processing method characterized by comprising:

The seventh aspect of the present invention is
A program for causing a computer to execute each step of the image processing method.

  According to the present invention, the frame rate of a moving image with image distortion can be increased without breaking the moving image.

1 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to a first embodiment. Diagram showing an example of input image signal The figure which shows an example of the image signal after frame rate conversion The figure which shows an example of the image signal after frame rate conversion The figure which shows an example of the image of 1 frame of an input image signal The figure which shows an example of the image of 1 frame of an input image signal The figure for demonstrating an example of the calculation method of the reduction rate by the reduction rate calculation part. The figure which shows an example of the block obtained by dividing | segmenting an image The figure which shows an example of the search field in block matching The figure which shows an example of the conversion from an interpolation vector to a distortion interpolation vector The figure for demonstrating an example of the method of a scaling ratio calculation process The figure for demonstrating an example of the method of a scaling ratio calculation process FIG. 9 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to a second embodiment. The figure for demonstrating the process of the interpolation frame production | generation part of Example 2. FIG. FIG. 9 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to a third embodiment. FIG. 9 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to a fourth embodiment.

<Example 1>
An image processing apparatus and an image processing method according to Embodiment 1 of the present invention will be described with reference to the drawings. The image processing apparatus according to the present embodiment performs frame rate conversion (FRC) to increase the frame rate of the input image signal by interpolating between frames of the input image signal.
FIG. 1 is a block diagram illustrating an example of a functional configuration of the image processing apparatus according to the present embodiment. The image processing apparatus according to the present embodiment includes an vignetting detection unit 101, a reduction rate calculation unit 102, an interpolation method determination unit 103, a reduction & distortion correction unit 104, a frame memory control unit 105, a motion vector detection unit 106, a distortion interpolation vector & An enlargement / reduction ratio calculation unit 107, an interpolation frame generation unit 108, and the like.
1 is realized by a CPU (Central Processing Unit; computer) (not shown) reading and executing a program stored in a recording medium such as a memory (not shown).
In this embodiment, the nth frame of the input image signal is denoted as F (n). An interpolated frame between F (n) and F (n + 1) is denoted as AF (n). Also, coordinates in a non-distorted space corresponding to an image without distortion are called normal coordinates, and coordinates in a distorted space corresponding to a distorted image photographed with a fisheye lens are called distorted coordinates.
As shown in FIG. 2, the input image signal is input to the image processing apparatus in units of frames. Specifically, the frames of the input image signal are sequentially input as F (n), F (n + 1), F (n + 2).

The frame memory control unit 105 will be described.
The frame memory control unit 105 stores an image signal (input image signal) input from the input terminal, specifically, a frame F input from the input terminal. In addition, the frame memory control unit 105 stores the interpolation frame AF generated by the interpolation frame generation unit 108. When the interpolation method m (FRC interpolation method) determined by the interpolation method determination unit 103 is “normal motion vector interpolation” or “fisheye motion vector interpolation”, the frame memory control unit 105 As described above, the interpolation frame AF is inserted between the frames of the input image signal. Then, the frame memory control unit 105 outputs the input image signal in which the interpolation frame AF is inserted from the output terminal at a frame rate that is twice the frame rate of the input image signal. Specifically, frames and interpolation frames are output in the order of F (n), AF (n), F (n + 1), AF (n + 1), F (n + 2). When the interpolation method m is “double interpolation”, the frame memory control unit 105 does not insert an interpolated frame between the frames, and converts each frame of the input image signal into the input image signal as shown in FIG. Outputs twice at a frame rate twice the frame rate. In other words, in the case of “double interpolation”, the frame of the input image signal is an interpolation frame. In this way, the number of frames is doubled and the frame rate is doubled.
The frame rate after the frame rate conversion is not limited to a frame rate that is twice the frame rate of the input image signal. When the frame rate of the input image signal is increased N times by frame rate conversion, N-1 interpolation frames are inserted between the frames of the input image signal, or each frame of the input image signal is output N times. That's fine.

The vignetting detection unit 101 will be described.
FIG. 5 is a diagram illustrating an example of an image of one frame of the input image signal. FIG. 5 shows an example in which the input image signal is a distorted image signal. Specifically, FIG. 5 is an example of an input image signal captured using a fisheye lens. A circular area in the center of FIG. 5 is an area of the captured image (imaging area). The hatched area is an area other than the area of the captured image (non-imaging area; typically a black area). The non-photographing area is generally called vignetting. The vignetting detection unit 101 detects the boundary between the imaging region and the non-imaging region. The boundary between the imaging region and the non-imaging region can be detected, for example, by searching for a position where the pixel value is not black from the top, bottom, left, and right edges of the image. The vignetting detection unit 101 outputs information kz indicating the boundary between the imaging region and the non-imaging region. In addition, when a boundary is detected, the vignetting detection unit 101 generates information representing a quadrangular area in which each side is a tangent to the photographing area as information representing the photographing area (photographing area information). For example, in the example of FIG. 5, the start point coordinates (xs, ys) and the end point coordinates (xe, ye) of the rectangular area are obtained as the shooting area information. In addition, as shown in FIG. 6, the shooting area may not be circular. In this case as well, the start point coordinates (xs, ys) and end point coordinates (xe, ye) of a rectangular area in which each side is a tangent to the imaging area are obtained.
Note that the shooting area information is not limited to these. For example, the shooting area information may be the center position, the horizontal size, and the vertical size of a rectangular area in which each side is a tangent to the shooting area.

The reduction rate calculation unit 102 will be described.
The reduction rate calculation unit 102 calculates a reduction rate k used by the reduction & distortion correction unit 104 (specifically, a candidate value for the reduction rate k used by the reduction & distortion correction unit 104).
When the input image signal is a distorted image signal, the reduction & distortion correction unit 104 performs distortion correction processing for correcting the image distortion on the input image signal. As a result, an image signal (motion vector detection frame) used when the motion vector detection unit 106 detects a motion vector is generated and stored in the frame memory control unit 105.
When the distortion is removed from the distorted image, the image size increases. However, the storage capacity and transfer speed of the frame memory control unit 105 and the time during which the motion vector detection unit 106 can execute the motion vector detection process are limited. Therefore, in this embodiment, a reduction process is performed on the image signal after the distortion correction process. Thereby, the data amount of the motion vector detection frame can be reduced, and the processing load of the motion vector detection process can be reduced.

  In addition, since there is no image motion in the vignetting portion, it is preferable to exclude the vignetting portion in the motion vector detection processing in the motion vector detection unit 106. By performing the motion vector detection process excluding the vignetting portion, it is possible to reduce erroneous detection of motion vectors. Therefore, in this embodiment, the reduction & distortion correction unit 104 extracts the image signal of the area represented by the shooting area information from the input image signal. The reduction & distortion correction unit 104 generates a motion vector detection frame by performing distortion correction processing and reduction processing on the extracted image signal. Then, the reduction rate calculation unit 102 obtains the reduction rate k so that the image size of the motion vector detection frame is a size that can be stored in the frame memory control unit 105. That is, the reduction ratio k is determined so that the image size after the distortion correction process and the reduction process are performed on the area excluding the vignetting portion is a size that can be stored in the frame memory control unit 105.

A specific method for obtaining the reduction ratio k will be described with reference to FIG. FIG. 7 shows an example in which the vertical size of frame F (interpolated frame AF) is v_len and the horizontal size is h_len. FIG. 7 shows a frame F and regions (regions of start point coordinates (xs, ys) and end point coordinates (xe, ye)) represented by the shooting region information obtained by the vignetting detection unit 101. Here, when the distortion correction processing is performed on the start point coordinates (xs, ys) and the end point coordinates (xe, ye), the start point coordinates (xs, ys) and the end point coordinates (xe, ye) are respectively coordinates (xs ′). , Ys ′) and coordinates (xe ′, ye ′). Coordinate conversion (conversion from distorted coordinates to normal coordinates) is performed using distortion aberration information. The distortion aberration information is information determined by a lens used for photographing and the like, and is information indicating how the image is distorted (degree of image distortion). The distortion aberration information is information including, for example, a conversion table from distortion coordinates to normal coordinates and a conversion table from normal coordinates to distortion coordinates. In this embodiment, the transformation from the distorted coordinate to the normal coordinate is (x ′, y ′) = g (x, y), and the transformation from the normal coordinate to the distorted coordinate is (x, y) = f (x ′, y). '). Here, (x, y) is a distorted coordinate. (X ′, y ′) is a normal coordinate corresponding to the distorted coordinate (x, y), that is, a coordinate obtained by performing distortion correction processing on the distorted coordinate (x, y). g and f are conversion coefficients.
When the data amount of one motion vector detection frame SF is (v_len / 2) × (h_len / 2), the reduction rate k is obtained by the following equation 1.

Reduction ratio k = {(v_len / 2) × (h_len / 2)}
/ {(Xe′−xs ′ + 1) × (ye′−ys ′ + 1)} (Formula 1)

The reduction ratio k is the ratio of the image size after the reduction process to the original image size, and is a value greater than 0 and less than 1.
Note that the distortion aberration information may be input from the outside, or may be stored in advance in the image processing apparatus. For example, the distortion information may be added as metadata to the input image signal. Distortion aberration information to be used may be selected according to a user operation from a plurality of distortion aberration information stored in advance in the image processing apparatus.
Note that the reduction ratio k is not limited to the value obtained by Equation 1. The reduction ratio may be a predetermined fixed value.

The interpolation method determination unit 103 will be described.
The interpolation method determination unit 103 determines the FRC interpolation method m from the lens type and the reduction ratio k. Information indicating the type of lens (lens information) may be input from the outside, or may be stored in advance in the image processing apparatus. For example, the lens information may be added as metadata to the input image signal or may be input by the user. Lens information to be used may be selected from a plurality of lens information stored in advance in the image processing apparatus in response to a user operation. The lens information represents the type of lens used when photographing the input image signal. As the interpolation method m, there are “normal motion vector interpolation”, “fisheye motion vector interpolation”, “double out interpolation”, and the like.
When the lens type is other than the fisheye lens (that is, when the input image signal is not a distorted image signal), “normal motion vector interpolation” is set as the interpolation method m. If the input image signal is not a distorted image signal, the entire area of the image represented by the input image signal is the shooting area. Therefore, the image size (data amount) of the motion vector detection frame before the reduction process is v_len × h_len. Based on Equation 1, the interpolation method determination unit 103 sets 1/4 as the reduction ratio k and outputs the result.
When the lens type is a fisheye lens, the interpolation method determination unit 103 compares the reduction ratio k with a predetermined threshold th_k. When k ≧ th_k, “fisheye motion vector interpolation” is set as the interpolation method m, and the reduction rate k obtained by the reduction rate calculation unit 102 is output to the reduction & distortion correction unit 104 as it is. In the case of k <th_k, “double interpolation” is set as the interpolation method m, and ¼ is output to the reduction & distortion correction unit 104 as the reduction ratio k. However, in the case of “double interpolation”, since the reduction rate k is not used, a value other than ¼ may be output as the reduction rate k (the reduction rate k may not be output). If the image is reduced too much, the motion vector detection accuracy is lowered. In this embodiment, an image is divided into a plurality of blocks (divided regions), and a motion vector is detected for each block. If the image is reduced too much, blocks that can detect the motion vector correctly and blocks that cannot detect the motion vector are mixed, and when the interpolation frame is generated using the motion vector, the image of the interpolation frame is distorted. There is a risk of it. The reason why “double interpolation” is set in the case of k <th_k is to suppress the disturbance of the interpolated frame image due to the decrease in the motion vector detection accuracy. As th_k, for example, the same value as the reduction ratio k set in the case of “normal motion vector interpolation” is used. In this embodiment, th_k = 1/4 is used.

The reduction & distortion correction unit 104 will be described.
The reduction & distortion correction unit 104 performs a distortion correction process on the input image signal when the input image signal is a distorted image signal. Then, the reduction & distortion correction unit 104 performs a reduction process on the image signal after the distortion correction process. Accordingly, the reduction & distortion correction unit 104 generates a motion vector detection frame SF. The reduction & distortion correction unit 104 converts the frame F (n) into the motion vector detection frame SF according to the shooting area information, distortion aberration information, lens type, and reduction rate k (reduction rate input from the interpolation method determination unit 103). Convert to (n).
Specifically, when the type of lens is a fisheye lens, the reduction and distortion correction unit 104 extracts an image signal of an area represented by the shooting area information from F (n). Then, the reduction & distortion correction unit 104 performs distortion correction processing and reduction processing on the extracted image signal. The distortion correction process is performed in accordance with (x ′, y ′) = g (x, y) of the distortion aberration information. The reduction process is performed according to the reduction rate k input from the interpolation method determination unit 103. Specifically, the image size in the vertical direction and the vertical direction of the image signal that has been subjected to the distortion correction processing is reduced by a factor of ^0.5 . Any reduction method may be used. For example, the reduction & distortion correction unit 104 performs a reduction process using a bilinear method. Then, the image signal after the distortion correction process and the reduction process is SF (n).
When the lens type is other than the fisheye lens, the reduction & distortion correction unit 104 performs only the reduction process on F (n). Thereby, SF (n) is generated.
The reduction & distortion correction unit 104 outputs the generated SF (n) to the frame memory control unit 105. The frame memory control unit 105 stores SF (n) output from the reduction & distortion correction unit 104.

The features of the present invention will be described.
When the input image signal is not a distorted image signal, an interpolation frame can be generated using the motion vector detected from F (n) as it is. Specifically, the pixel of the interpolation frame AF (n) can be generated by setting the pixel of F (n) to a pixel at a position half that of the motion vector from the position of the pixel. However, when the input image signal is a distorted image signal, the trajectory of the moving object does not become linear, but becomes a distorted trajectory. Therefore, if an interpolation frame is generated using the motion vector detected from F (n) as it is, a broken image is generated as an image of the interpolation frame. Therefore, in this embodiment, when the input image signal is a distorted image signal, a motion vector is detected from the image signal after the distortion correction processing. Then, the detected motion vector is distorted according to how the image of the input image signal is distorted (distortion processing). By using the motion vector so distorted, the pixel of the object in F (n) can be applied to a position according to the movement trajectory of the object, and as an image of the interpolation frame AF (n) An accurate image (an image that has not failed) can be generated.

The motion vector detection unit 106 will be described.
The motion vector detection unit 106 detects a motion vector from an image signal without distortion. That is, when the input image signal is a distorted image signal, a motion vector is detected from the image signal after the distortion correction processing. Specifically, the motion vector detection unit 106 detects a motion vector from the motion vector detection frames SF (n) and SF (n + 1), and obtains an interpolation vector from the detected motion vector. The interpolation vector is a vector indicating the position (interpolation position) in the interpolation frame of the region in the motion vector detection frame.
Specifically, the motion vector detection unit 106 detects a motion vector using a block matching method. In block matching, first, as shown in FIG. 8, the frame SF (n) is divided into a plurality of blocks. In this embodiment, the size of one block is 16 pixels in the horizontal direction × 16 pixels in the vertical direction. Then, for each block, an image area (corresponding area) having the highest similarity to the image in the block (image of a partial area in SF (n)) is searched from SF (n + 1). In this embodiment, an area (FIG. 9) of 112 pixels in the horizontal direction × 48 pixels in the vertical direction from the center position of the block to be processed is set as the search area. Then, a corresponding area corresponding to the processing target block is searched from the search area. In this example,
Sum of absolute differences (Sum of) with pixel value (pixel value of SF (n)) in the block to be processed
The area with the smallest Absolute Differences (SAD) is set as the corresponding area. When SAD is smaller than a predetermined threshold, a vector from the position of the block to be processed to the position of the corresponding area corresponding to the block is set as the motion vector of the block. When the SAD is equal to or greater than a predetermined threshold, it is determined that there is no image motion for the block to be processed, and a motion vector having a size of 0 (0 vector) is assigned. The above process is performed for all blocks, and a motion vector for each block is obtained. Here, the motion vector of the block blk in SF (n) is expressed as V (n, blk). A set of motion vectors for each block in SF (n) is denoted as {V (n)}.
Next, the motion vector detection unit 106 calculates an interpolation vector {E (n)} from the motion vector {V (n)}. The interpolation vector {E (n)} is a set of interpolation vectors for each block. The interpolation position is the midpoint of a straight line connecting the start point and end point of the motion vector. However, the motion vector {V (n)} is detected from the image signal after the reduction process. Therefore, in the present embodiment, the motion vector {V (n)} is expanded according to the reduction ratio k, and the size of the expanded motion vector is halved, so that the interpolation vector {E (n)} Ask for. Note that the starting point of the interpolation vector {E (n)} is assumed to be the same position as the starting point of the motion vector {V (n)}. Specifically, the interpolation vector {E (n)} is obtained by the following equation 2.

{E (n)} = {V (n)} / k ^0.5 / 2 (Formula 2)

The distortion interpolation vector & scaling factor calculation unit 107 will be described.
When the input image signal is a distorted image signal, the distortion interpolation vector & enlargement / reduction ratio calculation unit 107 performs a distortion interpolation vector calculation process and an enlargement / reduction ratio calculation process.
First, the distortion interpolation vector calculation process will be described.
The distortion interpolation vector calculation process is a process of calculating a distortion interpolation vector {Z (n)} from the interpolation vector {E (n)}. The interpolation vector {E (n)} is a vector in normal coordinates. The distortion interpolation vector {Z (n)} is a vector in distortion coordinates. The distortion interpolation vector calculation process obtains a distortion interpolation vector {Z (n)} by converting the coordinates of the start point and end point of the interpolation vector {E (n)} from normal coordinates to distortion coordinates according to the distortion aberration information (interpolation). Convert the vector {E (n)} into a distorted interpolation vector {Z (n)}).
The conversion method will be specifically described with reference to FIG. In FIG. 10, the normal coordinate A ′ is the center coordinate of the block blk ′. The normal coordinate B ′ is a position indicated by the interpolation vector E (n, blk ′) of the block blk ′ (the size of the block blk ′ is 16 × 16 pix). The distortion interpolation vector & scaling factor calculation unit 107 converts the normal coordinates A ′ and B ′ into the distortion coordinates A and B using (x, y) = f (x ′, y ′) of the distortion aberration information. Then, the distortion interpolation vector & enlargement / reduction ratio calculation unit 107 sets a block hblk (size is 16 × 16 pix) centered on the distortion coordinate A. The distortion interpolation vector & scaling factor calculation unit 107 sets a vector Z (n, hblk) from the distortion coordinate A to the distortion coordinate B as a distortion interpolation vector of the block hblk. The distortion interpolation vector & enlargement / reduction ratio calculation unit 107 performs the above process on all the interpolation vectors to obtain the distortion interpolation vector {Z (n)} of F (n).

Next, the enlargement / reduction ratio calculation process will be described.
The degree of image distortion varies depending on the position of the distortion coordinates. Therefore, if the image of the block hblk is applied to the distorted coordinates B as it is, the image of the interpolation frame will be broken. Therefore, in this embodiment, when the image of the block at the start point position of the distortion interpolation vector is applied to the end point position of the distortion interpolation vector, the enlargement process or the reduction process is performed on the image of the block. In the enlargement / reduction ratio calculation process, the magnification (enlargement ratio / reduction ratio) of the enlargement process or reduction process is obtained. Here, the enlargement ratio and the reduction ratio are collectively referred to as “enlargement / reduction ratio”.
First, as shown in FIG. 11, the distortion interpolation vector & scaling factor calculation unit 107 performs normal coordinates A ′.
And a block blk′2 centered on the normal coordinate B ′. Both the size of the block blk ′ and the size of the block blk′2 are 16 × 16 pix. Then, the distortion interpolation vector & scaling factor calculation unit 107 converts the block blk ′ and the block blk′2 into a block of distortion coordinates using distortion aberration information (x, y) = f (x ′, y ′). . In the example of FIG. 11, the block blk ′ is converted to the block blk, and the block blk′2 is converted to the block blk2.
Next, the distortion interpolation vector & scaling factor calculation unit 107 obtains the sizes of the block blk and the block blk2. Specifically, as shown in FIG. 12, the horizontal length of the block blk at the vertical position of the distorted coordinate A (the center of the block blk) is obtained as the horizontal size blk_h_len of the block blk. The vertical length of the block blk at the horizontal position of the distorted coordinates A is obtained as the vertical size blk_v_len of the block blk. Similarly, the horizontal size blk2_h_len and the vertical size blk2_v_len of the block blk2 are also obtained.
Then, the distortion interpolation vector & scaling factor calculation unit 107 calculates the horizontal scaling factor gh and the vertical scaling factor gv of the block hblk using the following equations 3-1 and 3-2.

gh = blk2_h_len / blk_h_len (formula 3-1)
gv = blk2_v_len / blk_v_len (Equation 3-2)

Here, gh and gv of the block hblk are combined and expressed as a scaling factor L (n, blkh). Then, the distortion interpolation vector & enlargement / reduction ratio calculation unit 107 performs the above process on all distortion interpolation vectors (blocks set as the start points of the distortion interpolation vectors in the distortion interpolation vector calculation process), and the enlargement / reduction ratio of F (n) { L (n)} is obtained.
In practice, when a square block on normal coordinates is converted to a block on distorted coordinates, the converted block does not become a square block. In the present embodiment, the shape of the converted block is approximated to a quadrangle in order to simplify the calculation.

  In this embodiment, the interpolation vector {E (n)} is converted into the distortion interpolation vector {Z (n)}. However, the present invention is not limited to this configuration. For example, the trajectory of the motion vector {V (n)} may be distorted according to the image distortion method of the input image signal, and the distorted interpolation vector {Z (n)} may be obtained from the distorted trajectory.

The interpolation frame generation unit 108 will be described.
The interpolation frame generation unit 108 generates an interpolation frame for interpolating between frames of the input image signal. In this embodiment, when the input image signal is a distorted image signal, an interpolation frame for interpolating between the frames of the input image signal is generated from the input image signal and the distorted motion vector.
Specifically, the interpolation frame generation unit 108 includes frames F (n) and F (n + 1), an interpolation method m, boundary information kz, an interpolation vector {E (n)}, a distortion interpolation vector {Z (n)}, An interpolation frame AF (n) is generated using the scaling factor {L (n)}.
When the interpolation method m is “fisheye motion vector interpolation”, the image on F (n) of each block corresponding to the distortion interpolation vector {Z (n)} is enlarged / reduced according to the enlargement / reduction ratio {L (n)}. Reduced. Then, the image after the enlargement / reduction processing is pasted at the position indicated by the distortion interpolation vector {Z (n)} as the image of the interpolation frame AF (n). The enlargement / reduction process is performed using, for example, a bilinear method. That is, in this embodiment, the image of a part of the region in the interpolation frame is determined by changing the position of the image of the part of the region in the frame of the input image signal according to the distorted motion vector. . Then, when changing the position of the image, the size of the image whose position is changed is changed according to how the image of the input image signal is distorted. When the images after enlargement / reduction processing are pasted, the average value of the pixel values of the overlapping images is applied to the position where the plurality of images overlap. The average value of the pixel values of F (n) and F (n + 1) at the same position as the position is applied to the position where the image is not pasted. Further, the interpolation frame generation unit 108 determines the vignetting portion (non-imaging area) from the boundary information kz. Then, the interpolation frame generation unit 108 applies the pixel value of F (n) without performing vector interpolation (image pasting according to the distortion interpolation vector {Z (n)}) for the vignetting portion. That is, in this embodiment, the pixel value of the interpolation frame is determined using the distorted motion vector for the shooting region, and the pixel position of the interpolation frame is input at the same position for the non-shooting region. The pixel value of the image signal is applied. As a result, it is possible to prevent the non-photographing area (vignetting portion) and the boundary between the non-photographing area and the photographing area from being disturbed by erroneous interpolation. As described above, an interpolation frame is generated.
When the interpolation method m is “normal motion vector interpolation”, an interpolation frame is generated according to the interpolation vector {E (n)}. Specifically, the image on F (n) of each block corresponding to the interpolation vector {E (n)} is positioned at the position indicated by the interpolation vector {E (n)} as the image of the interpolation frame AF (n). It is pasted. A position where a plurality of images overlap and a position where images are not pasted are processed in the same manner as in the case of “fisheye motion vector interpolation”.
When the interpolation method m is “double interpolation”, the interpolation frame AF (n) is not output from the output terminal, and therefore the interpolation frame AF (n) is not generated.

As described above, according to the present embodiment, when the input image signal is a distorted image signal, a motion vector is detected from the image signal after the distortion correction processing, and according to how the image of the input image signal is distorted. , The detected motion vector is distorted. Then, an interpolation frame for interpolating between frames of the input image signal is generated from the input image signal and the distorted motion vector. As a result, when the input image signal is a distorted image signal, the frame of the input image signal can be pasted at an accurate interpolation position to generate an interpolation frame. Thereby, the frame rate of a moving image with intentional image distortion can be increased without breaking the moving image.
Further, according to the present embodiment, when the position of the image of the partial area in the frame of the input image signal is changed to determine the image of the partial area in the interpolation frame, the image of the input image signal The size of the image whose position is changed is changed according to the distortion method. Thereby, the disturbance of the image of the interpolation frame can be further suppressed. Note that such a size change need not be performed. Even in such a configuration, by using the distorted motion vector, it is possible to suppress the disturbance of the image of the interpolation frame as compared with the conventional case.
Also, according to the present embodiment, since a motion vector is detected from the image signal after the distortion correction processing that has been subjected to the reduction processing, the data amount of the motion vector detection frame can be reduced, and the motion vector The processing load of the detection process can be reduced. In addition, the range of the degree of distortion that can be handled can be expanded. If the data storage capacity is sufficiently large and the data processing capability is sufficiently high, the reduction process may not be performed.
Further, according to the present embodiment, when the reduction rate of the reduction process is less than the predetermined threshold, the frame of the input image signal is an interpolation frame that interpolates between the frames of the input image signal. Thereby, it is possible to suppress the failure of the image caused by the erroneous detection of the motion vector. When the input image signal is a distorted image signal, “fisheye motion vector interpolation” may be set regardless of the reduction rate of the reduction process.

In this embodiment, the motion vector is detected in units of blocks. However, the motion vector may be detected in units of pixels.
In the present embodiment, the configuration is such that the motion vector is detected from the image signal in the imaging region, but the present invention is not limited to this configuration. For example, the distortion correction process may be performed on the entire image area of the input image signal, and the motion vector may be detected from the image signal after the distortion correction process corresponding to the entire image area.
In this embodiment, the pixel value of the interpolation frame is determined using the distorted motion vector for the shooting region, and the same position is input as the pixel value of the interpolation frame for the non-shooting region. Although the pixel value of the image signal is applied, the present invention is not limited to this configuration. For the entire image region, the interpolated frame pixel values may be determined using the distorted motion vectors.

<Example 2>
An image processing apparatus and an image processing method according to Embodiment 2 of the present invention will be described with reference to the drawings. In the first embodiment, the interpolation frame is generated starting from the coordinates in the frame of the input image signal. In the second embodiment, the interpolation frame is generated starting from the coordinates in the interpolation frame to be generated.
FIG. 13 is a block diagram illustrating an example of a functional configuration of the image processing apparatus according to the present embodiment. The image processing apparatus according to the present embodiment includes an vignetting detection unit 101, a reduction rate calculation unit 102, an interpolation method determination unit 103, a reduction & distortion correction unit 104, a frame memory control unit 105, a motion vector detection unit 106, and a distortion interpolation vector calculation. A unit 201, an interpolation frame generation unit 202, and the like. The same functional parts as those in the first embodiment are denoted by the same reference numerals and description thereof is omitted.

The distortion interpolation vector calculation unit 201 will be described.
The distortion interpolation vector calculation unit 201 performs processing similar to the distortion interpolation vector calculation processing of the distortion interpolation vector & scaling factor calculation unit 107 of the first embodiment. The distortion interpolation vector calculation unit 201 does not perform the enlargement / reduction rate calculation processing of the distortion interpolation vector & enlargement / reduction rate calculation unit 107 of the first embodiment.

The interpolation frame generation unit 202 will be described.
The interpolation frame generation unit 202 interpolates from the frames F (n) and F (n + 1), the interpolation method m, the boundary information kz, the interpolation vector {E (n)}, the distortion interpolation vector {Z (n)}, and distortion aberration information. A frame AF (n) is generated.
A case where the interpolation method m is “fisheye motion vector interpolation” will be described.
In the first embodiment, the pixel position in the interpolation frame AF (n) corresponding to the pixel position in the frame F (n) is determined from the distortion interpolation vector, and a partial region (block) in the frame F (n) is determined. An interpolated frame was generated by changing the position of the image. In the second embodiment, the pixel position in the frame F (n) corresponding to the pixel position in the interpolation frame AF (n) to be generated is determined from the distortion interpolation vector, and the pixels in the frame F (n) are determined according to the determination result. An interpolation frame is generated using the value.
This will be specifically described with reference to FIG.
From the distortion interpolation vector Z (n, blk1), the correspondence between the distortion coordinate A on F (n) and the distortion coordinate B on AF (n) is established. In order to determine an image in the block blk2 on AF (n), the interpolation frame generation unit 202 obtains a pixel value at the pixel position for each pixel position in the block blk2. The block blk2 may be a distorted coordinate block obtained by distorting a block (standard coordinate block) set around the standard coordinate B ′ corresponding to the distorted coordinate B, or centered on the distorted coordinate B. It may be a set block (for example, a block of 16 pixels in the horizontal direction × 16 pixels in the vertical direction).
When obtaining the pixel value of the distorted coordinate p0 at the upper left end of the block blk2, first, the distorted coordinate p0 is converted into the normal coordinate p0 ′ according to the distortion aberration information (x ′, y ′) = g (x, y).
Next, using the interpolation vector E (n, blk1 ′) corresponding to the distortion interpolation vector Z (n, blk1), the normal coordinate q0 ′ indicated by the vector −E (n, blk1 ′) starting from the normal coordinate p0 ′. Is required. The vector -E (n, blk1 ′) is a vector having the same size and the opposite direction as the interpolation vector E (n, blk1 ′).
Then, in accordance with the distortion aberration information (x, y) = f (x ′, y ′), the normal coordinate q0 ′ is converted into the distortion coordinate q0. The vector from the distortion coordinate q0 to the distortion coordinate p0 is a distortion interpolation vector for the distortion coordinate q0 (distortion coordinate p0).
The vector from the distortion coordinate q0 to the distortion coordinate p0 is a distortion interpolation vector corresponding to the distortion coordinate q0 (distortion coordinate p0). Therefore, in this embodiment, the pixel value of the distorted coordinate q0 on the frame F (n) is the pixel value to be interpolated to the distorted coordinate p0 on the interpolated frame AF (n). This process is performed for all pixel positions (all pixel positions in all blocks), and an interpolation frame AF (n) is generated.
When the interpolation method m is “normal motion vector interpolation” and when the interpolation method m is “double interpolation”, the same processing as in the first embodiment is performed. In the case of “normal motion vector interpolation”, an interpolated frame is generated by judging the position in the frame of the input image signal corresponding to the position in the interpolation frame to be generated, as in “fisheye motion vector interpolation”. May be.

As described above, according to the present embodiment, an interpolated frame is generated using a distorted motion vector. Therefore, as in the first embodiment, the frame rate of a moving image with image distortion is changed to a moving image. It can be enhanced without breaking the image.
Specifically, according to the present embodiment, the pixel position of the frame of the input image signal corresponding to the pixel position of the interpolation frame is obtained using the distortion aberration information. This is equivalent to obtaining a distortion interpolation vector for each pixel position (obtaining a motion vector for each pixel position as a distorted motion vector). An interpolation frame is generated according to the distortion interpolation vector for each pixel position. Thereby, it is possible to suppress the failure of the image of the interpolation frame as compared with the case where the distortion interpolation vector for each of the plurality of pixel positions is used.
Note that the processing for obtaining the pixel position of the frame of the input image signal corresponding to the pixel position of the interpolation frame, that is, the processing for obtaining the distortion interpolation vector for each pixel position is performed by the distortion interpolation vector calculation unit 201 instead of the interpolation frame generation unit. It may be done.

<Example 3>
An image processing apparatus and an image processing method according to Embodiment 1 of the present invention will be described with reference to the drawings. In the first and second embodiments, an interpolation frame for interpolating between frames of an input image signal by generating a distorted motion vector is generated. In the third embodiment, an interpolation frame (ordinary interpolation frame) for interpolating between image signals whose image distortion has been corrected is generated, and the normal interpolation frame is distorted to interpolate between frames of the input image signal (distortion). Interpolation frame).
FIG. 15 is a block diagram illustrating an example of a functional configuration of the image processing apparatus according to the present embodiment. The image processing apparatus according to the present embodiment includes an vignetting detection unit 101, a reduction rate calculation unit 102, an interpolation method determination unit 103, a reduction & distortion correction unit 104, a frame memory control unit 105, a motion vector detection unit 106, and an interpolation frame generation unit. 301, an enlargement & distortion unit 302, and the like. The same functional parts as those in the first and second embodiments are denoted by the same reference numerals and description thereof is omitted.

The interpolation frame generation unit 301 will be described.
The interpolation frame generation unit 301 interpolates between the frames of the image signal after the distortion correction processing from the image signal after the distortion correction processing and the motion vector detected by the motion vector detection unit 106 (normal interpolation frame). Is generated.
Specifically, the interpolation frame generation unit 301 includes frames F (n), F (n + 1), SF (n), SF (n + 1), an interpolation method m, a reduction ratio k, an interpolation vector {E (n)}, To generate an interpolation frame AF (n) or ASF (n).
When the interpolation method m is “fisheye motion vector interpolation”, the interpolation frame generation unit 301 uses the motion vector detection frames SF (n), SF (n + 1), and the interpolation vector {E (n)} to interpolate frames. ASF (n) is generated. The method for generating the interpolation frame is the same as in the case of “normal motion vector interpolation” in the first and second embodiments. For example, the image on SF (n) of each block corresponding to the interpolation vector {E (n)} is pasted at the position indicated by the interpolation vector {E (n)} as the image of the interpolation frame ASF (n). .
If SF (n) and SF (n + 1) are images with reduced normal coordinates, the interpolation frame ASF (n) is also an image with reduced normal coordinates. In such a case, since the interpolation vector {E (n)} is not a reduced vector, the interpolation vector {E (n)} is used after being multiplied by k ^0.5 .
Note that the interpolation frame generation unit 301 preferably determines the pixel value of the interpolation frame using the motion vector detected by the motion vector detection unit 106 for the imaging region. And it is preferable that the interpolation frame production | generation part 301 applies the pixel value of the image signal after the distortion correction process of the same position as a pixel value of an interpolation frame with respect to a non-photographing area | region. Thereby, it is possible to prevent the non-photographing area and the boundary between the non-photographing area and the photographing area from being disturbed by erroneous interpolation.
Note that the image of the motion vector detection frame SF may be an image of a shooting region or an image including a shooting region and a non-shooting region. For example, the motion vector detection frame SF may be an image obtained by performing distortion correction processing on the entire region of the frame of the input image signal. When the image of the motion vector detection frame SF is an image of a shooting area (an image in which the non-shooting area is excluded), the image of the non-shooting area is added to the image of the motion vector detection frame SF to interpolate the frame ASF. (N) is generated.
When the interpolation method m is “normal motion vector interpolation”, the interpolation frame generation unit 301 uses the frames F (n), F (n + 1), and the interpolation vector {E (n)} to generate the interpolation frame AF (n). Generate. The method for generating the interpolation frame is the same as in the first and second embodiments.
When the interpolation method m is “double interpolation”, the interpolation frame generation unit 301 does not generate an interpolation frame as in the first and second embodiments.

The enlargement & distortion unit 302 will be described.
The interpolation frame ASF (n) is an interpolation frame without distortion (normal interpolation frame). The enlargement & distortion unit 302 distorts the interpolation frame ASF (n) according to how the input image signal is distorted, thereby interpolating between frames of the input image signal (distortion interpolation frame). Is generated.
Specifically, the enlargement & distortion unit 302 generates an interpolation frame AF (n) from the interpolation frame ASF (n), the interpolation method m, the reduction ratio k, and distortion aberration information.
When the interpolation method m is “fisheye vector interpolation”, the enlargement & distortion unit 302 performs an enlargement process and a distortion process on the interpolation frame ASF (n). The enlargement process is a process of enlarging the horizontal direction size and the hydraulic direction size of the interpolation frame ASF (n) to 1 / k ^0.5 times, respectively. The enlargement process is performed using, for example, a bilinear method. The distortion process is a process for distorting the image of the image signal after the enlargement process. The distortion processing is performed according to (x, y) = f (x ′, y ′) of distortion aberration information. Use to convert to a realistic image. An interpolation frame AF (n) is generated by performing enlargement processing and distortion processing on the interpolation frame ASF (n).
When the interpolation method m is “normal vector interpolation”, since the interpolation frame AF (n) generated by the interpolation frame generation unit 301 is already stored in the frame memory control unit, the enlargement & distortion unit 302 The interpolation frame AF (n) is not generated.
When the interpolation method m is “double interpolation”, the enlargement & distortion unit 302 does not generate the interpolation frame AF (n).

As described above, according to the present embodiment, a motion vector is detected from an image signal after distortion correction processing, and an image after distortion correction processing is detected from the image signal after distortion correction processing and the detected motion vector. An interpolation frame (normal interpolation frame) for interpolating between signal frames is generated. Then, the normal interpolation frame is distorted in accordance with how the input image signal is distorted, thereby generating an interpolation frame (distortion interpolation frame) for interpolating between the frames of the input image signal. Thereby, the frame rate of a moving image with intentional image distortion can be increased without breaking the moving image.
Note that the interpolation frame generation unit 301 may generate an unreduced image as the interpolation frame ASF. For example, when the motion vector detection frame SF is a reduced image, the interpolation frame generation unit 301 performs an enlargement process on the motion vector detection frame SF, and an interpolation frame ASF is generated from the enlarged SF. Also good. In addition, the interpolation frame generation unit 301 may generate an interpolation frame ASF by performing an enlargement process on the interpolation frame (reduced image) generated from the motion vector detection frame SF. Further, the image of the motion vector detection frame SF may be an image that has not been subjected to reduction processing. In those cases, enlargement processing by the frame enlargement & distortion unit 302 is not necessary.

<Example 4>
An image processing apparatus and an image processing method according to Embodiment 4 of the present invention will be described with reference to the drawings.
FIG. 16 is a block diagram illustrating an example of a functional configuration of the image processing apparatus according to the present embodiment. The image processing apparatus according to the present embodiment includes a reduction unit 401, an interpolation method determination unit 402, a frame memory control unit 105, a motion vector detection unit 403, an interpolation frame generation unit 404, and the like. The same functional units as those in the first to third embodiments are denoted by the same reference numerals and description thereof is omitted.

The reduction unit 401 will be described.
The reduction unit 401 reduces the frame F (n) of the input image signal and generates a motion vector detection frame SF (n). In the present embodiment, it is assumed that the horizontal size and the vertical size of the frame F (n) are reduced by a factor of 1/2. The reduction rate may be larger or smaller than 1/2. The reduction ratio may be changed according to the image size of the frame F (n). The reduction process is performed using, for example, a bilinear method.

The motion vector detection unit 403 will be described.
The motion vector detection unit 403 obtains a motion vector {V (n)} from the motion vector detection frames SF (n) and SF (n + 1). The motion vector detection method is the same as in the other embodiments. In this embodiment, the motion vector detection frames SF (n) and SF (n + 1) are obtained by reducing the horizontal size and the vertical size of the frame F (n) by 1/2 times, respectively. Therefore, in this embodiment, the motion vector {V (n)} is directly used as the interpolation vector {E (n)}.

The interpolation frame generation unit 404 will be described.
The interpolation frame generation unit 404 generates an interpolation frame for interpolating between frames of the input image signal using the motion vector detected by the motion vector detection unit 403 when the interpolation method m is “normal motion vector interpolation”. (First generation process). Specifically, when the interpolation method m is “normal motion vector interpolation”, an interpolation frame AF (n) is generated from the frames F (n), F (n + 1), and the interpolation vector {E (n)}. . The method for generating the interpolation frame AF (n) is the same as in the other embodiments.
When the interpolation method m is “double interpolation”, the interpolation frame generation unit 404 does not generate the interpolation frame AF. When the interpolation method m is “double interpolation”, as described in the first embodiment, the frame of the input image signal is an interpolation frame for interpolating between the frames of the input image signal (second generation). processing).

The interpolation method determination unit 402 will be described.
The interpolation method determination unit 402 determines the interpolation method m according to lens information or distortion information. Thereby, whether to generate an interpolation frame in the first generation process or to generate an interpolation frame in the second generation process is controlled.

A case where the interpolation method m is determined according to the lens information will be described.
The interpolation method determination unit 402 determines from the lens information whether the input image signal is a distorted image signal. When the input image signal is a distorted image signal, the interpolation method determination unit 402 sets “double interpolation” as the interpolation method m. If the input image signal is not a distorted image signal, the interpolation method determination unit 402 sets “normal motion vector interpolation” as the interpolation method m. For example, when the lens information indicates a standard lens, a telephoto lens, or a macro lens,
The interpolation method determination unit 402 determines that the input image signal is not a distorted image signal. When the lens information indicates a fisheye lens or a wide-angle lens, the interpolation method determination unit 402 determines that the input image signal is a distorted image signal.
That is, an interpolation frame is generated in the first generation process when the input image signal is not a distorted image signal, and an interpolation frame is generated in the second generation process when the input image signal is a distorted image signal. In addition, the generation of the interpolation frame is controlled.

The case where the interpolation method m is determined according to distortion aberration information will be described.
The interpolation method determination unit 402 determines the degree of distortion of the image represented by the input image signal from the distortion aberration information. When the degree of image distortion is larger than a predetermined threshold, the interpolation method determination unit 402 sets “double interpolation” as the interpolation method m. When the degree of image distortion is less than a predetermined threshold, the interpolation method determination unit 402 sets “normal motion vector interpolation” as the interpolation method m. The degree of image distortion is, for example, the maximum value, minimum value, or average value of the difference between the normal coordinates and the distortion coordinates for each pixel of the input image signal.
That is, an interpolation frame is generated in the first generation process when the degree of image distortion is equal to or less than a predetermined threshold, and an interpolation frame is generated in the second generation process when the degree of image distortion is greater than the predetermined threshold. As described above, the interpolation frame generation process is controlled.

As described above, according to the present embodiment, when the degree of image distortion is small (no image distortion), an interpolation frame is generated in the first generation process. When the degree of image distortion is large (there is image distortion), an interpolation frame is generated in the second generation process. Thereby, the frame rate of a moving image with intentional image distortion can be increased without causing the moving image to break down significantly.
Specifically, when there is no distortion of the image, the interpolation position indicated by the motion vector is an accurate position, so that the image of the interpolation frame is not disturbed. In such a case, by generating the interpolation frame according to the motion vector, it is possible to increase the smoothness of the motion of the moving image and reduce the flicker. In addition, when there is distortion of the image, the interpolation position indicated by the motion vector is not an accurate position, so that the image of the interpolation frame may be disturbed. In such a case, by setting the frame of the input image signal as an interpolation frame, it is possible to suppress the image of the interpolation frame from failing, and to prevent the moving image from failing.
Further, when the degree of distortion of the image is small, even if the interpolation position indicated by the motion vector is deviated from the correct position, the deviation is small, so that the disturbance of the image of the interpolation frame is small. In such a case, by generating the interpolation frame according to the motion vector, it is possible to preferentially obtain the effects of improving the smoothness of the moving image and reducing flicker. Further, when the degree of distortion of the image is large, the image of the interpolation frame may be greatly disturbed. In such a case, by setting the frame of the input image signal as an interpolation frame, it is possible to suppress the image of the interpolation frame from failing, and to prevent the moving image from failing.

The frame described in the first to fourth embodiments may be a field in an interlaced image signal. That is, the frame rate conversion may be performed on a frame basis or on a field basis.
In addition, the block size, the size of the search area in block matching, the data size of the motion vector detection frame, and the like are not limited to the values exemplified in the embodiment, and are set and changed as appropriate.
Further, the reduction process and the enlargement process are not limited to the method using the bilinear method. The reduction process and the enlargement process may be performed using a method other than the bilinear method, for example, a bicubic method.
The image signal with distortion is not limited to an image signal photographed using a fisheye lens. As long as the image signal is intentionally distorted, it may be an image signal photographed using a lens other than a fisheye lens. For example, the image signal with distortion may be an image signal captured using a wide-angle lens.

104 Reduction & Distortion Correction Unit 106 Motion Vector Detection Unit 107 Distortion Interpolation Vector & Expansion / Reduction Ratio Calculation Unit 108 Interpolation Frame Generation Unit 201 Distortion Interpolation Vector Calculation Unit 202 Interpolation Frame Generation Unit 301 Interpolation Frame Generation Unit 302 Distortion Unit 402 Interpolation Method Determination Unit 403 Vector detection unit 404 Interpolated frame generation unit

Claims (23)

An image processing apparatus that interpolates between frames of an input image signal to increase a frame rate of the input image signal,
When the input image signal is a distorted image signal, distortion correction means for performing a distortion correction process for correcting image distortion on the input image signal;
Detecting means for detecting a motion vector from the image signal after the distortion correction processing;
Distortion processing means for distorting the motion vector detected by the detection means according to how the input image signal is distorted;
Generating means for generating an interpolated frame for interpolating between frames of the input image signal from the input image signal and the motion vector distorted by the distortion processing means;
An image processing apparatus comprising: When the input image signal is a signal of an image including a shooting region that is a region of a captured image and a non-shooting region that is a region other than that,
The generating unit determines a pixel value of an interpolation frame using the motion vector distorted by the distortion processing unit for the imaging region, and a pixel value of the interpolation frame for the non-imaging region. The image processing apparatus according to claim 1, wherein a pixel value of the input image signal at the same position is applied. The generating means changes the position of the image of the partial area in the frame of the input image signal in accordance with the motion vector distorted by the distortion processing means, whereby the image of the partial area in the interpolation frame The size of the image whose position is to be changed is changed according to a method of distortion of the image of the input image signal when the position of the image is changed. Image processing device. The image processing apparatus according to claim 1, wherein the distortion processing unit obtains a motion vector for each pixel position as a distorted motion vector. An image processing apparatus that interpolates between frames of an input image signal to increase a frame rate of the input image signal,
When the input image signal is a distorted image signal, distortion correction means for performing a distortion correction process for correcting image distortion on the input image signal;
Detecting means for detecting a motion vector from the image signal after the distortion correction processing;
Generating means for generating an interpolated frame for interpolating between frames of the image signal after the distortion correction processing, from the image signal after the distortion correction processing and the motion vector detected by the detection means;
Distortion processing means for generating an interpolation frame for interpolating between the frames of the input image signal by distorting the interpolation frame generated by the generation means according to a method of distortion of the image of the input image signal;
An image processing apparatus comprising: When the input image signal is a signal of an image including a shooting region that is a region of a captured image and a non-shooting region that is a region other than that,
The generation unit determines a pixel value of an interpolation frame using the motion vector detected by the detection unit for the shooting region, and is the same as a pixel value of the interpolation frame for the non-shooting region. 6. The image processing apparatus according to claim 5, wherein a pixel value of an image signal after the position correction processing is applied. The image processing apparatus according to claim 1, wherein the detection unit detects a motion vector from the image signal after the distortion correction process that has been subjected to the reduction process. 8. The frame of the input image signal is an interpolation frame for interpolating between frames of the input image signal when the reduction rate of the reduction process is less than a predetermined threshold. Image processing apparatus. When the input image signal is a signal of an image including a shooting region that is a region of a captured image and a non-shooting region that is a region other than that,
The image processing apparatus according to claim 1, wherein the detection unit detects a motion vector from an image signal of the shooting area. An image processing apparatus that interpolates between frames of an input image signal to increase a frame rate of the input image signal,
First generating means for detecting a motion vector from the input image signal and generating an interpolation frame for interpolating between frames of the input image signal using the detected motion vector;
A second generation means for setting the frame of the input image signal as an interpolation frame for interpolating between frames of the input image signal;
When the input image signal is not a distorted image signal, an interpolation frame is generated by the first generation means, and when the input image signal is a distorted image signal, an interpolation frame is generated by the second generation means. Control means for controlling the processing of the first generation means and the processing of the second generation means,
An image processing apparatus comprising: The control means generates an interpolated frame by the first generation means when the degree of image distortion is equal to or less than a predetermined threshold, and the second generation means when the degree of image distortion is greater than the predetermined threshold. The image processing apparatus according to claim 10, wherein the processing of the first generation unit and the processing of the second generation unit are controlled such that an interpolation frame is generated in step S 11. An image processing method for interpolating between frames of an input image signal to increase a frame rate of the input image signal,
When the computer is a signal of a distorted image, the distortion correction step of performing a distortion correction process to correct the distortion of the image to the input image signal;
A detection step in which a computer detects a motion vector from the image signal after the distortion correction processing;
A distortion processing step in which the computer distorts the motion vector detected in the detection step according to a method of distortion of the image of the input image signal;
A generation step of generating an interpolated frame for interpolating between frames of the input image signal from the input image signal and the motion vector distorted in the distortion processing step;
An image processing method comprising: When the input image signal is a signal of an image including a shooting region that is a region of a captured image and a non-shooting region that is a region other than that,
In the generation step, a pixel value of an interpolation frame is determined using the motion vector distorted in the distortion processing step for the shooting region, and a pixel value of the interpolation frame for the non-shooting region. The image processing method according to claim 12, wherein pixel values of the input image signal at the same position are applied. In the generating step, an image of a partial region in the interpolation frame is obtained by changing a position of an image of the partial region in the frame of the input image signal according to the motion vector distorted in the distortion processing step. 14. The image processing according to claim 12 or 13, wherein when the position of the image is determined and the position of the image is changed, the size of the image whose position is changed is changed according to how the image of the input image signal is distorted. Method. 15. The image processing method according to claim 12, wherein in the distortion processing step, a motion vector for each pixel position is obtained as a distorted motion vector. An image processing method for interpolating between frames of an input image signal to increase a frame rate of the input image signal,
When the computer is a signal of a distorted image, the distortion correction step of performing a distortion correction process to correct the distortion of the image to the input image signal;
A detection step in which a computer detects a motion vector from the image signal after the distortion correction processing;
A computer that generates an interpolated frame for interpolating between frames of the image signal after the distortion correction processing, from the image signal after the distortion correction processing and the motion vector detected in the detection step;
A distortion processing step for generating an interpolation frame for interpolating between frames of the input image signal by distorting the interpolation frame generated in the generation step according to a method of distortion of the image of the input image signal;
An image processing method comprising: When the input image signal is a signal of an image including a shooting region that is a region of a captured image and a non-shooting region that is a region other than that,
In the generation step, the pixel value of the interpolation frame is determined using the motion vector detected in the detection step for the shooting region, and the same as the pixel value of the interpolation frame for the non-shooting region. The image processing method according to claim 16, wherein a pixel value of an image signal after the distortion correction processing of a position is applied. The image processing method according to any one of claims 12 to 17, wherein in the detection step, a motion vector is detected from an image signal after distortion correction processing that has been subjected to reduction processing. 19. The frame of the input image signal is an interpolation frame for interpolating between frames of the input image signal when the reduction rate of the reduction process is less than a predetermined threshold. Image processing method. When the input image signal is a signal of an image including a shooting region that is a region of a captured image and a non-shooting region that is a region other than that,
The image processing method according to claim 12, wherein in the detection step, a motion vector is detected from an image signal in the shooting area. An image processing method for interpolating between frames of an input image signal to increase a frame rate of the input image signal,
A first generation step in which a computer detects a motion vector from the input image signal and generates an interpolated frame for interpolating between frames of the input image signal using the detected motion vector;
A second generation step in which the computer sets the frame of the input image signal as an interpolation frame for interpolating between frames of the input image signal;
An interpolation frame is generated in the first generation step when the input image signal is not a distorted image signal, and an interpolation frame is generated in the second generation step when the input image signal is a distorted image signal. A control step for controlling the processing of the first generation step and the processing of the second generation step, so that
An image processing method comprising: In the control step, an interpolation frame is generated in the first generation step when the degree of image distortion is equal to or less than a predetermined threshold, and the second generation step when the degree of image distortion is greater than the predetermined threshold. The image processing method according to claim 21, wherein the processing of the first generation step and the processing of the second generation step are controlled so that an interpolation frame is generated in step (a).   A program for causing a computer to execute each step of the image processing method according to any one of claims 12 to 22.

Images