JP2004234637A - Moving image compositing device and method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a high resolution frame where the deterioration of picture quality is reduced regardless of the movement of an object included in the frame at the time of acquiring one high resolution frame from a plurality of frames sampled from moving image data. <P>SOLUTION: Patches are arranged in a plurality of frames FrN, FrN+1, and the patches are moved and/or deformed so that the corresponding relations of those respective frames can be estimated, and the frame FrN+1 is coordinate transformed based on the estimated corresponding relation, and a correlation value between that and the frame FrN is calculated. The number of divisions of the patches is changed, and the correlation value is calculated in the same way. Then, a composited frame FrG is prepared from the frames FrN and FrN+1 based on the corresponding relation estimated from the number of divisions leading to the maximum correlation. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

  The present invention relates to a moving image synthesizing apparatus and method, and a moving image synthesizing method for synthesizing a plurality of consecutive frames obtained by sampling moving images to create one synthesized frame having a higher resolution than the sampled frames. The present invention relates to a program for causing a computer to execute.

  With the recent spread of digital video cameras, it is possible to handle moving images in units of frames. When printing out a frame of such a moving image, it is necessary to make the frame high resolution in order to improve the image quality. For this reason, a method of creating one composite frame having a higher resolution than those frames from a plurality of frames sampled from a moving image has been proposed (for example, see Patent Document 1). In this method, a motion vector between a plurality of frames is obtained, and a signal value to be interpolated between pixels is calculated based on the motion vector when a synthesized frame is synthesized from a plurality of frames. In particular, in the method described in Patent Document 1, each frame is divided into a plurality of blocks, the orthogonal coordinate coefficients of the corresponding blocks between the frames are calculated, and high-frequency information in the orthogonal coordinate coefficients is used as low-frequency information in other blocks. Since the pixel value to be interpolated by combining with this information is calculated, a high-quality synthesized frame can be obtained without reducing necessary information. Further, in this method, since the motion vector is calculated with a resolution finer than the inter-pixel distance, it is possible to obtain a synthesized frame with higher image quality by accurately compensating for the motion between frames.

  Also, one of the plurality of frames is set as a reference frame, a reference patch composed of a plurality of rectangular areas is arranged in the reference frame, and a patch similar to the reference patch is arranged in a frame other than the reference frame. Move and / or deform the patch on the other frame so that the image matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch, the pixels in the patch on the other frame A method has also been proposed in which a correspondence relationship with pixels in a reference patch on a reference frame is estimated to synthesize a plurality of frames more accurately (see, for example, Non-Patent Document 1).

In the method of Non-Patent Document 1, the correspondence between the reference frame and other frames is estimated, and after the estimation, the other frames and the reference frame are finally allocated on the integrated image having the necessary resolution. A high-definition composite frame can be obtained.
JP 2000-354244 A Yuji Nakazawa, Takashi Komatsu, Takahiro Saito, “Acquisition of high-definition digital images by interframe integration”, Journal of the Television Society, 1995, Vol. 49, no. 3, p299-308

  However, in the method described in Non-Patent Document 1, when the movement of the subject included in another frame is very large, or the subject included locally moves in a complicated manner or moves at a very high speed. In some cases, the movement and / or deformation of the patch cannot follow the movement of the subject. In this way, if the movement and / or deformation of the patch cannot follow the movement and / or deformation of the subject, the entire composite frame will be blurred, or the subject with large movement included in the frame will be blurred, so high-quality composition is possible. There is a problem that a frame cannot be obtained. In this case, by dividing the patch into more rectangular areas, the movement and / or deformation of the subject can follow the movement and / or deformation of the subject. However, if the number of divisions is too large, the rectangular area becomes too small, and it becomes difficult to make the movement and / or deformation of the subject follow the movement and / or deformation of the subject.

  The present invention has been made in view of the above circumstances, and an object thereof is to obtain a high-quality composite frame with an optimal number of patch divisions.

A first moving image synthesizing apparatus according to the present invention comprises sampling means for sampling two consecutive frames from a moving image;
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch A correspondence estimation means for estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame;
Based on the correspondence, coordinate conversion means for converting the image in the patch of the other frame to the coordinate space of the reference frame to obtain a coordinate-converted frame;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame;
The reference patch and the number of divisions of the rectangular area in the patch are changed in stages, and the correlation is estimated, the coordinate-transformed frame is obtained, and the correlation value is calculated in each stage. Control means for controlling the correspondence estimation means, the coordinate conversion means, and the correlation value calculation means so as to obtain a plurality of correlation values according to the patch and the number of divisions of the patch;
Comparing means for comparing the magnitude of the correlation for each of the number of divisions based on the plurality of correlation values;
Based on the correspondence estimated in the number of divisions that maximizes the correlation, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame, and And a synthesizing unit that creates a synthetic frame having a higher resolution than each frame.

  “Change the number of divisions stepwise” means that the number of divisions is gradually increased from a small number of divisions to a predetermined number of divisions, and stepwise from a large number of divisions to a predetermined number of divisions. Includes gradually reducing the number of divisions.

  The “correlation value” may be calculated in units of a rectangular area or a frame of a patch between the coordinate-converted frame and the image in the reference patch of the reference frame.

  “Comparing the magnitude of correlation” means comparing the magnitude of correlation between frames when the correlation value is calculated in units of frames, and the correlation value is calculated in units of rectangular areas. In this case, it means that the magnitude of correlation between the rectangular areas is compared. Note that the comparison is performed between the correlation value calculated in a certain division number and the correlation value calculated in another division number. Here, the larger the number of divisions, the smaller the size of the rectangular area. Therefore, the comparison between the correlation value calculated for one number of divisions and the correlation value calculated for other number of divisions is for the rectangular area with the larger size. The correlation value is calculated based on the average value and intermediate value of the correlation values of a plurality of rectangular areas having smaller sizes corresponding to the rectangular area.

  In the first moving image synthesizing apparatus according to the present invention, the correspondence used for creating the synthesized frame may be estimated for each rectangular area.

  Further, in the first moving image synthesizing apparatus according to the present invention, the correspondence relationship estimation means, the coordinate conversion means, the correlation value calculation means, the control means, and the composition means are used for the correspondence relationship estimation, the coordinate conversion, and the like. As means for performing acquisition of a completed frame, calculation of the correlation value, acquisition of the plurality of correlation values, comparison of the magnitude of the correlation, and creation of the composite frame using at least one component constituting the frame Good.

  “At least one component constituting the frame” is, for example, at least one component of RGB color components when the frame is composed of three RGB color data, and when composed of YCC luminance color difference components, At least one of the luminance and color difference components, preferably the luminance component.

A second moving image synthesizing apparatus according to the present invention comprises sampling means for sampling three or more consecutive frames from a moving image;
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Correspondence estimation means for estimating a correspondence between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame based on the reference patch;
Coordinate conversion means for converting the image in the patch of the other one frame into the coordinate space of the reference frame to obtain a coordinate-converted frame based on the correspondence relationship;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame;
The reference patch and the number of divisions of the rectangular area in the patch are changed in stages, and the correlation is estimated, the coordinate-transformed frame is obtained, and the correlation value is calculated in each stage. Control means for controlling the correspondence estimation means, the coordinate conversion means, and the correlation value calculation means so as to obtain a plurality of correlation values according to the patch and the number of divisions of the patch;
Comparing means for comparing the magnitude of the correlation for each of the number of divisions based on the plurality of correlation values;
Based on the correspondence estimated at the number of divisions that maximizes the correlation, an interpolation operation is performed on the image in the patch of the other one frame and the image in the reference patch of the reference frame. , Obtaining an intermediate composite frame having a higher resolution than each frame, and for all the other frames, estimating the correspondence, obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the plurality of correlation values. A synthesizing unit that obtains, compares the correlation magnitudes, and obtains the intermediate synthesized frame, and synthesizes a plurality of the intermediate synthesized frames obtained for all the other frames; It is characterized by that.

  In the second moving image synthesizing apparatus according to the present invention, the correspondence used for creating the intermediate synthesized frame may be estimated for each rectangular area.

  In the second moving image synthesizing apparatus according to the present invention, the correspondence relationship estimation means, the coordinate conversion means, the correlation value calculation means, the control means, and the composition means are connected to the correspondence relationship estimation, the coordinate conversion, and the like. Acquisition of completed frames, calculation of the correlation values, acquisition of the plurality of correlation values, comparison of the magnitudes of the correlations, acquisition of the intermediate composite frame and creation of the composite frame, at least one component constituting the frame It is good also as a means to perform using.

A first moving image composition method according to the present invention samples two consecutive frames from a moving image,
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch Estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame,
Based on the correspondence relationship, the image in the patch of the other frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame,
Calculating a correlation value representing a correlation between the coordinate-transformed frame and an image in the reference patch of the reference frame;
The reference patch and the number of divisions of the rectangular area in the patch are changed in stages, and the correlation is estimated, the coordinate-transformed frame is obtained, and the correlation value is calculated in each stage. Obtain a plurality of correlation values according to the patch and the number of divisions of the patch,
Comparing the correlation magnitude for each of the number of divisions based on the plurality of correlation values;
Based on the correspondence estimated in the number of divisions that maximizes the correlation, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame, and A composite frame having a higher resolution than each frame is created.

  In the first moving image composition method according to the present invention, the correspondence used for creating the composite frame may be estimated for each rectangular area.

  In the first moving image composition method according to the present invention, the correspondence is estimated, the coordinate-transformed frame is acquired, the correlation value is calculated, the plurality of correlation values are acquired, and the correlation magnitude is compared. The composite frame may be created using at least one component constituting the frame.

The second moving image composition method according to the present invention samples three or more consecutive frames from a moving image,
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Based on the reference patch, a correspondence relationship between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame is estimated,
Based on the correspondence, the image in the patch of the other one frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame;
Calculating a correlation value representing a correlation between the coordinate-transformed frame and an image in the reference patch of the reference frame;
The reference patch and the number of divisions of the rectangular area in the patch are changed in stages, and the correlation is estimated, the coordinate-transformed frame is obtained, and the correlation value is calculated in each stage. Obtain a plurality of correlation values according to the patch and the number of divisions of the patch,
Comparing the correlation magnitude for each of the number of divisions based on the plurality of correlation values;
Based on the correspondence estimated at the number of divisions that maximizes the correlation, an interpolation operation is performed on the image in the patch of the other one frame and the image in the reference patch of the reference frame. , Obtain an intermediate composite frame having a higher resolution than each of the frames,
For all the other frames, estimation of the correspondence, acquisition of the coordinate-transformed frame, calculation of the correlation value, acquisition of the plurality of correlation values, comparison of the correlation magnitudes, and acquisition of the intermediate composite frame And
A synthesized frame is created by synthesizing a plurality of the intermediate synthesized frames acquired for all the other frames.

  In the second moving image composition method according to the present invention, the correspondence used for creating the intermediate composition frame may be estimated for each rectangular area.

  In the second moving image composition method according to the present invention, the correspondence is estimated, the coordinate-transformed frame is acquired, the correlation value is calculated, the plurality of correlation values are acquired, and the correlation magnitude is compared. The acquisition of the intermediate composite frame and the creation of the composite frame may be performed using at least one component constituting the frame.

  In addition, you may provide as a program for making a computer perform the 1st and 2nd moving image composition method by this invention.

  According to the present invention, a plurality of continuous frames are acquired by sampling a moving image, and a reference patch having one or more rectangular areas is arranged on one reference frame as a reference among the plurality of frames. . A patch similar to the reference patch is arranged on a frame other than the reference frame. Then, the image in the patch is moved and / or deformed so as to match the image in the reference patch, and based on the moved and / or deformed patch and the reference patch, A correspondence relationship with the pixels in the reference patch on the reference frame is estimated.

  Further, based on the correspondence, the image in the patch of the other frame is transformed into the coordinate space of the reference frame to obtain the coordinate-converted frame, and the correlation between the coordinate-converted frame and the image in the reference patch of the reference frame is obtained. Is calculated.

  Then, the division number of the reference patch and the rectangular area in the patch is changed in stages, and in each stage, the correspondence is estimated, the coordinate converted frame is obtained, and the correlation value is calculated. Is obtained.

  Further, the magnitude of the correlation is compared for each number of divisions based on a plurality of correlation values, and the images in the patches of the other frames and the reference frame are determined based on the correspondence relationship estimated in the number of divisions that maximizes the correlation. Interpolation is performed on the image in the patch to create a composite frame having a higher resolution than each frame.

  When three or more frames are sampled, for all other frames other than the reference frame, correspondence estimation, acquisition of coordinate-transformed frames, calculation of correlation values, acquisition of a plurality of correlation values, and correlation The size is compared, a plurality of intermediate composite frames are acquired from the reference frame and each of the other frames, and a composite frame is created from the plurality of intermediate composite frames.

  For this reason, it is possible to create a composite frame with an optimal number of divisions that can follow the movement and / or deformation of a moving subject within the frame, thereby enabling high image quality regardless of the movement of the subject included in the frame. Can be obtained.

  In addition, by estimating the correspondence used for creating a composite frame or an intermediate composite frame for each rectangular area, a composite frame or an intermediate composite frame is created for each rectangular area divided within the reference patch and the patch. Become. For this reason, a composite frame can be created with an optimal number of divisions that follow the movement of the subject included in each part of the frame, thereby enabling a higher quality composite regardless of the movement of the subject included in the frame. You can get a frame.

  Also, estimation of correspondence, acquisition of coordinate-converted frames, calculation of correlation values, acquisition of a plurality of correlation values, comparison of correlation magnitudes, acquisition of intermediate composite frames (when three or more frames are sampled) and By creating a composite frame using at least one component constituting the frame, it is possible to obtain a composite frame with reduced image quality degradation for each component. A composite frame with high image quality can be obtained.

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of a moving image synthesis apparatus according to the first embodiment of the present invention. As shown in FIG. 1, the moving image synthesizing apparatus according to the present embodiment includes a sampling unit 1 that samples a plurality of frames from input moving image data M0, and one reference frame that is a reference among the plurality of frames. Correspondence relationship estimation means 2 for estimating the correspondence relationship between the pixels and the pixels of other frames other than the reference frame, and other frames on the coordinate space of the reference frame based on the correspondence relation estimated by the correspondence relationship estimation means 2 The coordinate conversion means 3 for obtaining a plurality of coordinate-converted frames (hereinafter referred to as coordinate conversion frames) FrT0 and FrT1 according to the number of divisions of the patch whose relationship is estimated as described later, The number of patches divided to estimate the correlation as will be described later with respect to the correlation value representing the correlation between the converted frames FrT0 and FrT1 and the reference frame Correlation value calculation means 4 for calculating a plurality of values in response, comparison means 5 for comparing the magnitudes of the correlations for each number of divisions based on the plurality of correlation values, and estimation by the correspondence relationship estimation means 2 for the number of divisions with the maximum correlation And a synthesizing unit 6 that performs an interpolation operation on the reference frame and other frames based on the correspondence relationship and creates a synthesized frame FrG having a higher resolution than each frame. In the present embodiment, it is assumed that the synthesized frame FrG has twice as many pixels as the sampled frame. In the following description, the composite frame FrG is described as having twice as many pixels as the sampled frame, but it may be n times (n: positive number).

  The sampling means 1 samples a plurality of frames from the moving image data M0. In this embodiment, the sampling means 1 samples two frames FrN and FrN + 1 from the moving image data M0. Note that the frame FrN is a reference frame. Here, the moving image data M0 represents a color moving image, and the frames FrN and FrN + 1 are composed of luminance, color difference components of Y, Cb, and Cr. In the following description, processing is performed for each component of Y, Cb, and Cr. Since the processing to be performed is the same for all components, the processing of the luminance component Y is described in detail in the present embodiment. A description of the processing for the color difference components Cb and Cr will be omitted.

  The correspondence estimation means 2 estimates the correspondence between the frame FrN + 1 and the reference frame FrN as follows. FIG. 2 is a diagram for explaining the estimation of the correspondence between the frame FrN + 1 and the reference frame FrN. In FIG. 2, it is assumed that the circular subject included in the reference frame FrN has moved slightly to the right in the drawing in the frame FrN + 1.

First, the correspondence estimation means 2 first arranges a reference patch P0 composed of one or a plurality of rectangular areas on the reference frame FrN. FIG. 2A shows a state in which the reference patch P0 is arranged on the reference frame FrN. As shown in FIG. 2A, in the present embodiment, the reference patch P0 is assumed to be composed of a 2 × 2 rectangular area. Next, as shown in FIG. 2B, a patch P1 similar to the reference patch P0 is arranged at an appropriate position in the frame FrN + 1, and a correlation value representing the correlation between the image in the reference patch P0 and the image in the patch P1. Is calculated. The correlation value can be calculated as a mean square error by the following equation (1). The coordinate axes are assumed to be the x axis in the left and right direction on the paper and the y axis in the vertical direction on the paper.

Where E: correlation value pi, qi: pixel value of the corresponding pixel in each of the reference patches P0, P1 N: number of pixels in the reference patch P0 and patch P1

  Next, the patch P1 on the frame FrN + 1 is moved by fixed pixels ± Δx, ± Δy in four directions, up, down, left, and right. The correlation value between the image in the patch P1 and the image in the reference patch P0 on the reference frame FrN is calculate. Here, the correlation value is calculated for each of the up, down, left, and right directions, and each correlation value is expressed as E (Δx, 0), E (−Δx, 0), E (0, Δy), E (0, −Δy), respectively. To do.

Then, the correlation value becomes smaller (that is, the correlation becomes larger) from the four correlation values E (Δx, 0), E (−Δx, 0), E (0, Δy), and E (0, −Δy) after the movement. The gradient direction is obtained as the correlation gradient, and the patch P1 is moved in this direction by a preset real value multiple as shown in FIG. Specifically, coefficients C (Δx, 0), C (−Δx, 0), C (0, Δy), C (0, −Δy) are calculated by the following equation (2), and these coefficients C Correlation gradients gx and gy are calculated from the following equations (3) and (4) from (Δx, 0), C (−Δx, 0), C (0, Δy), and C (0, −Δy).

  Then, the entire patch P1 is moved (−λ1gx, −λ1gy) based on the calculated correlation gradients gx, gy, and the same processing as described above is repeated, so that the patch P1 as shown in FIG. The patch P1 is repeatedly moved until it converges to a certain position. Here, λ1 is a parameter that determines the speed of convergence, and takes a real value. If λ1 is too large, the solution is diverged by iterative processing, so an appropriate value (for example, 10) needs to be selected.

  Further, the lattice point of the patch P1 is moved by a fixed pixel in four directions along the coordinate axis. At this time, the rectangular area including the moved grid point is deformed as shown in FIG. 3, for example. Then, a correlation value between the deformed rectangular area and the corresponding rectangular area of the reference patch P0 is calculated. The correlation values are defined as E1 (Δx, 0), E1 (−Δx, 0), E1 (0, Δy), and E1 (0, −Δy), respectively.

  Similarly to the above, the correlation value is small from the four correlation values E1 (Δx, 0), E1 (−Δx, 0), E1 (0, Δy), and E1 (0, −Δy) after deformation (ie, The gradient direction (in which the correlation is large) is obtained, and the grid point of the patch P1 is moved in this direction by a preset real value multiple. This is performed for all the grid points of the patch P1, and this is regarded as one process. This process is repeated until the coordinates of the grid points converge.

  Thereby, the movement amount and the deformation amount of the patch P1 with respect to the reference patch P0 are obtained, and the correspondence between the pixels in the reference patch P0 and the pixels in the patch P1 can be estimated based on this.

  Subsequently, the correspondence estimation means 2 further increases the number of divisions (here, 4 × 4) as shown in FIG. 4A, and sets the reference patch P01 and the patch P11 to appropriate positions of the reference frame FrN and the frame FrN + 1. In the same manner as described above, the movement amount and deformation amount of the patch P11 with respect to the reference patch P01 are obtained, and the correspondence between the pixels in the reference patch P01 and the pixels in the patch P11 is estimated.

  As shown in FIG. 4B, the reference patch P02 and the patch P12 having a larger number of divisions (here, 8 × 8) may be used, but in this embodiment, 2 × 2 and 4 × 4 are used. Assume that the correspondence relationship is estimated by the number of divisions in two stages. In this embodiment, the correspondence estimated by the reference patches P0 and P1 having the division number of 2 × 2 is estimated by the first correspondence, and the reference patches P01 and patch P11 having the division number of 4 × 4. This correspondence is referred to as a second correspondence.

  The coordinate conversion means 3 performs coordinate conversion of the frame FrN + 1 into the coordinate space of the reference frame FrN for each division number based on the estimated first and second correspondences as follows, and converts the coordinate conversion frames FrT0 and FrT1. To get. First, acquisition of the coordinate conversion frame FrT0 based on the first correspondence will be described. In the following description, conversion and synthesis are performed only for the regions in the reference patches P0 and P01 of the reference frame FrN and the regions in the patches P1 and P11 of the frame FrN + 1.

In the present embodiment, coordinate transformation is performed using bilinear transformation. Coordinate transformation by bilinear transformation is defined by the following equations (5) and (6).

  Expressions (5) and (6) express coordinates in the patch P1 given by four points (xn, yn) (1 ≦ n ≦ 4) on a two-dimensional coordinate in a normalized coordinate system (u, v) ( 0 ≦ u, v ≦ 1), and coordinate transformation in any two rectangles is performed by combining the inverse transformations of equations (5) and (6) and equations (5) and (6). It can be carried out.

  Here, as shown in FIG. 5, the position in the reference patch P0 (x′n, y′n) corresponding to the point (x, y) in the patch P1 (xn, yn) is considered. . First, normalized coordinates (u, v) are obtained for the point (x, y) in the patch P1 (xn, yn). This is obtained by inverse transformation of equations (5) and (6). Based on the reference patch P0 (x′n, y′n) corresponding to (u, v) at this time, the coordinates (x ′) corresponding to the point (x, y) from the equations (5) and (6) , Y ′). Here, since the point (x, y) is an integer coordinate where the pixel value originally exists, the point (x ′, y ′) may be a real number coordinate where the pixel value originally does not exist. As for the pixel value in the integer coordinates, an area surrounded by integer coordinates in the vicinity of 8 adjacent to the integer coordinates of the reference patch P0 is set, and the pixel value of the converted coordinates (x ′, y ′) is set in this area. It shall be obtained as the load sum.

Specifically, as shown in FIG. 6, for integer coordinates b (x, y) on the reference patch P0, integer coordinates b (x-1, y-1) and b (x, y-1) in the vicinity of the eight coordinates. ), B (x + 1, y-1), b (x-1, y), b (x + 1, y), b (x-1, y + 1), b (x, y + 1), b (x + 1, y + 1) Calculation is performed based on the pixel value of the frame FrN + 1 converted in the enclosed area. Here, m pixel values of the frame FrN + 1 are converted into an area surrounded by eight neighboring pixels, and the converted pixel value of each pixel is Itj (x °, y °) (1 ≦ j ≦ m). Then, the pixel value It (x ^, y ^) at the integer coordinates b (x, y) can be calculated by the following equation (7). In Expression (7), φ is a function representing the load sum calculation.

  However, Wj (1 ≦ j ≦ m): product of the internal ratio of coordinates as viewed from neighboring integer pixels at the position where the pixel value Itj (x °, y °) is assigned.

Here, for the sake of simplicity, consider the case where the two pixel values It1 and It2 of the frame FrN + 1 are transformed into the area surrounded by the eight neighboring pixels with reference to FIG. 6, at the integer coordinate b (x, y). The pixel value It (x ^, y ^) can be calculated by the following equation (8).

  However, W1 = u × v, W2 = (1-s) × (1-t)

  By performing the above processing for all the pixels in the patch P1, the image in the patch P1 is converted into the coordinate space of the reference frame FrN, and the coordinate conversion frame FrT0 is acquired.

  Similarly, based on the second correspondence relationship, the image in the patch P11 is converted into the coordinate space of the reference frame FrN to obtain the coordinate conversion frame FrT1.

The correlation value calculation means 4 calculates a correlation value d0 between the coordinate conversion frame FrT0 and the reference frame FrN and a correlation value d1 between the coordinate conversion frame FrT1 and the reference frame FrN. Specifically, as shown in the following formula (9), the absolute value of the difference between the pixel values FrT0 (x, y) and FrN (x, y) in the corresponding pixels of the coordinate conversion frame FrT0 and the reference frame FrN. A value obtained by dividing the addition value for all the pixels in the reference patch P0 by the number m of pixels in the reference patch P0, that is, the average value for all the pixels in the reference patch P0 of the absolute difference is calculated as the correlation value d0. . Further, as shown in the following formula (10), the reference of the absolute value of the difference between the pixel values FrT1 (x, y) and FrN (x, y) in the corresponding pixels of the coordinate conversion frame FrT1 and the reference frame FrN The average value for all pixels in the patch P0 is calculated as the correlation value d1. The correlation values d0 and d1 are smaller as the correlation between the coordinate conversion frames FrT0 and FrT1 and the reference frame FrN is larger.

  In the present embodiment, the correlation values d0 and d1 are calculated from the absolute value of the difference between the pixel values of the corresponding pixels in the coordinate conversion frames FrT0 and FrT1 and the reference frame FrN, but the correlation value is calculated from the square of the difference. May be. Further, an addition value of absolute values of pixel value differences in corresponding pixels between the coordinate conversion frames FrT0 and FrT1 and the reference frame FrN may be used as a correlation value. Also, the histograms of the coordinate transformation frames FrT0, FrT1 and the reference frame FrN are calculated, respectively, and the average value, median value or standard deviation difference value of the histograms of the coordinate transformation frames FrT0, FrT1 and the reference frame FrN, or the difference value of the histogram are calculated. A cumulative sum may be used as a correlation value. Further, a motion vector representing the motion of the coordinate conversion frames FrT0 and FrT1 with respect to the reference frame FrN is calculated for each pixel or small region of the reference frame FrN, and the calculated average value, median value, or standard deviation of the motion vectors is a correlation value. Or a cumulative sum of histograms of motion vectors may be used as a correlation value.

  The comparing means 5 compares the correlation value d0 and the correlation value d1, and selects the correlation value having the smaller correlation value, that is, the correlation having the larger correlation value. In the present embodiment, it is assumed that the correlation value d1 calculated based on the reference patch P01 and the patch P11 divided into 4 × 4 is selected.

The synthesizing unit 6 performs an interpolation operation on the reference frame FrN and the frame FrN + 1 on the basis of the second correspondence estimated by the number of divisions obtained from the correlation value d1 selected by the comparing unit 5, and combines the frame FrG. Create Specifically, first, as shown in FIG. 7, an integrated image having a finally required number of pixels (in the present embodiment, a case where the number of pixels is doubled in both the vertical and horizontal directions of the frames FrN and FrN + 1 will be described. , Which may have a pixel number n times (n: positive number)), and based on the second correspondence obtained by the correspondence estimation means 2, the frame FrN + 1 (in the patch P11) The pixel values of the pixels in (region) are assigned on the integrated image. Assuming that the function for performing this assignment is Π, the pixel value of each pixel of the frame FrN + 1 is assigned on the integrated image by the following equation (11).

However, I1N + 1 (x °, y °): the pixel value of the frame FrN + 1 allocated on the integrated image FrN + 1 (x, y): the pixel value of the frame FrN + 1

  Thus, by assigning the pixel value of the frame FrN + 1 on the integrated image, the pixel value I1N + 1 (x °, y °) is obtained, and I1 (x °, y °) (= I1N + 1 (x °, y °)) for each pixel. A first interpolation frame having a pixel value of) is acquired.

  Here, when assigning pixel values to the integrated image, depending on the relationship between the number of pixels in the integrated image and the number of pixels in the frame FrN + 1, each pixel on the frame FrN + 1 has an integer coordinate (that is, a pixel value exists). (Coordinates) may not correspond. In this embodiment, the pixel value in the integer coordinates of the integrated image is set to an area surrounded by integer coordinates in the vicinity of 8 adjacent to the integer coordinates of the integrated image, and each of the frames on the frame FrN + 1 assigned in this area is set. Obtained as a weighted sum of pixel values of pixels.

That is, as shown in FIG. 8, for integer coordinates p (x, y) in the integrated image, integer coordinates p (x-1, y-1), p (x, y-1), p (8) in the vicinity of the eight. x + 1, y-1), p (x-1, y), p (x + 1, y), p (x-1, y + 1), p (x, y + 1), p (x + 1, y + 1) Is calculated based on the pixel value of the frame FrN + 1 assigned to. Here, k pixel values of the frame FrN + 1 are allocated in a region surrounded by eight neighboring pixels, and the pixel values of the allocated pixels are I1N + 1i (x °, y °) (1 ≦ i ≦ k). Then, the pixel value I1N + 1 (x ^, y ^) at the integer coordinates p (x, y) can be calculated by the following equation (12). In Expression (12), Φ is a function representing the load sum calculation.

  However, Mi (1 ≦ i ≦ k): product of the internal ratio of coordinates as viewed from neighboring integer pixels at the position where the pixel value I1N + 1i (x °, y °) is assigned.

Here, for the sake of simplicity, consider the case where the two pixel values I1N + 11 and I1N + 12 of the frame FrN + 1 are assigned to the region surrounded by the pixels in the vicinity of 8 with reference to FIG. 8 at the integer coordinates p (x, y). Pixel value I1N + 1 (x ^, y ^) can be calculated by the following equation (13).

  However, M1 = u × v, M2 = (1-s) × (1-t)

  The pixel value I1N + 1 (x ^, y ^) can be obtained by assigning the pixel value of the frame FrN + 1 for all integer coordinates of the integrated image. In this case, each pixel value I1 (x ^, y ^) of the first interpolation frame is I1N + 1 (x ^, y ^).

  Note that the reference frame FrN is acquired by interpolating and directly assigning the pixels of the reference frame FrN to the integer coordinates of the integrated image. A pixel value obtained by assigning the reference frame FrN to the integer coordinates of the integrated image is I2 (x ^, y ^), and a frame having the pixel value I2 (x ^, y ^) is a second interpolation frame. To do.

  Then, by adding or weighting the pixel values I1 (x ^, y ^) and I2 (x ^, y ^) in the corresponding pixels of the first and second interpolation frames, the pixel value FrG (x ^, A composite frame FrG having y ^) is created. Note that the composite frame FrG having the pixel value FrG (x ^, y ^) may be created only from the pixel value I1 (x ^, y ^).

  Note that pixel values may not be assigned to all integer coordinates of the integrated image. In such a case, various interpolation operations such as a linear interpolation operation and a spline interpolation operation are performed on the assigned pixel value or the pixel value of the reference frame (corresponding to the coordinates not assigned), and the pixel value is What is necessary is just to calculate the pixel value of the integer coordinate which was not allocated.

  In the above description, the process for obtaining the composite frame FrG for the luminance component Y has been described. However, the composite frame FrG is also obtained for the color difference components Cb and Cr. Then, a synthesized frame FrG (Y) obtained from the luminance component Y and synthesized frames FrG (Cb) and FrG (Cr) obtained from the color difference components Cb and Cr are synthesized to obtain a final synthesized frame. It will be. In order to increase the processing speed, the correspondence relationship between the reference frame FrN and the frame FrN + 1 is estimated only for the luminance component Y, and the color difference components Cb and Cr are processed based on the correspondence relationship estimated for the luminance component Y. It is preferable to carry out.

  Next, the operation of this embodiment will be described. FIG. 9 is a flowchart showing processing performed in the present embodiment. First, the moving image data M0 is input to the sampling means 1 (step S1), and here, the reference frame FrN and the frame FrN + 1 are sampled from the moving image data M0 (step S2). Subsequently, a plurality of correspondence relationships between the reference frame FrN and the frame FrN + 1 are estimated by the correspondence estimation means 2 according to the number of divisions of the patch (step S3).

  Then, based on each of the estimated plurality of correspondences, the coordinate conversion means 3 converts the frame FrN + 1 into the coordinate space of the reference frame FrN to obtain a plurality of coordinate conversion frames FrT0 and FrT1 (step S4). Then, the correlation value calculation means 4 calculates the correlation value d0 between the coordinate conversion frame FrT0 and the reference frame FrN and the correlation value d1 between the coordinate conversion frame FrT1 and the reference frame FrN (step S5). Furthermore, the correlation value with the larger correlation is selected by the comparison means 5 (step S6).

  Then, based on the correspondence relationship (second correspondence relationship) estimated by the number of divisions (4 × 4 in this case) from which the selected correlation value d1 was obtained, the synthesis means 6 uses the reference frame FrN and the frame FrN + 1 as a synthesis frame. FrG is created (step S7), and the process ends.

  Thus, in the present embodiment, since the composite frame FrG is created from the frame FrN and the frame FrN + 1 based on the correspondence estimated by the number of divisions having a large correlation, the movement of the moving subject within the frame is generated. In addition, a composite frame FrG can be created with an optimal number of divisions that can follow the deformation, whereby a high-quality composite frame FrG can be obtained regardless of the movement of the subject included in the frame.

  Next, a second embodiment of the present invention will be described. In the first embodiment, the composite frame FrG is created based on the correspondence between the entire images in the patch of the frames FrN and FrN + 1 estimated by the number of divisions having a large correlation. In the second embodiment, In this example, a correlation value is calculated for each rectangular area obtained by dividing the patch, and a composite frame FrG is created based on the correspondence relationship for each rectangular area.

  FIG. 10 is a diagram for explaining the calculation of the correlation value for each rectangular area. As shown in FIG. 10A, the rectangular areas of the reference patch P0 and the patch P1 having the number of divisions of 2 × 2 are defined as rectangular areas A1 to A4. On the other hand, as shown in FIG. 10B, the rectangular areas of the reference patch P01 and the patch P11 having a division number of 4 × 4 are defined as rectangular areas A11 to A14, A21 to A24, A31 to A34, and A41 to A44. The rectangular area A1 corresponds to the rectangular areas A1 to A14, the rectangular area A2 corresponds to the rectangular areas A21 to A24, the rectangular area A3 corresponds to the rectangular areas A31 to A34, and the rectangular area A4 corresponds to the rectangular areas A41 to A44.

  When the number of divisions is 2 × 2, the correlation value between the reference frame FrN and the frame FrN + 1 is calculated for each of the rectangular areas A1 to A4. Assume that the correlation values in each of the rectangular areas A1 to A4 are d01, d02, d03, and d04.

  On the other hand, when the number of divisions is 4 × 4, the correlation value is calculated for each of the rectangular areas A11 to A14, A21 to A24, A31 to A34, A41 to A44, and the average value of the correlation values of the rectangular areas A11 to A14 is calculated. Is the correlation value d11, the average value of the correlation values of the rectangular areas A21 to A24 is the correlation value d12, the average value of the correlation values of the rectangular areas A31 to A34 is the correlation value d13, and the correlation values of the rectangular areas A41 to A44 The average value is calculated as the correlation value d14.

  Next, the correlation values are compared for each corresponding region. That is, the correlation value d01 of the rectangular area A1 and the correlation value d11 of the rectangular areas A11 to A41, the correlation value d02 of the rectangular area A2, the correlation value d12 of the rectangular areas A21 to A24, and the correlation value d03 of the rectangular area A3 The correlation value d13 of the rectangular areas A31 to A34 is compared with the correlation value d04 of the rectangular area A4 and the correlation value d14 of the rectangular areas A41 to A44, respectively. Then, a composite frame FrG is created from the reference frame FrN and the frame FrN + 1 for each rectangular region based on the correspondence estimated by the number of divisions of the region having the larger correlation according to the comparison result.

  For example, in the present embodiment, the smaller the correlation value, the larger the correlation. Therefore, when the correlation value d01 <correlation value d11, the region corresponding to the rectangular region A1 of the reference frame FrN and the frame FrN + is 2 ×. Interpolation is performed based on the correspondence estimated by the number of divisions of 2. If correlation value d02 <correlation value d12, interpolation calculation is performed on the area corresponding to rectangular area A2 of reference frame FrN and frame FrN + based on the correspondence estimated by the number of divisions of 2 × 2. .

  On the other hand, in the case of correlation value d03> correlation value d13, an interpolation calculation is performed on the area corresponding to the rectangular area A3 of the reference frame FrN and the frame FrN + based on the correspondence estimated by the number of divisions of 4 × 4. . Further, in the case of correlation value d04> correlation value d14, an interpolation calculation is performed on the area corresponding to the rectangular area A4 of the reference frame FrN and the frame FrN + based on the correspondence estimated by the number of 4 × 4 divisions. . Then, a composite frame FrG composed of pixels obtained by the interpolation calculation is created.

  In this way, by estimating the correspondence used for creating the synthesized frame FrG for each rectangular area, the synthesized frame FrG is created for each reference patch and each divided rectangular area in the patch. For this reason, the composite frame FrG can be created with the optimal number of divisions following the movement of the subject included in each part in the frame, and thus, higher image quality can be achieved regardless of the movement of the subject included in the frame. A composite frame FrG can be obtained.

  In the first and second embodiments, the synthesized frame FrG is acquired for each of the luminance color difference components Y, Cb, and Cr of the reference frame FrN and the frame FrN + 1. However, the synthesized frame FrG is obtained only for the luminance component Y. For the color difference components Cb and Cr, the color difference components Cb and Cr of the reference frame FrN may be linearly interpolated to obtain a color difference component composite frame.

  If the frames FrN and FrN + 1 are composed of RGB color data, the composite frame FrG may be created by performing processing for each RGB color data.

In the first and second embodiments, the correlation values d0 and d1 are calculated for each of the luminance and chrominance components Y, Cb, and Cr. As shown in the following equation (14), for example, the correlation value is calculated. For d0, the correlation value d0 ′ of luminance component correlation value d0Y and the correlation values d0Cb and d0Cr of color difference component may be weighted and added by weighting coefficients a1, b1, and c1 to calculate one correlation value d0 ′.

Further, as shown in the following equation (15), the luminance component FrTiY (x, y) and the color difference components FrTiCb (x, y), FrTiCr (x, y) of the coordinate conversion frame FrTi, and FrNY ( x, y) and the Euclidean distance using the weight coefficients a2, b2, c2 with the color difference components FrNCb (x, y), FrNCr (x, y) are calculated for each pixel in the reference patch P0. The correlation value di ″ (x, y) of the pixel is used, and in the first embodiment, the average value of all the pixels in the reference patch P0 of the correlation value di ″ (x, y) is used in the second embodiment. The correlation value di ″ (x, y) may be an average value for all pixels in each rectangular area as a correlation value used for comparison.

  In the first and second embodiments, the composite frame FrG is created from the two frames FrN and FrN + 1. However, the composite frame FrG may be created from a plurality of three or more frames. For example, when creating a composite frame FrG from T frames FrN + t ′ (0 ≦ t ′ ≦ T−1), other frames FrN + t (1 ≦ t ≦ T−1) other than the reference frame FrN (= FrN + 0) For each, a plurality of correspondences with the reference frame FrN are estimated according to the number of divisions of the patch, and a plurality of coordinate conversion frames are acquired according to the plurality of correspondences, and a plurality of coordinate conversion frames and the reference frame FrN are obtained. A correlation value is calculated. Then, based on the correlation estimated based on the correspondence estimated by which division number, the composite frame is selected, and the first embodiment is selected based on the correspondence estimated by the selected division number. Create a composite frame in the same way as This synthesized frame is referred to as an intermediate synthesized frame FrGt. Note that the pixel value at integer coordinates of the integrated image of the intermediate composite frame FrGt is FrGt (x ^, y ^).

Then, the intermediate composite frame FrGt is obtained for all other frames FrN + t, and the intermediate composite frame FrGt is added between corresponding pixels by the following equation (16), whereby the pixel value FrG (x ^, y ^) is obtained. A composite frame FrG is created.

  Note that pixel values may not be assigned to all integer coordinates of the integrated image. In such a case, various interpolation operations such as a linear interpolation operation and a spline interpolation operation are performed on the assigned pixel value or the pixel value of the reference frame (corresponding to the coordinates not assigned), and the pixel value is What is necessary is just to calculate the pixel value of the integer coordinate which was not allocated.

  Further, in the case where the composite frame FrG is created from a plurality of three or more frames, when the intermediate composite frame FrGt is acquired, the correlation value is calculated for each rectangular area constituting the patch, as in the second embodiment. Then, the intermediate composite frame FrGt may be created based on the correspondence relationship for each rectangular area.

1 is a schematic block diagram showing the configuration of a moving image synthesis apparatus according to a first embodiment of the present invention. The figure for demonstrating estimation of the correspondence of frame FrN + 1 and the reference | standard flame | frame FrN Diagram for explaining patch deformation Diagram showing patches with different number of divisions The figure for demonstrating the correspondence of patch P1 and reference | standard patch P0 Diagram for explaining bilinear interpolation The figure for demonstrating allocation to the integrated image of frame FrN + 1 The figure for demonstrating the calculation of the pixel value of the integer coordinate in an integrated image A flowchart showing processing performed in the present embodiment The figure for demonstrating the process performed in the 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Sampling means 2 Correspondence relation estimation means 3 Coordinate conversion means 4 Correlation value calculation means 5 Comparison means 6 Composition means

Claims (10)

Sampling means for sampling two consecutive frames from a moving image;
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch A correspondence estimation means for estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame;
Based on the correspondence, coordinate conversion means for converting the image in the patch of the other frame to the coordinate space of the reference frame to obtain a coordinate-converted frame;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame;
The reference patch and the number of divisions of the rectangular area in the patch are changed in stages, and the correlation is estimated, the coordinate-transformed frame is obtained, and the correlation value is calculated in each stage. Control means for controlling the correspondence estimation means, the coordinate conversion means, and the correlation value calculation means so as to obtain a plurality of correlation values according to the patch and the number of divisions of the patch;
Comparing means for comparing the magnitude of the correlation for each of the number of divisions based on the plurality of correlation values;
Based on the correspondence estimated in the number of divisions that maximizes the correlation, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame, and A moving image synthesizing apparatus comprising: synthesizing means for creating a synthetic frame having a higher resolution than each frame.   2. The moving image synthesizing apparatus according to claim 1, wherein the correspondence used for creating the synthesized frame is estimated for each rectangular area.   The correspondence estimation means, the coordinate conversion means, the correlation value calculation means, the control means, and the synthesis means are configured to estimate the correspondence, acquire the coordinate-converted frame, calculate the correlation value, and calculate the plurality of correlations. The moving image composition according to claim 1 or 2, wherein said means is a means for performing acquisition of a value, comparison of the magnitude of said correlation and creation of said composite frame using at least one component constituting said frame. apparatus. Sampling means for sampling three or more consecutive frames from a moving image;
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Correspondence estimation means for estimating a correspondence between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame based on the reference patch;
Coordinate conversion means for converting the image in the patch of the other one frame into the coordinate space of the reference frame to obtain a coordinate-converted frame based on the correspondence relationship;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame;
The reference patch and the number of divisions of the rectangular area in the patch are changed in stages, and the correlation is estimated, the coordinate-transformed frame is obtained, and the correlation value is calculated in each stage. Control means for controlling the correspondence estimation means, the coordinate conversion means, and the correlation value calculation means so as to obtain a plurality of correlation values according to the patch and the number of divisions of the patch;
Comparing means for comparing the magnitude of the correlation for each of the number of divisions based on the plurality of correlation values;
Based on the correspondence estimated at the number of divisions that maximizes the correlation, an interpolation operation is performed on the image in the patch of the other one frame and the image in the reference patch of the reference frame. , Obtaining an intermediate composite frame having a higher resolution than each frame, and for all the other frames, estimating the correspondence, obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the plurality of correlation values. A synthesizing unit that obtains, compares the correlation magnitudes, and obtains the intermediate synthesized frame, and synthesizes a plurality of the intermediate synthesized frames obtained for all the other frames; An apparatus for synthesizing moving images.   5. The moving image synthesizing apparatus according to claim 4, wherein the correspondence used for creating the intermediate synthesized frame is estimated for each rectangular area.   The correspondence estimation means, the coordinate conversion means, the correlation value calculation means, the control means, and the synthesis means are configured to estimate the correspondence, acquire the coordinate-converted frame, calculate the correlation value, and calculate the plurality of correlations. 5. The means for performing acquisition of a value, comparison of the magnitude of the correlation, acquisition of the intermediate composite frame, and creation of the composite frame using at least one component constituting the frame. Or the moving image synthesizing device according to 5. Sampling two consecutive frames from a video,
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch Estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame,
Based on the correspondence relationship, the image in the patch of the other frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame,
Calculating a correlation value representing a correlation between the coordinate-transformed frame and an image in the reference patch of the reference frame;
The reference patch and the number of divisions of the rectangular area in the patch are changed in stages, and the correlation is estimated, the coordinate-transformed frame is obtained, and the correlation value is calculated in each stage. Obtain a plurality of correlation values according to the patch and the number of divisions of the patch,
Comparing the correlation magnitude for each of the number of divisions based on the plurality of correlation values;
Based on the correspondence estimated in the number of divisions that maximizes the correlation, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame, and A moving image synthesizing method, characterized in that a synthetic frame having a higher resolution than each frame is created. Sampling three or more consecutive frames from a video,
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Based on the reference patch, a correspondence relationship between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame is estimated,
Based on the correspondence, the image in the patch of the other one frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame;
Calculating a correlation value representing a correlation between the coordinate-transformed frame and an image in the reference patch of the reference frame;
The reference patch and the number of divisions of the rectangular area in the patch are changed in stages, and the correlation is estimated, the coordinate-transformed frame is obtained, and the correlation value is calculated in each stage. Obtain a plurality of correlation values according to the patch and the number of divisions of the patch,
Comparing the correlation magnitude for each of the number of divisions based on the plurality of correlation values;
Based on the correspondence estimated at the number of divisions that maximizes the correlation, an interpolation operation is performed on the image in the patch of the other one frame and the image in the reference patch of the reference frame. , Obtain an intermediate composite frame having a higher resolution than each of the frames,
For all the other frames, estimation of the correspondence, acquisition of the coordinate-transformed frame, calculation of the correlation value, acquisition of the plurality of correlation values, comparison of the correlation magnitudes, and acquisition of the intermediate composite frame And
A moving image synthesizing method, wherein a synthesized frame is created by synthesizing a plurality of the intermediate synthesized frames acquired for all the other frames. A procedure for sampling two consecutive frames from a moving image;
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch Estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame;
A procedure for obtaining a coordinate-converted frame by converting the image in the patch of the other frame into the coordinate space of the reference frame based on the correspondence relationship;
Calculating a correlation value representing a correlation between the coordinate-transformed frame and the image in the reference patch of the reference frame;
The reference patch and the number of divisions of the rectangular area in the patch are changed in stages, and the correlation is estimated, the coordinate-transformed frame is obtained, and the correlation value is calculated in each stage. A procedure for obtaining a plurality of correlation values according to the patch and the number of divisions of the patch;
A procedure for comparing the magnitude of the correlation for each of the number of divisions based on the plurality of correlation values;
Based on the correspondence estimated in the number of divisions that maximizes the correlation, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame, and A program for causing a computer to execute a moving image synthesizing method including a procedure for creating a synthesized frame having a higher resolution than each frame. A procedure for sampling three or more consecutive frames from a moving image;
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Estimating a correspondence relationship between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame based on the reference patch;
A procedure for obtaining a coordinate-converted frame by converting the image in the patch of the other one frame into the coordinate space of the reference frame based on the correspondence relationship;
Calculating a correlation value representing a correlation between the coordinate-transformed frame and the image in the reference patch of the reference frame;
The reference patch and the number of divisions of the rectangular area in the patch are changed in stages, and the correlation is estimated, the coordinate-converted frame is obtained, and the correlation value is calculated in each stage. A procedure for obtaining a plurality of correlation values according to the patch and the number of divisions of the patch;
A procedure for comparing the magnitude of the correlation for each of the number of divisions based on the plurality of correlation values;
Based on the correspondence estimated at the number of divisions that maximizes the correlation, interpolation is performed on the image in the patch of the other one frame and the image in the reference patch of the reference frame. Obtaining an intermediate composite frame having a higher resolution than each of the frames;
For all the other frames, estimation of the correspondence, acquisition of the coordinate-transformed frame, calculation of the correlation value, acquisition of the plurality of correlation values, comparison of the magnitude of the correlation, and acquisition of the intermediate composite frame The steps to do
A program for causing a computer to execute a moving image synthesizing method including a procedure for creating a synthesized frame by synthesizing a plurality of the intermediate synthesized frames acquired for all the other frames.

Images