JP2004236291A - Motion image compositing apparatus, method and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To obtain a high resolution frame wherein picture quality deterioration is suppressed regardless of a motion of a subject included in the frame when obtaining one high-resolution frame from a plurality of frames sampled from motion image data. <P>SOLUTION: A patch divided into one or more rectangular regions is disposed in each of the plurality of frames FrN, FrN+1, the patch is moved and/or deformed to estimate the corresponding relationship of the frames, coordinates of the frame FrN+1 are transformed on the basis of the estimated corresponding relationship, and a correlative value of this frame and the frame FrN is calculated. The number of divided regions in the patch is increased for one stage to similarly calculate the correlative value. A quantity of changes between the two correlative values is calculated and compared with a threshold value and until the quantity of the changes becomes less than or equal to the threshold value, the number of divided regions is increased to similarly calculate the quantity of changes. On the basis of the corresponding relationship estimated with the smaller number of divided regions between the two numbers of divided regions in the two correlative values wherein the quantity of changes less than or equal to the threshold value is calculated, a synthetic frame FrG is prepared from the frames FrN, FrN+1. <P>COPYRIGHT: (C)2004,JPO&NCIPI

Description

  The present invention relates to a moving image synthesizing apparatus and method, and a moving image synthesizing method for synthesizing a plurality of consecutive frames obtained by sampling moving images to create one synthesized frame having a higher resolution than the sampled frames. The present invention relates to a program for causing a computer to execute.

  With the recent spread of digital video cameras, it is possible to handle moving images in units of frames. When printing out a frame of such a moving image, it is necessary to make the frame high resolution in order to improve the image quality. For this reason, a method of creating one composite frame having a higher resolution than those frames from a plurality of frames sampled from a moving image has been proposed (for example, see Patent Document 1). In this method, a motion vector between a plurality of frames is obtained, and a signal value to be interpolated between pixels is calculated based on the motion vector when a synthesized frame is synthesized from a plurality of frames. In particular, in the method described in Patent Document 1, each frame is divided into a plurality of blocks, the orthogonal coordinate coefficients of the corresponding blocks between the frames are calculated, and high-frequency information in the orthogonal coordinate coefficients is used as low-frequency information in other blocks. Since the pixel value to be interpolated by combining with this information is calculated, a high-quality synthesized frame can be obtained without reducing necessary information. Further, in this method, since the motion vector is calculated with a resolution finer than the inter-pixel distance, it is possible to obtain a synthesized frame with higher image quality by accurately compensating for the motion between frames.

  Also, one of the plurality of frames is set as a reference frame, a reference patch composed of a plurality of rectangular areas is arranged in the reference frame, and a patch similar to the reference patch is arranged in a frame other than the reference frame. Move and / or deform the patch on the other frame so that the image matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch, the pixels in the patch on the other frame A method has also been proposed in which a correspondence relationship with pixels in a reference patch on a reference frame is estimated to synthesize a plurality of frames more accurately (see, for example, Non-Patent Document 1).

In the method of Non-Patent Document 1, the correspondence between the reference frame and other frames is estimated, and after the estimation, the other frames and the reference frame are finally allocated on the integrated image having the necessary resolution. A high-definition composite frame can be obtained.
JP 2000-354244 A Yuji Nakazawa, Takashi Komatsu, Takahiro Saito, “Acquisition of high-definition digital images by interframe integration”, Journal of the Television Society, 1995, Vol. 49, no. 3, p299-308

  However, in the method described in Non-Patent Document 1, when the movement of the subject included in another frame is very large, or the subject included locally moves in a complicated manner or moves at a very high speed. In some cases, the movement and / or deformation of the patch cannot follow the movement of the subject. In this way, if the movement and / or deformation of the patch cannot follow the movement and / or deformation of the subject, the entire composite frame will be blurred, or the subject with large movement included in the frame will be blurred, so high-quality composition is possible. There is a problem that a frame cannot be obtained. In this case, by dividing the patch into more rectangular areas, the movement and / or deformation of the subject can follow the movement and / or deformation of the subject. However, if the number of divisions is too large, the rectangular area becomes too small, and it becomes difficult to make the movement and / or deformation of the subject follow the movement and / or deformation of the subject.

  The present invention has been made in view of the above circumstances, and an object thereof is to obtain a high-quality composite frame with an optimal number of patch divisions.

A first moving image synthesizing apparatus according to the present invention comprises sampling means for sampling two consecutive frames from a moving image;
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch A correspondence estimation means for estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame;
Based on the correspondence, coordinate conversion means for converting the image in the patch of the other frame to the coordinate space of the reference frame to obtain a coordinate-converted frame;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame;
The correspondence estimation means for increasing the number of divisions of the reference patch and the rectangular area in the patch by one step so as to estimate the correspondence, acquire the coordinate-transformed frame, and calculate the correlation value; Control means for controlling the coordinate conversion means and the correlation value calculation means;
A change amount calculating means for calculating a change amount of the two correlation values;
The number of divisions of the reference patch and the rectangular area in the patch is increased stepwise until the amount of change becomes a predetermined threshold value or less, and the correspondence is estimated at each step, and the coordinate-converted frame The correlation estimating means, the coordinate converting means, the correlation value calculating means, and the correlation value calculating means, so as to calculate the amount of change of two correlation values calculated by the acquisition of the correlation value and the number of divisions at adjacent stages. Comparison means for controlling the change amount calculation means;
Based on the correspondence relationship estimated in the number of divisions in which one of the two correlation values when the change amount is equal to or less than the predetermined threshold is calculated, the other frame And a synthesizing unit that performs an interpolation operation on the image in the patch and the image in the reference patch of the reference frame to create a synthesized frame having a higher resolution than the frames. It is.

  “Increase the number of divisions stepwise” means to gradually increase the number of divisions from a small number of divisions until the correlation reaches a predetermined threshold value or more. If the number of divisions is too large, the rectangular area becomes too small, and there are many rectangular areas that represent similar images. On the contrary, the movement and / or deformation of the patch follows the movement and / or deformation of the subject. There are cases where it is not possible. For this reason, it is preferable to set the upper limit of the number of divisions.

  “Increasing the number of divisions by one step” means increasing the number of one-step divisions in the number of divisions increasing stepwise.

  The “correlation value” may be calculated as representing the correlation between the entire coordinate-converted frame and the entire image in the reference patch of the reference frame. Specifically, all the pixels in the patch of the correlation value of each pixel An average value or an addition value can be used.

  The “change amount of two correlation values” is calculated from a correlation value calculated at a stage of a certain number of divisions (including an initial stage at which the correspondence is first estimated) and the number of divisions increased by one stage from there. This represents how much the correlation value differs. Specifically, the difference value between the two correlation values, the absolute value of the difference value, or the like can be used.

A second moving image synthesizing apparatus according to the present invention comprises sampling means for sampling two consecutive frames from a moving image;
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch A correspondence estimation means for estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame;
Based on the correspondence, coordinate conversion means for converting the image in the patch of the other frame to the coordinate space of the reference frame to obtain a coordinate-converted frame;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame in units of the rectangular area;
The correspondence estimation means for increasing the number of divisions of the reference patch and the rectangular area in the patch by one step so as to estimate the correspondence, acquire the coordinate-transformed frame, and calculate the correlation value; Control means for controlling the coordinate conversion means and the correlation value calculation means;
A change amount calculating means for calculating a change amount of the two correlation values in units of the rectangular area;
The change in a predetermined number of rectangular areas or rectangular areas at predetermined positions among all rectangular areas or rectangular areas other than the rectangular areas in which the amount of change is equal to or greater than a predetermined value among all the rectangular areas. The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the representative value of the amount or the change amount is equal to or less than a predetermined threshold, and the correspondence is estimated at each step. The correspondence estimation means, obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the amount of change of the two correlation values calculated based on the number of adjacent divisions in the rectangular area unit, Comparison means for controlling the coordinate conversion means, the correlation value calculation means, and the change amount calculation means;
The change amount or the representative value of the change amount is equal to or less than the predetermined threshold value for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated. Based on the correspondence relationship, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame in units of areas corresponding to the rectangular areas, and from each frame And a synthesizing unit for creating a synthetic frame with a high resolution.

  The “correlation value” may be calculated as representing the correlation of the corresponding rectangular area between the coordinate-converted frame and the image in the reference patch of the reference frame. Specifically, the correlation value is within the rectangular area of the correlation value of each pixel. An average value or an addition value for all previous pixels can be used.

  "A rectangular area in which the amount of change exceeds a predetermined value among all rectangular areas" means that the correlation between the rectangular areas is very low, and the amount of change is a predetermined threshold value even if the number of divisions is increased. A rectangular area that does not become the following.

  The “predetermined number of rectangular areas” means “all rectangular areas” or “rectangular areas other than rectangular areas in which the amount of change exceeds a predetermined value among all rectangular areas” (hereinafter referred to as target rectangular areas) At least one rectangular area of the target rectangular area, or one or a plurality of rectangular areas of the target rectangular area, or all of the target rectangular areas.

  In addition, in the case where the ratio of the rectangular area in which the amount of change is equal to or less than a predetermined threshold or the area in the reference patch in the target rectangular area is equal to or greater than a predetermined ratio (for example, 50%), The number of rectangular areas where the amount of change is less than or equal to a predetermined threshold may be a predetermined number of rectangular areas.

  The “rectangular area at a predetermined position” means a rectangular area at a position where a main subject is included in the reference patch or the patch. Specifically, a rectangular area at a position including a main subject such as a human face or a complex texture, a rectangular area positioned near the center of the reference patch or patch can be used.

  “Representative value of change amount” is a value representative of the change amount of a predetermined number of rectangular areas or rectangular areas at predetermined positions, and specifically, the average value, median value, maximum value, and minimum value of the change amount Etc. can be used.

  “Of the rectangular areas other than the rectangular areas in which all the rectangular areas or all the rectangular areas have the amount of change equal to or greater than a predetermined value, the predetermined number of rectangular areas or the rectangular areas at predetermined positions The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the change amount or the representative value of the change amount becomes a predetermined threshold value or less, and the correspondence is estimated at each step. `` Acquisition of the coordinate-converted frame, calculation of the correlation value, and calculation of the amount of change in the unit of the rectangular area of two correlation values calculated by the number of divisions in adjacent stages '' In the stage, a change amount or a representative value of the change amount is a predetermined threshold value for a predetermined number of rectangular regions or a rectangular region at a predetermined position among the reference patch and the target rectangular regions in the patch. The amount of change may be calculated until the following is satisfied, and a predetermined number of rectangular areas or a part of a rectangular area at a predetermined position among the reference patch and the target rectangular area in the patch in a certain number of divisions. If the change amount or the representative value of the change amount is less than or equal to a predetermined threshold value for only the rectangular area, the next stage rectangular area is divided only for other rectangular areas other than the rectangular area. The amount of change may be calculated by increasing the number of divisions only for other rectangular areas. In the latter case, the calculation time for calculating the amount of change can be shortened compared to the former.

  Here, in the second moving image synthesizing apparatus according to the present invention, when the change amount or the representative value of the change amount is equal to or less than a predetermined threshold, the number of divisions is different between the reference patch and the local region in the patch. There is. For this reason, "interpolation calculation is performed in units of areas corresponding to rectangular areas" means that the number of divisions when the amount of change is equal to or less than a predetermined threshold is different between the reference patch and the local area in the patch. Using the correspondence estimated by the division number of the rectangular area corresponding to the reference patch and the local area in the patch, the interpolation calculation is performed based on the different correspondence for each area corresponding to the reference patch and the rectangular area in the patch. Means that.

  In the first and second moving image synthesizing apparatuses according to the present invention, the control means increases the predetermined threshold value as the number of divisions of the rectangular area increases beyond a certain value. It is good also as a means to do.

  In the first and second moving image synthesizing apparatuses according to the present invention, the correspondence estimation means, the coordinate conversion means, the correlation value calculation means, the control means, the change amount calculation means, the comparison means, and the The synthesizing unit performs estimation of the correspondence, acquisition of the coordinate-transformed frame, calculation of the correlation value, calculation of the change amount, and creation of the synthesized frame using at least one component constituting the frame. It may be a means.

  “At least one component constituting the frame” is, for example, at least one component of RGB color components when the frame is composed of three RGB color data, and when composed of YCC luminance color difference components, At least one of the luminance and color difference components, preferably the luminance component.

A third moving image synthesizing apparatus according to the present invention comprises sampling means for sampling three or more consecutive frames from a moving image;
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Correspondence estimation means for estimating a correspondence between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame based on the reference patch;
Coordinate conversion means for converting the image in the patch of the other one frame into the coordinate space of the reference frame to obtain a coordinate-converted frame based on the correspondence relationship;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame;
The correspondence estimation means for increasing the number of divisions of the reference patch and the rectangular area in the patch by one step so as to estimate the correspondence, acquire the coordinate-transformed frame, and calculate the correlation value; Control means for controlling the coordinate conversion means and the correlation value calculation means;
A change amount calculating means for calculating a change amount of the two correlation values;
The number of divisions of the reference patch and the rectangular area in the patch is increased stepwise until the amount of change becomes a predetermined threshold value or less, and the correspondence is estimated at each step, and the coordinate-converted frame The correlation estimating means, the coordinate converting means, the correlation value calculating means, and the correlation value calculating means, so as to calculate the amount of change of two correlation values calculated by the acquisition of the correlation value and the number of divisions at adjacent stages. Comparison means for controlling the change amount calculation means;
Based on the correspondence relationship estimated in the number of divisions in which one of the two correlation values when the change amount is equal to or less than the predetermined threshold is calculated, the other frame Interpolation is performed on the image in the patch and the image in the reference patch of the reference frame to obtain an intermediate composite frame having a resolution higher than that of each frame, and the correspondence for all the other frames Estimate the relationship, acquire the coordinate-transformed frame, calculate the correlation value, calculate the amount of change, and acquire the intermediate composite frame, and obtain the plurality of intermediate composite frames acquired for all the other frames. And a synthesizing unit that creates a synthesized frame by synthesizing.

A fourth moving image synthesizing apparatus according to the present invention comprises sampling means for sampling three or more consecutive frames from a moving image;
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Correspondence estimation means for estimating a correspondence between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame based on the reference patch;
Coordinate conversion means for converting the image in the patch of the other one frame into the coordinate space of the reference frame to obtain a coordinate-converted frame based on the correspondence relationship;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame in units of the rectangular area;
The correspondence estimation means for increasing the number of divisions of the reference patch and the rectangular area in the patch by one step so as to estimate the correspondence, acquire the coordinate-transformed frame, and calculate the correlation value; Control means for controlling the coordinate conversion means and the correlation value calculation means;
A change amount calculating means for calculating a change amount of the two correlation values in units of the rectangular area;
The change in a predetermined number of rectangular areas or rectangular areas at predetermined positions among all rectangular areas or rectangular areas other than the rectangular areas in which the amount of change is equal to or greater than a predetermined value among all the rectangular areas. The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the representative value of the amount or the change amount is equal to or less than a predetermined threshold, and the correspondence is estimated at each step. The correspondence estimation means, obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the amount of change of the two correlation values calculated based on the number of adjacent divisions in the rectangular area unit, Comparison means for controlling the coordinate conversion means, the correlation value calculation means, and the change amount calculation means;
The change amount or the representative value of the change amount is equal to or less than the predetermined threshold value for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated. Based on the correspondence relationship, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame in units of areas corresponding to the rectangular areas, and from each frame Get an intermediate composite frame with high resolution,
For all the other frames, the correspondence is estimated, the coordinate-transformed frame is acquired, the correlation value is calculated, the change amount is calculated, and the intermediate composite frame is acquired. And a synthesizing unit for synthesizing the plurality of acquired intermediate synthesis frames to create a synthesis frame.

  In the third and fourth video synthesizing apparatuses according to the present invention, the control means increases the predetermined threshold value as the number of divisions of the rectangular area increases beyond a certain value. It is good also as a means to do.

  In the third and fourth moving image synthesizing apparatuses according to the present invention, the correspondence estimation means, the coordinate conversion means, the correlation value calculation means, the control means, the change amount calculation means, the comparison means, and the Combining means for estimating the correspondence, acquiring the coordinate-transformed frame, calculating the correlation value, calculating the amount of change, acquiring the intermediate combined frame, and creating the combined frame at least configure the frame It is good also as a means to perform using one component.

A first moving image composition method according to the present invention samples two consecutive frames from a moving image,
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch Estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame,
Based on the correspondence relationship, the image in the patch of the other frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame,
Calculating a correlation value representing a correlation between the coordinate-transformed frame and an image in the reference patch of the reference frame;
Increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value,
A change amount of the two correlation values is calculated;
The number of divisions of the reference patch and the rectangular area in the patch is increased stepwise until the amount of change becomes a predetermined threshold value or less, and the correspondence is estimated at each step, and the coordinate-converted frame Obtaining the correlation value, calculating the correlation value, and calculating the amount of change between the two correlation values calculated by the number of divisions in the adjacent stages,
Based on the correspondence relationship estimated in the number of divisions in which one of the two correlation values when the change amount is equal to or less than the predetermined threshold is calculated, the other frame An interpolation operation is performed on the image in the patch and the image in the reference patch of the reference frame to create a composite frame having a higher resolution than each frame.

A second moving image composition method according to the present invention samples two consecutive frames from a moving image,
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch Estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame,
Based on the correspondence relationship, the image in the patch of the other frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame,
Calculating a correlation value representing the correlation between the coordinate-converted frame and the image in the reference patch of the reference frame in units of the rectangular area;
Increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value,
The amount of change of the two correlation values is calculated in units of the rectangular area,
The change in a predetermined number of rectangular areas or rectangular areas at predetermined positions among all rectangular areas or rectangular areas other than the rectangular areas in which the amount of change is equal to or greater than a predetermined value among all the rectangular areas. The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the representative value of the amount or the change amount is equal to or less than a predetermined threshold, and the correspondence is estimated at each step. Obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the amount of change in units of the rectangular area of two correlation values calculated by the number of divisions in adjacent stages,
The change amount or the representative value of the change amount is equal to or less than the predetermined threshold value for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated. Based on the correspondence relationship, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame in units of areas corresponding to the rectangular areas, and from each frame Also, a composite frame having a high resolution is created.

A third moving image composition method according to the present invention samples three or more consecutive frames from a moving image,
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Based on the reference patch, a correspondence relationship between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame is estimated,
Based on the correspondence, the image in the patch of the other one frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame;
Calculating a correlation value representing a correlation between the coordinate-transformed frame and an image in the reference patch of the reference frame;
Increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value,
A change amount of the two correlation values is calculated;
The number of divisions of the reference patch and the rectangular area in the patch is increased stepwise until the amount of change becomes a predetermined threshold value or less, and the correspondence is estimated at each step, and the coordinate-converted frame Obtaining the correlation value, calculating the correlation value, and calculating the amount of change between the two correlation values calculated by the number of divisions in the adjacent stages,
Based on the correspondence relationship estimated in the number of divisions in which one of the two correlation values when the change amount is equal to or less than the predetermined threshold is calculated, the other frame Interpolating the image in the patch and the image in the reference patch of the reference frame to obtain an intermediate composite frame having a higher resolution than each of the frames,
For all the other frames, the correspondence is estimated, the coordinate-transformed frame is acquired, the correlation value is calculated, the change amount is calculated, and the intermediate composite frame is acquired.
A synthesized frame is created by synthesizing a plurality of the intermediate synthesized frames acquired for all the other frames.

A fourth moving image composition method according to the present invention samples three or more consecutive frames from a moving image,
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Based on the reference patch, a correspondence relationship between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame is estimated,
Based on the correspondence, the image in the patch of the other one frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame;
Calculating a correlation value representing the correlation between the coordinate-converted frame and the image in the reference patch of the reference frame in units of the rectangular area;
Increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value,
The amount of change of the two correlation values is calculated in units of the rectangular area,
The change in a predetermined number of rectangular areas or rectangular areas at predetermined positions among all rectangular areas or rectangular areas other than the rectangular areas in which the amount of change is equal to or greater than a predetermined value among all the rectangular areas. The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the representative value of the amount or the change amount becomes a predetermined threshold value or less, and the correspondence is estimated at each step. Obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the amount of change in units of the rectangular area of two correlation values calculated by the number of divisions in adjacent stages,
The change amount or the representative value of the change amount is equal to or less than the predetermined threshold value for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated. Based on the correspondence relationship, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame in units of areas corresponding to the rectangular areas, and from each frame Get an intermediate composite frame with high resolution,
For all the other frames, the correspondence is estimated, the coordinate-transformed frame is acquired, the correlation value is calculated, the change amount is calculated, and the intermediate composite frame is acquired.
A synthesized frame is created by synthesizing a plurality of the intermediate synthesized frames acquired for all the other frames.

  In addition, you may provide as a program for making a computer perform the 1st-4th moving image composition method by this invention.

  According to the first moving image synthesizing apparatus and method of the present invention, a moving image is sampled to obtain a plurality of continuous frames, and one or a plurality of frames on a reference frame serving as a reference among the plurality of frames. A reference patch having a rectangular area is arranged. A patch similar to the reference patch is arranged on a frame other than the reference frame. Then, the image in the patch is moved and / or deformed so as to match the image in the reference patch, and based on the moved and / or deformed patch and the reference patch, A correspondence relationship with the pixels in the reference patch on the reference frame is estimated.

  Further, based on the correspondence, the image in the patch of the other frame is transformed into the coordinate space of the reference frame to obtain the coordinate-converted frame, and the correlation between the coordinate-converted frame and the image in the reference patch of the reference frame is obtained. Is calculated. Then, the number of divisions of the reference patch and the rectangular area in the patch is increased by one stage, the correspondence is estimated, the coordinate converted frame is obtained, and the correlation value is calculated. The amount of change between the two correlation values is calculated.

  Then, the number of divisions of the reference patch and the rectangular area in the patch is increased step by step until the change amount becomes a predetermined threshold value or less, and in each step, the correspondence is estimated, the coordinate converted frame is obtained, and the correlation value And the amount of change of two correlation values calculated by the number of divisions at adjacent stages is calculated.

  Based on the correspondence estimated in the number of divisions for which one of the two correlation values when the amount of change is equal to or less than a predetermined threshold is calculated, Interpolation is performed on the image and the image in the reference frame patch to create a composite frame having a higher resolution than each frame.

  Note that, when three or more frames are sampled as in the third moving image synthesizing apparatus and method according to the present invention, the amount of change is less than or equal to a predetermined threshold value for all other frames other than the reference frame. Correspondence estimation, acquisition of coordinate-transformed frames, calculation of correlation value and calculation of change amount are performed until a plurality of intermediate composite frames are acquired from each of the reference frame and other frames, and a plurality of intermediate frames are further acquired. A composite frame is created from the composite frame.

  For this reason, it is possible to create a composite frame with an optimal number of divisions that can follow the movement and / or deformation of a moving subject within the frame, thereby enabling high image quality regardless of the movement of the subject included in the frame. Can be obtained.

  Further, according to the second moving image synthesizing apparatus and method of the present invention, a moving image is sampled to obtain a plurality of continuous frames, and one or more of the plurality of frames is set on one reference frame as a reference. A reference patch having a plurality of rectangular areas is arranged. A patch similar to the reference patch is arranged on a frame other than the reference frame. Then, the image in the patch is moved and / or deformed so as to match the image in the reference patch, and based on the moved and / or deformed patch and the reference patch, A correspondence relationship with the pixels in the reference patch on the reference frame is estimated.

  Further, based on the correspondence, the image in the patch of the other frame is transformed into the coordinate space of the reference frame to obtain the coordinate-converted frame, and the correlation between the coordinate-converted frame and the image in the reference patch of the reference frame is obtained. Is calculated in units of rectangular areas. Then, the number of divisions of the reference patch and the rectangular area in the patch is increased by one stage, the correspondence is estimated, the coordinate converted frame is obtained, and the correlation value is calculated. The amount of change between the two correlation values is calculated in units of rectangular areas.

  Then, among the rectangular areas other than the rectangular areas where the amount of change is equal to or greater than a predetermined value among all the rectangular areas or all the rectangular areas, the amount of change in the predetermined number of rectangular areas or the rectangular areas at the predetermined positions, or The reference patch and the number of divisions of the rectangular area in the patch are changed in stages until the representative value of the amount of change is equal to or less than the predetermined threshold. Correspondence estimation, acquisition of coordinate-converted frames, and correlation are performed at each stage. Value calculation and change calculation are performed.

  Then, among the two correlation values when the change amount or the representative value of the change amount is equal to or less than a predetermined threshold value, the correspondence relationship for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated Based on this, an interpolation operation is performed on the image in the patch of the other frame and the image in the patch of the reference frame in units of regions corresponding to the rectangular regions, and a synthesized frame having a higher resolution than each frame is created. .

  As in the fourth moving image synthesizing apparatus and method according to the present invention, when three or more frames are sampled, the change amount is set to a predetermined threshold value or less for all other frames other than the reference frame. Correspondence estimation, acquisition of coordinate-transformed frames, calculation of correlation value and calculation of change amount are performed until a plurality of intermediate composite frames are acquired from each of the reference frame and other frames, and a plurality of intermediate frames are further acquired. A composite frame is created from the composite frame.

  For this reason, a composite frame can be created with an optimal number of divisions that follow the movement of the subject included in each part of the frame, thereby enabling a higher quality composite regardless of the movement of the subject included in the frame. You can get a frame.

  In addition, the predetermined threshold value to be compared with the amount of change or the representative value of the amount of change is increased as the number of divisions of the rectangular area increases beyond a certain value, so that the number of divisions of the rectangular area is appropriately set. Can be set to

  Also, the frame is composed of estimation of correspondence, acquisition of coordinate-converted frames, calculation of correlation values, calculation of change amounts, acquisition of intermediate composite frames (when three or more frames are sampled), and generation of composite frames By using at least one component, it is possible to obtain a composite frame with reduced image quality degradation for each component, thereby obtaining a high-quality composite frame composed of composite frames for each component. .

  Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 is a schematic block diagram showing the configuration of a moving image synthesis apparatus according to the first embodiment of the present invention. As shown in FIG. 1, the moving image synthesizing apparatus according to the first embodiment includes sampling means 1 for sampling a plurality of frames from input moving image data M0, and one reference serving as a reference among the plurality of frames. Correspondence relationship estimation means 2 for estimating the correspondence relationship between the pixels of the frame and the pixels of other frames other than the reference frame, and based on the correspondence relationship estimated by the correspondence relationship estimation means 2, each of the other frames is represented by the coordinates of the reference frame. Coordinate conversion means 3 for obtaining a coordinate-converted frame (hereinafter referred to as a coordinate conversion frame) FrTi in space, and a correlation value calculation for calculating a correlation value representing a correlation between the coordinate conversion frame FrTi and the reference frame Based on the correlation value calculated by stepwise increasing the number of patch divisions for estimating the correspondence relationship and means 4 as will be described later. Then, the change amount calculating means 5 for calculating the change amount of the two correlation values calculated by the number of divisions at the preceding and following stages is compared with the threshold value Th1, and the change amount is equal to or less than the threshold value Th1. As described later, the number of divisions of the patch for estimating the correspondence is increased step by step until the correspondence is estimated in the correspondence estimation unit 2, the coordinate conversion unit 3, the correlation value calculation unit 4, and the change amount calculation unit 5. Comparison means 6 for performing estimation, acquisition of coordinate conversion frame, calculation of correlation value and calculation of change amount, and correlation value of one of two correlation values when the change amount becomes equal to or less than threshold value Th1. Based on the correspondence estimated by the correspondence estimation means 2 in the calculated number of divisions, a composite frame Fr having higher resolution than each frame by performing an interpolation operation on the reference frame and other frames. And a synthesizing unit 7 to create. In the present embodiment, it is assumed that the synthesized frame FrG has twice as many pixels as the sampled frame. In the following description, the composite frame FrG is described as having twice as many pixels as the sampled frame, but it may be n times (n: positive number).

  The sampling means 1 samples a plurality of frames from the moving image data M0. In this embodiment, the sampling means 1 samples two frames FrN and FrN + 1 from the moving image data M0. Note that the frame FrN is a reference frame. Here, the moving image data M0 represents a color moving image, and the frames FrN and FrN + 1 are composed of luminance, color difference components of Y, Cb, and Cr. In the following description, processing is performed for each component of Y, Cb, and Cr. Since the processing to be performed is the same for all components, the processing of the luminance component Y is described in detail in the present embodiment. A description of the processing for the color difference components Cb and Cr will be omitted.

  The correspondence estimation means 2 estimates the correspondence between the frame FrN + 1 and the reference frame FrN as follows. FIG. 2 is a diagram for explaining the estimation of the correspondence between the frame FrN + 1 and the reference frame FrN. In FIG. 2, it is assumed that the circular subject included in the reference frame FrN has moved slightly to the right in the drawing in the frame FrN + 1.

First, the correspondence estimation means 2 first arranges a reference patch P0 composed of one or a plurality of rectangular areas on the reference frame FrN. FIG. 2A shows a state in which the reference patch P0 is arranged on the reference frame FrN. As shown in FIG. 2A, in this embodiment, the reference patch P0 is divided into 2 × 2 rectangular areas. Next, as shown in FIG. 2B, a patch P1 similar to the reference patch P0 is arranged at an appropriate position in the frame FrN + 1, and a correlation value representing the correlation between the image in the reference patch P0 and the image in the patch P1. Is calculated. The correlation value can be calculated as a mean square error by the following equation (1). The coordinate axes are assumed to be the x axis in the left and right direction on the paper and the y axis in the vertical direction on the paper.

Where E: correlation value pi, qi: pixel value of the corresponding pixel in each of the reference patches P0, P1 N: number of pixels in the reference patch P0 and patch P1

  Next, the patch P1 on the frame FrN + 1 is moved by fixed pixels ± Δx, ± Δy in four directions, up, down, left, and right. The correlation value between the image in the patch P1 and the image in the reference patch P0 on the reference frame FrN is calculate. Here, the correlation value is calculated for each of the up, down, left, and right directions, and each correlation value is expressed as E (Δx, 0), E (−Δx, 0), E (0, Δy), E (0, −Δy), respectively. To do.

Then, the correlation value becomes smaller (that is, the correlation becomes larger) from the four correlation values E (Δx, 0), E (−Δx, 0), E (0, Δy), and E (0, −Δy) after the movement. The gradient direction is obtained as the correlation gradient, and the patch P1 is moved in this direction by a preset real value multiple as shown in FIG. Specifically, coefficients C (Δx, 0), C (−Δx, 0), C (0, Δy), C (0, −Δy) are calculated by the following equation (2), and these coefficients C Correlation gradients gx and gy are calculated from the following equations (3) and (4) from (Δx, 0), C (−Δx, 0), C (0, Δy), and C (0, −Δy).

  Then, the entire patch P1 is moved (−λ1gx, −λ1gy) based on the calculated correlation gradients gx, gy, and the same processing as described above is repeated, so that the patch P1 as shown in FIG. The patch P1 is repeatedly moved until it converges to a certain position. Here, λ1 is a parameter that determines the speed of convergence, and takes a real value. If λ1 is too large, the solution is diverged by iterative processing, so an appropriate value (for example, 10) needs to be selected.

  Further, the lattice point of the patch P1 is moved by a fixed pixel in four directions along the coordinate axis. At this time, the rectangular area including the moved grid point is deformed as shown in FIG. 3, for example. Then, a correlation value between the deformed rectangular area and the corresponding rectangular area of the reference patch P0 is calculated. The correlation values are defined as E1 (Δx, 0), E1 (−Δx, 0), E1 (0, Δy), and E1 (0, −Δy), respectively.

  Similarly to the above, the correlation value is small from the four correlation values E1 (Δx, 0), E1 (−Δx, 0), E1 (0, Δy), and E1 (0, −Δy) after deformation (ie, The gradient direction (in which the correlation is large) is obtained, and the grid point of the patch P1 is moved in this direction by a preset real value multiple. This is performed for all the grid points of the patch P1, and this is regarded as one process. This process is repeated until the coordinates of the grid points converge.

  Thereby, the movement amount and the deformation amount of the patch P1 with respect to the reference patch P0 are obtained, and the correspondence between the pixels in the reference patch P0 and the pixels in the patch P1 can be estimated based on this.

  Based on the estimated correspondence, the coordinate conversion unit 3 performs coordinate conversion of the frame FrN + 1 into the coordinate space of the reference frame FrN as follows to obtain a coordinate conversion frame FrT0. In the following description, conversion and composition are performed only for the region in the reference patch P0 of the reference frame FrN and the region in the patch P1 of the frame FrN + 1.

In the present embodiment, coordinate transformation is performed using bilinear transformation. Coordinate transformation by bilinear transformation is defined by the following equations (5) and (6).

  Expressions (5) and (6) express coordinates in the patch P1 given by four points (xn, yn) (1 ≦ n ≦ 4) on a two-dimensional coordinate in a normalized coordinate system (u, v) ( 0 ≦ u, v ≦ 1), and coordinate transformation in any two rectangles is performed by combining the inverse transformations of equations (5) and (6) and equations (5) and (6). It can be carried out.

  Here, as shown in FIG. 4, it is considered which position in the reference patch P0 (x′n, y′n) the point (x, y) in the patch P1 (xn, yn) corresponds to. . First, normalized coordinates (u, v) are obtained for the point (x, y) in the patch P1 (xn, yn). This is obtained by inverse transformation of equations (5) and (6). Based on the reference patch P0 (x′n, y′n) corresponding to (u, v) at this time, the coordinates (x ′) corresponding to the point (x, y) from the equations (5) and (6) , Y ′). Here, since the point (x, y) is an integer coordinate where the pixel value originally exists, the point (x ′, y ′) may be a real number coordinate where the pixel value originally does not exist. As for the pixel value in the integer coordinates, an area surrounded by integer coordinates in the vicinity of 8 adjacent to the integer coordinates of the reference patch P0 is set, and the pixel value of the converted coordinates (x ′, y ′) is set in this area. It shall be obtained as the load sum.

Specifically, as shown in FIG. 5, for integer coordinates b (x, y) on the reference patch P0, integer coordinates b (x-1, y-1) and b (x, y-1) in the vicinity of the eight coordinates. ), B (x + 1, y-1), b (x-1, y), b (x + 1, y), b (x-1, y + 1), b (x, y + 1), b (x + 1, y + 1) Calculation is performed based on the pixel value of the frame FrN + 1 converted in the enclosed area. Here, m pixel values of the frame FrN + 1 are converted into an area surrounded by eight neighboring pixels, and the converted pixel value of each pixel is Itj (x °, y °) (1 ≦ j ≦ m). Then, the pixel value It (x ^, y ^) at the integer coordinates b (x, y) can be calculated by the following equation (7). In Expression (7), φ is a function representing the load sum calculation.

  However, Wj (1 ≦ j ≦ m): product of the internal ratio of coordinates as viewed from neighboring integer pixels at the position where the pixel value Itj (x °, y °) is assigned.

Here, for the sake of simplicity, consider the case where the two pixel values It1 and It2 of the frame FrN + 1 are transformed into the region surrounded by the eight neighboring pixels with reference to FIG. 5 at the integer coordinate b (x, y). The pixel value It (x ^, y ^) can be calculated by the following equation (8).

  However, W1 = u × v, W2 = (1-s) × (1-t)

  By performing the above processing for all the pixels in the patch P1, the image in the patch P1 is converted into the coordinate space of the reference frame FrN, and the coordinate conversion frame FrT0 is acquired.

The correlation value calculation means 4 calculates a correlation value d0 between the coordinate conversion frame FrT0 and the reference frame FrN. Specifically, as shown in the following formula (9), the absolute value of the difference between the pixel values FrT0 (x, y) and FrN (x, y) in the corresponding pixels of the coordinate conversion frame FrT0 and the reference frame FrN. The value obtained by dividing the added value for all the pixels m in the reference patch P0 by the number m of pixels in the reference patch P0, that is, the average value of all the pixels in the reference patch P0 of the absolute value of the difference for each pixel is correlated. Calculated as the value d0. The correlation value d0 becomes smaller as the correlation between the coordinate conversion frame FrT0 and the reference frame FrN increases.

  In the present embodiment, the correlation value d0 is calculated from the absolute value of the difference between the pixel values of the corresponding pixels of the coordinate conversion frame FrT0 and the reference frame FrN. However, the correlation value may be calculated from the square of the difference. . Further, an addition value of absolute values of pixel value differences in corresponding pixels of the coordinate conversion frame FrT0 and the reference frame FrN may be used as the correlation value. Further, the histograms of the coordinate conversion frame FrT0 and the reference frame FrN are calculated, respectively, and the average value of the histograms of the coordinate conversion frame FrT0 and the reference frame FrN, the difference value of the median value or the standard deviation, or the cumulative sum of the difference values of the histograms are correlated. It may be used as a value. Also, a motion vector representing the motion of the coordinate conversion frame FrT0 with respect to the reference frame FrN is calculated for each pixel or small region of the reference frame FrN, and the calculated average value, median value, or standard deviation of the motion vectors is used as a correlation value. Alternatively, a cumulative sum of motion vector histograms may be used as the correlation value.

  When the number of divisions is an initial value (2 × 2 in the present embodiment), the change amount calculation unit 5 increases the number of divisions by one level (4 × 4 in the present embodiment), and correlates according to the increased number of divisions. A change amount Δd0 between the correlation value d1 calculated by the value calculation means 4 and the correlation value d0 acquired by the number of divisions at the previous stage is calculated.

Specifically, in the correspondence estimation means 2, as shown in FIG. 6A, the number of divisions of the reference patch P0 and the patch P1 is increased from 2 × 2 to 4 × 4, and the reference patch P0 and the patch P1 are set. It is arranged at appropriate positions of the reference frame FrN and the frame FrN + 1, and the movement amount and deformation amount of the patch P1 with respect to the reference patch P0 are obtained in the same manner as described above, and the correspondence relationship between the pixels in the reference patch P0 and the pixels in the patch P1 is obtained. presume. Then, the coordinate conversion unit 3 performs coordinate conversion of the frame FrN + 1 into the coordinate space of the reference frame FrN based on the correspondence estimated by the correspondence estimation unit 2 to obtain a coordinate conversion frame FrT1. Further, the correlation value calculation means 4 calculates a correlation value d1 between the coordinate conversion frame FrT1 and the reference frame FrN. Specifically, as shown in the following formula (10), the absolute value of the difference between the pixel values FrT1 (x, y) and FrN (x, y) in the corresponding pixels of the coordinate conversion frame FrT1 and the reference frame FrN The average value for all pixels in the reference patch P0 is calculated as the correlation value d1. Further, as shown in the following equation (11), the absolute value of the difference between the correlation value d0 and the correlation value d1 is calculated as the change amount Δd0.

  Since the change amount Δd0 cannot be calculated simply by calculating the correlation value d0, the change amount calculation unit 5 calculates the correlation value d1 when the division number 1 is increased by one step from the initial stage. This is done by controlling the coordinate conversion means 3 and the correlation value calculation means 4.

  The comparison means 6 compares the change amount Δd0 with the threshold value Th1. This threshold value Th1 is determined in advance and stored in a memory (not shown) of the comparison means 6. Then, when the change amount Δd0 is equal to or less than the threshold value Th1, the smaller division number from which the correlation value d0 is obtained out of the two correlation values d0 and d1 for which the change amount Δd0 is calculated will be described later. As described above, the composition unit 7 determines the composite division number that estimates the correspondence used to create the composite frame FrG from the frames FrN and FrN + 1.

  On the other hand, when the change amount Δd0 exceeds the threshold value Th1, the comparison unit 6 increases the number of divisions of the reference patch P0 and the patch P1 by one step to estimate the correspondence, obtain the coordinate conversion frame, and obtain the correlation value. The correspondence relationship estimation unit 2, the coordinate conversion unit 3, the correlation value calculation unit 4, and the change amount calculation unit 5 are controlled so as to calculate the change amount and the change amount.

  Specifically, in the correspondence estimation means 2, as shown in FIG. 6B, the number of divisions of the reference patch P0 and the patch P1 is increased from 4 × 4 to 8 × 8, and the reference patch P0 and the patch P1 are set. It is arranged at appropriate positions of the reference frame FrN and the frame FrN + 1, and the movement amount and deformation amount of the patch P1 with respect to the reference patch P0 are obtained in the same manner as described above, and the correspondence relationship between the pixels in the reference patch P0 and the pixels in the patch P1 is obtained. presume. Then, the coordinate conversion unit 3 performs coordinate conversion of the frame FrN + 1 into the coordinate space of the reference frame FrN based on the correspondence estimated by the correspondence estimation unit 2 to obtain a coordinate conversion frame FrT2. Further, the correlation value calculation means 4 calculates a correlation value d2 between the coordinate conversion frame FrT2 and the reference frame FrN. Further, the change amount calculating means 5 calculates the absolute value of the difference value between the correlation value d1 calculated by the 4 × 4 division number and the correlation value d2 calculated by the 8 × 8 division number as the change amount Δd1. .

Then, the comparing means 6 compares the calculated change amount Δd1 with the threshold value Th1, and if the change amount Δd1 exceeds the threshold value Th1, the number of divisions of the reference patch P0 and the patch P1 is further increased to 16 × 16. And the correlation value di + 1 (−1 ≦ i) (hereinafter, i is used for generalization) is calculated in the increased number of divisions, and the amount of change between the correlation value di and the correlation value d1 + 1 is calculated. Correspondence estimation, coordinate conversion frame acquisition, and correlation value calculation are performed in correspondence relationship estimation means 2, coordinate conversion means 3, correlation value calculation means 4, and change amount calculation means 5 until Δdi becomes equal to or less than threshold value Th1. And the amount of change is calculated. In this embodiment, the number of divisions at the i-th stage of the reference patch P0 and the patch P1 is 2 ^{i + 1} × 2 ^{i + 1} , and the coordinate conversion frame obtained by the number of divisions at the i-th stage is FrTi, Let the correlation value be di. Then, a change amount Δdi between the correlation value di + 1 calculated by the number of divisions at the i + 1 stage and the correlation value di calculated by the number of divisions at the i stage is calculated, and the change amount Δdi becomes equal to or less than the threshold value Th1. The division number of the reference patch P0 and the patch P1 is determined as the composite division number. Note that the composite division number is the smaller division number of the two division values obtained by calculating the two correlation values obtained from the variation Δdi.

  If the number of divisions is too large, the rectangular area becomes too small, and there are many rectangular areas that represent similar images. On the other hand, the movement and / or deformation of the patch follows the movement and / or deformation of the subject. There are cases where it is not possible. For this reason, it is preferable to set the upper limit of the number of divisions.

The synthesizing unit 7 performs an interpolation operation on the reference frame FrN and the frame FrN + 1 based on the correspondence estimated by the number of synthesis divisions determined by the comparing unit 6 to create a synthesized frame FrG. Specifically, first, as shown in FIG. 7, an integrated image having a finally required number of pixels (in the present embodiment, a case where the number of pixels is doubled in both the vertical and horizontal directions of the frames FrN and FrN + 1 will be described. , Which may have a pixel number n times (n: positive number)), and the correspondence estimation means 2 determines the frame FrN + 1 (region in the patch P1) based on the correspondence in the composite division number. Are assigned on the integrated image. Assuming that the function for performing this assignment is Π, the pixel value of each pixel of the frame FrN + 1 is assigned on the integrated image by the following equation (12).

However, I1N + 1 (x °, y °): the pixel value of the frame FrN + 1 allocated on the integrated image FrN + 1 (x, y): the pixel value of the frame FrN + 1

  Thus, by assigning the pixel value of the frame FrN + 1 on the integrated image, the pixel value I1N + 1 (x °, y °) is obtained, and I1 (x °, y °) (= I1N + 1 (x °, y °)) for each pixel. A first interpolation frame having a pixel value of) is acquired.

  Here, when assigning pixel values to the integrated image, depending on the relationship between the number of pixels in the integrated image and the number of pixels in the frame FrN + 1, each pixel on the frame FrN + 1 has an integer coordinate (that is, a pixel value exists). (Coordinates) may not correspond. In this embodiment, the pixel value in the integer coordinates of the integrated image is set to an area surrounded by integer coordinates in the vicinity of 8 adjacent to the integer coordinates of the integrated image, and each of the frames on the frame FrN + 1 assigned in this area is set. Obtained as a weighted sum of pixel values of pixels.

That is, as shown in FIG. 8, for integer coordinates p (x, y) in the integrated image, integer coordinates p (x-1, y-1), p (x, y-1), p (8) in the vicinity of the eight. x + 1, y-1), p (x-1, y), p (x + 1, y), p (x-1, y + 1), p (x, y + 1), p (x + 1, y + 1) Is calculated based on the pixel value of the frame FrN + 1 assigned to. Here, k pixel values of the frame FrN + 1 are allocated in a region surrounded by eight neighboring pixels, and the pixel values of the allocated pixels are I1N + 1i (x °, y °) (1 ≦ i ≦ k). Then, the pixel value I1N + 1 (x ^, y ^) at the integer coordinates p (x, y) can be calculated by the following equation (13). In Expression (12), Φ is a function representing the load sum calculation.

  However, Mi (1 ≦ i ≦ k): product of the internal ratio of coordinates as viewed from neighboring integer pixels at the position where the pixel value I1N + 1i (x °, y °) is assigned.

Here, for the sake of simplicity, consider the case where the two pixel values I1N + 11 and I1N + 12 of the frame FrN + 1 are assigned to the region surrounded by the pixels in the vicinity of 8 with reference to FIG. 8 at the integer coordinates p (x, y). The pixel value I1N + 1 (x ^, y ^) can be calculated by the following equation (14).

  However, M1 = u × v, M2 = (1-s) × (1-t)

  The pixel value I1N + 1 (x ^, y ^) can be obtained by assigning the pixel value of the frame FrN + 1 for all integer coordinates of the integrated image. In this case, each pixel value I1 (x ^, y ^) of the first interpolation frame is I1N + 1 (x ^, y ^).

  Note that the reference frame FrN is acquired by interpolating and directly assigning the pixels of the reference frame FrN to the integer coordinates of the integrated image. A pixel value obtained by assigning the reference frame FrN to the integer coordinates of the integrated image is I2 (x ^, y ^), and a frame having the pixel value I2 (x ^, y ^) is a second interpolation frame. To do.

  Then, by adding or weighting the pixel values I1 (x ^, y ^) and I2 (x ^, y ^) in the corresponding pixels of the first and second interpolation frames, the pixel value FrG (x ^, A composite frame FrG having y ^) is created. Note that the composite frame FrG having the pixel value FrG (x ^, y ^) may be created only from the pixel value I1 (x ^, y ^).

  Note that pixel values may not be assigned to all integer coordinates of the integrated image. In such a case, various interpolation operations such as a linear interpolation operation and a spline interpolation operation are performed on the assigned pixel value or the pixel value of the reference frame (corresponding to the coordinates not assigned), and the pixel value is What is necessary is just to calculate the pixel value of the integer coordinate which was not allocated.

  In the above description, the process for obtaining the composite frame FrG for the luminance component Y has been described. However, the composite frame FrG is also obtained for the color difference components Cb and Cr. Then, a synthesized frame FrG (Y) obtained from the luminance component Y and synthesized frames FrG (Cb) and FrG (Cr) obtained from the color difference components Cb and Cr are synthesized to obtain a final synthesized frame. It will be. In order to increase the processing speed, the correspondence relationship between the reference frame FrN and the frame FrN + 1 is estimated only for the luminance component Y, and the color difference components Cb and Cr are processed based on the correspondence relationship estimated for the luminance component Y. It is preferable to carry out.

  Next, the operation of the first embodiment will be described. FIG. 9 is a flowchart showing processing performed in the first embodiment. First, the moving image data M0 is input to the sampling means 1 (step S1), and here, the reference frame FrN and the frame FrN + 1 are sampled from the moving image data M0 (step S2). Subsequently, the correspondence relationship estimation means 2 estimates the correspondence relationship between the reference frame FrN and the frame FrN + 1 (step S3).

  Based on the estimated correspondence, the coordinate conversion unit 3 converts the frame FrN + 1 into the coordinate space of the reference frame FrN to obtain a coordinate conversion frame FrTi (step S4). Then, the correlation value calculation means 4 calculates the correlation value di between the coordinate conversion frame FrT0 and the reference frame FrN (step S5).

  Subsequently, it is determined whether or not the correlation value is calculated based on the number of divisions at the 0th stage of the reference patch P0 and the patch P1, that is, the initial number of divisions (step S6). When step S6 is affirmed, The number of divisions of the reference patch P0 and the patch P1 is increased by one level (step S7), the process returns to step S3, and the processing from step S3 to step S5 is repeated with the increased number of divisions, and the coordinate conversion frame FrT0 and the reference frame FrN The correlation value di + 1 is calculated.

  When the correlation value di + 1 is calculated based on the increased number of divisions, step S6 is negated, and the amount of change Δdi between the correlation value di calculated based on the previous number of divisions and the correlation value di + 1 calculated based on the current number of divisions. Is calculated (step S8). Then, it is determined by the comparison means 6 whether or not the change amount Δdi is equal to or less than the threshold value Th1 (step S9).

  If step S9 is negative, the process proceeds to step S7, the number of divisions of the reference patch P0 and the patch P1 is increased by one step (step S7), and the process returns to step S3, and processing from step S3 to step S9 is performed according to the increased number of divisions. Is repeated.

  If the determination in step S9 is affirmative, the number of divisions of the reference patch P0 and the patch P1 that has obtained the smaller correlation value di of the two correlation values di and di + 1 that have obtained the change amount Δdi becomes the combined division number. The synthesized frame FrG is created from the reference frame FrN and the frame FrN + 1 by the synthesizing means 7 based on the correspondence relationship determined (step S10) and estimated by the determined number of synthesized divisions (step S11), and the process is terminated.

  As described above, in the first embodiment, based on the correspondence estimated by the number of divisions for which the correlation value obtained by calculating the change amount equal to or less than the threshold value Th1 is calculated, the frame FrN and the frame FrN + 1 are combined with the combined frame FrG. Therefore, the composite frame FrG can be created with an optimal number of divisions that can follow the movement and / or deformation of the moving subject in the frame, and thus the motion of the subject included in the frame can be determined. Regardless, a high-quality composite frame FrG can be obtained.

  Next, a second embodiment of the present invention will be described. In the first embodiment, the synthesized frame FrG is calculated based on the correspondence between the entire images in the patches of the frames FrN and FrN + 1 estimated by the number of divisions for which the correlation value that causes the change amount to be equal to or less than the threshold Th1. In the second embodiment, the change amount of the correlation value is calculated for each rectangular area obtained by dividing the reference patch P0 and the patch P1, and the change amount for each rectangular area is set as a threshold value. Compared to Th1, the composite division number is determined in units of rectangular areas to create a composite frame FrG. In the second embodiment, using the above equation (9), the pixel values FrT0 (x, y) and FrN (x, F) in the corresponding pixels of the coordinate conversion frame FrT0 and the reference frame FrN for each rectangular area. y) The value obtained by dividing the addition value for all the pixels m in the rectangular area with the absolute value of the difference from y) by the number m of pixels in the rectangular area, that is, for all the pixels in the rectangular area with the absolute value of the difference for each pixel Is calculated as a correlation value for determining the amount of change for each rectangular area.

  FIG. 10 is a diagram for explaining the calculation of the correlation value for each rectangular area. As shown in FIG. 10A, the rectangular areas of the reference patch P0 and the patch P1 having the number of divisions of 2 × 2 are defined as rectangular areas A1 to A4. On the other hand, as shown in FIG. 10B, the rectangular areas of the reference patch P0 and the patch P1 having a division number of 4 × 4 are defined as rectangular areas A11 to A14, A21 to A24, A31 to A34, and A41 to A44. The rectangular area A1 corresponds to the rectangular areas A1 to A14, the rectangular area A2 corresponds to the rectangular areas A21 to A24, the rectangular area A3 corresponds to the rectangular areas A31 to A34, and the rectangular area A4 corresponds to the rectangular areas A41 to A44.

  When the number of divisions is 2 × 2, the correlation value between the reference frame FrN and the frame FrN + 1 is calculated for each of the rectangular areas A1 to A4. Assume that the correlation values in each of the rectangular areas A1 to A4 are d01, d02, d03, and d04.

  In addition, for the calculation of the amount of change, the addition value of the correlation values of the rectangular areas A11 to A14, A21 to A24, A31 to A34, A41 to A44 when the number of divisions is 4 × 4 is calculated. The sum of the correlation values is set as correlation values d11, d12, d13, and d14.

Then, the absolute value of the difference value of the correlation value between the corresponding rectangular areas is calculated as the change amount. That is, when the number of divisions is 2 × 2, change amounts Δd01, Δd02, Δd03, and Δd04 respectively corresponding to the rectangular areas A1 to A4 are calculated by the following equations (15) to (18).

  Subsequently, the calculated change amounts Δd01, Δd02, Δd03, and Δd04 are compared with the threshold value Th1, and the reference patch P0 and the region in the patch P1 corresponding to the rectangular region in which the change amount is equal to or less than the threshold value Th1. Determines the smaller division number (that is, 2 × 2) among the division numbers for which the two correlation values for which the amount of change has been calculated is calculated as the composite division number. On the other hand, for the rectangular area where the amount of change exceeds the threshold value Th1, the number of divisions is increased by one step, and the correspondence is estimated, the coordinate conversion frame is acquired, the correlation value is calculated, and the amount of change is calculated again. Then, the correlation value and the change amount are calculated by gradually increasing the number of divisions until the change amount becomes equal to or less than the threshold value Th1 for all regions in the reference patch P0 and the patch P1, and the change amount is the threshold value Th1. The smaller division number of the two division values for which the two correlation values are calculated is determined as the combined division number for the areas in the reference patch P0 and the patch P1 corresponding to the rectangular area.

  For example, when the number of divisions is 2 × 2, the change amounts Δd01 and Δd02 are less than or equal to the threshold value Th1 for the rectangular regions A1 and A2, and the change amounts Δd03 and Δd04 exceed the threshold value Th1 for the rectangular regions A3 and A4. In this case, for the areas corresponding to the rectangular areas A1 and A2 of the reference patch P0 and the patch P1, the smaller division number (that is, 2 × 2) of the two correlation value division numbers for which the amount of change is calculated is synthesized. Determine the number of divisions.

  On the other hand, for the rectangular areas A3 and A4, the rectangular areas A31 to A34 and A41 to A44 corresponding to the rectangular areas A3 and A4 so that the number of divisions is the same as when the reference patch P0 and the patch P1 are divided into 8 × 8. Are further divided, and according to the number of divisions, only for the regions corresponding to the rectangular regions A31 to A34 and A41 to A44 of the reference patch P0 and the patch P1, the correspondence is estimated, the coordinate conversion is performed, and the correlation value is calculated. And the addition value of the correlation value about four rectangular areas corresponding to each of rectangular area A31-A34, A41-A44 is calculated as a correlation value respectively corresponding to rectangular area A31-A34, A41-A44, and rectangular area A31 The amount of change between the correlation values of A34 and A41 to A44 and the correlation values corresponding to the rectangular areas A31 to A34 and A41 to A44 of each rectangular area calculated by the number of divisions of 8 × 8 was calculated. The amount of change is compared with a threshold value Th1.

  When the amount of change for all the rectangular areas A31 to A34 and A41 to A44 is equal to or less than the threshold value Th1, the areas corresponding to the rectangular areas A31 to A34 and A41 to A44 of the reference patch P0 and the patch P1. For, the smaller division number (that is, 4 × 4) of the two correlation value division numbers for which the amount of change is calculated is determined as the composite division number.

  In this case, the composite division number of the images in the reference patch P0 and the patch P1 is 2 × 2 in the left half area and 4 × 4 in the right half area as shown in FIG. When the amount of change for the rectangular areas A31 to A34 and A41 to A44 again exceeds the threshold value Th1, the correlation value is equal to or less than the threshold value Th1 for all areas in the reference patch P0 and the patch P1. Up to the region of the reference patch P0 and patch P1 corresponding to the rectangular region where the correlation value exceeds the threshold value Th1, the number of divisions is increased step by step, the correspondence is estimated, the coordinate conversion frame is obtained, and the correlation value Calculation and change amount calculation are performed.

  Next, the operation of the second embodiment will be described. FIG. 12 is a flowchart showing processing performed in the second embodiment. First, the moving image data M0 is input to the sampling means 1 (step S21). Here, the reference frame FrN and the frame FrN + 1 are sampled from the moving image data M0 (step S22). Subsequently, the correspondence relationship estimation means 2 estimates the correspondence relationship between the reference frame FrN and the frame FrN + 1 (step S23).

  Then, based on the estimated correspondence, the coordinate conversion means 3 converts the frame FrN + 1 into the coordinate space of the reference frame FrN to obtain the coordinate conversion frame FrTi (step S24). Then, the correlation value calculation means 4 calculates the correlation value between the coordinate conversion frame FrT0 and the reference frame FrN for each rectangular area (step S25).

  Subsequently, it is determined whether or not the correlation value is calculated based on the number of divisions of the 0th stage of the reference patch P0 and the patch P1, that is, the initial number of divisions (step S26). When step S26 is affirmed, The number of divisions of the reference patch P0 and the patch P1 is increased by one level (step S27), the process returns to step S23, and the processing from step S23 to step S25 is repeated with the increased number of divisions, and the coordinate conversion frame FrT0 and the reference frame FrN Are calculated for each rectangular area.

  When the correlation value is calculated based on the increased number of divisions, step S26 is denied, and the amount of change between the correlation value calculated based on the previous number of divisions and the correlation value calculated based on the current number of divisions is determined for each rectangular area. (Step S28). Then, with respect to one rectangular area (corresponding to the smaller division number) of the plurality of rectangular areas, the comparison unit 6 determines whether or not the amount of change for each rectangular area is equal to or less than the threshold value Th1. (Step S29).

  If step S29 is negative, the number of divisions of the reference patch P0 and the patch P1 is increased by one step for the rectangular area (step S30), and the process proceeds to step S32 described later. On the other hand, when step S29 is affirmed, for the rectangular area, the smaller one of the two correlation value division numbers for which the amount of change is calculated is determined as the composite division number (step S31). Then, it is determined whether or not the determination in step S29 is completed for all the rectangular areas (step S32). If step S32 is negative, the determination target is changed to the next rectangular area (step S33), and the process proceeds to step S29. Return and the subsequent processing is repeated.

  If step S32 is affirmed, it is determined whether or not the number of composite divisions has been determined for all regions in the reference patch P0 and the patch P1 (step S34), and if step S34 is negative, the process returns to step S23 to return to the reference patch. Only in the area where the composite division number is not determined in P0 and the patch P1, the processes after step S23 are repeated. On the other hand, if step S34 is affirmed, a synthesized frame FrG is created from the reference frame FrN and the frame FrN + 1 by the synthesizing unit 7 based on the correspondence estimated by the determined number of synthesized divisions (step S35), and the process is performed. finish.

  As described above, in the second embodiment, since the correspondence used to create the composite frame FrG is determined for each rectangular area, the reference patch and the area unit corresponding to the divided rectangular area in the patch are combined. A frame FrG is created. For this reason, the composite frame FrG can be created with the optimal number of divisions following the movement of the subject included in each part in the frame, and thus, higher image quality can be achieved regardless of the movement of the subject included in the frame. A composite frame FrG can be obtained.

  In the second embodiment, the number of divisions is stepwise for all the regions in the reference patch P0 and the patch P1 until the amount of change for all rectangular regions obtained by the division is equal to or less than the threshold value Th1. Although increasing, the representative value of the change amount (for example, the average value of the change amount, the median value, the maximum value, the minimum value, etc.) for all the rectangular regions obtained in each division stage is a predetermined threshold value ( The number of divisions of the regions in the reference patch P0 and the patch P1 when it becomes equal to or less than Th2 may be determined as the composite division number.

  In the second embodiment, the number of divisions is increased stepwise until the amount of change in all rectangular areas obtained by division for all areas in the reference patch P0 and patch P1 is equal to or less than the threshold value Th1. However, the division number of the rectangular area in the reference patch P0 and the patch P1 when the change amount of one rectangular area becomes equal to or less than the threshold Th1 may be determined as the composite division number. For example, if the change amount Δd01 is equal to or less than the threshold value Th1 in only one rectangular area (for example, A1) in the initial stage of division into 2 × 2, 2 × 2 may be set as the composite division number.

  Further, the division number when the ratio of the rectangular area in which the amount of change is equal to or less than the threshold value Th1 to the reference patch P0 and the area in the patch P1 exceeds a predetermined ratio (for example, 50%) is determined as the composite division number. May be. For example, when the number of divisions is 4 × 4 as shown in FIG. 13, 4 × 4 is synthesized when the change amount is equal to or less than the threshold value Th1 in nine rectangular regions indicated by diagonal lines among 16 rectangular regions. The number of divisions may be determined.

  Further, in the reference patch P0 and the patch P1, it is located in a region including a main subject such as a human face or a complex texture or a region near the center where the main subject is likely to be included (hereinafter referred to as a main region). For the rectangular area to be processed, the division number of the rectangular area in the reference patch P0 and the patch P1 when the change amount is equal to or less than the threshold Th1 may be determined as the composite division number. For example, in the case where the number of divisions is 4 × 4 as shown in FIG. 14, when four rectangular areas near the center are located in the face portion of a person, the change amount is limited only for these four rectangular areas. What is necessary is just to determine the division | segmentation number when it becomes below value Th1 as a synthetic | combination division | segmentation number. In this case, an average value of the amount of change calculated for the rectangular area located in the main area may be calculated, and this average value may be compared with the threshold value Th1.

Further, when calculating the average value of the change amount, as shown in the following equation (19), the change calculated in each rectangular area according to the ratio of the main subject in each rectangular area located in the main area The amount may be weighted, and an average value Δdm of the weighted change amount may be calculated.

However, Δd0i: Change amount of each rectangular area Si: Weighting factor that increases in proportion to the ratio of the main subject in each rectangular area In the second embodiment, all rectangular areas obtained in each stage of division The amount of change is compared with the threshold value Th1, but the amount of change is very large in the local region in the reference patch P0 and the patch P1, and even if the number of divisions is increased, In some cases, the amount of change in the rectangular area obtained in the stage does not fall below the threshold value Th1. For this reason, in the initial number of divisions (2 × 2) in the reference patch P0 and the patch P1, the amount of change for the rectangular area is compared with a predetermined constant value. After that, the region may not be used for calculating the change amount. Thereby, the amount of calculation can be reduced and processing can be performed at high speed. The constant value may be a value that is several times the threshold value Th1.

In the first and second embodiments, the threshold value Th1 to be compared with the change amount may be changed according to the number of divisions. Specifically, as shown in FIG. 15, the number of divisions does not change up to a constant value (here 2 ⁸ ), and when it exceeds a certain value, the threshold value Th1 is changed so as to increase as the number of divisions increases. You can do it. Thereby, the number of divisions can be set appropriately. Note that the threshold Th1 is increased when the number of divisions exceeds a certain value because the rectangular area becomes too small when the number of divisions is too large, and there are many rectangular areas representing similar images. On the contrary, the movement and / or deformation of the subject may not be able to follow the movement and / or deformation of the subject.

  In the first and second embodiments, the synthesized frame FrG is acquired for each of the luminance color difference components Y, Cb, and Cr of the reference frame FrN and the frame FrN + 1. However, the synthesized frame FrG is obtained only for the luminance component Y. For the color difference components Cb and Cr, the color difference components Cb and Cr of the reference frame FrN may be linearly interpolated to obtain a color difference component composite frame.

  If the frames FrN and FrN + 1 are composed of RGB color data, the composite frame FrG may be created by performing processing for each RGB color data.

In the first and second embodiments, the correlation value di is calculated for each of the luminance and chrominance components Y, Cb, and Cr. For example, as shown in the following equation (20), the correlation value di is calculated. The correlation value diY of the luminance component and the correlation values diCb and diCr of the chrominance component are weighted and added by the weighting coefficients a1, b1, and c1, thereby calculating one correlation value di ′, and the correlation value di thus calculated The amount of change Δdi ′ may be calculated from ′ and the correlation value di + 1 ′ obtained by increasing the number of divisions by one step.

Further, as shown in the following equation (21), the luminance component FrTiY (x, y) and the color difference components FrTiCb (x, y), FrTiCr (x, y) of the coordinate conversion frame FrTi, and FrNY ( x, y) and the color difference components FrNCb (x, y), FrNCr (x, y) and the Euclidean distance using the weight coefficients a2, b2, c2 are calculated for each pixel in the reference patch P0, and this is calculated for each pixel. The correlation value di ″ (x, y) of the pixel is used, and in the first embodiment, the average value of all the pixels in the reference patch P0 of the correlation value di ″ (x, y) is used in the second embodiment. The correlation value di ″ (x, y) may be an average value for all pixels in each rectangular area as a correlation value for calculating a change amount for determining the number of divisions.

  In the first and second embodiments, the composite frame FrG is created from the two frames FrN and FrN + 1. However, the composite frame FrG may be created from a plurality of three or more frames. For example, in the first embodiment, when the composite frame FrG is created from three or more T frames FrN + t ′ (0 ≦ t ′ ≦ T−1), other frames FrN + t other than the reference frame FrN (= FrN + 0) For each (1 ≦ t ≦ T−1), a correspondence relationship with the reference frame FrN is estimated, a coordinate conversion frame is acquired according to the estimated correspondence relationship, and a correlation between the coordinate conversion frame and the reference frame FrN is obtained. Calculate the value. Then, the change amount with the correlation value obtained by increasing the number of divisions by one step is calculated, and the correspondence is estimated, the coordinate conversion frame is acquired, the correlation value is calculated, and the change amount is calculated until the change amount is equal to or less than the threshold Th1. And a composite frame is created based on the correspondence estimated by the number of divisions when the change amount is equal to or less than the threshold value Th1. This synthesized frame is referred to as an intermediate synthesized frame FrGt. Note that the pixel value at integer coordinates of the integrated image of the intermediate composite frame FrGt is FrGt (x ^, y ^).

Then, the intermediate composite frame FrGt is obtained for all other frames FrN + t, and the intermediate composite frame FrGt is added between corresponding pixels by the following equation (22), whereby the pixel value FrG (x ^, y ^) is obtained. A composite frame FrG is created.

  Note that pixel values may not be assigned to all integer coordinates of the integrated image. In such a case, various interpolation operations such as a linear interpolation operation and a spline interpolation operation are performed on the assigned pixel value or the pixel value of the reference frame (corresponding to the coordinates not assigned), and the pixel value is What is necessary is just to calculate the pixel value of the integer coordinate which was not allocated.

  In addition, when the composite frame FrG is created from a plurality of three or more frames, when the intermediate composite frame FrGt is acquired, the amount of change is calculated for each rectangular area constituting the patch, as in the second embodiment. Then, the intermediate composite frame FrGt may be created based on the correspondence estimated in units of regions corresponding to the rectangular regions.

  In the above embodiment, the division number for which the smaller correlation value di is calculated among the division numbers for which the two correlation values di and di + 1 for which the change amount Δdi has been calculated is the combined division number. Alternatively, the composite frame FrG may be created using the number of divisions for which the correlation value di + 1 is calculated as the composite division number.

1 is a schematic block diagram showing the configuration of a moving image synthesis apparatus according to a first embodiment of the present invention. The figure for demonstrating estimation of the correspondence of frame FrN + 1 and the reference | standard flame | frame FrN Diagram for explaining patch deformation The figure for demonstrating the correspondence of patch P1 and reference | standard patch P0 Diagram for explaining bilinear interpolation Diagram showing patches with different number of divisions The figure for demonstrating allocation to the integrated image of frame FrN + 1 The figure for demonstrating the calculation of the pixel value of the integer coordinate in an integrated image The flowchart which shows the process performed in 1st Embodiment. The figure for demonstrating the process performed in the 2nd Embodiment of this invention (the 1) The figure for demonstrating the process performed in the 2nd Embodiment of this invention (the 2) The flowchart which shows the process performed in 2nd Embodiment. The figure for demonstrating the other process performed in the 2nd Embodiment of this invention (the 1) The figure for demonstrating the other process performed in the 2nd Embodiment of this invention (the 2) Diagram showing the relationship between the number of divisions and the threshold

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Sampling means 2 Correspondence relationship estimation means 3 Coordinate conversion means 4 Correlation value calculation means 5 Change amount calculation means 6 Comparison means 7 Composition means

Claims (16)

Sampling means for sampling two consecutive frames from a moving image;
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch A correspondence estimation means for estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame;
Based on the correspondence, coordinate conversion means for converting the image in the patch of the other frame to the coordinate space of the reference frame to obtain a coordinate-converted frame;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame;
The correspondence estimation means for increasing the number of divisions of the reference patch and the rectangular area in the patch by one step so as to estimate the correspondence, acquire the coordinate-transformed frame, and calculate the correlation value; Control means for controlling the coordinate conversion means and the correlation value calculation means;
A change amount calculating means for calculating a change amount of the two correlation values;
The number of divisions of the reference patch and the rectangular area in the patch is increased stepwise until the amount of change becomes a predetermined threshold value or less, and the correspondence is estimated at each step, and the coordinate-converted frame The correlation estimating means, the coordinate converting means, the correlation value calculating means, and the correlation value calculating means, so as to calculate the amount of change of two correlation values calculated by the acquisition of the correlation value and the number of divisions at adjacent stages. Comparison means for controlling the change amount calculation means;
Based on the correspondence relationship estimated in the number of divisions in which one of the two correlation values when the change amount is equal to or less than the predetermined threshold is calculated, the other frame A moving image comprising: a synthesizing unit that performs an interpolation operation on the image in the patch and the image in the reference patch of the reference frame to create a synthesized frame having a higher resolution than each of the frames. Image synthesizer. Sampling means for sampling two consecutive frames from a moving image;
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch A correspondence estimation means for estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame;
Based on the correspondence, coordinate conversion means for converting the image in the patch of the other frame to the coordinate space of the reference frame to obtain a coordinate-converted frame;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame in units of the rectangular area;
The correspondence estimation means for increasing the number of divisions of the reference patch and the rectangular area in the patch by one step so as to estimate the correspondence, acquire the coordinate-transformed frame, and calculate the correlation value; Control means for controlling the coordinate conversion means and the correlation value calculation means;
A change amount calculating means for calculating a change amount of the two correlation values in units of the rectangular area;
The change in a predetermined number of rectangular areas or rectangular areas at predetermined positions among all rectangular areas or rectangular areas other than the rectangular areas in which the amount of change is equal to or greater than a predetermined value among all the rectangular areas. The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the representative value of the amount or the change amount is equal to or less than a predetermined threshold, and the correspondence is estimated at each step. The correspondence estimation means, obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the amount of change of the two correlation values calculated based on the number of adjacent divisions in the rectangular area unit, Comparison means for controlling the coordinate conversion means, the correlation value calculation means, and the change amount calculation means;
The change amount or the representative value of the change amount is equal to or less than the predetermined threshold value for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated. Based on the correspondence relationship, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame in units of areas corresponding to the rectangular areas, and from each frame A moving image synthesizing apparatus comprising a synthesizing unit for generating a synthesized frame having a high resolution.   3. The moving image according to claim 1, wherein the control unit is a unit that increases the predetermined threshold value as the number of divisions of the rectangular area increases beyond a certain value. Synthesizer.   The correspondence relationship estimation means, the coordinate conversion means, the correlation value calculation means, the control means, the change amount calculation means, the comparison means, and the synthesis means are configured to estimate the correspondence relation, acquire the coordinate-converted frame, The calculation of the correlation value, the calculation of the amount of change, and the creation of the composite frame using at least one component that constitutes the frame, 4. The moving image composition apparatus described. Sampling means for sampling three or more consecutive frames from a moving image;
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Correspondence estimation means for estimating a correspondence between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame based on the reference patch;
Coordinate conversion means for converting the image in the patch of the other one frame into the coordinate space of the reference frame to obtain a coordinate-converted frame based on the correspondence relationship;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame;
The correspondence estimation means for increasing the number of divisions of the reference patch and the rectangular area in the patch by one step so as to estimate the correspondence, acquire the coordinate-transformed frame, and calculate the correlation value; Control means for controlling the coordinate conversion means and the correlation value calculation means;
A change amount calculating means for calculating a change amount of the two correlation values;
The number of divisions of the reference patch and the rectangular area in the patch is increased stepwise until the amount of change becomes a predetermined threshold value or less, and the correspondence is estimated at each step, and the coordinate-converted frame The correlation estimating means, the coordinate converting means, the correlation value calculating means, and the correlation value calculating means, so as to calculate the amount of change of two correlation values calculated by the acquisition of the correlation value and the number of divisions at adjacent stages. Comparison means for controlling the change amount calculation means;
Based on the correspondence relationship estimated in the number of divisions in which one of the two correlation values when the change amount is equal to or less than the predetermined threshold is calculated, the other frame Interpolation is performed on the image in the patch and the image in the reference patch of the reference frame to obtain an intermediate composite frame having a resolution higher than that of each of the frames, and for all the other frames, the correspondence Estimate the relationship, acquire the coordinate-transformed frame, calculate the correlation value, calculate the amount of change, and acquire the intermediate composite frame, and obtain the plurality of intermediate composite frames acquired for all the other frames. A moving image synthesizing apparatus comprising a synthesizing unit that creates a synthesized frame by synthesizing. Sampling means for sampling three or more consecutive frames from a moving image;
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Correspondence estimation means for estimating a correspondence between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame based on the reference patch;
Coordinate conversion means for converting the image in the patch of the other one frame into the coordinate space of the reference frame to obtain a coordinate-converted frame based on the correspondence relationship;
Correlation value calculating means for calculating a correlation value representing a correlation between the coordinate-converted frame and an image in the reference patch of the reference frame in units of the rectangular area;
The correspondence estimation means for increasing the number of divisions of the reference patch and the rectangular area in the patch by one step so as to estimate the correspondence, acquire the coordinate-transformed frame, and calculate the correlation value; Control means for controlling the coordinate conversion means and the correlation value calculation means;
A change amount calculating means for calculating a change amount of the two correlation values in units of the rectangular area;
The change in a predetermined number of rectangular areas or rectangular areas at predetermined positions among all rectangular areas or rectangular areas other than the rectangular areas in which the amount of change is equal to or greater than a predetermined value among all the rectangular areas. The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the representative value of the amount or the change amount is equal to or less than a predetermined threshold, and the correspondence is estimated at each step. The correspondence estimation means, obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the amount of change of the two correlation values calculated based on the number of adjacent divisions in the rectangular area unit, Comparison means for controlling the coordinate conversion means, the correlation value calculation means, and the change amount calculation means;
The change amount or the representative value of the change amount is equal to or less than the predetermined threshold value for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated. Based on the correspondence relationship, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame in units of areas corresponding to the rectangular areas, and from each frame Get an intermediate composite frame with high resolution,
For all the other frames, the correspondence is estimated, the coordinate-transformed frame is acquired, the correlation value is calculated, the change amount is calculated, and the intermediate composite frame is acquired. A moving image synthesizing apparatus comprising: a synthesizing unit that creates a synthesized frame by synthesizing the plurality of acquired intermediate synthesized frames.   7. The moving image according to claim 5, wherein the control means is means for increasing the predetermined threshold value in accordance with an increase in the number of divisions of the rectangular area exceeding a predetermined value. Synthesizer.   The correspondence relationship estimation means, the coordinate conversion means, the correlation value calculation means, the control means, the change amount calculation means, the comparison means, and the synthesis means are configured to estimate the correspondence relation, acquire the coordinate-converted frame, 6. The means for calculating the correlation value, calculating the change amount, acquiring the intermediate composite frame, and creating the composite frame using at least one component constituting the frame. 8. The moving image synthesizing apparatus according to any one of items 1 to 7. Sampling two consecutive frames from a video,
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch Estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame,
Based on the correspondence relationship, the image in the patch of the other frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame,
Calculating a correlation value representing a correlation between the coordinate-transformed frame and an image in the reference patch of the reference frame;
Increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value,
A change amount of the two correlation values is calculated;
The number of divisions of the reference patch and the rectangular area in the patch is increased stepwise until the amount of change becomes a predetermined threshold value or less, and the correspondence is estimated at each step, and the coordinate-converted frame Obtaining the correlation value, calculating the correlation value, and calculating the amount of change between the two correlation values calculated by the number of divisions in the adjacent stages,
Based on the correspondence relationship estimated in the number of divisions in which one of the two correlation values when the change amount is equal to or less than the predetermined threshold is calculated, the other frame A moving image synthesizing method, comprising: interpolating an image in the patch and an image in the reference patch of the reference frame to generate a synthesized frame having a higher resolution than each frame. Sampling two consecutive frames from a video,
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch Estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame,
Based on the correspondence relationship, the image in the patch of the other frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame,
Calculating a correlation value representing the correlation between the coordinate-converted frame and the image in the reference patch of the reference frame in units of the rectangular area;
Increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value,
The amount of change of the two correlation values is calculated in units of the rectangular area,
The change in a predetermined number of rectangular areas or rectangular areas at predetermined positions among all rectangular areas or rectangular areas other than the rectangular areas in which the amount of change is equal to or greater than a predetermined value among all the rectangular areas. The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the representative value of the amount or the change amount is equal to or less than a predetermined threshold, and the correspondence is estimated at each step. Obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the amount of change in units of the rectangular area of two correlation values calculated by the number of divisions in adjacent stages,
The change amount or the representative value of the change amount is equal to or less than the predetermined threshold value for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated. Based on the correspondence relationship, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame in units of areas corresponding to the rectangular areas, and from each frame A method for synthesizing a moving image, wherein a synthetic frame having a high resolution is created. Sampling three or more consecutive frames from a video,
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Based on the reference patch, a correspondence relationship between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame is estimated,
Based on the correspondence, the image in the patch of the other one frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame;
Calculating a correlation value representing a correlation between the coordinate-transformed frame and an image in the reference patch of the reference frame;
Increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value,
A change amount of the two correlation values is calculated;
The number of divisions of the reference patch and the rectangular area in the patch is increased stepwise until the amount of change becomes a predetermined threshold value or less, and the correspondence is estimated at each step, and the coordinate-converted frame Obtaining the correlation value, calculating the correlation value, and calculating the amount of change between the two correlation values calculated by the number of divisions in the adjacent stages,
Based on the correspondence relationship estimated in the number of divisions in which one of the two correlation values when the change amount is equal to or less than the predetermined threshold is calculated, the other frame Interpolating the image in the patch and the image in the reference patch of the reference frame to obtain an intermediate composite frame having a higher resolution than each of the frames,
For all the other frames, the correspondence is estimated, the coordinate-transformed frame is acquired, the correlation value is calculated, the change amount is calculated, and the intermediate composite frame is acquired.
A moving image synthesizing method, wherein a synthesized frame is created by synthesizing a plurality of the intermediate synthesized frames acquired for all the other frames. Sampling three or more consecutive frames from a video,
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Based on the reference patch, a correspondence relationship between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame is estimated,
Based on the correspondence, the image in the patch of the other one frame is coordinate-transformed into the coordinate space of the reference frame to obtain a coordinate-transformed frame;
Calculating a correlation value representing the correlation between the coordinate-converted frame and the image in the reference patch of the reference frame in units of the rectangular area;
Increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value,
The amount of change of the two correlation values is calculated in units of the rectangular area,
The change in a predetermined number of rectangular areas or rectangular areas at predetermined positions among all rectangular areas or rectangular areas other than the rectangular areas in which the amount of change is equal to or greater than a predetermined value among all the rectangular areas. The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the representative value of the amount or the change amount is equal to or less than a predetermined threshold, and the correspondence is estimated at each step. Obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the amount of change in units of the rectangular area of two correlation values calculated by the number of divisions in adjacent stages,
The change amount or the representative value of the change amount is equal to or less than the predetermined threshold value for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated. Based on the correspondence relationship, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame in units of areas corresponding to the rectangular areas, and from each frame Get an intermediate composite frame with high resolution,
For all the other frames, the correspondence is estimated, the coordinate-transformed frame is acquired, the correlation value is calculated, the change amount is calculated, and the intermediate composite frame is acquired.
A moving image synthesizing method, wherein a synthesized frame is created by synthesizing a plurality of the intermediate synthesized frames acquired for all the other frames. A procedure for sampling two consecutive frames from a moving image;
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch Estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame;
A procedure for obtaining a coordinate-converted frame by converting the image in the patch of the other frame into the coordinate space of the reference frame based on the correspondence relationship;
Calculating a correlation value representing a correlation between the coordinate-transformed frame and the image in the reference patch of the reference frame;
A step of increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value;
A procedure for calculating a change amount of the two correlation values;
The number of divisions of the reference patch and the rectangular area in the patch is increased stepwise until the amount of change becomes a predetermined threshold value or less, and the correspondence is estimated at each step, and the coordinate-converted frame Obtaining the correlation value, calculating the correlation value, and calculating the amount of change of the two correlation values calculated by the number of divisions in the adjacent stages;
Based on the correspondence relationship estimated in the number of divisions in which one of the two correlation values when the change amount is equal to or less than the predetermined threshold is calculated, the other frame The computer executes a moving image synthesizing method including a step of performing an interpolation operation on the image in the patch and the image in the reference patch of the reference frame to create a synthesized frame having a higher resolution than each frame. Program to let you. A procedure for sampling two consecutive frames from a moving image;
Of the two frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is arranged on a frame other than the reference frame. , Moving and / or deforming the patch on the other frame so that the image in the patch matches the image in the reference patch, and based on the moved and / or deformed patch and the reference patch Estimating a correspondence between a pixel in the patch on the other frame and a pixel in the reference patch on the reference frame;
A procedure for obtaining a coordinate-converted frame by converting the image in the patch of the other frame into the coordinate space of the reference frame based on the correspondence relationship;
Calculating a correlation value representing the correlation between the coordinate-converted frame and the image in the reference patch of the reference frame in units of the rectangular area;
A step of increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value;
Calculating a change amount of the two correlation values in units of the rectangular area;
The change in a predetermined number of rectangular areas or rectangular areas at predetermined positions among all rectangular areas or rectangular areas other than the rectangular areas in which the amount of change is equal to or greater than a predetermined value among all the rectangular areas. The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the representative value of the amount or the change amount is equal to or less than a predetermined threshold, and the correspondence is estimated at each step. A procedure for obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the amount of change in units of the rectangular area of two correlation values calculated by the number of divisions in adjacent stages;
The change amount or the representative value of the change amount is equal to or less than the predetermined threshold value for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated. Based on the correspondence relationship, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame in units of areas corresponding to the rectangular areas, and from each frame A program for causing a computer to execute a moving image synthesizing method having a procedure for creating a synthesized frame having a high resolution. A procedure for sampling three or more consecutive frames from a moving image;
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Estimating a correspondence relationship between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame based on the reference patch;
A procedure for obtaining a coordinate-converted frame by converting the image in the patch of the other one frame into the coordinate space of the reference frame based on the correspondence relationship;
Calculating a correlation value representing a correlation between the coordinate-transformed frame and the image in the reference patch of the reference frame;
A step of increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value;
A procedure for calculating a change amount of the two correlation values;
The number of divisions of the reference patch and the rectangular area in the patch is increased stepwise until the amount of change becomes a predetermined threshold value or less, and the correspondence is estimated at each step, and the coordinate-converted frame Obtaining the correlation value, calculating the correlation value, and calculating the amount of change of the two correlation values calculated by the number of divisions in the adjacent stages;
Based on the correspondence relationship estimated in the number of divisions in which one of the two correlation values when the change amount is equal to or less than the predetermined threshold is calculated, the other frame A step of performing an interpolation operation on the image in the patch and the image in the reference patch of the reference frame to obtain an intermediate composite frame having a higher resolution than each frame;
A procedure for estimating the correspondence, acquiring the coordinate-transformed frame, calculating the correlation value, calculating the amount of change, and acquiring the intermediate composite frame for all the other frames;
A program for causing a computer to execute a moving image synthesizing method including a procedure for creating a synthesized frame by synthesizing a plurality of the intermediate synthesized frames acquired for all the other frames. A procedure for sampling three or more consecutive frames from a moving image;
Among the three or more frames, a reference patch having one or more rectangular areas is arranged on one reference frame serving as a reference, and a patch similar to the reference patch is placed on one other frame other than the reference frame And moving and / or deforming the patch on the other one frame so that the image in the patch matches the image in the reference patch, and the patch after the movement and / or deformation and Estimating a correspondence relationship between a pixel in the patch on the other one frame and a pixel in the reference patch on the reference frame based on the reference patch;
A procedure for obtaining a coordinate-converted frame by converting the image in the patch of the other one frame into the coordinate space of the reference frame based on the correspondence relationship;
Calculating a correlation value representing the correlation between the coordinate-converted frame and the image in the reference patch of the reference frame in units of the rectangular area;
A step of increasing the number of divisions of the reference patch and the rectangular area in the patch by one step, estimating the correspondence, obtaining the coordinate-transformed frame, and calculating the correlation value;
Calculating a change amount of the two correlation values in units of the rectangular area;
The change in a predetermined number of rectangular areas or rectangular areas at predetermined positions among all rectangular areas or rectangular areas other than the rectangular areas in which the amount of change is equal to or greater than a predetermined value among all the rectangular areas. The number of divisions of the rectangular area in the reference patch and the patch is increased step by step until the representative value of the amount or the change amount is equal to or less than a predetermined threshold, and the correspondence is estimated at each step. A procedure for obtaining the coordinate-transformed frame, calculating the correlation value, and calculating the amount of change in units of the rectangular area of two correlation values calculated by the number of divisions in adjacent stages;
The change amount or the representative value of the change amount is equal to or less than the predetermined threshold value for each rectangular area estimated in the number of divisions for which one of the correlation values is calculated. Based on the correspondence relationship, an interpolation operation is performed on the image in the patch of the other frame and the image in the reference patch of the reference frame in units of areas corresponding to the rectangular areas, and from each frame Get an intermediate composite frame with high resolution,
A procedure for estimating the correspondence, acquiring the coordinate-transformed frame, calculating the correlation value, calculating the amount of change, and acquiring the intermediate composite frame for all the other frames;
A program for causing a computer to execute a moving image synthesizing method including a procedure for creating a synthesized frame by synthesizing a plurality of the intermediate synthesized frames acquired for all the other frames.

Images