JP2005129996A - Efficiency enhancement for generation of high-resolution image from a plurality of low resolution images - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To shorten the processing time, when a static image of high resolution is generated from a plurality of low-resolution images. <P>SOLUTION: A correction amount for correcting a positional deviation between respective images represented by a plurality of first image data is estimated. A plurality of the first image data are corrected, on the basis of the estimated correction amount, and second image data representing a higher resolution image than the resolution of the image represented by the first image data are generated as static image data on the basis of a plurality of the corrected first image data are generated as static image data. However, when estimating the correction amount, first designation of a prescribed partial shared region by a user is permitted, with respect to the respective images represented by a plurality of the first image data. A correction amount estimate range to be utilized for estimating the correction amount is set, on the basis of the designated prescribed partial region, with respect to a plurality of the first images. Then the correction amount is estimated, on the basis of image data included in the correction amount estimate range, among a plurality of the first image data. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for generating a high-resolution image from a plurality of low-resolution images.

  Conventionally, a moving image shot by a digital video camera or the like (hereinafter also simply referred to as “video camera”) is reproduced to capture one scene, and the captured one frame image (hereinafter also referred to as “frame image”). .)) Is generated. Here, the resolution means the density or the number of pixels constituting one image.

  As a method for generating such a high-resolution image, a method of generating a plurality of low-resolution images by superimposing and synthesizing them is considered. In this way, the method of superimposing and synthesizing a plurality of images can be expected to have higher image quality than the method of simply converting the resolution of one image.

  For example, in Patent Document 1, one frame image is selected as a reference frame image from consecutive (n + 1) frame images, and the movement of other n frame images (target frame images) with respect to the reference frame image. A technique is disclosed in which vectors are calculated and (n + 1) frame images are synthesized based on each motion vector to generate one high-resolution image.

Japanese Patent Laid-Open No. 2000-244851

  However, when a single high-resolution image is generated by synthesizing a plurality of low-resolution images as in the prior art, it is generated as compared with a case where a single high-resolution image is generated from one image. There is a problem that the amount of data of an image used for the process increases, and the processing time increases accordingly.

  The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a technique capable of shortening the processing time when a high-resolution image is generated from a plurality of low-resolution images. And

  In order to solve at least a part of the above-described problems, the present invention performs the following processing when generating a second image having a relatively high resolution from a plurality of first images having a relatively low resolution. First, a correction amount for correcting a positional shift between the plurality of first images is estimated. Then, the positional deviation of the plurality of first images is corrected based on the estimated correction amount, and the second image is generated by synthesizing the plurality of corrected first images. However, when estimating the correction amount, first, the user is allowed to specify a predetermined partial area in the plurality of first images, and is specified for the range of the plurality of first images. A correction amount estimation range to be used for estimation of the correction amount is set based on the predetermined partial area. Then, the correction amount is estimated based on the plurality of first images corresponding to the set correction amount estimation range.

  According to the above configuration, the correction amount can be estimated based on the plurality of first images corresponding to the set correction amount estimation range. Therefore, the correction amount can be calculated based on all of the plurality of first images. The processing time can be shortened compared to the case of estimation.

  Note that the plurality of first images may be a plurality of images that are acquired in time series from a moving image.

  In this way, it is possible to shorten the processing time when a high-resolution image is generated from a plurality of low-resolution images included in the moving image.

  The first image includes a frame image and a field image. A frame image means a still image that can be displayed by a progressive method (referred to as a non-interlace method), and a field image constitutes an image corresponding to a frame image of a non-interlace method in an interlace method. This means an odd field still image and an even field still image.

  Here, the predetermined partial area may be set as the correction amount estimation range. An area other than the predetermined partial area may be set as the correction amount estimation range. Furthermore, the user may be able to select which of the predetermined partial area and an area other than the predetermined partial area is set as the correction amount estimation range.

  In particular, when a region other than the partial region is set as the correction amount estimation range, for example, when a local image change, that is, a so-called “movement” occurs between the plurality of first images, this movement is performed. By designating this part as a partial area, it is possible to estimate the correction amount with high accuracy using only the image data of the area without motion.

Note that the present invention can be realized in various modes as described below.
(1) An image generation method, an image processing method, and an image data generation method.
(2) An image generation device, an image processing device, and an image data generation device.
(3) A computer program for realizing the above apparatus and method.
(4) A storage medium storing a computer program for realizing the above-described apparatus and method.
(5) A data signal embodied in a carrier wave including a computer program for realizing the above apparatus and method.

  When the present invention is configured as a computer program or a storage medium storing the program, the entire program for controlling the operation of the apparatus may be configured, or only the portion that performs the functions of the present invention is configured. It may be a thing. The storage medium includes a flexible disk, a CD-ROM, a DVD-ROM / RAM, a magneto-optical disk, an IC card, a ROM cartridge, a punch card, a printed matter on which a code such as a barcode is printed, an internal storage device of a computer ( A variety of computer-readable media such as a memory such as a RAM and a ROM and an external storage device can be used.

Hereinafter, the embodiment of the present invention will be described based on the following procedure.
A. Configuration of still image generator:
B. Still image generation procedure:
B1. Frame image acquisition processing:
B2. Correction amount estimation processing B2.1. Outline of displacement correction:
B2.2. Correction amount estimation method:
B2.3. Example correction amount estimation:
B3. Motion compatible composition processing:
B3.1. Non-motion compatible composition processing:
B3.2. Motion compatible composition processing:
C. effect:
D. Variation:

A. Configuration of still image generator:
FIG. 1 is an explanatory diagram showing a schematic configuration of a still image generating apparatus as an embodiment of the present invention. This still image generation device is a general-purpose personal computer, and includes a keyboard 130 and a mouse 140 as devices for inputting information to the computer 100, and a display 120 and a printer 200 as devices for outputting information. Further, a digital video camera 300 and a CD-R / RW drive 150 are provided as devices for inputting image data representing moving images (hereinafter also referred to as “moving image data”) to the computer 100. As a device for inputting moving image data, it is possible to provide a drive device capable of reading data from various information storage media such as a DVD drive and a hard disk drive in addition to a CD-R / RW drive.

  The computer 100 functions as a still image generation device by executing an application program for generating a still image under a predetermined operating system. In particular, as shown in the figure, it functions as a display control unit 104, a frame image acquisition unit 106, a correction amount estimation unit 108, a resolution enhancement processing unit 110, a print control unit 112, and a storage control unit 114. The correction amount estimation unit 108 includes a correction amount estimation range setting unit 108a and a correction amount estimation processing unit 108b.

  When an instruction to select a moving image desired by the user is input from the keyboard 130 or the mouse 140, the frame image acquisition unit 106 receives a moving image in a memory (not shown) from the CD-R / RW drive 150 or the video camera 300. Is read. This moving image data includes a plurality of frames of image data each representing a still image (hereinafter also referred to as “frame image data”).

  When the user inputs a moving image playback instruction from the keyboard 130 or mouse 140, the frame image acquisition unit 106 sequentially reads the frame image data of each frame stored in the memory. At this time, the display control unit 104 sequentially displays still images represented by the read frame image data of each frame on the display 120. Thereby, a moving image is displayed on the display 120.

  Also, when a user inputs a frame image acquisition instruction from the keyboard 130 or mouse 140, the frame image acquisition unit 106 acquires a plurality of frame image data and stores them in a memory area separate from the moving image data. . Further, when a still image generation instruction is input by the user from the keyboard 130 or the mouse 140, the correction amount estimation unit 108 estimates the correction amount of the positional deviation between the plurality of frame images and increases the resolution as will be described later. The processing unit 110 corrects misalignment between a plurality of frame images based on the estimated correction amount, and also uses a plurality of frame image data representing the corrected low-resolution frame images as a high-resolution still image ( Hereinafter, image data (hereinafter also referred to as “high resolution still image data”) representing “high resolution still image”) is generated. The display control unit 104 displays a high-resolution still image represented by the generated high-resolution still image data on the display 120. Furthermore, when a printing instruction is input by the user from the keyboard 130 or the mouse 140, the print control unit 112 can generate print data based on the generated high-resolution still image data, and can cause the printer 200 to print the print data. In addition, when a save instruction is input from the keyboard 130 or the mouse 140 by the user, the save control unit 114 can store the generated high-resolution still image data in a storage medium such as a CD-RW or a hard disk.

B. Still image generation procedure:
When an application program for generating a high-resolution still image is executed, first, moving image data supplied from the digital video camera 300 as a moving image including a frame image specified as a still image generation target. Is selected, or a moving image represented by moving image data stored in a storage medium such as a CD-RW is selected as a moving image file. When one of the selections is made by the user, processing corresponding to this is started. In the following, an example of processing when a moving image represented by moving image data stored in a CD-RW is selected will be described.

  FIG. 2 is an explanatory diagram showing a procedure for generating high-resolution still image data. In this high-resolution still image data generation procedure, frame image acquisition (step S10), correction amount estimation (step S20), and motion correspondence synthesis (step S30) are sequentially executed to generate high-resolution still image data. Is done. The high-resolution still image represented by the generated high-resolution still image data can be displayed and printed. In addition, the generated high-resolution still image data can be saved. Below, each process is demonstrated in order.

B1. Frame image acquisition processing:
In the frame image acquisition process in step S10, a frame image (hereinafter, also referred to as “high resolution target image”) for which a high resolution still image is to be generated from a moving image to be reproduced and displayed is designated, and a plurality of frames are described as described below. Image data (frame image data) representing a frame image of the frame is acquired.

  FIG. 3 is an explanatory diagram showing a user interface screen for the user to specify a high resolution target image during playback of a moving image. The user interface screen NW0 (hereinafter also simply referred to as “high resolution target designation screen NW0”) is displayed on the display 120 by the display control unit 104 (see FIG. 1).

  When the user operates the cursor Cs through the keyboard 130 or the mouse 140 and presses the “file” button A0, an operation screen (not shown) for opening the file is displayed. In the following, the description “operating the cursor Cs through the keyboard 130 or the mouse 140” is simply described as “operating the cursor Cs”. In this operation screen, when a moving image showing a moving image to be reproduced is selected and specified from the moving image file list, moving image data of the specified moving image file is read from a storage medium (here, a CD-RW). And stored in a memory (not shown). Then, the moving image represented by the read moving image data is automatically or when the user operates the cursor Cs and presses the “play” button A1, the image display field SW0 in the high resolution target designation screen NW0. Will be displayed. In this example, a case where a moving image represented by moving image data stored in a CD-RW as a moving image file is selected as an example is described. However, moving image data supplied from the digital video camera 300 is selected. When the moving image to be displayed is selected, as described above, the process of selecting and specifying the moving image file indicating the moving image to be reproduced from the moving image file list is omitted.

  Here, when the reproduction of the moving image is stopped or paused by the user operating the cursor Cs and pressing the “stop” button A2 or the “pause” button A3 during the reproduction of the moving image, The frame image displayed in the image display field SW0 is designated as the resolution enhancement target image.

  Then, the frame image designated as the resolution enhancement target image is used as a reference frame image (hereinafter also referred to as “reference frame image”), and a plurality of frames of frame image data continuous in time series are still images to be described later. Obtained for generation and stored in memory. In the following embodiments, it is assumed that four frames of frame image data are acquired in chronological order starting with the reference frame image.

  Note that when the user operates the cursor Cs and presses the “frame advance” button A4 or the “frame return” button A5, the frame displayed on the image display field SW0 at the time of pause or stop is moved forward or backward. An image is displayed. Also in this case, a frame image displayed after frame advance or frame return can be designated as a high resolution target image, and frame image data of a plurality of frames can be acquired based on this.

  Further, the image data representing the designated resolution enhancement target image can be stored in a storage device such as a CD-RW or a hard disk by the save control unit 114 by pressing the “Save” button A12. When the “print” button A13 is pressed, the print control unit 112 generates print data based on the image data representing the designated resolution-enhanced image and prints it with the printer 200 (see FIG. 1). You can also

  As described above, the function of displaying a moving image in the image display field SW0 is executed by the display control unit 104, and the function of acquiring a plurality of frame image data during playback of a moving image is the frame image acquiring unit 106 (see FIG. 1). ) Is executed.

  Note that the frame image data is composed of gradation data (hereinafter also referred to as “pixel data”) indicating the gradation value (hereinafter also referred to as “pixel value”) of each pixel in the dot matrix form. The pixel data is YCbCr data composed of Y (luminance), Cb (blue color difference), Cr (red color difference), RGB data composed of R (red), G (green), and B (blue).

  After designating the image to be increased in resolution, when the cursor Cs is operated and the “enhance resolution” button A11 is pressed, processing for generating high-resolution still image data is started, and correction amount estimation processing (step S20 in FIG. 2). And a motion corresponding | compatible synthesis process (step S30) is performed in order.

B2. Correction amount estimation processing:
In the correction amount estimation process in step S20, a correction amount estimation process for correcting a deviation (positional deviation) occurring between the acquired four frames is executed. In the estimation of the correction amount, the other three frames other than one reference frame among the four frames are set as target frames, and the correction amount for correcting the positional deviation with respect to the reference frame is estimated for each target frame. As described above, the function of estimating the correction amount is executed by the correction amount estimation unit 108 (see FIG. 1).

  In the following, in order to facilitate the description of the correction amount estimation processing of the embodiment, first, an outline of the shift correction and the correction amount estimation method will be described, and then the correction amount estimation processing of the embodiment will be described.

B2.1. Outline of displacement correction:
FIG. 4 is an explanatory diagram showing a positional shift between the frame image of the reference frame and the frame image of the target frame. FIG. 5 is an explanatory diagram showing correction of positional deviation between the reference frame image and the target frame image.

  In the following description, the acquired number of four frames (hereinafter also referred to as “frame number”) is a (a = 0, 1, 2, 3), the frame with frame number a is called frame a, The image of frame a is referred to as frame image Fa. For example, a frame with frame number a 0 is referred to as frame 0, and its image is referred to as frame image F0. Note that frame 0 is a reference frame, and frames 1 to 3 are target frames. The frame image F0 of the reference frame is also referred to as a reference frame image, and the frame images F1 to F3 of the target frame are also referred to as target frame images.

  4 and 5, a description will be given taking as an example the positional deviation and the correction of the positional deviation of the target frame image F3 relative to the reference frame image F0 among the target frame images F1 to F3.

  The positional deviation of the image is represented by a combination of translational (horizontal or vertical) deviation and rotational deviation. In FIG. 4, in order to easily show the shift amount of the target frame image F3 with respect to the reference frame image F0, the edge of the reference frame image F0 and the edge of the target frame image F3 are shown in an overlapping manner, and on the reference frame image F0. Assuming that a virtual cross image X0 is added at the center position and this cross image X0 is shifted in the same manner as the target frame image F3, the cross image X3, which is the shifted image, is shown on the target frame image F3. ing. Further, in order to show the shift amount more easily, the reference frame image F0 and the cross image X0 are indicated by thick solid lines, and the target frame image F3 and the cross image X3 are indicated by thin broken lines.

  In the present embodiment, the translational displacement amount is represented by “um” in the horizontal direction, “vm” in the vertical direction, “δm” in the vertical direction, and the target frame image Fa (a = 1, 2, 3). The deviation amounts are represented as “uma”, “vma”, and “δma”. For example, as shown in FIG. 4, the target frame image F3 has a translational shift and a rotational shift with respect to the reference frame image F0, and the shift amounts are expressed as um3, vm3, and δm3.

  Here, in order to synthesize the target frame images F1 to F3 with the reference frame image F0, the positional displacement of each pixel of the target frame images F1 to F3 so that the target frame images F1 to F3 coincide with the reference frame image F0. Will be corrected. As the translation correction amount used for this purpose, the horizontal direction is “u”, the vertical direction is “v”, the rotation correction amount is “δ”, and the correction amount for the target frame image Fa (a = 1, 2, 3). Are represented as “ua”, “va”, and “δa”. For example, the correction amounts for the target frame image F3 are expressed as um3, vm3, and δm3.

  Here, the correction means that the position of each pixel of the target frame image Fa (a = 1, 2, 3) is a position where the movement of ua in the horizontal direction, the movement of va in the vertical direction, and the rotation of δa are performed. It means to move. Therefore, the correction amounts ua, va, δa for the target frame image Fa (a = 1, 2, 3) are expressed by the relationship of ua = −uma, va = −vma, δa = −δma. For example, the correction amounts u3, v3, and δ3 for the target frame image F3 are represented by u3 = −um3, v3 = −vm3, and δ3 = −δm3.

  From the above, for example, as shown in FIG. 5, by correcting the position of each pixel of the target frame image F3 using the correction amounts u3, v3, and δ3, the target frame image F3 is changed to the reference frame image F0. Can be matched. At this time, when the corrected target frame image F3 and the reference frame image F0 are displayed on the display 120, as shown in FIG. 5, it is estimated that the target frame image F3 partially matches the reference frame image F0. Is done. In order to show the correction result in an easy-to-understand manner, the same virtual cross image X0 and cross image X3 as in FIG. 4 are shown in FIG. 5, and as shown in FIG. X3 coincides with the cross image X0.

  Similarly, the target frame images F1 and F2 are also corrected using the values of the correction amounts u1, v1, δ1, and u2, v2, δ2, and the positions of the pixels of the target frame images F1 and F2 are determined. Can be replaced.

  Note that the above-mentioned “partial match” means the following. That is, as shown in FIG. 5, for example, the hatched area P1 is an image of an area that exists only in the target frame image F3, and no image of the corresponding area exists in the reference frame image F0. As described above, even if the above correction is performed, an image of an area that exists only in the reference frame image F0 or only in the target frame image F3 is generated due to the shift. The reference frame image F0 is not completely matched but is partially matched.

  By the way, the correction amounts ua, va, and δa for each target frame image Fa (a = 1, 2, 3) are the reference frame in the correction amount estimation unit 108 (see FIG. 1), in particular, the correction amount estimation processing unit 108b. Based on the image data of the image F0 and the image data of the target frame images F1 to F3, it is calculated as an estimated amount using a predetermined calculation formula such as a pattern matching method or a gradient method. The calculated correction amounts ua, va, δa are stored in a predetermined area in a memory (not shown) as translation correction amount data and rotation correction amount data. Hereinafter, a correction amount estimation method will be described.

B2.2. Correction amount estimation method:
FIG. 6 is an explanatory diagram showing the correction amount of the target frame image with respect to the reference frame image. The figure shows a state where the target frame image Ft is arranged so as to be superimposed on the reference frame image Fr by correcting translational deviation and rotational deviation. Orthogonal coordinates (x2, y2) with the center of the target frame image Ft as the origin, the horizontal direction as the x2 axis, and the vertical direction as the y2 axis, the center of the reference frame image Fr as the origin, the horizontal direction as the x1 axis, and the vertical direction Is offset with respect to the Cartesian coordinates (x1, y1) with the y1 axis as the horizontal translation correction amount at this time, u is the translation correction amount in the vertical direction, v is the translation correction amount in the vertical direction, and the rotation correction amount is the center of the target frame image Ft. Let δ be the origin. At this time, the conversion equation for converting the coordinates of the target frame image Ft into the coordinates on the reference frame image Fr is expressed by the following equation using the translation correction amount (u, v) and the rotation correction amount δ as variables.

x1 ＝ cosδ ・ (x2 + u) −sinδ ・ (y2 + v) ... (1)
y1 ＝ sinδ ・ (x2 + u) + cosδ ・ (y2 + v) ... (2)

  Since the time difference between the reference frame and the target frame is very small, δ is considered to be a minute amount. At this time, since COSδ≈1 and sinδ≈δ, the above equation can be replaced as follows.

x1 = (x2 + u) −δ · (y2 + v) (3)
y1 = δ · (x2 + u) + (y2 + v) (4)

  Therefore, as described above, in order to arrange the target frame image Ft and the reference frame image Fr so as to overlap each other, the above formulas (3) and (4) are set so that the target frame image Ft matches the reference frame image Fr. ) To convert the coordinate data of each pixel of the target frame image Ft into coordinate data on the reference frame image Fr.

  Here, the variables U, V, and δ in the above formulas (3) and (4) are the pixel values (pixel data) of each pixel between the reference frame image Fr and the target frame image Ft, as described below. For example, it can be estimated by the least square method based on a gradient method (gradient method) that estimates the position of a pixel in units smaller than one pixel using a gradation value (gradation data) of luminance.

  FIG. 7 is an explanatory diagram illustrating a translation correction amount estimation method using a gradient method. FIG. 7A shows the pixels and brightness of the reference frame image Fr and the target frame image Ft. In FIG. 7A, for example, (x1i, y1i) represents the coordinates of a pixel on the reference frame image Fr, and B1 (x1i, y1i) represents the luminance thereof. Here, i is a number for distinguishing each pixel used for estimating the correction amount. FIG. 7B shows the principle of the gradient method.

  Here, the pixel at the coordinates (x2i, y2i) on the target frame image Ft is between the coordinates (x1i to x1i + 1, y1i to y1i + 1) of the reference frame image Fr, that is, the coordinates (x1i + Δx, y1i + Δy) between the pixels. It will be explained as a thing.

As shown on the left side of FIG. 7B, the pixel of the coordinate (x2i, y2i) in the target frame image Ft is between the coordinates (x1i to x1i + 1, y1i) of the reference frame image Fr, that is, the coordinates between the pixels. Suppose that (x1i + Δx, y1i)
Px = B1 (x1i + 1, y1i)-B1 (x1i, y1i) ... (5)
Then,
Px · Δx = B2 (x2i, y2i)-B1 (x1i, y1i) ... (6)
This relationship should hold. In this case, B1 (x1i, y1i) and B2 (x2i, y2i) are simply represented by B1, B2,
{Px · Δx− (B2−B1)} ² = 0 (7)
If Δx that satisfies is obtained, the lateral translation correction amount (Δx = u) of the target frame image Ft can be obtained. Actually, Δx is calculated for each pixel, and an average is obtained as a whole.

Similarly, as shown on the left side of FIG. 7B, the pixel at the coordinates (x2i, y2i) in the target frame image Ft is between the coordinates (x1i, y1i to y1i + 1) of the reference frame image Fr, that is, between the pixels. And (xi1, y1i + Δy)
Py = B1 (x1i, y1i + 1)-B1 (x1i, y1i) ... (8)
Then,
Py · Δy = B2 (x2i, y2i)-B1 (x1i, y1i) ... (9)
This relationship should hold. Therefore, B1 (x1i, y1i) and B2 (x2i, y2i) are simply expressed as B1, B2,
{Py · Δy− (B2−B1)} ² = 0 (10)
If Δy that satisfies is obtained, the translational correction amount (Δy = v) in the vertical direction of the target frame image Ft can be obtained. Actually, Δy is calculated for each pixel, and the average is taken as a whole.

In the above description, the case where only one of the x-axis direction (horizontal direction) and the y-axis direction (vertical direction) is considered is described. When considering both the x-axis direction and the y-axis direction, expand this,
S ² = Σ {Px · Δx + Py · Δy− (B2−B1)} ² (11)
[Delta] x and [Delta] y that minimize the value may be obtained by the method of least squares. The (Δx, Δy) thus obtained corresponds to the translation correction amount (u, v).

  The method for obtaining the translational correction amount (u, v) when the target frame image Ft is overlapped with the reference frame image Fr by being translated in the x-axis direction and the y-axis direction by the gradient method is described above. did. The present invention further considers the case where the target frame image Ft is rotated and superimposed on the reference frame image Fr. Hereinafter, a method for obtaining the rotation correction amount will be described.

FIG. 8 is an explanatory diagram schematically showing the amount of pixel rotation correction. If the distance from the origin O of the coordinates (x1, y1) of the reference frame image Fr is r and the rotation angle from the x1 axis is θ, r and θ are expressed by the following equations.
r = (x1 ² + y1 ² ) ^1/2 (12)
θ = tan ⁻¹ (y1 / x1) ... (13)

Here, it is assumed that there is no translational shift of the target frame image Ft with respect to the reference frame image Fr, and only a rotational shift has occurred, and the pixel at the coordinates (x2, y2) in the target frame image Ft is on the reference frame image Fr. Assume that the coordinates (x1 ′, y1 ′) are rotated by the rotation correction amount δ from the position of the coordinates (x1, y1). The movement amount Δx in the x1 axis direction and the movement amount Δy in the y1 axis direction based on the rotation correction amount δ can be obtained by the following equations.
Δx = x1′−x1 ≒ −r · δ · sinθ = −δ · y1 (14)
Δy = y1′−y1 ≒ r · δ · cosθ = δ · x1 (15)

Therefore, Δx and Δy in the above equation (11) can be expressed as the following equations, considering not only the translation correction amount (u, v) but also the above equations (14) and (15) based on the rotation correction amount δ. it can.
Δx = u−δ · y1 (16)
Δy = v + δ · x1 (17)

Substituting the above equations (16) and (17) into the above equation (11), the following equation is obtained.
S ² = Σ {Px · (u−δ · y) + Py (v + δ · x) − (B2−B1)} ² (18)

That is, the coordinates of the reference frame image Fr (x1i, y1i) as the coordinate values and brightness values of all pixels of the subject frame image Ft when substituted into equation (18), the correction quantity u of the ^{S 2} to minimize, v and δ can be obtained by the method of least squares. Here, i is a number for distinguishing each pixel used for estimating the correction amount. The correction amounts u, v, and δ are expressed by the following equations.

また、M02，M11,M20，m_ｕ，m_ｖ，m_δ，c_ｕ，c_ｖ，c_δ，dは、以下の式で表される。

Further, M02, M11, M20, m _u , m _v , m _δ , c _u , c _v , c _δ , and d are expressed by the following equations.

  Therefore, the translational correction amount (u, v) and the rotational correction amount (δ) for correcting the translational deviation and rotational deviation of less than one pixel between the reference frame image Fr and the target frame image Ft are expressed by the above equation (18). ) To (33) can be obtained by substituting the pixel value (luminance value) of each pixel of the reference frame image Fr and the target frame image Ft.

  The translation correction amount and the rotation correction amount are estimated in the case where the translational deviation between the reference frame image Fr and the target frame image Ft is less than one pixel. Therefore, it is preferable that the translational deviation in units of pixels is roughly corrected using an estimation method such as a general pattern matching method before performing the above estimation.

B2.3. Example correction amount estimation:
As described above, the translation correction amount (u, v) and the rotation correction amount (δ) for correcting the translational deviation and rotational deviation of less than one pixel between the reference frame image Fr and the target frame image Ft are: It can be obtained by substituting the pixel values (luminance values) of the respective pixels of the reference frame image Fr and the target frame image Ft into the above equations (18) to (33).

  Here, in the conventional correction amount estimation processing, the correction amount is estimated using all the pixels of the target reference frame image and the target frame image, and information used for estimating the correction amount Since the amount (data amount) becomes very large, it takes time to estimate the correction amount. As a result, the processing time required for generating the high-resolution still image data is long.

  Therefore, in the correction amount estimation process of the present embodiment, the time required for the correction amount estimation process is shortened as described below.

  FIG. 9 is an explanatory diagram specifically showing the correction amount estimation processing in step S20 of FIG. In step S210, a range of image data used for estimation of the correction amount, that is, a range of pixel data representing an image value used for calculation of the correction amount according to the above formulas (18) to (33) (hereinafter, “ It is also referred to as “correction amount estimation range”.

  FIG. 10 is an explanatory diagram showing a user interface screen for the user to specify the correction amount estimation range. The user interface screen NW1 (hereinafter referred to as “correction amount estimation range designation screen NW1”) is displayed on the display 120 by the display control unit 104 (see FIG. 1).

  First, the user operates the cursor Cs and presses the selection button A23 in the range designation target selection field SW2, thereby selecting either “background” or “moving” as the range designation target. FIG. 10A shows a state in which “background” is designated, and FIG. 10B shows a state in which “moving thing” is designated.

  Next, by operating the cursor Cs and pressing one of the two selection buttons A21 and A22 shown in the range designation method selection field SW1, the “center designation” and “start / end designation” are designated as the range designation methods. ”Is selected. As a result, the cursor Cs can be operated in the image display field SW0, and the correction amount estimation range can be designated in accordance with the selected range designation method from the resolution enhancement target images displayed in the image display field SW0. . FIG. 10A shows a state in which “center designation” is selected, and FIG. 10B shows a state in which “start point / end point designation” is selected. Of course, “start / end point designation” may be selected in the state of “background” in FIG. 10A, and “center” in the state of “moving” in FIG. 10B. “Specify” may be selected. Note that “movement” is mainly a local image change in a moving image, and means a movement of an object in the moving image.

  In the range designation method selected by the first selection button A21, the center position of the range designation area to be designated is designated by operating the cursor Cs in the resolution enhancement target image displayed in the image display field SW0. Is the method. In this method, an area (hereinafter, also simply referred to as “rectangular area”) indicated by a rectangular frame having a predetermined size centered on a designated position is designated as a range designation area.

  In the range designation method selected by the second selection button A22, a rectangle indicating a rectangular area to be set as a range designation area by operating the cursor Cs in the high resolution target image displayed in the image display field SW0. This is a method for designating the start and end points on the diagonal of the frame. In this method, a rectangular area having an arbitrary size represented by two points on the diagonal line is designated as the range designation area.

  Here, as shown in FIG. 10A, when “background” is designated as the range designation target, the designated range designation area is set as the correction amount estimation range. In this case, for each of the four frame images, image data representing a portion of the image corresponding to the range designation region (pixel data representing the pixel value of each pixel) is used for correction amount estimation.

  On the other hand, as shown in FIG. 10B, when “moving” is designated, a region excluding the range designated region is set as the correction amount estimation range. In this case, for each of the four frame images, image data representing a portion of the image corresponding to the region excluding the range designation region (pixel data representing the pixel value of each pixel) is used for correction amount estimation.

  In the above example, the case where either “background” or “moving” can be selected as the range specification target is shown, but the present invention is not limited to this, and any one of the correction specification targets Only one side may be selected.

  As described above, the function of setting the correction amount estimation range by specifying the correction amount estimation range is executed by the correction amount estimation range setting unit 108a (see FIG. 1).

  Then, when the “execute” button A14 is pressed by operating the cursor Cs, in step S220 in FIG. 9, using the above equations (18) to (33), a shift occurring between the four frame images. Calculation of a correction amount for correcting (positional deviation) is executed. In the calculation of the correction amount, only the image data of the portion corresponding to the correction amount estimation range set in step S210 out of the image data of each frame image is substituted into the equations (18) to (33), respectively. Thus, the correction amount for correcting the positional deviation of each target frame with respect to the reference frame is calculated. As described above, the function of calculating the correction amount is executed by the correction amount estimation processing unit 108b (see FIG. 1).

B3. Motion compatible composition processing:
After completion of the correction amount estimation processing in step S20 in FIG. 2, in step S30, the four frame image data is superimposed and superimposed on the four frame image data by correcting each of the obtained correction amounts. Are combined to generate high-resolution still image data. However, as will be described later, it is determined whether or not “movement” has occurred between the frames, and generation of still image data corresponding to this is executed. As described above, the function of generating image data corresponding to the motion is executed by the high resolution processing unit 110 (see FIG. 1).

  In the following, in order to facilitate the description of the motion corresponding synthesis process of the embodiment, first, a synthesis process that does not correspond to motion (motion non-corresponding synthesis process) will be described, and then the motion corresponding synthesis process of the embodiment. Will be described.

B3.1. Non-motion compatible composition processing:
Basically, in the motion non-corresponding synthesis processing, the image data of the target frame is corrected based on the estimated correction amount so that there is no deviation from the image data of the reference frame, and the image data of the reference frame and the target frame are corrected. High-resolution still image data is generated by superimposing and synthesizing the image data. Further, among each pixel (hereinafter also referred to as “generated pixel”) that constitutes the generated high-resolution still image (hereinafter also referred to as “generated image”), it does not exist in either the reference frame or the target frame. For a pixel, a predetermined interpolation process is performed using pixel data representing pixel values of pixels existing around the generated pixel (gradation data representing gradation values). Hereinafter, this combining process will be briefly described with reference to FIGS. 11 and 12.

  FIG. 11 is an explanatory view showing, in an enlarged manner, a state in which the reference frame image F0 and the target frame images F1 to F3 are arranged with the deviation corrected. In FIG. 11, each pixel of the generated image G is indicated by a black circle, each pixel of the reference frame image F0 is indicated by a white quadrilateral, and each pixel of the target frame images F1 to F3 after correction is Shown as hatched quadrilaterals. Note that, in the motion non-corresponding synthesis processing and the motion correspondence synthesis processing described below, the pixel density of the generated image G is increased to a pixel density 1.5 times as high as that of the reference frame image F0. And Further, it is assumed that each pixel of the generated image G is in a position where it overlaps with each pixel of the reference frame image F0 every two pixels. However, the pixel of the generated image G does not necessarily need to be positioned so as to overlap each pixel of the reference frame image F0. For example, all the pixels of the generated image G0 may be located in the middle of the pixels of the reference frame image F0, and can be in various positions. Also, the magnification for increasing the resolution is not limited to 1.5 times in length and width, and various magnifications can be used.

  Hereinafter, description will be given focusing on a pixel G (j) in the generated image G. Here, the variable j indicates a number for distinguishing all the pixels of the generated image G. For example, the variable j starts from the upper left pixel and continues to the upper right pixel in order, and thereafter, from the lower left pixel in turn. To the rightmost pixel, and finally the lower right pixel. In each of the frame images F0, F1, F2, and F3, four pixels (hereinafter referred to as “neighboring pixels”) that are closest to this pixel (hereinafter referred to as “target pixel”) G (j). Distances L0, L1, L2, and L3 between F (0), F (1), F (2), and F (3) and the target pixel G (j) are calculated. Then, a neighboring pixel at the nearest distance (hereinafter referred to as “nearest neighbor pixel”) is determined. In the example of FIG. 11, since L3 <L1 <L0 <L2, the pixel F (3) of the target frame image F3 is determined as the nearest pixel of the target pixel G (j). Note that the nearest pixel to the target pixel G (j) is hereinafter referred to as the nearest pixel F (3, i) assuming that it is the i-th pixel of the target frame image F3. However, i here is different from the number i of the target pixel of the reference frame image Fr in the correction amount estimation processing.

  The above procedure is executed for all the pixels in the generated image G in the order of j = 1, 2, 3,... That is the number of the pixel of interest G (j), and the nearest neighbor for each pixel. The pixel will be determined.

  Pixel data of the target pixel G (j) is obtained by using pixel data of the determined nearest neighbor pixel and each of the other pixels surrounding the target pixel G (j) in the frame image including the nearest neighbor pixel. It is generated by various interpolation processes such as a linear method and a bi-cubic method.

  FIG. 12 is an explanatory diagram showing interpolation processing by the bi-linear method. Since the target pixel G (j) is a pixel that does not exist in any of the reference frame image F0 and the target frame images F1 to F3 after the positional deviation correction, there is no pixel data. Therefore, as described above, when the pixel F (3) of the target frame image F3 is determined as the nearest pixel F (3, i) of the target pixel G (j), as shown in FIG. In addition to F (3, i), an area defined by three pixels F (3, i + 1), F (3, k), and F (3, k + 1) surrounding the target pixel G (j) The pixel data of the target pixel G (j) can be interpolated by dividing into four sections in (j) and adding the weighted pixel data at the diagonal positions according to the area ratio. Here, k represents a pixel number obtained by adding the number of pixels in the horizontal direction of the frame image F3 to the i-th pixel.

  In addition to the bi-linear method, various interpolation methods such as a bi-cubic method and a two-arrest neighbor method can be used for the interpolation processing method.

  Here, the following points are not taken into consideration in the motion non-corresponding synthesis process described above. FIG. 13 is an explanatory diagram in a case where the motion non-corresponding synthesis process is executed when a motion has occurred between a plurality of frame images. The lower part of the figure shows a generated image G when the upper four frame images F0 to F3 are combined. The four frame images F0 to F3 show moving images in which a car moving from the left to the right of the screen is photographed, and the positions of the cars are moving sequentially. In the motion non-corresponding synthesis processing, since the nearest pixel is used to perform interpolation processing of the pixel of interest regardless of whether or not the determined nearest pixel is a pixel that moves between frames, FIG. As shown, the car image in the generated image G may be a multiple image.

  As described below, the motion-corresponding synthesis process synthesizes a plurality of frame images in consideration of “motion” that is a cause of generating the above-described multiple images.

B3.2. Motion compatible composition processing:
Also in the motion corresponding synthesis process, the generated image G is obtained by superimposing the corrected four frame images F0 to F3 by correcting the four frame images F0 to F3 in FIG. 13 based on the estimated correction amount. It is assumed that the resolution is increased to a pixel density 1.5 times as long and wide as the reference frame image F0.

  First, among the generated pixels of the generated image G, for each target image G (j), each neighboring pixel is obtained in each of the frame images F0 to F3, and from among the obtained four neighboring pixels. The nearest neighbor pixel is determined.

  Subsequently, a motion determination is performed on the obtained nearest neighbor pixel and the reference frame image F0. However, when the nearest pixel is the pixel of the reference frame image F0, the motion is not determined, and the pixel data of the target pixel G (j) is the pixel of the reference frame image F0 surrounding the target pixel G (j). And generated by various interpolation processes such as a bi-linear method, a bi-cubic method, and a nearest neighbor method. On the other hand, when the nearest pixel is a pixel of the target frame images F1 to F3, the motion is determined by the motion determination method described below.

  Hereinafter, for easy understanding of the motion determination method, the reference frame image is assumed to be Fr and the target frame image is assumed to be Ft. FIG. 14 is an explanatory diagram showing preconditions for explaining a motion determination method. In the figure, one filled quadrilateral indicates a pixel in which motion is determined in the target frame image Ft, that is, a pixel corresponding to the nearest pixel (hereinafter also referred to as “determination pixel”) Fpt. . Further, the four open quadrilaterals arranged in a grid form indicate the four pixels Fp1, Fp2, Fp3, and Fp4 surrounding the determination pixel Fpt in the reference frame image Fr. The luminance value of the determination pixel Fpt is Vtest, and the luminance values of the four pixels Fp1, Fp2, Fp3, and Fp4 are V1, V2, V3, and V4. The position (Δx, Δy) in the lattice formed by the four pixels Fp1, Fp2, Fp3, and Fp4 is based on the position of the upper left pixel Fp1, with the horizontal direction being the x axis and the vertical direction being the y axis. It is possible to take only values between 0 and 1, respectively.

  Further, in FIG. 15 described below, for easy understanding of the description, the position of the determination pixel Fpt is a one-dimensional position, that is, between two pixels Fp1 and Fp2 arranged in the x direction in the lattice. A case where the coordinates are at the coordinates (Δx, 0) is drawn as an example.

  FIG. 15 is an explanatory diagram illustrating an example of a motion determination method. When the determination pixel Fpt does not move, the luminance value Vtest of the determination pixel Fpt is an estimated luminance value V obtained by, for example, the bilinear method based on surrounding pixels surrounding the determination pixel Fpt in the reference frame image Fr. Equals'. On the other hand, if there is a motion in the determination pixel Fpt, the luminance value Vtest of the determination pixel Fpt is a value different from the estimated luminance value V ′ according to the amount of movement. Therefore, considering the above points, the movement of the determination pixel Fpt can be determined as described below.

First, the estimated luminance value V ′ can be obtained from the luminance values of the pixels Fp1 and Fp2 located on both sides of the determination pixel Fpt in the reference frame image Fr by, for example, the following expression representing one-dimensional linear interpolation.
V ′ = (1−Δx) · V1 + Δx · V2 (34)
However, V1 and V2 indicate the luminance values of the pixels Fp1 and Fp2 of the reference frame located on both sides of the determination pixel Fpt. Δx indicates a position in the x-axis direction with respect to the left pixel Fp1 in the determination pixel between the two pixels Fp1 and Fp2.

When the difference between the luminance value Vtest of the determination pixel Fpt and the estimated luminance value V ′ is larger than the threshold value ΔVth, it is determined that the determination pixel Fpt has a motion. That is, when the following expression is satisfied, it is determined that there is no movement, and when the following expression is not satisfied, it is determined that there is movement.
| Vtest−V ′ | <ΔVth (35)

  In addition, as in the above equation (35), when the absolute value of the difference between the luminance value Vtest and the estimated luminance value V ′ is smaller than the threshold value ΔVth, it is determined that there is no movement as follows. Depending on the reason. That is, as described above, for the target frame image Ft including the determination pixel, a correction amount for correcting the positional deviation with respect to the reference frame image Fr is obtained, the positional deviation is corrected according to the correction amount, and the reference frame image Fr is corrected. Are superimposed on each other. Ideally, the displacement between the frames is ideally corrected completely. However, in practice, it is often difficult to completely eliminate the displacement, and noise such as an overlay error occurs. . For this reason, even if the determination pixel does not move at all, a difference may occur between the estimated luminance value V ′ and the luminance value Vtest of the determination pixel.

  Therefore, in order not to detect a difference caused by this noise as a motion, an area having a threshold ΔVth in the vertical direction from the estimated luminance value V ′ is set as a range in which there is no motion.

  Note that the magnitude of the threshold ΔVth is preferably set to about ΔVth = 3, for example, when the luminance value can take a gradation of 8 bits, that is, a value of 0 to 255. However, the present invention is not limited to this, and it is preferable to set various values according to the number of gradations of the luminance value, the accuracy of positional deviation correction, and the like.

  In the above description, the case where the coordinate of the determination pixel Fpt is the coordinate (Δx, 0) with the pixel Fp1 of the reference frame image Fr as the origin is described as an example, but the coordinate of the determination pixel Fpt is (0, The same applies to Δy).

In addition, when the coordinates of the determination pixel Fpt are two-dimensional coordinates (Δx, Δy), the estimated luminance value V ′ is obtained by, for example, the following expression representing two-dimensional linear interpolation by the bilinear method. That's fine.
V ′ = (1−Δx) · (1−Δy) · V1 + Δx · (1−Δy) · V2 + (1−Δx) · Δy · V3 + Δx · Δy · V4 (36)

  If it is determined in the motion determination by the motion determination method described above that there is no motion in the nearest pixel, the pixel data of the pixel of interest G (j) is the nearest pixel and the target frame image including the nearest pixel. It is generated by various interpolation processes such as the bi-linear method, the bi-cubic method, and the two-arrest neighbor method using the pixel data of other pixels surrounding the pixel of interest G (j).

  On the other hand, if it is determined in the motion determination that the nearest pixel is in motion, the nearest neighbor pixel next to the nearest pixel with respect to the target pixel G (j) (hereinafter also referred to as the next neighborhood pixel). The above-described motion determination is performed. If it is determined that there is no movement in the next neighboring pixel, the pixel data of the pixel of interest G (j) is included in the next neighboring pixel and the target frame image including this next neighboring pixel as described above. Using each pixel data of other pixels surrounding the target pixel G (j), it is generated by various interpolation processes such as a bi-linear method, a bi-cubic method, and a two-arrest neighbor method.

  If it is determined in the motion determination for the next neighboring pixel that the next neighboring pixel has a motion, a pixel next to the next neighboring pixel is obtained, and the same motion determination is performed for the pixel.

  When the above procedure is repeated and it is determined that there is movement in all neighboring pixels of the target frame images F1 to F3 with respect to the target pixel G (j), the pixel data of the target pixel G (j) j) is generated by various interpolation processes such as the bi-linear method, the bi-cubic method, and the two-arrest neighbor method, using the pixel data of the pixels of the reference frame image F0 that surrounds j).

  In this way, the interpolation process is performed on the target pixel G (j), and the same interpolation process is performed in the order of j = 1, 2, 3,. This is performed for all the pixels in the generated image G.

  As described above, in the motion correspondence synthesis process, the motion determination is performed on the neighboring pixels of the target frame image located in the vicinity of the target pixel G (j) in the closest order, and only when it is determined that the pixel has no motion. Interpolation of the pixel of interest G (j) using each pixel data of the pixel determined not to move and the other pixels surrounding the pixel of interest G (j) in the target frame image including this pixel Processing is performed. Further, when it is determined that all the pixels of the target frame image located in the vicinity of the target pixel G (j) are in motion, the image data of the pixels of the reference frame image surrounding the target pixel G (j) is used. , Interpolation processing of the target pixel G (j) is performed.

  Therefore, in this motion correspondence synthesis process, the target frame image including the moving pixels can be excluded from the target of the frame used for the interpolation process, so that the multiplexing of the automobiles generated in the generated image G in FIG. It is possible to generate high-resolution still image data by preventing a moving part from becoming a multiple image like an image. can do.

  In the above-described motion-corresponding synthesis process, an example has been described in which selection is performed in the order of neighboring pixels close to the target pixel, and the movement of the selected pixel with respect to the reference image is determined. For example, the motion may be determined by simply selecting frames 0, 1, 2, and 3 in this order. That is, the movement may be determined in any order. Further, in the motion corresponding composition processing described above, when the frame including the selected nearest pixel is a reference frame, the motion is not determined. In this case, the motion is also determined. You may make it perform.

  Incidentally, FIG. 16 is an explanatory diagram showing an interface screen displayed during the high-resolution still image data generation process. As shown in FIG. 16, the correction amount calculation process in step S210 (see FIG. 9) during the correction amount estimation in step S20 (see FIG. 2) and the motion correspondence synthesis process in step S40 (see FIG. 2) are executed. If so, a progress bar SW3 indicating the progress of the process is displayed at any position in the correction amount estimation range designation screen NW1, as shown in FIG. Thereby, the user can know the progress of the high resolution processing.

  When the high resolution processing is completed, the display of the progress bar SW3 is stopped, and the generated high resolution still image data is stored in a memory (not shown). Then, the display controller 104 (FIG. 1) displays an image represented by the generated high-resolution still image data in the image display field SW0.

  Note that the storage controller 114 stores the generated high-resolution still image data in a storage medium such as a CD-RW or a hard disk by pressing the “save” button A12 in the correction amount estimation range designation screen NW1. Can do. In addition, by pressing the “print” button A13, the print control unit 112 can generate print data based on the generated high-resolution still image data and print it with the printer 200 (see FIG. 1).

C. effect:
As described above, in the still image generating apparatus according to the present embodiment, a part of the high resolution target image is set as the correction amount estimation range in the correction amount estimation processing. Then, the correction amount is obtained using image data of each frame image corresponding to the set correction amount estimation range. As a result, the amount of information used for estimating the correction amount can be reduced, and the processing time for estimating the correction amount can be shortened. Then, by shortening the processing time in the correction amount estimation processing, it is possible to shorten the processing time for generating high-resolution still image data based on a plurality of frame image data.

  In addition, as described above, when specifying the correction amount estimation range by specifying “moving” as the range specification target, the moving portion is specified by specifying the moving portion as the range specifying region. Since it can exclude from a correction amount estimation range, correction amount estimation can also be performed more accurately.

D. Variation:
The present invention is not limited to the above-described examples and embodiments, and can be implemented in various modes without departing from the gist thereof. For example, the following modifications are possible.

(1) In the above-described embodiment, the case where the range designation area in the high resolution target image is designated by a rectangular area represented by a rectangular frame is described as an example, but the present invention is not limited to this. You may make it designate the range designation | designated area | region with the area | region which the frame which has various shapes other than a rectangle represents. In addition, a user may arbitrarily select a plurality of types of frames (for example, rectangles, circles, stars, etc.) as frames for specifying the range specifying area.

(2) In the above embodiment, the case where the motion correspondence synthesis process is executed as an example of the process for generating the high resolution still image data executed by the high resolution processing unit 110 (see FIG. 1) has been described. However, the present invention is not limited to this, and motion non-corresponding synthesis processing may be executed. That is, the present invention acquires a plurality of relatively low-resolution images, corrects a shift between the acquired images, and generates a relatively high-resolution image by superimposing and combining the corrected images. It is applicable to what you do.

(3) In the above embodiment, when a specific moment in the moving image playback period is specified, the frame image acquisition unit 106 (see FIG. 1) uses the frame at that moment as a reference frame, and includes this reference frame. The frame image data of 4 frames arranged in time series on the rear side is acquired. Of course, frame image data of 4 frames arranged in time series on the front side may be acquired, or frame image data of 4 frames may be acquired in accordance with the front and back. Further, the number of frames is not necessarily four, and two or more frames may be acquired, and still image data may be generated from the acquired frame image data of the plurality of frames.

(4) In the above embodiment, a case where frame image data of a plurality of frames is acquired and still image data is generated from the acquired frame image data of the plurality of frames has been described as an example. However, the present invention is not limited to this. Absent. For example, image data for a plurality of frames is acquired in the form of image data of a plurality of fields constituting one frame (hereinafter also referred to as “field image data”), and still image data is obtained from the acquired field image data of the plurality of fields. You may make it produce | generate.

(5) In the above-described embodiment, in the process of synthesizing the generated image from a plurality of frame images, the motion is determined when calculating the pixel value of a certain pixel in the generated image and generating the pixel data. The frame image that can be used to generate the pixel data is determined, but the present invention is not limited to this. For example, for each pixel of the generated image, a frame image that can be used for each generation is first determined by motion determination, and then pixel data of each pixel of the generated image is calculated to generate pixel data. At this time, the pixel value may be calculated based on the previously determined frame images.

(6) In the above embodiment, time-sequential images of a plurality of frames acquired from a moving image (this moving image is represented by moving image data) (this image is represented by image data). In this example, a single high-resolution still image (this still image is represented by still image data) is generated. However, the present invention is not limited to this, and it is also possible to generate a single high-resolution image by simply synthesizing a plurality of low-resolution images that are continuous in time series. The plurality of low-resolution images that are continuous in time series may be, for example, a plurality of images continuously shot by a digital camera.
Further, it is not always necessary to use a plurality of low-resolution images (including frame images) that are continuous in time series, and may be a plurality of low-resolution images arranged in time series.

(7) The motion determination method in the above-described embodiment is an example, and is not limited to this, and various motion determination methods can be used. For example, the position (estimated position) of a pixel having an estimated brightness value V ′ equal to the brightness value of the determination pixel Fpt is obtained, and whether or not the deviation amount between the position of the determination pixel Fpt and the estimated position is within a predetermined range. It is also possible to use a method for determining the movement by the above.

(8) In the above embodiment, when the background portion is designated as the range designation region, the designated portion is set as the correction amount estimation range, and the correction amount estimation is executed, or the moving portion is designated as the range designation region. As an example, the case where the portion with the specified movement is excluded from the correction amount estimation range and the correction amount estimation is executed is specified as an example. However, the present invention is not limited to this. For example, only a portion having movement designated as the range designation region may be simply increased in resolution using a bi-linear method, a bi-cubic method, or the like based on the reference frame image. Further, it is possible to increase the resolution by synthesizing a part other than the part having a motion designated as the range designation area in a non-motion compatible manner. That is, a part designated as a range designation area or a part other than a part designated as a range designation area can be designated as a target area for various processes.

  In the above-described embodiment, the background portion is designated as the range designation region, the designated portion is set as the correction amount estimation range, and the correction amount estimation is executed as an example. However, the present invention is limited to this. Instead, various processes can be executed for the portion designated as the range designation region.

(9) In the above embodiment, a part of the configuration realized by hardware may be replaced by software, and conversely, a part of the configuration realized by software may be replaced by hardware. Good. For example, the processing by the still image generation control unit, the correction amount estimation unit, and the motion correspondence synthesis unit as illustrated in FIG. 1 may be performed by a hardware circuit.

(10) In the above embodiment, the case where the personal computer is operated as the still image generating apparatus of the present invention is described as an example, but the present invention is not limited to this. For example, various computer systems such as a digital video camera, a digital camera, and a printer can be used as the still image generation apparatus of the present invention. In particular, when a digital video camera is used as the still image generation apparatus of the present invention, a single image is taken from a plurality of frame images included in the moving image while shooting the moving image or confirming the shooting result. A resolution still image can be generated. Even when the digital camera is a still image generation device of the present invention, a single high-resolution still image is generated from a plurality of captured images while continuously shooting a subject or checking the result of continuous shooting. can do.

It is explanatory drawing which shows schematic structure of the still image generation device as one Example of this invention. It is explanatory drawing which shows the production | generation procedure of high resolution still image data. It is explanatory drawing which shows the user interface screen for a user to designate the high resolution object image during reproduction | regeneration of a moving image. It is explanatory drawing shown about the position shift between the frame image of a reference | standard frame, and the frame image of an object frame. It is explanatory drawing shown about correction | amendment of the position shift between a reference | standard frame image and an object frame image. It is explanatory drawing shown about the corrected amount of the object frame image with respect to a reference | standard frame image. It is explanatory drawing which shows the estimation method of the translation correction amount by the gradient method. It is explanatory drawing which shows typically the rotation correction amount of a pixel. It is explanatory drawing which shows the correction amount estimation process in step S20 of FIG. 2 concretely. It is explanatory drawing which shows the user interface screen for a user to specify the correction amount estimation range. It is explanatory drawing which expands and shows a mode that the reference | standard frame image F0 and object frame image F1-F3 were correct | amended and arrange | positioned. It is explanatory drawing shown about the interpolation process by a bilinear method. It is explanatory drawing at the time of performing a motion non-corresponding synthetic | combination process when the motion generate | occur | produced between several frame images. It is explanatory drawing which shows the precondition for demonstrating the determination method of a motion. It is explanatory drawing shown about an example of a motion determination method. It is explanatory drawing which shows the interface screen displayed during the production | generation process of high resolution still image data.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100 ... Computer 102 ... Control part 104 ... Display control part 106 ... Frame image acquisition part 108 ... Correction amount estimation part 108a ... Correction amount estimation range setting part 108b ... Correction amount Estimating processing unit 110 ... High resolution processing unit 112 ... Print control unit 120 ... Display 130 ... Keyboard 140 ... Mouse 200 ... Printer 300 ... Digital video camera A0 ... "File" button A1 ... "Play" button A2 ... "Stop" button A3 ... "Pause" button A4 ... "Frame advance" button A5 ... "Frame reverse" button A11 .. "Resolution" button A12 ... "Save" button A13 ... "Print" button A14 ... "Execute" button A23 ... Select button A21, A22, A23 ... Select button Cs .. Cursor Fa ... Frame image F0-F3 ... Frame image NW0 ... The interface screen NW1 ... User interface screen SW0 ... Image display field SW1 ... Range designation method selection field SW2 ... Range designation target selection field SW3 ... Progress bar Ft ... Target frame image Fr. ..Reference frame image

Claims (11)

A high-resolution image generation device that generates a relatively high-resolution second image from a plurality of relatively low-resolution first images,
A correction amount estimator for estimating a correction amount for correcting misalignment between the plurality of first images;
A high resolution processing for correcting the positional deviation of the plurality of first images based on the estimated correction amount and generating the second image by combining the corrected first images. And
With
The correction amount estimation unit
The user is allowed to designate a predetermined partial area in the plurality of first images, and the correction amount is based on the designated predetermined partial area for the range of the plurality of first images. A correction amount estimation range setting unit for setting a correction amount estimation range to be used for estimation of
A correction amount estimation processing unit that estimates the correction amount based on the plurality of first images corresponding to the set correction amount estimation range;
A high-resolution image generation apparatus comprising:   The high-resolution image generation apparatus according to claim 1, wherein the plurality of first images are a plurality of time-series consecutive images acquired from a moving image.   The high-resolution image generation apparatus according to claim 1, wherein the correction amount estimation range setting unit sets the predetermined partial region as the correction amount estimation range.   The high-resolution image generation apparatus according to claim 1, wherein the correction amount estimation range setting unit sets a region other than the predetermined partial region as the correction amount estimation range.   The correction amount estimation range setting unit allows a user to select which of the predetermined partial region and a region other than the predetermined partial region is set as the correction amount estimation range. The high-resolution image generation apparatus according to claim 2. A high-resolution image generation method for generating a relatively high-resolution second image from a plurality of relatively low-resolution first images,
(A) estimating a correction amount for correcting a positional shift between the plurality of first images;
(B) correcting the positional deviation of the plurality of first images based on the estimated correction amount and generating the second image by combining the corrected first images. When,
With
The step (b)
(B1) The user is allowed to specify a predetermined partial area in the plurality of first images, and based on the specified predetermined partial area with respect to the range of the plurality of first images. Setting a correction amount estimation range to be used for estimating the correction amount;
(B2) estimating the correction amount based on the plurality of first images corresponding to the set correction amount estimation range;
A high-resolution image generation method comprising:   The high-resolution image generation method according to claim 6, wherein the plurality of first images are a plurality of images consecutive in time series acquired from a moving image.   The high-resolution image generation method according to claim 6 or 7, wherein the step (b1) sets the predetermined partial region as the correction amount estimation range.   The high-resolution image generation method according to claim 6 or 7, wherein the step (b1) sets an area other than the predetermined partial area as the correction amount estimation range.   The step (b1) allows the user to select which of the predetermined partial area and an area other than the predetermined partial area is set as the correction amount estimation range. 8. The high-resolution image generation method according to 7. A program for generating a relatively high-resolution second image from a plurality of relatively low-resolution first images,
A first function for estimating a correction amount for correcting a positional shift between the plurality of first images;
A second function of correcting the positional deviation of the plurality of first images based on the estimated correction amount and generating the second image by combining the corrected first images. When,
Is realized on a computer,
The first function is:
The user is allowed to specify a predetermined partial region in the plurality of first images, and the range of the plurality of first images to be used for the estimation of the correction amount is specified as the specified predetermined region. A function for setting a correction amount estimation range based on the partial area of
A function of estimating the correction amount based on the plurality of first images corresponding to the set correction amount estimation range;
The program characterized by including.

Images