JP2007257042A - Image magnifying device and image magnifying method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable high resolutions with high image quality even for an irreversible/deformable photographic object. <P>SOLUTION: This device is constituted by a reference frame input means for inputting a plurality of reference frames with low resolutions, a first displacement amount calculation means for calculating displacement amount from each pixel of a standard frame as a standard of high resolutions to the remaining reference frames by comparing pixel values, a second displacement amount calculation means for calculating displacement amount from each pixel of the reference frames to the standard frame by comparing the pixel values, a difference frame calculation means for preparing the plurality of estimated reference frames by deforming/reducing a temporary estimated high resolution frame created from the standard frame using second displacement amount and for calculating a plurality of difference frames between the reference frames and the estimated reference frames, and a difference addition means for enlarging/deforming the difference frames using first displacement amount and for adding the temporary estimated high resolution frame. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image enlargement apparatus and an image enlargement method.

  In recent years, pixel density has increased for devices that output video and images, such as televisions, personal computer displays, and printers. Images for converting low-resolution images to sharp high-resolution images for these devices are output. Enlarging devices have been proposed.

As such an image enlarging apparatus, Non-Patent Document 1 first inputs a plurality of low resolution images in which a subject has been deformed, and among these low resolution images, an image to be enlarged is set as a reference image and the remaining images. Is a reference image. Subsequently, a deformation value between the standard image and the reference image is calculated. Next, a high-resolution image temporarily estimated from the standard image is generated, and a difference image between this and the reference image is calculated. In addition, there is described a method in which a pixel value of a high-resolution image is determined and a high-resolution image is generated by repeating a process of inversely deforming the difference image and adding it to the temporarily estimated high-resolution image.
Michal Irani and Shmuel Peleg, “Motion Analysis for image enhancement: resolution, occlusion, and transparency,” Journal of Visualization and Communication. 4, no. 4, pp. 324-335, 1993.

  The conventional image enlarging apparatus described above cannot increase the resolution when the deformation model of the subject is a complex deformation that cannot be expressed by a reversible Affine deformation or when an occlusion area exists in continuous images. There was a problem.

  The present invention has been made in order to solve the above-described problems of the prior art, and it is possible to increase the resolution even when the deformation model of the subject is not a deformation such as a reversible Affine deformation or an occlusion area exists. An object of the present invention is to provide an image enlarging apparatus and an image enlarging method that enable the above.

In order to achieve the above object, an image enlarging apparatus of the present invention includes:
A reference frame as a reference for increasing the resolution of one of a plurality of frames, the remaining frame is input as a reference frame, and a first positional shift amount from each pixel of the reference frame to the reference frame, A first misregistration amount calculating means for calculating and comparing pixel values;
A reference frame serving as a reference for increasing the resolution of one of a plurality of frames, the remaining frame is input as a reference frame, and a second positional shift amount from each pixel of the reference frame to the reference frame, A second misregistration amount calculating means for calculating and comparing pixel values;
A temporary estimated high resolution frame generating means for generating a temporary estimated high resolution frame using the reference frame;
A motion compensation frame is generated from the provisional estimated high-resolution frame using the second positional shift amount, and a motion compensation frame is reduced to generate an estimated reference frame corresponding to the reference frame. The reference frame and the estimated reference frame Difference frame calculation means for calculating a difference frame between and
Difference addition means for generating a motion compensation frame from the difference frame using the first misregistration amount, expanding the frame and adding it to the temporary estimated high resolution frame to generate an estimated high resolution frame; An image enlarging apparatus is provided.

  At this time, the difference frame calculation means generates a motion compensation frame from the temporary estimated high resolution frame using the second positional shift amount, and reduces the reduced frame to generate an estimated reference frame corresponding to the reference frame Instead, after the estimated high-resolution frame is reduced, a motion compensation frame may be generated using the second displacement amount, and this may be used as the estimated reference frame.

  Further, the difference adding means generates a motion compensation frame from the difference frame using the first positional deviation amount and expands the generated frame to generate the provisional estimated high resolution frame. After the enlargement, a motion compensation frame may be generated from the difference frame using the first displacement amount, and this may be used as the estimated high resolution frame.

  Further, the estimated high resolution frame is input to the difference frame calculation unit, a motion compensation frame is generated from the estimated high resolution frame using the second positional shift amount, and the frame is reduced to correspond to the reference frame. An estimated reference frame is generated, a difference frame between the reference frame and the estimated reference frame is calculated, a motion compensation frame is generated using the difference frame and the first positional deviation amount, and is expanded. An estimated high resolution frame may be generated by adding to the estimated high resolution frame.

In the present invention, a reference frame serving as a reference for increasing the resolution of one of a plurality of frames is input as a reference frame, and the remaining frames are input as reference frames, and a first frame from each pixel of the reference frame to the reference frame is input. First misregistration amount calculating means for calculating the misregistration amount by comparing pixel values;
A reference frame serving as a reference for increasing the resolution of one of a plurality of frames, the remaining frame is input as a reference frame, and a second positional shift amount from each pixel of the reference frame to the reference frame, A second misregistration amount calculating means for calculating and comparing pixel values;
A temporary estimated high resolution frame generating means for generating a temporary estimated high resolution frame using the reference frame;
A first occlusion area where there is no corresponding point from the base frame to the reference frame, and a corresponding point from the reference frame to the base frame, using the first positional shift amount and the second positional shift amount An occlusion area calculating means for calculating a second occlusion area having no
In a region that is not the first and second occlusion regions, a motion compensation frame is generated from the provisional estimated high-resolution frame using the second positional shift amount, and the motion compensation frame is reduced and estimated corresponding to the reference frame Difference frame calculation means for generating a reference frame and calculating a difference frame between the reference frame and the estimated reference frame;
In a region that is not the first and second occlusion regions, a motion compensation frame is generated from the difference frame using the first displacement amount, and is expanded and added to the temporary estimated high resolution frame. An image enlarging apparatus comprising: a difference adding unit that generates an estimated high resolution frame.

  At this time, the difference frame calculation means generates a motion compensation frame from the temporary estimated high resolution frame using the second positional shift amount, and reduces the reduced frame to generate an estimated reference frame corresponding to the reference frame Instead, after the estimated high-resolution frame is reduced, a motion compensation frame may be generated using the second displacement amount, and this may be used as the estimated reference frame.

  Further, the difference adding means generates a motion compensation frame from the difference frame using the first positional deviation amount and expands the generated frame to generate the provisional estimated high resolution frame. After the enlargement, a motion compensation frame may be generated from the difference frame using the first displacement amount, and this may be used as the estimated high resolution frame.

  Further, the estimated high resolution frame is input to the difference frame calculation unit, a motion compensation frame is generated from the estimated high resolution frame using the second positional shift amount, and the frame is reduced to correspond to the reference frame. An estimated reference frame is generated, a difference frame between the reference frame and the estimated reference frame is calculated, a motion compensation frame is generated using the difference frame and the first positional deviation amount, and is expanded. An estimated high resolution frame may be generated by adding to the estimated high resolution frame.

The present invention also includes a step of inputting a reference frame as a reference for increasing the resolution of one of a plurality of frames, and the remaining frames as reference frames;
Calculating a first displacement amount from each pixel of the base frame to the reference frame by comparing pixel values with a first displacement amount calculation unit;
A second misregistration amount calculation means for calculating a second misregistration amount from each pixel of the reference frame to the base frame by comparing pixel values;
In the temporary estimated high resolution frame generating means, generating a temporary estimated high resolution frame using the reference frame;
In the difference frame calculation means, a motion compensation frame is generated from the temporary estimated high resolution frame using the second misregistration amount, and this is reduced to generate an estimated reference frame corresponding to the reference frame, Calculating a difference frame between a reference frame and the estimated reference frame;
A step of generating an estimated high resolution frame by generating a motion compensation frame from the difference frame using the first misregistration amount, adding the motion compensation frame to the provisional estimated high resolution frame by the difference adding means; And an image enlarging method.

The present invention also includes a step of inputting a reference frame as a reference for increasing the resolution of one of a plurality of frames, and the remaining frames as reference frames;
Calculating a first displacement amount from each pixel of the base frame to the reference frame by comparing pixel values with a first displacement amount calculation unit;
A second misregistration amount calculation means for calculating a second misregistration amount from each pixel of the reference frame to the base frame by comparing pixel values;
In the temporary estimated high resolution frame generating means, generating a temporary estimated high resolution frame using the reference frame;
In the occlusion area calculation means, the first occlusion area having no corresponding point from the reference frame to the reference frame using the first positional deviation amount and the second positional deviation amount, and the reference frame Calculating a second occlusion region having no corresponding point to the reference frame;
The difference frame calculation means generates a motion compensation frame from the temporary estimated high resolution frame using the second positional deviation amount in an area that is not the first and second occlusion areas, and reduces the frame. Generating an estimated reference frame corresponding to the reference frame and calculating a difference frame between the reference frame and the estimated reference frame;
The difference adding means generates a motion compensation frame from the difference frame using the first positional deviation amount in an area that is not the first and second occlusion areas, and expands the motion compensation frame to increase the temporary estimated height. And a step of generating an estimated high resolution frame by adding to the resolution frame.

  According to the present invention, the first positional deviation amount from each pixel of the reference frame to the reference frame is calculated by comparing the pixel values, and the second positional deviation amount from each pixel of the reference frame to the reference frame is further calculated. Is calculated by comparing pixel values, so that high resolution with high image quality can be realized even in complicated cases such as reversible Affine deformation. In addition, since an area where occlusion occurs is calculated in advance, and the area where no occlusion occurs is processed, it is possible to achieve high image quality and high resolution even with a subject that is not reversible and causes occlusion.

  Hereinafter, embodiments of the present invention will be described.

(First embodiment)
FIG. 1 is a block diagram showing an image enlargement apparatus according to the first embodiment of the present invention.

  The image enlargement apparatus according to the first embodiment calculates a first positional shift amount from each pixel of a reference frame serving as a reference for high resolution to a reference frame, and a frame input unit 101 that inputs a plurality of frames. A first misregistration amount calculation unit 102, a second misregistration amount calculation unit 103 that calculates a second misregistration amount from each pixel of the reference frame to the base frame, and a temporary estimated high resolution from the base frame A temporary estimated high-resolution frame generation unit 108 that generates a frame, and a plurality of estimated reference frames are generated by deforming and reducing the temporary estimated high-resolution frame using the second positional deviation amount, and the reference frame and the estimated frame A difference frame calculation unit 104 that calculates a plurality of difference frames with respect to the reference frame, and the difference frame is enlarged and deformed by using the first positional deviation amount to temporarily estimate the high-resolution frame. And a difference adding section 105 to be added to the arm.

  The frame input unit 101 is means for inputting a plurality of low-resolution image frames. The reason why the low-resolution image is used here is to increase the resolution later. A plurality of frames are input from the reference frame input unit 101. One of the plurality of frames is used as a standard reference frame as a reference for increasing the resolution, and the remaining frames are used as reference frames for increasing the resolution. Is entered as

  For example, the reference frame is a time-series image of VGA size (640 × 480 pixels) obtained by digitizing the NTSC size of the current television content, image data of SD size that can be played back on a PC, or a digital video camera It may be an image sequence photographed by a CCD (Charge Coupled Device) element such as a digital camera or a camera-equipped mobile phone, or may be a part of an image area to be enlarged during digital zoom.

  The first misregistration amount calculation unit 102 receives the misregistration amount (first misregistration amount) from each pixel of the reference frame, which is input by the frame input unit 101 and serves as a reference for high resolution, to the remaining reference frames. ) Is calculated up to decimal precision, that is, sub-pixel precision, by comparing the pixel values.

  The second misregistration amount calculation unit 103 is a misregistration amount (second misregistration amount) from each pixel of the reference frame, which is input by the frame input unit 101, to a reference reference frame that is a reference for high resolution. The pixel value is compared to calculate how much the position is shifted to the decimal system, that is, to the subpixel accuracy.

  The first misregistration amount calculation unit 102 and the second misregistration amount calculation unit 103 constitute a deformation estimation unit 106.

  The temporary estimated high-resolution frame generation unit 108 temporarily expands the reference frame input from the frame input unit 101 to generate a temporary estimated high-resolution frame. This temporary estimated high-resolution frame is used as an initial value for increasing the resolution, and bilinear interpolation (Bi-linear interpolation), bicubic interpolation (Bi-cubic interpolation), cubic The image can be generated by a general conventional image interpolation method such as a convolution interpolation method (Cubic Convolution interpolation method).

  The difference frame calculation unit 104 uses the second positional shift amount calculated by the second positional shift amount calculation unit 103 for the temporary estimated high resolution frame generated by the temporary estimated high resolution frame generation unit 108. A plurality of estimated reference frames are generated by deformation / reduction, and a plurality of difference frames between the reference frame input by the frame input unit 101 and the estimated reference frame are calculated. Here, the deformation / reduction means that a motion compensation frame is generated from a temporary estimated high-resolution frame using the second positional deviation amount and is reduced to generate an estimated reference frame. However, instead of this, after the estimated high-resolution frame is reduced, a motion compensation frame may be generated using the second positional deviation amount, and this may be used as the estimated reference frame.

  The difference addition unit 105 uses the first positional deviation amount calculated by the first positional deviation amount calculation unit 103 to enlarge and deform the difference frame calculated by the difference frame calculation unit 104 to temporarily estimate the height. Add to the resolution frame to update the temporary estimated high-resolution frame. Here, the enlargement / deformation means that a motion compensation frame is generated from the difference frame using the first positional deviation amount, is expanded, and is added to a temporary estimated high resolution frame to generate an estimated high resolution frame. To tell. However, instead of enlarging the difference frame, a motion compensation frame may be generated from the difference frame using the first positional shift amount, and this may be used as the estimated high resolution frame.

  The difference frame calculating unit 104 and the difference adding unit 105 constitute a high resolution unit 107.

  Note that the resolution enhancement unit 107 inputs the estimated high resolution frame to the difference frame calculation unit 104, generates a motion compensation frame from the estimated high resolution frame using the second positional deviation amount, reduces the frame, and reduces the reference frame. An estimated reference frame corresponding to is generated, a difference frame between the reference frame and the estimated reference frame is calculated, and the motion compensation frame is calculated by using the difference frame and the first positional deviation amount in the difference or triple part 105. Is generated, enlarged, and added to the estimated high resolution frame to generate an estimated high resolution frame, thereby resolving the resolution.

  FIG. 2 is a flowchart for explaining the operation of the image enlargement apparatus according to the first embodiment of the present invention. The operation of the image enlargement apparatus according to the first embodiment of the present invention will be described with reference to FIGS.

  First, as shown in FIGS. 1 and 2, a plurality of frames that are low-resolution images are input from the frame input unit 101. One of the plurality of frames is a reference frame serving as a reference for increasing the resolution, and the remaining is a reference frame used as a reference for the reference frame.

  Next, the first misregistration amount calculation unit 102 determines how much the misregistration from each pixel of the base frame input from the frame input unit 101 to the remaining reference frame is to a decimal accuracy, that is, a sub-pixel accuracy. Calculate (step S201).

  This will be described with reference to FIG. FIG. 3 is a diagram in which one of the four frames 1, 2, 3, 4 is designated as a reference frame 3, and the remaining frames are designated as reference frames 1, 2, 4.

  As shown in FIG. 3, how much each pixel arranged in an integer pixel unit in the grid pattern of the base frame 3 is displaced with respect to each of the reference frame 1, the reference frame 2, and the reference frame 4, The subpixel accuracy is calculated by comparing the plotting measures.

  Specifically, a block of 7 × 7 pixels is arranged around each pixel of the reference frame 3. Subsequently, the similarity between the block in the reference frame 3 and the candidate block in the reference frames 1, 2, and 4 is calculated with pixel accuracy, that is, integer accuracy.

  As the similarity between frames, here, SSD (Sum of Square Distance), which is the sum of squares of pixel value differences in a block, is used. The similarity is calculated while shifting the positions of the candidate blocks in the reference frames 1, 2, and 4 with integer precision, and an integer precision similarity map is calculated. A symmetric continuous function, for example, a parabola, is fitted around the integer precision misregistration amount that is most similar (when SSD is SSD, the SSD is the smallest).

  FIG. 4 is a graph showing this state. The vertical axis represents the similarity, and the horizontal axis represents the integer system positional deviation amount.

  As shown in FIG. 4, the most similar block from the position of the estimated displacement amount with subpixel accuracy indicated by the white arrow indicates that the displacement amount with subpixel accuracy has a discrete similarity map with integer accuracy. Expressed as the apex position of the fitted parabola. Since the image is two-dimensional, a two-dimensional positional shift amount can be calculated by performing the positional shift amount calculation process independently in the vertical direction and the horizontal direction, or by applying a paraboloid.

  In FIG. 4, the most similar block from the position indicated by the white arrow is n.

  For all pixels in the standard reference frame, the position deviation calculation process between all the reference frames is performed, and the position with sub-pixel accuracy between the standard reference frame and the reference frame, which is the standard for high resolution. The amount of deviation can be calculated.

  Here, as a unit for calculating the positional deviation amount, the positional deviation amount is expressed by a dense optical flow (the highest density) in which all pixels have independent positional deviation components. However, all the pixels in the frame have the same positional deviation amount. It may be expressed as a full screen motion having a certain full screen motion, and may be a block unit or a subject object unit.

  Further, here, as a method of calculating the positional deviation amount with sub-pixel accuracy, the similarity with integer accuracy is obtained by the sum of squares of the pixel value differences within the unit of calculating the positional deviation, and the similarity with integer accuracy is obtained. The method of calculating the amount of subpixel accuracy displacement by applying a parabola was explained, but the similarity in integer accuracy is not only the sum of squares of the luminance difference, but also the absolute value sum of the luminance difference and the normalized cross-correlation A conventional similarity between images may be used.

  The similarity interpolation method may use not only parabolic fitting but also equiangular straight line fitting and other symmetric functions.

  In addition, the calculation of the positional deviation amount with sub-pixel accuracy is not only a method of interpolating by applying a symmetric function to the similarity with integer accuracy, but also calculating the similarity by density gradient method or interpolation enlargement of the image itself. An existing subpixel-accurate position deviation calculation method such as a method, a method using FFT, or a method for extracting feature points may be used.

  Here, since the amount of positional deviation between reference frames, which are a plurality of low-resolution images, is calculated, the calculated dense optical flow is calculated for each pixel of the low-resolution image. In order to increase the resolution of an image, a dense optical flow in units of pixels of the high-resolution image is necessary. Each reference frame is preliminarily obtained by bilinear interpolation (Bi-linear interpolation), bi-3. The displacement amount is calculated by enlarging to the size of a high resolution image by a general conventional image interpolation method such as a secondary interpolation method (Bi-cubic interpolation method) or a cubic convolution interpolation method (Cubic Convolution interpolation method). In addition, by interpolating the amount of displacement calculated for each pixel of the low-resolution image, a dense optical flow can be obtained for each pixel of the high-resolution image.

  Next, as shown in FIGS. 1 and 2, how much the second displacement amount calculation unit 103 is displaced from each pixel of the reference frame to the reference reference frame that is a reference for high resolution. Calculation is performed with sub-pixel accuracy (step S202).

  FIG. 5 compares pixel values of the amount of positional deviation from three reference frames 1, 2, 2, and 4 to the base frame 3, in contrast to the original described in FIG. 3. It is a figure explaining the example calculated by this.

  Subpixels are compared by comparing pixel values with respect to how much each pixel arranged in an integer pixel unit in the reference frame 1, reference frame 2, and reference frame 4 is shifted to the base frame 3. Calculate to accuracy. A specific calculation method can be calculated by a method similar to that of the first misregistration amount calculation unit 102.

  Here, when calculating the second misregistration amount, the first misregistration amount calculated by the first misregistration amount calculation unit 102 is used as the initial value of the search range, or the first misregistration amount is calculated. By limiting the search range when calculating the second misregistration amount using the amount, the second misregistration amount can be calculated at high speed.

  Next, as shown in FIGS. 1 and 2, a temporary estimated high resolution frame is generated by a temporary estimated high resolution frame generation unit 108 from the reference frame input from the frame input unit 101. Next, the difference frame calculation unit 104 generates a motion compensation frame from the temporary estimated high resolution frame using the second positional deviation amount, reduces the motion compensation frame, and generates an estimated reference frame corresponding to the previous reference frame, A difference frame between the reference frame and the estimated reference frame is calculated.

  FIG. 6 is a diagram for explaining this situation.

  As illustrated in FIG. 6, the temporary estimated high resolution frame generation unit 108 generates a temporary estimated high resolution frame, and the difference frame calculation unit 104 calculates the temporary estimated high resolution frame as a second positional deviation amount. A plurality of estimated reference frames are generated by deformation / reduction using the second positional deviation amount calculated by the unit 103, and reference frames 1, 2, 3 (here, the reference frame 3) input by the frame input unit 101 are generated. Are also used as reference frames), and a plurality of difference frames 1, 2, 3, 4 between 4 and the estimated reference frame are calculated (step S203).

  The estimated high resolution frame is converted into a provisional estimated high resolution frame (initial value) at the beginning of the resolution enhancement process, and the reference frame 3 is converted into a bilinear interpolation method (Bi-linear interpolation method) which is a conventional interpolation method. ), A bicubic interpolation method (Bi-cubic interpolation method), and a cubic convolution interpolation method (Cubic Convolution interpolation method).

  The temporary estimated high resolution frame is updated by the iterative process of the difference frame calculation unit 104 and the difference addition unit 105, thereby generating a final output high resolution frame.

  The estimated high-resolution frame is deformed / reduced using the second positional deviation amount calculated by the second positional deviation amount calculation unit 103. In the deformation process, since the second positional deviation amount can be used to determine the positional deviation amount with sub-pixel accuracy from each pixel of the high-resolution reference frame to the base frame 3, the positional deviation amount in the base frame 3 around the destination. By interpolating the pixel values by bilinear interpolation (Bi-linear interpolation), high-resolution pixel values in each reference frame are determined. In the reduction process, each estimated reference frame with a low resolution is generated by performing an averaging process.

  The difference frame corresponding to each reference frame is generated by subtracting each pixel of the reference frame input from the frame input unit 101 and the estimated reference frame. This difference frame indicates an error between the estimated reference frame generated from the estimated high resolution frame and the actually observed / captured reference frame. The smaller the error, the higher the estimated high resolution frame is estimated. Indicates that

  Next, as shown in FIGS. 1 and 2, the difference addition unit 105 is calculated by the difference frame calculation unit 104 using the first position shift amount calculated by the first position shift amount calculation unit 102. The difference frame is enlarged and deformed and added to the temporary estimated high resolution frame to update the temporary estimated high resolution frame (step S204).

  The differential frame enlarging process is performed by bilinear interpolation (Bi-linear interpolation), bicubic interpolation (Bi-cubic interpolation), or cubic convolution (concubation interpolation). A general image interpolation method such as an interpolation method is used. In the deformation process, the first positional deviation amount can be used to determine the positional deviation amount with sub-pixel accuracy from each pixel of the high-resolution base frame to the reference frame. Therefore, the pixel values around the destination of the positional deviation amount in the difference frame Is interpolated by bilinear interpolation (Bi-linear interpolation). That is, the update amount (minus error) of the estimated high resolution frame of the high resolution reference frame is calculated. By adding the average value of the update amounts to the estimated high resolution frame at the current stage, the updated high precision frame is updated.

  Next, as shown in FIGS. 1 and 2, when the high-resolution processing is not completed (step S <b> 205: No), the process returns to the difference frame calculation unit 104 and the estimated high-resolution frame updated by the difference addition unit 105. Based on the above, the second positional deviation amount calculated by the second positional deviation amount calculation unit 103 is used to generate a plurality of estimated reference frames by deformation and reduction, and the reference input by the frame input unit 101 The process of calculating a plurality of difference frames between the frame and the estimated reference frame is repeated.

  The determination as to whether or not the resolution enhancement processing has been completed is made based on whether or not the number of times of resolution enhancement updating by the iterative processing of the difference frame calculation unit 104 and the difference addition unit 105 has been performed more than a certain number. Judgment can be made based on whether the frequency has become sufficiently small, or whether the high-frequency component of the estimated high-resolution frame has exceeded a certain level. Here, it is determined that the resolution has been updated when the number of times of high resolution update is repeated 20 to 30 times.

  When the high resolution processing is completed (step S205: Yes), the estimated high resolution frame updated by the difference adding unit 105 is output as an enlarged frame of the standard reference frame, and the processing ends.

  As described above, according to the image enlargement apparatus according to the first embodiment, the second positional deviation amount is calculated by comparing the pixel values, so that even in the case of a deformation that is not a reversible Affine deformation, high image quality can be achieved. High resolution can be achieved.

(Second Embodiment)
Next, an image enlarging apparatus according to the second embodiment of the present invention will be described. This image enlarging apparatus calculates the occlusion area having no corresponding point between frames using the first positional deviation amount and the second positional deviation amount, thereby increasing the resolution of the reference reference frame with higher accuracy. Is to do.

  FIG. 7 is a block diagram showing an image enlargement apparatus according to the second embodiment of the present invention.

  As shown in FIG. 7, this image enlargement apparatus is different from the first embodiment in that an occlusion area calculation unit 706 is added and the functions of a difference frame calculation unit 704 and a difference addition unit 705 are changed. Yes. Other configurations and functions are the same as those in FIG. 1 which is a block diagram showing the configuration according to the first embodiment, and therefore, the same reference numerals are given and description thereof is omitted here.

  The occlusion area calculation unit 706 compares the pixel value in the same manner with the second positional deviation amount calculation unit 103 and the first positional deviation amount calculated by comparing the pixel values with the first positional deviation amount calculation unit 102. Occlusion region calculation that calculates an occlusion region that does not have a corresponding point from the reference frame to the reference frame and an occlusion region that does not have a corresponding point from the reference frame to the reference frame, using the calculated second positional deviation amount. Means. The first misregistration amount and the second misregistration amount are calculated in the same manner as described in the first embodiment.

  The difference frame calculation unit 704 generates a temporary estimated high-resolution frame generated from the reference frame in the region that is not the occlusion region calculated by the occlusion region calculation unit 706, and is calculated by the second positional deviation amount calculation unit 103. A plurality of estimated reference frames are generated by being deformed / reduced using the position shift amount of 2, and a plurality of difference frames between the reference frame input by the reference frame input unit 101 and the estimated reference frame are calculated. Although the difference frame is calculated in an area that is not an occlusion area, the difference frame is calculated in the same manner as in the first embodiment.

  The difference addition unit 705 uses the second positional deviation amount calculated by the second positional deviation amount calculation unit 103 in the region that is not the occlusion area calculated by the occlusion region calculation unit 706 to use the difference frame calculation unit 704. The calculated difference frame is enlarged / deformed and added to the temporary estimated high resolution frame to update the temporary estimated high resolution frame. The difference between the first embodiment and the first embodiment is that the difference frame is enlarged / deformed in an area other than the occlusion area, but the method for calculating the difference frame enlargement / deformation is the same as in the first embodiment.

  Next, the operation of the image enlargement apparatus according to the second embodiment of the present invention will be described with reference to FIGS. FIG. 8 is a flowchart showing the operation of the image enlargement apparatus according to the second embodiment of the present invention.

  First, as shown in FIG. 7 and FIG. 8, the frame input unit 101 receives a plurality of frames, one of which is a reference frame and the other is a reference frame.

  Next, the first misregistration amount calculation unit 102 determines how much the misregistration from each pixel of the base frame that is the reference for high resolution input by the frame input unit 101 to the remaining reference frames. The pixel values are compared and calculated to decimal precision, that is, sub-pixel precision (step S801).

  Next, the second misregistration amount calculation unit 103 determines how much the misregistration from each pixel of the reference frame input by the frame input unit 101 to a reference frame that is a reference for high resolution is a pixel value. Are calculated with sub-pixel accuracy (step S802).

  Next, the occlusion area calculation unit 706 includes the first positional deviation amount calculated by the first positional deviation amount calculation unit 102 and the second positional deviation amount calculated by the second positional deviation amount calculation unit 103. Are used to calculate an occlusion area having no corresponding point from the base frame to the reference frame and an occlusion area having no corresponding point from the reference frame to the reference frame (step S803).

  As shown in FIG. 9, the occlusion area is an area that is captured in one certain frame but not captured in the remaining frames between two certain frames. It may be an area around a tree in the frame A (an area hidden by a car in the frame B) or an area around a house in the frame B (an area hidden by a car in the frame A). Considering the positional shift amount (dense optical flow) of each pixel from frame A to frame B, each pixel in the occlusion area (area around the tree) of frame A does not have a corresponding point in frame B. For this reason, the amount of misalignment cannot be defined and should be indefinite. Similarly, when considering the positional shift amount (dense optical flow) of each pixel from the frame B to the frame A, each pixel in the occlusion area (area around the house) of the frame B corresponds to the frame A. Since there is no point to perform, the amount of misalignment should be indefinite.

  However, when the first misregistration amount calculation unit 102 and the second misregistration amount calculation unit 103 calculate the first misregistration amount and the second misregistration amount, what is the occlusion area? No information is given, and an uncertain misregistration amount is calculated even in an indefinite occlusion area.

  Therefore, the occlusion area calculation unit 706 performs the following processing on all the pixels of the frame A in order to calculate an occlusion area that does not have a corresponding point from the frame A to the frame B.

  First, based on the amount of positional deviation from the frame A to the frame B, a corresponding pixel Pb indicating which pixel Pa in the frame A is located near is determined.

  Next, the corresponding pixel Pa ′ indicating which pixel in the frame A has been displaced is obtained based on the second displacement amount based on the second displacement amount. When the first positional deviation amount and the second positional deviation amount are correct, Pa and Pa ′ are equal or close to each other. If either is incorrect, Pa and Pa 'are in different positions.

  Therefore, when Pa and Pa ′ are equal or close to each other, it is determined that the pixel Pa in the frame A is not an occlusion area. When Pa and Pa ′ are at different positions, the pixel Pa in the frame A is an occlusion area. Judge that there is.

  By performing this process on all the pixels in the frame A, an occlusion area having no corresponding point from the frame A to the frame B can be calculated. Similarly, in the occlusion area where there is no corresponding point from frame B to frame A, the pixel Pb in which frame B is located in frame A is determined based on the second positional shift amount from frame B to frame A. The corresponding pixel Pa that is misaligned in the vicinity of the pixel is obtained, and the corresponding pixel Pa in the frame A is located near which pixel in the frame B based on the first misalignment amount. By determining the corresponding pixel Pb ′, it can be determined whether or not it is an occlusion region.

  Next, the difference frame calculation unit 704 calculates a temporary estimated high-resolution frame generated from the reference frame in the region that is not the occlusion region calculated by the occlusion region calculation unit 706 by the second positional deviation amount calculation unit 103. The plurality of estimated reference frames are generated by being deformed / reduced by using the second positional deviation amount, and a plurality of difference frames between the reference frame input by the reference frame input unit 101 and the estimated reference frame are calculated ( Step S804).

  First, the estimated high-resolution frame is deformed / reduced using the second positional deviation amount calculated by the second positional deviation amount calculation unit 103. In the deformation process, the second positional deviation amount can be used to determine the positional deviation amount with sub-pixel accuracy from each pixel of the high-resolution reference frame to the standard reference frame. By interpolating the pixel values by bilinear interpolation (Bi-linear interpolation), high-resolution pixel values in each reference frame are determined.

  However, since the second misregistration amount at the pixel determined to be the occlusion area is low in reliability, the interpolation using the second misregistration amount is not performed, and the pixel in the reference frame is copied as it is.

  In the reduction process, each estimated reference frame with a low resolution is generated by performing an averaging process. The difference frame corresponding to each reference frame is generated by subtracting each pixel of the reference frame input from the frame input unit 101 and the estimated reference frame. This difference frame indicates an error between the estimated reference frame generated from the estimated high resolution frame and the actually observed / captured reference frame. The smaller the error, the higher the estimated high resolution frame is estimated. Indicates that

  Next, the difference addition unit 705 calculates a difference frame using the second positional shift amount calculated by the second positional shift amount calculation unit 103 in the region that is not the occlusion region calculated by the occlusion region calculation unit 706. The difference frame calculated by the unit 704 is enlarged and deformed and added to the temporary estimated high resolution frame to update the temporary estimated high resolution frame (step S805).

  The differential frame enlarging process is performed by bilinear interpolation (Bi-linear interpolation), bicubic interpolation (Bi-cubic interpolation), or cubic convolution (concubation interpolation). A general image interpolation method such as an interpolation method is used.

  In the difference frame deformation process, since the first positional deviation amount indicates the positional deviation amount with sub-pixel accuracy from each pixel of the high-resolution base frame to the reference frame, around the destination of the positional deviation amount in the differential frame. Are interpolated by bilinear interpolation (Bi-linear interpolation).

  That is, the update amount (minus error) of the estimated high resolution frame of the high resolution reference frame is calculated. However, since the positional deviation amount at the pixel determined to be the occlusion region is low in reliability, interpolation using the first positional deviation amount is not performed, and the update amount is set to zero. By adding the average value of the update amounts to the estimated high resolution frame at the current stage, the updated high precision frame is updated.

  Next, as in step S205, if the high resolution processing has not been completed (step S806: No), the process returns to the difference frame calculation unit 704 and is based on the estimated high resolution frame updated by the difference addition unit 705. The second misregistration amount calculated by the second misregistration amount calculation unit 103 is deformed / reduced to generate a plurality of estimated reference frames, and the reference frame and the estimated reference input by the frame input unit 101 are generated. The process of calculating a plurality of difference frames with the frame is repeated.

  When the high resolution processing is completed (step S806: Yes), the estimated high resolution frame updated by the difference adding unit 705 is output as an enlarged frame of the reference frame, and the processing ends.

  As described above, according to the image enlargement apparatus according to the second embodiment, by calculating the occlusion area having no corresponding point between frames using the first positional deviation amount and the second positional deviation amount, Even with a subject that deforms in a non-reversible manner that causes occlusion, the resolution of the reference frame can be increased with higher accuracy.

  Note that this image enlargement apparatus can also be realized by using, for example, a general-purpose computer apparatus as basic hardware. That is, the first misregistration amount calculation unit 102, the second misregistration amount calculation unit 103, the difference frame calculation units 104 and 704, the difference addition units 105 and 705, and the occlusion region calculation unit 706 This can be realized by causing a processor mounted on the computer to execute a program. At this time, the image enlarging apparatus may be realized by installing the above program in a computer device in advance, or stored in a storage medium such as a CD-ROM or distributing the above program via a network. Thus, this program may be realized by appropriately installing it in a computer device.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

1 is a block diagram showing a configuration of an image enlargement apparatus according to a first embodiment of the present invention. The flowchart which shows the operation | movement of the 1st Embodiment of this invention. The figure which shows the correspondence of 1st position shift amount calculation. The figure which shows the calculation method of the amount of position shift of subpixel accuracy The figure which shows the correspondence of 2nd position shift amount calculation. The figure which shows the process of calculating a difference frame. The block diagram which shows the structure of the image expansion apparatus concerning the 2nd Embodiment of this invention. The flowchart which shows the operation | movement of the 2nd Embodiment of this invention. The figure which shows the example of an occlusion area | region.

Explanation of symbols

101... Frame input unit 102... First misregistration amount calculation unit 103... Second misregistration amount calculation units 104 and 704... Difference frame calculation units 105 and 705. 706 ... Occlusion area calculation unit

Claims (10)

A reference frame as a reference for increasing the resolution of one of a plurality of frames, the remaining frame is input as a reference frame, and a first positional shift amount from each pixel of the reference frame to the reference frame, A first misregistration amount calculating means for calculating and comparing pixel values;
A reference frame serving as a reference for increasing the resolution of one of a plurality of frames, the remaining frame is input as a reference frame, and a second positional shift amount from each pixel of the reference frame to the reference frame, A second misregistration amount calculating means for calculating and comparing pixel values;
A temporary estimated high resolution frame generating means for generating a temporary estimated high resolution frame using the reference frame;
A motion compensation frame is generated from the provisional estimated high-resolution frame using the second positional shift amount, and a motion compensation frame is reduced to generate an estimated reference frame corresponding to the reference frame. The reference frame and the estimated reference frame Difference frame calculation means for calculating a difference frame between and
Difference addition means for generating a motion compensation frame from the difference frame using the first misregistration amount, expanding the frame and adding it to the temporary estimated high resolution frame to generate an estimated high resolution frame; An image enlarging apparatus characterized by:   The difference frame calculation means generates a motion compensation frame from the provisional estimated high resolution frame using the second positional shift amount and reduces the reduced frame to generate an estimated reference frame corresponding to the reference frame. 2. The image enlarging apparatus according to claim 1, wherein after the estimated high-resolution frame is reduced, a motion compensation frame is generated using the second positional deviation amount and is used as an estimated reference frame.   The difference adding means expands the difference frame instead of generating a motion compensation frame from the difference frame using the first positional deviation amount and expanding the frame to generate the provisional estimated high resolution frame. The image enlarging apparatus according to claim 1 or 2, wherein a motion compensation frame is generated from the difference frame using the first misregistration amount and is used as the estimated high resolution frame. .   The estimated high resolution frame is input to the difference frame calculation unit, a motion compensation frame is generated from the estimated high resolution frame using the second positional deviation amount, and the estimated reference corresponding to the reference frame is reduced. A frame is generated, a difference frame between the reference frame and the estimated reference frame is calculated, a motion compensation frame is generated using the difference frame and the first positional shift amount, and the frame is expanded to perform the estimation The image enlarging apparatus according to any one of claims 1 to 3, wherein an estimated high resolution frame is further generated by adding to the high resolution frame. A reference frame as a reference for increasing the resolution of one of a plurality of frames, the remaining frame is input as a reference frame, and a first positional shift amount from each pixel of the reference frame to the reference frame, A first misregistration amount calculating means for calculating and comparing pixel values;
A reference frame serving as a reference for increasing the resolution of one of a plurality of frames, the remaining frame is input as a reference frame, and a second positional shift amount from each pixel of the reference frame to the reference frame, A second misregistration amount calculating means for calculating and comparing pixel values;
A temporary estimated high resolution generating means for generating a temporary estimated high resolution frame using the reference frame;
A first occlusion area where there is no corresponding point from the base frame to the reference frame, and a corresponding point from the reference frame to the base frame, using the first positional shift amount and the second positional shift amount An occlusion area calculating means for calculating a second occlusion area having no
In a region that is not the first and second occlusion regions, a motion compensation frame is generated from the provisional estimated high resolution frame using the second positional shift amount, and the motion compensation frame is reduced and estimated corresponding to the reference frame Difference frame calculation means for generating a reference frame and calculating a difference frame between the reference frame and the estimated reference frame;
In a region that is not the first and second occlusion regions, a motion compensation frame is generated from the difference frame using the first displacement amount, and is expanded and added to the temporary estimated high resolution frame. An image enlarging device comprising: difference addition means for generating an estimated high resolution frame.   The difference frame calculation means generates a motion compensation frame from the provisional estimated high resolution frame using the second positional shift amount and reduces the reduced frame to generate an estimated reference frame corresponding to the reference frame. 6. The image enlarging apparatus according to claim 5, wherein after the estimated high-resolution frame is reduced, a motion compensation frame is generated using the second positional deviation amount and is used as an estimated reference frame.   The difference adding means expands the difference frame instead of generating a motion compensation frame from the difference frame using the first positional deviation amount and expanding the frame to generate the provisional estimated high resolution frame. The image enlarging apparatus according to claim 5 or 6, wherein a motion compensation frame is generated from the difference frame using the first displacement amount later and is used as the estimated high resolution frame. .   The estimated high resolution frame is input to the difference frame calculation unit, a motion compensation frame is generated from the estimated high resolution frame using the second positional deviation amount, and the estimated reference corresponding to the reference frame is reduced. A frame is generated, a difference frame between the reference frame and the estimated reference frame is calculated, a motion compensation frame is generated using the difference frame and the first positional shift amount, and the frame is expanded to perform the estimation The image enlarging apparatus according to claim 5, wherein an estimated high resolution frame is further generated by adding to the high resolution frame. Inputting a reference frame as a reference for increasing the resolution of one of a plurality of frames, and the remaining frames as reference frames;
Calculating a first displacement amount from each pixel of the base frame to the reference frame by comparing pixel values with a first displacement amount calculation unit;
A second misregistration amount calculation means for calculating a second misregistration amount from each pixel of the reference frame to the base frame by comparing pixel values;
In the temporary estimated high resolution frame generating means, generating a temporary estimated high resolution frame using the reference frame;
In the difference frame calculation means, a motion compensation frame is generated from the temporary estimated high resolution frame using the second misregistration amount, and this is reduced to generate an estimated reference frame corresponding to the reference frame, Calculating a difference frame between a reference frame and the estimated reference frame;
A step of generating an estimated high resolution frame by generating a motion compensation frame from the difference frame using the first misregistration amount, adding the motion compensation frame to the provisional estimated high resolution frame by the difference adding means; And a method for enlarging an image. Inputting a reference frame as a reference for increasing the resolution of one of a plurality of frames, and the remaining frames as reference frames;
Calculating a first displacement amount from each pixel of the base frame to the reference frame by comparing pixel values with a first displacement amount calculation unit;
A second misregistration amount calculation means for calculating a second misregistration amount from each pixel of the reference frame to the base frame by comparing pixel values;
In the temporary estimated high resolution frame generating means, generating a temporary estimated high resolution frame using the reference frame;
In the occlusion area calculation means, the first occlusion area having no corresponding point from the reference frame to the reference frame using the first positional deviation amount and the second positional deviation amount, and the reference frame Calculating a second occlusion region having no corresponding point to the reference frame;
The difference frame calculation means generates a motion compensation frame from the temporary estimated high resolution frame using the second positional deviation amount in an area that is not the first and second occlusion areas, and reduces the frame. Generating an estimated reference frame corresponding to the reference frame and calculating a difference frame between the reference frame and the estimated reference frame;
The difference adding means generates a motion compensation frame from the difference frame using the first positional deviation amount in an area that is not the first and second occlusion areas, and expands the motion compensation frame to increase the temporary estimated height. And a step of generating an estimated high resolution frame by adding to the resolution frame.

Images