JP4837615B2 - Image processing method and image processing apparatus - Google Patents

Description

  The present invention relates to an image processing method and an image processing apparatus for enlarging the resolution of an image using a plurality of low resolution images as input.

  There are many efforts to increase the resolution of still images taken with a digital still camera to obtain clearer images. As one of such methods, a method of synthesizing a plurality of images having positional shifts to restore a high frequency component has attracted attention as a method of acquiring a clear high resolution image close to the original image. According to this method, it is possible to increase the resolution by using a plurality of temporally continuous images in a moving image, and a wide range of applications such as increasing the resolution of a moving image taken with a video camera is expected. . Hereinafter, the process of generating a high resolution image from a plurality of low resolution images is referred to as super-resolution.

  There are many super-resolution techniques, but reconfigurable super-resolution, which sequentially updates pixel values of high-resolution images by iterative processing, is widely used to obtain high-resolution images with high image quality stably. Yes. The reconstruction type super-resolution includes an alignment process for each of a plurality of low-resolution images and an iterative process in which an update process for obtaining pixel values of a high-resolution image is repeated. However, reconstruction-type super-resolution requires alignment of multiple low-resolution images, and because it uses iterative processing, the amount of processing involved is large. Reduction is necessary. As an example, Patent Document 1 proposes a method of reducing the number of iterations by optimizing the parameter value of the evaluation function during the iteration process.

  The following describes a MAP (Maximum A Posteriori) method, which is a reconstruction-type super-resolution process, as a conventional technique for reducing the amount of calculation without degrading image quality in reconstruction-type super-resolution.

  The MAP method uses a high-resolution image generated by the bicubic method or the nearest neighbor method as an initial value, and obtains a high-resolution image that maximizes the posterior probability when multiple low-resolution images that are observed images are used as conditions. It is. The posterior probability is expressed by an evaluation function, and the evaluation function includes an error term and a convergence term. The error term indicates a square error between the pixel value estimated based on the imaging model from the high resolution image and the pixel value of the aligned low resolution image when the imaging model is assumed, and the convergence term is The image shows prior information based on the assumption that it is smooth everywhere. An example of the evaluation function is shown in (Formula 1).

  Here, h_vec (i) in (Equation 1) is a vector representation of the pixel value of the high-resolution image (hereinafter h_vec (i) after the nth update is denoted as HR (n)), and fi is the value after alignment Pixel value of low-resolution image, b_vec (i) is a kernel element indicating the imaging model corresponding to the pixel position of fi, C is a function indicating prior information of smoothness, α is a weight between the error term and the convergence term, Nl Indicates the number of pixels of the low-resolution image used for updating h_vec (i). Note that Σ is the sum of Nl elements from 0th to (Nl-1) th (the total number of pixels of the aligned low-resolution image), || is the second-order norm, and * is the inner product of the vectors.

  At the time of iterative calculation, the evaluation function I of (Equation 1) is minimized, and at that time, optimization calculation such as the steepest descent method or the conjugate gradient method can be used. In these methods, it is necessary to obtain the gradient I ′ (Equation 2) of the evaluation function I.

Here, ∇ indicates the differentiation of the element.
As apparent from (Equation 2), when calculating the gradient I ′, calculation in units of pixels depending on Nl is necessary, and Nl is preferably in the same order as the number of pixels of the high-resolution image. The amount of calculation becomes very large.

  In addition, it is necessary to estimate the amount of motion in block units or pixel units between the super-resolution target image and the reference image for alignment when obtaining fi. Also increases depending on the number of reference images.

  1A and 1B show an application example of the MAP method to moving images. Here, the application destination is not limited to a moving image. For example, when the same subject is photographed from a plurality of positions (multiview), super-resolution using still images at each photographing position is possible. In the case of not a moving image, the amount of motion in the following description corresponds to the amount of positional deviation between the images, and the motion estimation corresponds to estimation of the amount of positional deviation. FIG. 1A shows that the kth picture is a super-resolution target picture, and the pictures before and after the temporal succession are reference pictures. Here, one picture corresponds to one frame or one field. FIGS. 1B (a) and 1B (b) show the pixels of the target picture and the reference picture to be super-resolved, respectively, and the result of performing the motion estimation between these pictures is shown in FIG. 1B (c). become that way. FIG. 1B (c) shows the result of aligning the pixel of the super-resolution target picture and the reference picture with respect to the pixel position of the high-resolution image. The gray circle and the white circle in FIG. The total number of gray circles and white circles corresponds to Nl in fi in 1) or (Formula 2). FIG. 1B (d) shows the relationship between the pixels of the high resolution image and the pixels of the low resolution image after alignment, and the black circle corresponds to the pixel position of the high resolution image. Here, the gray circle in which the super-resolution target picture is aligned is in the same position as the black circle that is a pixel of the high-resolution image in FIG. 1B (d). Note that the initial image of the high resolution image is generated by interpolating the super-resolution target image shown in FIG. 1B (a) by the bicubic method. The error term in (Equation 2) is calculated based on the difference between the pixel value of the white circle (and gray circle) and the value obtained by estimating the pixel value of the white circle (and gray circle) from the pixel value of the surrounding black circle. Then, the convergence term of (Expression 2) is calculated from the pixel value of the black circle. In the iterative process, the pixel values of the black circles are updated sequentially.

  From the above, in the MAP method, it has been found that the operations for alignment and iterative processing occupy most, and it is important to reduce the amount of calculation of these two processes.

  FIG. 2 is a block diagram showing a configuration of an image processing apparatus 500 that performs a conventional reconstruction-type super-resolution process. The image processing apparatus 500 includes an image input unit 501, a motion estimation unit 502, a registration unit 503, an initial image determination unit 504, a reconstruction unit 505, and a memory 510. The image input unit 501 stores input image data in the memory 510. The motion estimation unit 502 acquires image data necessary for motion estimation from the memory 510, performs motion estimation, and inputs the acquired motion vector information 511 to the alignment unit 503. Subsequently, the alignment unit 503 performs alignment based on the motion vector information 511 and outputs the result as position information 512. The initial image determination unit 504 generates an initial image 513 of a high resolution image according to the specified magnification. Further, the reconstruction unit 505 performs repetitive processing based on the position information 512 and the initial image 513 to generate reconstructed image data and output it.

  FIG. 3 is a block diagram illustrating a configuration of the reconfiguration unit 505. The reconstruction unit 505 includes an update calculation unit 601 and a picture unit update determination unit 602. When an update is instructed by the update instruction 611 input from the picture unit update determination unit 602, the update calculation unit 601 updates the pixel values for all the pixels of the high-resolution image based on the position information 512 and the initial image 513. . The picture unit update determination unit 602 determines whether to end the iterative process from the update result 612 of the high resolution image, and outputs the high resolution image data if it is determined to end, and continues the iterative process without ending. If it is determined, the update operation unit 601 is instructed to update the high resolution image by the update instruction 611.

  FIG. 4 is a flowchart showing the operation of a conventional reconstruction type super-resolution process. Position alignment is performed from step S001 to step S004, an initial image of a high resolution image is generated in step S005, and a high resolution image is reconstructed by iterative processing in step S006. Hereinafter, details of each step will be described in order. First, in step S001, image data of a super-resolution target picture and N reference pictures are input. Here, N is assumed to be a predetermined number. Next, in step S002, it is determined whether motion estimation and alignment have been completed for all N reference pictures. If processing has been completed for all reference pictures, the process proceeds to step S005, and if not completed, the process proceeds to step S003. In step S003, motion estimation is performed between the super-resolution target picture pic_cur and the reference picture pic_ref (k), and alignment is performed in step S004 based on the acquired motion amount. Here, k is an integer of 1 or more and N or less. Subsequently, in step S005, an initial image 513 of a high-resolution image is generated according to the specified magnification based on the pixel value of the super-resolution target picture pic_cur. In step S006, the initial image 513 is updated by iterative processing, and a reconstructed image is output.

  The motion estimation process in step S003 and the iterative process in step S006 will be further described.

  FIG. 5 is a flowchart showing the operation of the motion estimation process in step S003. In addition, motion estimation shall be performed per block and the size can be set arbitrarily. First, in step S0031, (i, j), which is a set of index numbers for designating a block, is both set to zero. Next, in step S0032, it is determined whether or not motion estimation has been completed for all blocks in the super-resolution target picture pic_cur. If motion estimation has been completed for all blocks, the motion estimation process is terminated. If not completed, the process proceeds to step S0033, and motion estimation is performed between the (i, j) th block and the kth reference picture pic_ref (k). Thereafter, (i, j) is updated in step S0034, and the process returns to step S0032. As described above, in the conventional motion estimation processing, motion estimation is always performed between all the blocks in the super-resolution target picture pic_cur and N reference pictures.

FIG. 6 is a flowchart showing the operation of the iterative process in step S006. First, in step S0061, the number of iterations n is set to 0. In step S0062, the end of the iterative process is determined. At this time, if the quadratic norm of the gradient I ′ of the evaluation function is smaller than the predetermined threshold ε, the iterative process is terminated, and the process proceeds to step S0065 to output the high resolution image HR (n + 1) as a reconstructed image. . If it is equal to or greater than the threshold ε, the process proceeds to step S0063. In step S0063, all pixel values of the high resolution image HR (n) are updated, and an updated high resolution image HR (n + 1) is generated. Here, the high resolution image HR (0) matches the initial image 513 of the high resolution image generated in step S005. Subsequently, in step S0064, 1 is added to the number of iterations n, and the process returns to step S0062. Thus, in the conventional iterative process, all the pixels of the high-resolution image HR (n) are always updated.
JP 2000-339450 A

  In the motion estimation process in the conventional reconfiguration-type super-resolution, motion estimation is always performed between all the blocks in the super-resolution target picture and N reference pictures. However, if the low-resolution images necessary for updating the high-resolution image are aligned, the super-resolution processing can be performed without degrading the image quality even if the number of reference pictures is less than N. FIG. 7 is an example showing a problem of conventional motion estimation processing. Here, it is assumed that the number of reference pictures is set to eight. (A), (b), and (c) of FIG. 7 show the alignment results of the low-resolution image at each stage where the reference number is 2, 4, and 8, respectively. White circles indicate pixel positions of the aligned low-resolution image, and black circles indicate pixel positions of the high-resolution image. Here, in the block A, the motion estimation process is continued although the pixels of the low-resolution image necessary for the update of the high-resolution image are prepared at the time of FIG. On the other hand, the block B does not have a sufficient number of low-resolution image pixels at the time of FIG. 7B, and is only necessary for updating the high-resolution image at the time of FIG. 7C. The pixels of the low resolution image are aligned. As described above, in the conventional motion estimation process, the motion estimation process is continued until the reference number reaches a predetermined number even in an area where there is no problem even if the motion estimation process is finished, as in the block A. As a result, there is a first problem that the amount of calculation increases.

  In the repetitive processing in the conventional reconstruction type super-resolution, all the pixels of the high-resolution image HR (n) are always updated. However, it may be considered that the pixel value has converged for a pixel with a small update amount of the pixel value. FIG. 8 is an example showing a problem of conventional iterative processing. FIG. 8A shows the relationship between the increase in the number of iterations and the region in which the pixel value of the high-resolution image is updated, and FIG. 8B shows the increase in the number of iterations and the update amount of the pixel value. Indicates a relationship with a region where the value exceeds a predetermined threshold value REP_STOP. As shown in FIGS. 8A and 8B, the pixel value of the high-resolution image is reduced in the region where the update amount of the pixel value exceeds the predetermined threshold value REP_STOP as the number of repetitions increases. The updated region is always constant regardless of the number of iterations. Here, for pixels whose pixel value update amount is less than or equal to the predetermined threshold value REP_STOP, the pixel value can be considered to have converged, so the image quality does not improve even if the update is continued, and the pixel value is not changed in the subsequent iterative processing. There is no need to update. As described above, in the conventional iterative process, there is a second problem that the amount of calculation increases because the update process is continued even for the pixel whose pixel value is considered to have converged.

  Therefore, the present invention has been made in view of the above circumstances, and an object thereof is to provide an image processing apparatus and an image processing method capable of reducing the amount of calculation without deteriorating the image quality after super-resolution. And

  In order to achieve the above object, an image processing apparatus according to the present invention uses a representative image of a first resolution and a plurality of reference images of a first resolution related thereto, and a high resolution of a second resolution higher than the first resolution. An image processing apparatus for generating an image, wherein a first iterative processing unit that repeats alignment processing while switching a reference image, and update processing that updates pixel values of a high-resolution image to be obtained after completion of the alignment processing A second repetitive processing unit that repeats the above, and the registration processing estimates a reference image misregistration amount with respect to the representative image, and determines the reference image pixel based on the estimated misregistration amount. The update processing is arranged at a sub-pixel position of the enlarged high-resolution image, and the update processing includes a pixel value of the aligned low-resolution image and a high solution to be obtained. Updating the pixel value of the high-resolution image so as to reduce the value of the evaluation function including the difference value with the pixel value of the image as a variable, the first iteration processing means and the second iteration processing means At least one is a determination unit that determines a pixel that satisfies the end condition from the result of the alignment process or the update process, and an exclusion unit that excludes the pixel determined to satisfy the end condition from the alignment process or the update process. Is provided. According to this configuration, by excluding pixels that satisfy the end condition from the subsequent alignment process or update process, the amount of calculation required for repeated alignment process or update process is reduced, and the excluded pixel ends. Since the condition is satisfied, deterioration in image quality can be prevented.

  Here, the first iterative processing means includes, for each block of the representative image, motion estimation means for estimating the motion of the reference image with respect to the representative image, and a plurality of the high resolution images corresponding to the pixels of the high resolution image. An alignment means for dividing the reference image pixel into sub-pixel positions of the high-resolution image for each block according to the estimated motion, and one or more reference image pixels for all the grids in the block When the ratio of the grid to include exceeds a predetermined value, the determination unit that determines that the block satisfies the end condition and the pixel of the block that is determined to satisfy the end condition are excluded from the alignment process. You may provide the said exclusion means. According to this configuration, it is possible to align a sufficient number of reference pixels for obtaining high image quality by the update process according to the end condition, and to reduce the calculation amount in the alignment process.

  Here, the determination unit estimates a motion for each block of the representative image, and the alignment unit divides the high-resolution image into a plurality of grids corresponding to pixels of the high-resolution image, and the estimated motion In accordance with the above, for each block, the pixels of the reference image are aligned at the sub-pixel positions of the high-resolution image, and the determination unit is a grid including one or more pixels of the reference image with respect to all the grids in the high-resolution image If the ratio exceeds the predetermined value, it is determined that the reference image satisfies the end condition, and the exclusion unit positions pixels of the remaining reference image when it is determined that the end condition is satisfied. You may make it exclude from a matching process.

  Here, when the ratio of the number of blocks determined to satisfy the termination condition with respect to the total number of blocks in the high-resolution image further exceeds a second predetermined value, the determination unit further determines the remaining reference images. These pixels may be excluded from the alignment process.

  Here, the motion estimation means may estimate motion in order from a reference image having a display order close to that of the representative image.

  Here, the image processing apparatus further includes determination means for determining whether the reference image belongs to the same scene as the representative image or a different scene, and the motion estimation means moves the reference image belonging to a different scene. You may make it exclude from estimation.

  Here, the second iterative processing means updates the pixel value so as to decrease the value of the evaluation function, and an update amount which is a difference between the pixel value before the update and the pixel value after the update. When the value is a value, the determination unit may determine that the pixel satisfies the end condition, and the exclusion unit may exclude the pixel determined to satisfy the end condition from the update process. According to this configuration, the termination condition sufficiently converges the high-resolution pixel estimation value in the update process, so that the image quality is not deteriorated. In addition, the calculation amount of the update process can be reduced.

Here, the threshold value may be a predetermined value.
Here, the threshold value may be determined such that a ratio of the number of pixels not excluded by the determination unit to the total number of pixels of the high-resolution image is equal to or less than a predetermined value.

  Here, the updating unit calculates a gradient value of the evaluation function using only pixel values of pixels not excluded by the determining unit, and updates the pixel estimated value of the high-resolution image based on the gradient value. May be.

  Here, the updating unit ends updating of the pixel estimated values for all the pixels when the reduction ratio of the sum of the difference values for the pixels not excluded by the determining unit is equal to or less than a predetermined value. Also good.

  The image processing method, program, and semiconductor integrated circuit of the present invention have the same means as described above.

  As described above, according to the present invention, based on the pixel arrangement of the aligned low-resolution image, a region requiring motion estimation is selectively determined in the high-resolution image, or motion estimation is performed for the entire high-resolution image. By deciding whether or not to end the processing, the amount of calculation related to the motion estimation processing can be reduced without degrading the image quality after super-resolution, and its practical value is high.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(Embodiment 1)
First, an overview of the image processing apparatus 100 according to Embodiment 1 of the present invention will be described.

  The image processing apparatus 100 is different from the conventional image processing apparatus in that the block for performing motion estimation is selectively determined in the super-resolution target picture.

  The image processing apparatus according to the present embodiment is an image processing apparatus that generates a high-resolution image having a second resolution higher than the first resolution by using a representative image having the first resolution and a plurality of reference images having the first resolution. Thus, a first iterative processing unit that repeats the alignment process while switching the reference image and a second iterative processing unit that repeats the update process for updating the pixel estimation value of the high-resolution image to be obtained are provided.

  Here, the first iterative processing means includes a determination means for determining a pixel satisfying the end condition from the result of the alignment process or the update process, and a subsequent alignment process or update process for the pixel determined to satisfy the end condition. And excluding means for excluding from.

  As described above, the alignment process is not performed on all blocks of the representative image for all the reference images, but only on a block including pixels that are excluded by the exclusion unit. As a result, the amount of calculation required for repeated alignment processing is reduced by excluding pixels that satisfy the end condition from the subsequent alignment processing. Since the excluded pixels satisfy the end condition, it is possible to prevent the image quality from being deteriorated. In the present embodiment, there are two end conditions: a picture end condition and a block end condition.

  FIG. 9 is a block diagram illustrating a configuration of the image processing apparatus 100. The image processing apparatus 100 includes an image input unit 101, a motion estimation unit 102, a registration unit 103, a reference image determination unit 104, an initial image determination unit 105, a reconstruction unit 106, and a memory 110.

  The image input unit 101 stores input image data in the memory 110. The motion estimation unit 102 acquires image data necessary for motion estimation from the memory 110, performs motion estimation, and inputs the acquired motion vector information 111 to the alignment unit 103. Subsequently, the alignment unit 103 performs alignment based on the motion vector information 111 and outputs the result as position information 112. The reference image determination unit 104 determines a reference picture based on the pixel position information 114 after alignment of the low-resolution image input from the alignment unit 103, and performs motion estimation in the picture to be super-resolved. A region to be implemented is determined, and these pieces of information are input to the motion estimation unit 102 as an estimation instruction 115. Therefore, the motion estimation unit 102 performs motion estimation according to the estimation instruction 115. The initial image determination unit 105 generates an initial image 113 of a high resolution image according to the specified magnification. Further, the reconstruction unit 106 performs repetitive processing based on the position information 112 and the initial image 113 to generate and output reconstructed image data.

  Here, the reference picture determination unit 104 determines one next reference picture from the remaining reference pictures if the target picture does not satisfy the picture end condition, and if the target picture satisfies the picture end condition, Reference pictures are excluded from motion estimation. Also, the reference image determination unit 104 determines a block that does not satisfy the block end condition as a region for performing motion estimation, and excludes a block that satisfies the block end condition as a region for performing motion estimation.

  FIG. 10 is a flowchart showing the operation of the image processing apparatus 100. Here, the operations in step S005 and step S006, which are processes after the alignment of the low-resolution image, are the same as those in the conventional image processing apparatus 500, and therefore, the same reference numerals are given and the description thereof is omitted. Note that the initial values of the picture end flag for each reference picture and the block end flag for each block are reset to 0. The picture end flag indicates whether or not the picture satisfies a picture end condition. The block end flag indicates whether or not the block satisfies a block end condition.

  First, in step S101, image data of a super-resolution target picture and N reference pictures are input. Here, N is a predetermined number of pictures, and N pictures are used as reference pictures in order from the picture whose display order is closest to the super-resolution target picture. Next, in step S105, it is determined whether the super-resolution target picture pic_cur satisfies the motion estimation termination condition. This determination depends on whether the picture end flag provided for each reference picture is set to 1. If the picture end flag corresponding to the target picture pic_cur is 1, that is, if the end condition for motion estimation is satisfied, the motion estimation and alignment process is terminated and the process proceeds to step S005. Whether or not the super-resolution target picture pic_cur satisfies the motion estimation termination condition depends on the output result in the subsequent step S103. However, in the first loop, it is assumed that the process proceeds to step S102. If the motion estimation end condition is not satisfied, the process proceeds to step S102, and it is determined whether or not motion estimation and alignment have been completed for all N reference pictures. If the process has been completed for all reference pictures, the process proceeds to step S005, and if not completed, the process proceeds to step S103. In step S103, motion estimation is performed between the super-resolution target picture pic_cur and the reference picture pic_ref (k), and alignment is performed in step S104 based on the acquired motion amount. At this time, motion estimation and alignment are selectively performed based on whether or not the pixels of the low-resolution image that satisfy the conditions necessary for updating the high-resolution image are prepared. Here, k is an integer of 1 or more and N or less.

  FIG. 11 is a flowchart showing details of the motion estimation operation in step S103. First, in step S1031, i and j, which are index numbers for identifying a block for motion estimation, are both set to 0. Next, in step S1032, it is determined whether or not the processing in steps S1033 and S1034 has been completed for all blocks of the super-resolution target picture pic_cur. If it is determined that the processing has been completed, the process proceeds to step S1036. If it is determined that the processing has not ended, the process proceeds to step S1033, and it is determined whether the (i, j) -th block satisfies the motion estimation end condition.

  FIG. 12A to FIG. 12D are examples illustrating conditions for ending motion estimation when the resolution of a low-resolution image is doubled. FIG. 12A shows the block position in the super-resolution target picture pic_cur, the portion shown in gray corresponds to the (4, 4) -th block, and the x mark indicates the pixel of the super-resolution target picture pic_cur. FIG. 12B shows the pixel positions in the high-resolution image, and the black circles correspond to the pixels in the high-resolution image. Here, in order to increase the resolution by a factor of 2, the pixels of the original super-resolution target picture pic_cur exist every other pixel in both the horizontal direction and the vertical direction, as indicated by the x mark. Further, as shown in region C, one pixel of the high resolution image is divided into a plurality of grids. Here, an example in which an area corresponding to one pixel is divided into 2 × 2 grids is shown. 12C and 12D show a state after the reference picture is aligned in the same area as FIG. 12B, and white circles indicate the pixels of the reference picture that have been aligned. FIG. 12C shows a case where, after alignment, each grid includes one or more pixels of the super-resolution target picture pic_cur or reference picture pic_ref. At this time, motion estimation for the (4, 4) -th block It is assumed that the termination condition is satisfied. On the other hand, FIG. 12D shows a case where there is a grid that does not include pixels of the super-resolution target picture pic_cur or the reference picture pic_ref. At this time, the motion estimation termination condition for the (4, 4) -th block is I do not meet.

  In this way, in the high-resolution image block corresponding to the (i, j) -th block of the super-resolution target picture pic_cur, all the grids in all the pixels are one or more low-resolution image pixels after alignment. If it is filled with, it is determined that the motion estimation ends for the (i, j) -th block.

  When it is determined in step S1033 that the motion estimation of the (i, j) -th block is to be ended, the block end flag corresponding to the block is set to 1 among the block end flags provided for each block. Then, the process returns to step S1032, and if it is determined not to end, the process proceeds to step S1034. In step S1034, motion estimation processing is performed with respect to the reference picture pic_ref (k) for the (i, j) -th block. A block matching method, a phase only correlation method, or the like can be applied as a motion estimation method. Then, the process proceeds to step S1035. After updating (i, j), the process returns to step S1032. Subsequently, in step S1036, it is determined whether all blocks satisfy the motion estimation end condition. If the picture end condition is satisfied, the process proceeds to step S1038, and the motion estimation process for the super-resolution target picture pic_cur is ended. Judge that. As a result of this determination, the picture end flag corresponding to the target picture pic_cur is set to 1. If there is a block that does not satisfy the picture motion estimation termination condition, the process advances to step S1037 to determine that the motion estimation process for the super-resolution target picture pic_cur is not terminated. As a result of this determination, the picture end flag corresponding to the target picture pic_cur remains 0.

  Here, the termination condition in step S1033 is not limited to whether or not the grids of the pixels of the high resolution image are all filled, and may be any one of 1 to 4 below, for example.

  1. Among the grids corresponding to each pixel of the high resolution image, a grid of a predetermined ratio or more is filled with one or more pixels of the low resolution image. For example, if pixels of a high resolution image are divided into 4 × 4 grids, at least 16 reference pictures are required to fill all the grids. If the number of reference pictures increases, not only will the amount of computation increase, but the memory required to store the reference pictures will also increase, and if a grid with a predetermined ratio or more is filled, sufficient image quality will be obtained when a high-resolution image is reconstructed. Thus, this method is particularly effective.

  Furthermore, the predetermined ratio may be switched according to the property of the image in the block. For example, in a portion containing a lot of high-frequency components such as edges, the effect of improving the image quality depending on the increase in the number of buried grids is greater than in a flat region with few high-frequency components. Therefore, based on the results of edge detection in the block, if there are high-frequency components such as edges or the ratio of high-frequency components is large, motion estimation will continue until more grids are filled. May be.

  2. Of the grid corresponding to each pixel of the high resolution image or the grid close to the pixel position of the high resolution image, a grid of a predetermined ratio or more is filled with one or more pixels of the low resolution image. For example, the determination may be made based on whether a grid adjacent to the grid including the pixel position of the high-resolution image is filled.

  3. Among the grids corresponding to each pixel of the high resolution image, a specific grid such as a grid adjacent to the pixel position of the high resolution image is filled with N or more pixels of the low resolution image.

  4). Of the grid corresponding to each pixel of the high resolution image, an increase amount of the ratio of the grid filled with one or more pixels of the low resolution image satisfies a predetermined condition. For example, when the size of the increase amount or the ratio of the number of newly filled grids to the total number of grids filled with pixels of one or more low-resolution images is less than a certain amount The motion estimation is terminated assuming that there is no reference picture having a motion that fills another grid.

  Further, the shape of the grid may not be rectangular. For example, the grid may be set concentrically according to the distance from the pixel position of the high resolution image.

  In step S101, all the pictures that can be referred to are input in advance. However, if it is determined in steps S105 and S102 that a new reference picture is necessary, the reference pictures may be input sequentially. FIG. 13 is a flowchart illustrating an operation when sequentially inputting necessary reference pictures. In this operation, first, the super-resolution target picture pic_cur is input in step S106, and the reference picture is input in step S107. In step S107, it is determined whether or not the next reference picture has not been acquired. If it has not been acquired, the reference picture is input. In the case of super-resolution of moving pictures, the pictures used as reference pictures in the super-resolved pictures are stored in the memory, and these pictures are used when super-resolution of subsequent pictures. Can also be used as a reference picture. Therefore, it is only necessary to input a reference picture when the next reference picture is not acquired. Note that the reference pictures are acquired in order from the picture whose display order is close to the super-resolution target picture pic_cur.

  FIG. 14 is a diagram for explaining the effect of the image processing method of the present embodiment. (A), (b), and (c) in FIG. 14 correspond to (a), (b), and (c) in FIG. 7, respectively. As shown in FIG. 7, in the conventional method, the motion estimation process is continued until the reference number reaches a predetermined number even for an area where the motion estimation process may be terminated, as in block A. However, according to the method of the present embodiment, in block A, since the end condition for motion estimation is satisfied at the stage where the alignment with the four reference pictures is completed in FIG. No motion estimation process is performed for. As a result, it is possible to reduce the motion estimation process of block A from the fifth reference picture to the eighth reference picture.

Below, the modification of this Embodiment is described.
The motion estimation in step S103 may be performed after increasing the resolution of the low-resolution image by interpolation processing. As an example, motion estimation is performed after interpolating the target picture pic_cur and the reference picture pic_ref of the super-resolution by the bicubic method or the 0th-order hold method, respectively.

  Furthermore, the motion estimation end determination in step S103 may be performed only in units of pictures. At this time, in step S1034, the motion estimation is performed regardless of the result of the motion estimation end determination of the (i, j) -th block in step S1033.

  In step S1036, the end determination is performed based on whether or not all the blocks satisfy the motion estimation end condition.However, more than a certain percentage of the blocks satisfy the motion estimation end condition, or there are many edges and high-frequency components. The motion estimation process for the super-resolution target picture pic_cur may be terminated if the included block satisfies a condition such as a motion estimation termination condition.

  In addition, the motion estimation is performed on the basis of a block in the super-resolution target picture, but may be performed on the basis of a block in the reference picture. At this time, since the selective motion estimation processing based on whether or not the motion estimation end condition is satisfied in the super-resolution target picture cannot be performed, the motion estimation end determination is performed only in units of pictures.

  In the above description, motion estimation in units of blocks has been described. However, motion estimation may be performed in units of objects after objects are extracted.

  In step S101, the maximum value of the number of reference pictures is fixed to N, but may be switched adaptively. For example, in the case of sequentially super-resolving pictures in a moving picture stream, the number of reference pictures used in the immediately preceding picture may be set as the number of necessary reference pictures.

  In addition, the generation of the super-resolution initial image 113 in step S105 may be performed at a stage before the step of performing the motion estimation process or at a stage before acquiring the reference picture.

  The reference picture pic_ref is preferably a picture in the same scene. This is because when a scene is switched, there is a high possibility that there is no correlation between images in the scene before switching and the scene after switching, and the importance as a reference picture decreases. FIG. 15 is a flowchart illustrating an operation when determining a reference picture in consideration of scene switching (scene change) in step S107. Here, it is assumed that the display order of the super-resolution target picture pic_cur is k-th. First, in step S201, variables k_minus and k_plus are both set to 0. Next, in step S202, it is determined that the display target is the reference object in order from the kth closest picture. In step S203, the next reference target picture is determined based on the result determined in step S202. In step S204, it is determined whether a scene change has been detected in the reference target picture. A scene change is a scene change information added to a video stream in DV or the like, a start position of a random access unit such as GOP (Group of Pictures) in an encoded stream such as MPEG, or a stream playback It can be detected by referring to the navigation data. If no scene change is detected in step S204, the process proceeds to step S205, the image of the reference target picture is input, and the process ends. If a scene change is detected, the process proceeds to step S206. In step S206, it is determined that the current reference target picture is not a reference picture, and the process proceeds to step S207. In step S207, it is determined whether or not the display order of the current reference target picture is after the k-th. If the display order is before, the process proceeds to step S208. If the display order is after, the process proceeds to step S209. In step S208, it is determined that only pictures whose display order is k + 1 and after (including k + 1) are to be referred to, and the process proceeds to step S210 where k_plus is set to 1. In step S209, it is determined that only pictures whose display order is before the k−1th (including the k−1th) are to be referenced, and the process proceeds to step S211 where k_minus is set to 1. After step S210 or step S211 ends, the process proceeds to step S212. In step S212, it is determined whether k_plus and k_minus are both 1. If both are 1, the process ends. If at least one of them is 0, the process returns to step S203. If the reference target picture can no longer be acquired at the beginning or end of the stream in the moving image, the process ends at that point. Note that step S202 may be performed only at the start of the super-resolution processing. Also, as in step S101 in FIG. 10, when N pictures to be referred to are input at a time, the processes from S201 to S212 are repeated until N reference pictures are determined. It should be noted that the processing is not continued until N reference pictures are determined, and the processing for N reference target pictures is completed, that is, for N pictures from step S203 to step S212. When the above process is completed, the reference picture input process may be terminated. Furthermore, when handling an encoded stream including pictures with different decoding order and display order, such as bi-predictive pictures in MPEG-2 video and bi-predictive pictures in MPEG-4 AVC (Advanced Video Coding), The display order of pictures is acquired in advance.

  Further, although iterative processing is used for image reconstruction, a high-resolution image may be generated by a method different from the iterative processing, such as applying an inverse filter after resampling processing.

  Further, although the low resolution image is used as the reference image, an image after super-resolution may be used as the reference image. In particular, when super-resolving a moving image stream in sequence, a super-resolution image can be used as a reference image.

  In addition, the number of reference images for which alignment is always performed may be set in advance, and it may be selectively determined whether or not to reference a reference image that exceeds the set number. For example, alignment is performed for at least four reference images, and it is determined whether or not the fifth and subsequent reference images are referred to by the method of the present embodiment. By doing so, it is possible to omit the process related to the determination of the end of motion estimation for the minimum reference image necessary for alignment.

  As described above, the image processing method according to the present embodiment includes step S105 for determining whether to end the motion estimation and alignment processing for each reference image in units of blocks or in units of pictures. Motion estimation and alignment processing can be performed only in the region where the above-mentioned alignment is necessary, and the amount of calculation related to motion estimation, which is a processing with a large calculation load, can be reduced. Furthermore, by sequentially inputting the necessary reference images, it is possible to reduce the memory necessary for holding the reference images, and it is possible to easily select the pictures in the same scene where the alignment is effective.

(Embodiment 2)
Compared with the first embodiment, the image processing apparatus 200 according to the present embodiment further includes a second iterative processing unit, a determination unit that determines a pixel that satisfies the end condition from the result of the update process, and an end condition Excluding means for excluding the pixels determined to be excluded from the subsequent update processing. In this way, the update process is not performed on all the pixels of the high-resolution image, but is performed on the remaining pixels that are excluded by the exclusion unit. By excluding pixels that satisfy the end condition from the update process, the amount of computation required for repeated update processes is reduced, and the excluded pixels satisfy the end condition, so that a reduction in image quality can be prevented.

  FIG. 16 is a block diagram illustrating a configuration of the image processing apparatus 200. The image processing apparatus 200 includes an image input unit 101, a motion estimation unit 102, a registration unit 103, a reference image determination unit 104, an initial image determination unit 105, a reconstruction unit 206, and a memory 110.

  The image input unit 101 stores input image data in the memory 110. The motion estimation unit 102 acquires image data necessary for motion estimation from the memory 110, performs motion estimation, and inputs the acquired motion vector information 111 to the alignment unit 103. Subsequently, the alignment unit 103 performs alignment based on the motion vector information 111 and outputs the result as position information 112. The initial image determination unit 105 generates an initial image 113 of a high resolution image according to the specified magnification. Further, the reconstruction unit 206 performs iterative processing based on the position information 112 and the initial image 113, selectively updates the pixel values of the high resolution image, generates reconstructed image data, and outputs the reconstructed image data.

  FIG. 17 is a block diagram illustrating a configuration of the reconstruction unit 206. The reconstruction unit 206 includes an update calculation unit 211, a picture unit update determination unit 212, and a pixel unit update determination unit 213. The update calculation unit 211 follows the position information 112 and the initial image in accordance with the update instruction information 311 for the picture unit input from the update unit for picture unit 212 and the update instruction 315 for the pixel unit input from the pixel unit update determination unit 213. The pixel value of the high resolution image is updated based on 113. The picture unit update determination unit 212 determines whether to end the iterative process from the update result 312 of the high resolution image, and outputs the high resolution image data when it is determined to end, and continues the iterative process without ending. If it is determined, the update operation information 211 is instructed to update the high-resolution image by the update instruction information 311. Further, the pixel unit update determination unit 213 determines whether or not to end the update in pixel units based on the pixel unit update information 314 of the high resolution image input from the update calculation unit 211, and updates the update instruction 315. Input to the unit 211.

  Next, the operation of the image processing apparatus 200 according to Embodiment 2 of the present invention will be described. The image processing apparatus 200 is different from the image processing apparatus 100 in that the pixel value to be updated is selectively determined when the pixel value of the high resolution image is updated by iterative processing in step S005.

  FIG. 18 is a flowchart showing the operation of the iterative process in the image processing apparatus 200. First, in step S0061, the number of iterations n is set to 0. In step S3062, the end of the iterative process is determined. At this time, if the quadratic norm of the gradient I ′ of the evaluation function is smaller than the predetermined threshold ε, the iterative process is terminated, and the process proceeds to step S0065 to output the high resolution image HR (n + 1) as a reconstructed image. . If the second-order norm of the gradient I ′ of the evaluation function is greater than or equal to the threshold value ε, the process proceeds to step S3063. In step S3063, the pixel value is updated only for pixels satisfying a predetermined condition in the high resolution image HR (n), and the updated high resolution image HR (n + 1) is generated. Here, the high resolution image HR (0) matches the initial image 113 of the high resolution image generated in step S005. Subsequently, in step S0064, 1 is added to the number of iterations n, and the process returns to step S3062.

  In step S3063, the pixel that satisfies the predetermined condition is a pixel whose update end flag RepFlag (i, j) provided for each pixel corresponds to zero. In the update process in step S3063, the pixel data corresponding to the update end flag RepFlag (i, j) of 1, that is, the pixel data that satisfies the update end condition is the vector element of h_vec (i) in (Expression 2). Excluded from. As a result, the calculation amount of the update process is greatly reduced.

  FIG. 19 is a flowchart showing details of the operation in step S3063. The operation is mainly composed of two parts. A step (step S4001 to step S4005) for calculating the gradient I ′ (formula 2) of the evaluation function I (formula 1) and a high-resolution image based on the obtained gradient I ′. And updating the pixel value (from step S4006 to step S4012).

  First, the part for obtaining the gradient I ′ of the evaluation function I will be described. In step S4001, both i and j, which are index numbers indicating each pixel of the high-resolution image HR (n), are set to 0, and the values of the update end flags RepFlag (i, j) for all the pixels are set to 0. To do. Next, in step S4002, it is determined whether or not the processing of all the pixels of the high resolution image HR (n) has been completed. If it is determined that the processing of all the pixels has been completed, the process proceeds to step S4006 and has not ended. If it is determined, the process proceeds to step S4003. In step S4003, it is determined whether or not the update end flag RepFlag (i, j) indicating whether or not to end the update of the pixel value is 1 in the pixel (i, j) of HR (n). The process proceeds to step S4005, and if it is 0, the process proceeds to step S4004. In step S4004, the value of the gradient I ′ (expression 2) of the evaluation function I (expression 1) is updated using the pixel value of the pixel (i, j) of HR (n), and the process proceeds to step S4005. In step S4005, (i, j) is updated, and the process returns to step S4002.

  Subsequently, the pixel value of the high resolution image is updated based on the obtained gradient I ′. First, in step S4006, i and j are both set to 0. Next, in step S4007, it is determined whether or not the processing of all the pixels of the high-resolution image HR (n) has been completed. If it is determined that the processing has ended, the pixel value update processing for HR (n) is terminated. To do. If it is determined that the update has not ended, the process advances to step S4008 to determine whether or not the update end flag RepFlag (i, j) in the pixel (i, j) of HR (n) is 1. The process proceeds to S4012, and if it is 0, the process proceeds to Step S4009. In step S4009, the pixel value of the pixel (i, j) is updated based on the gradient I ′ to obtain the updated pixel value HR (n + 1). (I, j), and then the process proceeds to step S4010. In step S4010, the absolute value of the difference between the updated pixel value HR (n + 1). (I, j) and the updated pixel value HR (n). (I, j) is updated to the pixel value. It is determined whether or not the threshold value REP_STOP is larger than the threshold value REP_STOP. If it is larger than the threshold value REP_STOP, the process proceeds to step S4012. Here, it is assumed that the threshold value REP_STOP is a predetermined value. In step S4011, the update end flag RepFlag (i, j) is set to 1 and the process proceeds to step S4012. Finally, (i, j) is updated in step S4012, and the process returns to step S4007.

  In the above description, the pixel value is updated after the calculation of the gradient I ′ (Equation 2) of the evaluation function I (Equation 1) is completed for all pixels, but the pixel ( The pixel value of the pixel (i, j) of the high-resolution image HR (n) may be updated when the calculation of the gradient I ′ is completed for the pixels that affect i, j). The pixel that affects the pixel (i, j) of the high-resolution image HR (n) is determined by the size of the kernel b_vec (i) indicating the imaging model in (Equation 1). For example, if the size of the kernel b_vec (i) is 7 × 7, the update of the gradient I ′ for the 7 × 7 pixels around (i, j) is completed, and then the pixel (i , j) can be updated. Thus, by sequentially updating the pixel values, the calculation process of the gradient I ′ and the update process of the pixel values can be parallelized.

  FIG. 20 is a diagram for explaining the effect of the image processing method of the present embodiment. 20A and 20B correspond to FIGS. 8A and 8B, respectively. As shown in FIG. 8A, in the conventional method, the pixel values are updated for all the pixels of the high-resolution image regardless of the number of iterations. However, in the method of the present embodiment, as shown in FIG. 20 (a), the pixel whose pixel value needs to be updated is determined for each iteration, and therefore the pixel whose pixel value is updated as the number of iterations increases. Decrease. For this reason, as compared with the conventional method, it is possible to reduce the calculation related to the update of the pixel value for the portion where the pixel value does not need to be updated (the white area in FIG. 20A).

Below, the modification of this Embodiment is described.
In step S4010, the threshold value REP_STOP indicating whether or not to end the update of the pixel value is a predetermined fixed value, but may be dynamically set as 1 to 3 below, for example.

  1. In the Mth iteration (M is an integer of 1 or more), the threshold value REP_STOP is set so that the ratio of the pixels whose pixel values are updated in the pixels of the high resolution image is close to a predetermined value. For example, the ratio of pixels updated in the second iterative process is set to be 80% or less. The setting of the threshold value may be fixed within the iterative process or may be switched for each loop of the iterative process. In addition, the same threshold value may be used in the same scene, and the threshold value may be reset according to scene switching. Furthermore, the setting method may be switched according to the property of the image, such as setting the predetermined ratio higher in a region having a high frequency component in the high resolution image. Thus, the amount of calculation can be effectively reduced by limiting the ratio of the pixels whose pixel values are updated.

  2. Switch according to the quality of the reconstructed image. For example, when there are a high image quality mode and a standard image quality mode, the threshold value REP_STOP is made smaller than the low image quality in the high image quality mode.

  3. Switch according to the nature of the image. For example, in an image having a high frequency component, the threshold REP_STOP is made smaller than an image having a low frequency component. Note that the unit of switching may be either for each image or for each region in the image.

  In step S3062, the iteration is terminated if the quadratic norm of the gradient I ′ of the evaluation function is smaller than the predetermined threshold ε. However, the high-resolution image HR (n) after the update and the high-resolution image before the update are displayed. Whether to end the iteration may be determined based on a difference value from HR (n-1), that is, a change amount of the update amount. Here, since iterative processing is executed at least once, n is 1 or more. FIG. 11 is a flowchart illustrating an operation example of determination based on the change amount of the update amount. First, in step S5001, variables i, j, DIFF, and SUM are all set to zero. In step S5002, it is determined whether or not the processing of all the pixels of the high resolution image HR (n) has been completed. If it is determined that the processing has ended, the process proceeds to step S5006. If it is determined that the processing has not ended, the process proceeds to step S5003. move on. In step S5003, it is determined whether or not the update end flag RepFlag (i, j) in the pixel (i, j) of the high-resolution image HR (n) is 1, and if the update end flag RepFlag (i, j) is 1. If the update end flag RepFlag (i, j) is 0, the process proceeds to step S5004. Subsequently, in step S5004, the pixel value HR (n). (I, j) of the updated high resolution image and the pixel value HR (n-1). The absolute value of the difference value of j) is added, and the pixel value HR (n−1). (i, j) of the high-resolution image before update is added to the variable SUM, and the process proceeds to step S5005. In step S5005, (i, j) is updated, and the process returns to step S5002. Next, in step S5006, it is determined whether the result of dividing DIFF by SUM is greater than the threshold value REP_THR. If so, the process proceeds to step S5007, and it is determined to end the iterative process for each picture. Here, it is assumed that the threshold value REP_THR is a predetermined value. In this method, the pixel value is updated with respect to the pixel value HR (n-1). (I, j) of the high-resolution image before the update by comparing the result of dividing the variable DIFF by the variable SUM with the threshold REP_THR. The iterative process is discontinued when the ratio of the quantity is equal to or less than a certain value. However, the iterative process may be discontinued when the value of the variable DIFF is equal to or less than a predetermined threshold. In this way, it is possible to determine the end of the iteration without depending on the pixel value HR (n−1). (I, j) of the high-resolution image before update.

  In the method of FIG. 21, only the pixels whose update end flag RepFlag (i, j) is 0 are used in the calculation of the variable DIFF and the variable SUM. However, the information on the pixels whose RepFlag (i, j) is 1 is used. May be reflected. At this time, if the portion related to the error term of the gradient I ′ is calculated for pixels whose RepFlag (i, j) is 0, the amount of calculation increases. Therefore, for these pixels, RepFlag (i, j) The pixel value immediately before 1 becomes 1 and the update amount of the pixel value can be used. This method can also be applied when updating the gradient I ′ in step S3063.

  In (Equation 1) and (Equation 2), error terms are calculated for all the pixels after alignment, but may be calculated in units of grids in which the pixels of the high-resolution image are partitioned. At this time, an average value or the like of the pixel values of the low-resolution image aligned in each grid is used as a representative value of the pixel value of the grid. Furthermore, with the center of the grid as a representative value of the pixel position of the grid, the result of applying the b_vec (i) kernel corresponding to the representative position of the pixel value to the vector h_vec (i) of the pixel value of the high resolution image, Each element of the error term is calculated from the difference value between the pixel value and the representative value.

  Further, the information of the update flag RepFlag indicating whether or not the pixel value needs to be updated may be held not in pixel units but in block units or object units. At this time, whether or not the pixel value needs to be updated in the area is determined from the sum of DIFF / SUM (the value obtained by dividing the variable DIFF by the variable SUM) or the sum of the DIFF. In this way, by holding update information for each area, it is possible to reduce the amount of memory necessary for holding update information.

  Further, a method different from the MAP method used for the iterative process may be used. That is, the evaluation function is not limited to the form of (Formula 1). For example, the ML (Maximum Likelihood) method may be used. In this case, only the error term of (Expression 1) is used as the evaluation function. Or you may weight with respect to the error term of (Formula 1).

  As described above, the image processing method according to the present embodiment includes the step S3063 for determining whether or not to end the iterative process in units of pixels, so that the pixel value is updated for pixels that can be regarded as having converged. Therefore, it is possible to reduce the processing amount related to the iterative processing without degrading the image quality of the reconstructed image.

Below, the modification regarding Embodiment 1 and Embodiment 2 is described.
First, in the first embodiment and the second embodiment, the resolution of the low resolution image and the high resolution image may be the same. At this time, the enlargement ratio is 1 and the resolution is not improved, but the high frequency component is restored by the reconstruction process, and the image quality is improved.

  In addition, the motion estimation and alignment method of the first embodiment and the iterative processing method of the second embodiment may be used in combination.

  Further, in Embodiments 1 and 2, the initial image of the high-resolution image is generated by interpolating the pixels of the super-resolution target picture. However, after the reference picture is aligned, the aligned pixel May be interpolated to generate an initial image of a high resolution image. If the alignment accuracy is high, an image closer to the original high-resolution image can be used as the initial value by using the pixel after alignment, so that the number of iterations can be reduced.

  The application destination is not limited to a moving image. For example, when the same subject is photographed from a plurality of positions (multiview), super-resolution using still images at each photographing position is possible. When the image is not a moving image, the amount of motion in the above description corresponds to the amount of displacement between images, and the motion estimation corresponds to estimation of the amount of displacement.

(Embodiment 3)
Further, by recording a program for realizing the image processing method shown in each of the above embodiments on a recording medium such as a flexible disk, the processing shown in each of the above embodiments can be performed by an independent computer. It can be easily implemented in the system.

  22A to 22C are explanatory diagrams when the image processing method of each of the above embodiments is implemented by a computer system using a program recorded on a recording medium such as a flexible disk.

  FIG. 22B shows an appearance, a cross-sectional structure, and a flexible disk as viewed from the front of the flexible disk, and FIG. 22A shows an example of a physical format of the flexible disk that is a recording medium body. The flexible disk FD is built in the case F, and on the surface of the disk, a plurality of tracks Tr are formed concentrically from the outer periphery toward the inner periphery, and each track is divided into 16 sectors Se in the angular direction. ing. Therefore, in the flexible disk storing the program, the program is recorded in an area allocated on the flexible disk FD.

  FIG. 22C shows a configuration for recording and reproducing the program on the flexible disk FD. When the program for realizing the image processing method is recorded on the flexible disk FD, the program is written from the computer system Cs via the flexible disk drive. When the image processing method for realizing the image processing method by the program in the flexible disk is constructed in the computer system, the program is read from the flexible disk by the flexible disk drive and transferred to the computer system.

  In the above description, a flexible disk is used as the recording medium, but the same can be done using an optical disk. Further, the recording medium is not limited to this, and any recording medium such as an IC card or a ROM cassette capable of recording a program can be similarly implemented.

  The image processing apparatus and the image processing method according to the present invention have been described based on the above embodiments, but the present invention is not limited to these embodiments. The present invention also includes modifications made to the present embodiment by those skilled in the art without departing from the scope of the present invention.

  For example, an optical disk recording apparatus, a moving image transmission apparatus, a digital television broadcast transmission apparatus, a Web server, a communication apparatus, a portable information terminal, and the like that include the image processing apparatus according to the present embodiment, and a moving image that includes the image processing apparatus according to the present embodiment. Needless to say, an image receiving device, a moving image recording device, a still image recording device, a digital television broadcast receiving device, a communication device, a portable information terminal, and the like are also included in the present invention. Here, the moving image recording device includes a camcorder, Web, and the like, and the still image recording device includes a digital still camera and the like.

Note that each functional block in the block diagrams (FIGS. 9, 16, 17 and the like) is typically realized as an LSI (Large Scale Integration) which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include a part or all of them. For example, functional blocks other than the memory (in FIGS. 9 and 16, a portion surrounded by a broken line in the figure) may be integrated into one chip.

  Although referred to as LSI here, it may be called IC (Integrated Circuit), system LSI, super LSI, or ultra LSI depending on the degree of integration.

  Further, the method of circuit integration is not limited to LSI, and implementation with a dedicated circuit or a general-purpose processor is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of the circuit cells inside the LSI may be used.

  Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied.

  Further, among the functional blocks, only the means for storing the data to be subjected to image processing may be configured separately instead of being integrated into one chip.

  According to the present invention, it is possible to reduce the amount of computation for obtaining a high-resolution and high-resolution image by super-resolution processing, so it is easy to reproduce a moving image recorded at a low resolution with high-resolution by super-resolution processing. The effectiveness is high.

It is explanatory drawing which shows the object picture of super-resolution, and the some reference picture before and behind temporally. It is explanatory drawing which shows the pixel of the object picture of a super resolution, the pixel of a reference picture, and the pixel of a high resolution image. It is a block diagram of the image processing apparatus which performs the conventional reconstruction type super-resolution. It is a block diagram of the reconstruction part in the image processing apparatus which performs the conventional reconstruction type super-resolution. It is a flowchart which shows the operation | movement of the conventional reconstruction type super-resolution. It is a flowchart which shows the operation | movement of the motion estimation process in the conventional reconstruction type super-resolution. It is a flowchart which shows the operation | movement of the iterative process in the conventional reconstruction type super-resolution. It is a figure which shows the subject of the conventional reconstruction type super-resolution motion estimation process. It is a figure which shows the subject of the repetitive process of the conventional reconstruction type super-resolution. 1 is a block diagram illustrating a configuration of an image processing apparatus according to a first embodiment. 3 is a flowchart illustrating an operation of the image processing apparatus according to the first embodiment. 4 is a flowchart illustrating an operation of motion estimation processing in the image processing apparatus according to the first embodiment. 3 is a diagram illustrating pixels of a high-resolution image in the image processing apparatus according to Embodiment 1. FIG. 3 is a diagram illustrating a grid of a high-resolution image in the image processing apparatus according to Embodiment 1. FIG. 6 is a diagram illustrating an example of an end condition in the image processing apparatus according to Embodiment 1. FIG. 6 is a diagram illustrating an example that does not satisfy an end condition in the image processing apparatus according to Embodiment 1. FIG. 3 is a flowchart illustrating a reference image input operation in the image processing apparatus according to the first embodiment. It is a figure explaining the effect of the image processing apparatus of Embodiment 1. FIG. 3 is a flowchart illustrating reference image determination operation in the image processing apparatus according to the first embodiment. 6 is a block diagram illustrating a configuration of an image processing apparatus according to a second embodiment. FIG. 10 is a block diagram illustrating a configuration of a reconstruction unit in the image processing apparatus according to Embodiment 2. FIG. 10 is a flowchart illustrating an iterative process operation in the image processing apparatus according to the second embodiment. 6 is a flowchart illustrating an operation for determining the end of update in pixel units in the image processing apparatus according to the second embodiment. It is a figure explaining the effect of the image processing apparatus of Embodiment 2. FIG. 10 is a flowchart illustrating an operation for determining the end of update in units of pictures in the image processing apparatus according to the second embodiment. It is a figure which shows the example of the physical format of the flexible disk which is a recording medium main body. It is a figure which shows the external appearance seen from the front of a flexible disk, sectional structure, and a flexible disk. It is a figure which shows the structure of the apparatus which records and reproduces the said program on flexible disk FD.

Explanation of symbols

DESCRIPTION OF SYMBOLS 100,200 Image processing apparatus 101 Image input part 102 Motion estimation part 103 Position alignment part 104 Reference image determination part 105 Initial image determination part 106,206 Reconstruction part 110 Memory 114 Pixel position information 115 Estimation instruction 211 Update calculation part 212 Picture unit Update determination unit 213 Pixel unit update determination unit

Claims (18)

An image processing apparatus that generates a high-resolution image having a second resolution higher than the first resolution by using a representative image having the first resolution and a plurality of reference images having the first resolution related thereto,
First iterative processing means for repeating the alignment process while switching the reference image;
A second iterative processing unit that repeats an update process for updating a pixel value of a high-resolution image to be obtained after the completion of the alignment process;
The alignment process estimates a positional deviation amount of the reference image with respect to the representative image, and arranges pixels of the reference image at subpixel positions of the high-resolution image obtained by enlarging the representative image based on the estimated positional deviation amount. Is to do
The update process is such that the pixel value of the high-resolution image is reduced so as to reduce the value of the evaluation function including the difference value between the pixel value of the aligned low-resolution image and the pixel value of the high-resolution image to be obtained as a variable. Is to update
At least one of the first iteration processing means and the second iteration processing means is:
Determination means for determining a pixel that satisfies the end condition from the result of the alignment process or the update process;
An image processing apparatus comprising: an excluding unit that excludes a pixel determined to satisfy an end condition from the alignment process or the update process. The first iterative processing means includes
Motion estimation means for estimating the motion of a reference image with respect to the representative image for each block of the representative image;
Alignment means for dividing the high-resolution image into a plurality of grids corresponding to the pixels of the high-resolution image, and arranging pixels of the reference image at sub-pixel positions of the high-resolution image for each block according to the estimated motion;
The determination means for determining that the block satisfies the end condition when a ratio of a grid including one or more pixels of the reference image with respect to all grids in the block exceeds a predetermined value;
The image processing apparatus according to claim 1, further comprising: an exclusion unit that excludes a pixel of a block determined to satisfy the end condition from the alignment process. The second iterative processing means includes
Updating means for updating the pixel value so as to decrease the value of the evaluation function;
The determination means for determining that the pixel satisfies the end condition when an update amount that is a difference between the pixel value before update and the pixel value after update is a threshold value;
The image processing apparatus according to claim 1, further comprising: an exclusion unit that excludes a pixel determined to satisfy the termination condition from the update process. An image processing apparatus that generates a high-resolution image having a second resolution higher than the first resolution by using a representative image having the first resolution and a plurality of reference images having the first resolution related thereto,
First iterative processing means for repeating the alignment process while switching the reference image;
A second iterative processing unit that repeats an update process for updating a pixel estimation value of a high-resolution image to be obtained after the completion of the alignment process;
The alignment process estimates a positional deviation amount of the reference image with respect to the representative image, and arranges pixels of the reference image at subpixel positions of the high-resolution image obtained by enlarging the representative image based on the estimated positional deviation amount. Is to do
The update processing is performed so that a value of an evaluation function including a difference value between a pixel value of the aligned low-resolution image and a pixel value of the high-resolution image model to be obtained as a variable is reduced. Update the pixel values,
The first iterative processing means includes
Motion estimation means for estimating the motion of a reference image with respect to the representative image;
Alignment means for arranging pixels of the reference image at sub-pixel positions of the high-resolution image according to the estimated movement;
Determination means for determining pixels satisfying the end condition from the result of the alignment processing;
An image processing apparatus comprising: the exclusion unit that excludes a pixel determined to satisfy the termination condition from the alignment process. The determination means estimates a motion for each block of the representative image,
The alignment unit divides the high-resolution image into a plurality of grids corresponding to the pixels of the high-resolution image, and arranges the reference image pixels at the sub-pixel positions of the high-resolution image for each block according to the estimated motion. Align and
The determination unit determines that the reference image satisfies the end condition when a ratio of a grid including one or more pixels of the reference image to a total grid in the high-resolution image exceeds a predetermined value. ,
The image processing apparatus according to claim 4, wherein when it is determined that the termination condition is satisfied, the exclusion unit excludes the remaining reference image pixels from the alignment process. The determination means estimates a motion for each block of the representative image,
The alignment unit divides the high-resolution image into a plurality of grids corresponding to the pixels of the high-resolution image, and arranges the reference image pixels at the sub-pixel positions of the high-resolution image for each block according to the estimated motion. Align and
The determination unit determines that the block satisfies the end condition when a ratio of a grid including one or more pixels of the reference image with respect to all grids in the block exceeds a first predetermined value;
The image processing apparatus according to claim 4, wherein the excluding unit excludes a pixel of a block that is determined to satisfy the end condition from the alignment process in a subsequent next iteration. The determination means further determines the remaining reference image pixels when the ratio of the number of blocks determined to satisfy the end condition to the total number of blocks in the high-resolution image exceeds a second predetermined value. The image processing apparatus according to claim 6, wherein the image processing apparatus is excluded from the alignment processing. The image processing apparatus according to claim 1, wherein the motion estimation unit estimates motion in order from a reference image whose display order is close to that of the representative image. The image processing apparatus further includes:
A determination means for determining whether a reference image belongs to the same scene as the representative image or a different scene;
The image processing method according to claim 1, wherein the motion estimation unit excludes reference images belonging to different scenes from motion estimation. An image processing apparatus that generates a high-resolution image having a second resolution higher than the first resolution by using a representative image having the first resolution and a plurality of reference images having the first resolution related thereto,
First iterative processing means for repeating the alignment process while switching the reference image;
A second iterative processing unit that repeats an update process for updating a pixel estimation value of a high-resolution image to be obtained after the completion of the alignment process;
The alignment process estimates a positional deviation amount of the reference image with respect to the representative image, and arranges pixels of the reference image at subpixel positions of the high-resolution image obtained by enlarging the representative image based on the estimated positional deviation amount. Is to do
The update processing is performed so that a value of an evaluation function including a difference value between a pixel value of the aligned low-resolution image and a pixel value of the high-resolution image model to be obtained as a variable is reduced. Update the pixel values,
The second iterative processing means includes
Updating means for updating the pixel estimated value so as to decrease the value of the evaluation function;
Determination means for determining a pixel that satisfies an end condition from a result of the update process;
An image processing apparatus comprising: the excluding unit that excludes a pixel determined to satisfy the end condition from a next update process. The said determination means determines that the said pixel satisfy | fills the said completion | finish conditions, when the update amount which is the difference of the pixel value before update and the pixel value after update is a threshold value. Image processing apparatus. The image processing apparatus according to claim 11, wherein the threshold value is a predetermined value. 12. The image according to claim 11, wherein the threshold is determined so that a ratio of the number of pixels not excluded by the determination unit to a total number of pixels of the high-resolution image is equal to or less than a predetermined value. Processing equipment. The update unit calculates a gradient value of the evaluation function using only pixel values of pixels not excluded by the determination unit, and updates a pixel estimated value of the high-resolution image based on the gradient value. The image processing apparatus according to claim 10. The updating means ends updating of the pixel estimated values for all pixels when the reduction ratio of the sum of the difference values for the pixels not excluded by the determining means is equal to or less than a certain value. The image processing apparatus according to claim 10. An image processing method for generating a high-resolution image of a second resolution higher than the first resolution using a representative image of the first resolution and a plurality of reference images of the first resolution related thereto,
A first step of repeating the alignment process while switching the reference image;
A second step of repeating an updating process for updating a pixel estimation value of a high-resolution image to be obtained after repeating the alignment process;
The alignment process estimates a positional deviation amount of the reference image with respect to the representative image, and arranges pixels of the reference image at subpixel positions of the high-resolution image obtained by enlarging the representative image based on the estimated positional deviation amount. Is to do
The update processing is performed such that the value of the evaluation function including a difference value between the pixel value of the aligned low-resolution image and the pixel value of the high-resolution image model to be obtained as a variable is reduced. Is to update the value,
At least one of the first step and the second step is:
A first sub-step for determining a pixel satisfying an end condition from a result of the alignment process or the update process;
And a second sub-step of excluding a pixel determined to satisfy an end condition from the alignment process or the update process.   A computer-readable program for executing the image processing method according to claim 16. A semiconductor integrated circuit that generates a high-resolution image having a second resolution higher than the first resolution by using a representative image having the first resolution and a plurality of reference images having the first resolution related thereto,
First iterative processing means for repeating the alignment process while switching the reference image;
A second iterative processing unit that repeats an update process for updating a pixel estimation value of a high-resolution image to be obtained after the completion of the alignment process;
The alignment process estimates a positional deviation amount of the reference image with respect to the representative image, and arranges pixels of the reference image at subpixel positions of the high-resolution image obtained by enlarging the representative image based on the estimated positional deviation amount. Is to do
The update processing is performed so that a pixel value of the high-resolution image is reduced so that a value of an evaluation function including a difference value between a pixel value of the aligned low-resolution image and a pixel value of the high-resolution image to be obtained as a variable is reduced. Is to update the value,
At least one of the first iteration processing means and the second iteration processing means is:
Determination means for determining a pixel that satisfies the end condition from the result of the alignment process or the update process;
A semiconductor integrated circuit comprising: exclusion means for excluding a pixel determined to satisfy an end condition from the alignment process or the update process.

Images