JP2006350562A - Image processor and image processing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor and an image processing program for preventing the blur of an edge even on a re-configured super-resolution image, and for holding sharpness when an edge exists in an observed low resolution image. <P>SOLUTION: The reference frame of a low resolution image frame string inputted to an image data input part 1 is selected by a reference frame selection part 4, and a high resolution image for edge decision is prepared by using the image data on the reference frame by a high resolution image generation part 5. Edge pixels are detected based on the high resolution image for edge decision, and edge information is generated. The luminance value of the reconfigured image is calculated by using the image data mapped by a projection converting part 8, and the reference range of the mapping data used when calculating a luminance value is changed to a direction where the range is reduced according to the edge information. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing program using super-resolution processing for generating a high-resolution image from a plurality of low-resolution images with positional deviation.

There is a method called super-resolution processing that constructs a high-resolution image from a plurality of low-resolution image sequences obtained by photographing the same scene while shifting the position. In widely used reconstruction-type super-resolution processing, an initial high-resolution image is first estimated from the observed low-resolution image sequence, and each pixel value of the low-resolution image sequence, which is an observed image, based on the camera model. Is estimated. The high resolution image is updated so as to minimize the error between the estimated pixel value and the actual observed pixel value. The final super-resolution image is obtained by repeating the update process until convergence. In the above reconstruction super-resolution processing, a back projection (back projection) method is generally used to obtain a final super-resolution image.
JP 2004-272895 A

  In the back projection method, a back projection function (BPF) is used when an initial high-resolution image is generated or when an error between an estimated pixel value and an actual observed pixel value is back-projected onto an image reconstruction area. This back-projection function performs weighting with reference to pixels in a predetermined area in the vicinity of the composition point set in the image reconstruction area to generate a high-resolution image, and calculates the luminance value of the composition point in the image reconstruction area. As a result, the pixel values of the low-resolution image sequence are averaged, and if an edge exists, the reconstructed super-resolution image has no clarity and is blurred. It was happening.

  The present invention was devised to solve the above-described problem. When an edge portion exists in the observed low-resolution image, the edge blur is prevented even on the reconstructed super-resolution image, and sharpness is achieved. An object of the present invention is to provide an image processing apparatus and an image processing program that can maintain the image quality.

  In order to achieve the above object, according to the first aspect of the present invention, an initial high-resolution image is estimated by back-projecting a low-resolution image frame sequence obtained by photographing the same scene onto an image reconstruction area. In the image processing apparatus for reconstructing an ultra-high resolution image by back projecting an error between a simulation low resolution image frame sequence estimated from a resolution image and the low resolution image frame sequence onto an image reconstruction area, the low resolution image frame A reference frame selection unit that selects an image serving as a reference of a column as a reference frame, and a high resolution image generation unit that generates a high resolution image for edge determination using image data on the reference frame selected by the reference frame selection unit And the edge determination target pixel in the edge determination high-resolution image generated by the high-resolution image generation unit and the brightness of pixels near the pixel. An edge detection unit that determines whether or not an edge determination target pixel is an edge pixel based on a degree of change in value and generates edge information; image data of a plurality of frames of the low-resolution image frame sequence; and the simulation low-resolution image frame A projection conversion unit that maps an error between a sequence and a low-resolution image frame sequence on a reference frame coordinate to form the image reconstruction area; and a luminance value of a component point in the reconstructed image set in the image reconstruction area Is generated using the image data mapped by the projective transformation unit, the luminance value of the component point is calculated using the luminance value of the mapped pixel existing in the reference region set around the component point. An image reconstructing unit to be generated, and a reconstructed image existing around the pixels mapped by the projective transformation unit with reference to the result of the edge detecting unit when reconstructing the image by the image reconstructing unit An image comprising a reconstructed reference area setting unit for changing a size of a reference area centered on the constituent point for a constituent point having edge information in the constituent point group It is a processing device.

  Further, the invention according to claim 2 is characterized in that the reconstructed reference area setting unit reduces the size of the reference area in the vertical direction when the edge information is a horizontal edge. The image processing apparatus according to claim 1.

  The invention according to claim 3 is characterized in that the reconstruction reference area setting unit reduces the horizontal size of the reference area when the edge information is an edge in the vertical direction. The image processing apparatus according to claim 1.

  According to the fourth aspect of the present invention, an initial high-resolution image is estimated by back projecting a low-resolution image frame sequence obtained by photographing the same scene onto an image reconstruction area, and is estimated from the initial high-resolution image. In an image processing program for reconstructing an ultra-high resolution image by back-projecting an error between a simulation low-resolution image frame sequence and the low-resolution image frame sequence onto an image reconstruction area, a computer is used as a reference for the low-resolution image frame sequence A reference frame selection unit that selects an image to be a reference frame, a high resolution image generation unit that generates a high resolution image for edge determination using image data on the reference frame selected by the reference frame selection unit, and Edge-determined pixel in the high-resolution image for edge determination generated by the high-resolution image generating means and pixels near this pixel Edge detection means for generating edge information by determining whether or not an edge determination target pixel is an edge pixel based on the degree of change in luminance value, image data of a plurality of frames of the low resolution image frame sequence, and the simulation low resolution Projection transformation means for mapping the error between the image frame sequence and the low-resolution image frame sequence on the reference frame coordinates to form the image reconstruction region, and the component points of the reconstructed image set in the image reconstruction region When the luminance value is generated using the image data mapped by the projective transformation means, the luminance value of the component point is calculated using the luminance value of the mapped pixel existing in the reference area set around the component point. An image reconstructing unit that generates a value, and the image reconstructing unit refers to the result of the edge detecting unit when reconstructing the image, and exists around the pixel mapped by the projective transforming unit. For the component points having edge information in the component point group of the reconstructed image to function as reconstruction reference region setting means for changing the size of the reference region around the component point to be reduced An image processing program.

  According to the present invention, when the luminance value of the constituent point of the reconstructed image set in the image reconstruction area is obtained in the process of reconstructing the super-resolution image, the low value used for calculating the luminance value of the constituent point is used. The pixel area of the resolution image is set as the reference area, but when the constituent points of the reconstructed image existing around the pixel of the low resolution image mapped on the standard frame coordinates have edge information, that is, the edge If the mapped pixel is included in the reference area of the constituent point, the reference area set around the constituent point is reduced. When calculating the luminance value of the constituent point, the contribution of the luminance value of the pixel of the low-resolution image can be reduced, and the sharpness of the edge can be preserved.

  If the edge direction of the edge information is a vertical edge, the horizontal length of the reference area is reduced. If the edge direction of the edge information is a horizontal edge, the vertical length of the reference area is reduced. By doing so, it is possible to perform appropriate processing according to the direction of the edge.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows the configuration of an image processing apparatus according to the present invention.

  The image processing apparatus includes an image data input unit 1, a storage unit 2, a processing condition setting unit 3, a reference frame selection unit 4, a high resolution image generation unit 5, an edge detection unit 6, a reconstruction reference region setting unit 7, and a projective conversion unit. 8. The image reconstruction unit 9 and the like are configured. The observed image sequence, the captured image frame sequence, and the like are input to the image data input unit 1 and subjected to predetermined image processing. Image data is output from the unit 9. Although not shown, each component of the image processing apparatus is controlled by a control unit such as a CPU or controlled by a control unit provided in each component. 1 can also be operated as a program, and in this case, is controlled by a computer.

  First, an image sequence (observation image sequence) with slight differences such as multiple frames of images of the same object (same scene) taken at different angles, distances, time, etc., or a series of sequences in video images To get. These image sequences are input to the image data input unit 1 as low-resolution image frame sequences. The input low resolution image frame sequence is stored in the storage unit 2.

  Next, the number N of low-resolution images used for the super-resolution processing is determined, and the reference frame is determined. In this embodiment, the frame (N / 2) located in the middle of the observed image sequence is set as the reference frame, but the image positioned at the beginning of the low resolution image sequence may be used as the reference frame. Either may be used as a reference frame.

  Further, the enlargement ratio r from the low resolution image to the high resolution image is determined. These data are input to the processing condition setting unit 3. The processing condition setting unit 3 transmits the number of frames N and the enlargement ratio r to the storage unit 2 and the reference frame number to the storage unit 2 and the reference frame selection unit 4. The storage unit 2 outputs N frames of the input low-resolution image frame sequence to the projective conversion unit 8. The storage unit 2 also outputs the enlargement ratio r, the reference frame number, and the like to the projective conversion unit 8.

  The upper part of FIG. 2 shows an example of N selected low-resolution images. As described above, the frame (N / 2) located in the middle of the low-resolution image sequence among the series of low-resolution images is used as the reference frame.

  The selected N low-resolution images and the enlargement ratio r are sent from the storage unit 2 to the reference frame selection unit 4. The reference frame selection unit 4 selects an image of a reference frame (low resolution image N / 2) from the N pieces of input low resolution images and outputs the selected image to the high resolution image generation unit 5.

  The high resolution image generation unit 5 creates a high resolution image for edge determination using only image data on the reference frame. First, the image of the reference frame is enlarged by the enlargement ratio r, and the pixels are interpolated by interpolation processing. FIG. 3 shows an example of creating a high resolution image for edge determination by interpolation processing. Black circles displayed in FIG. 3 are pixels on the reference frame, and white circles indicate pixels (interpolated pixels) generated by interpolation. Further, the enlargement ratio r = 2. As an interpolation method, linear interpolation in which interpolation is performed with a pixel value obtained by a ratio of a distance between an interpolation pixel and a pixel on a reference frame existing around the interpolation pixel may be used. In addition, bicubic interpolation, Sinc You may make it use the interpolation by a function.

  The edge determination high-resolution image generated as described above is output from the high-resolution image generation unit 5 to the edge detection unit 6 to detect an edge region. The edge detection unit 6 uses a predetermined edge detection method to perform edge detection for each pixel of the edge determination high-resolution image.

  For example, FIG. 4 shows a method for detecting an edge by performing template matching using a Prewitt filter. Of the nine pixels (black circles) shown in FIG. 4A, the center pixel is the edge determination target pixel. FIG. 4B shows a Prewitt filter for edge detection in the horizontal direction, and FIG. 4C shows a Prewitt filter for edge detection in the vertical direction, and the positions of nine numerical values displayed on each filter. Corresponds to the positions of the nine pixels in FIG.

  The luminance values (pixel values) of the nine pixels in FIG. 4A are multiplied by the numerical values of the filter in FIG. 4B corresponding to the pixel positions, and the sum of the nine calculated values is calculated. Also, the luminance values (pixel values) of the nine pixels in FIG. 4A are multiplied by the numerical values of the filter in FIG. 4C corresponding to each pixel position, and the sum of the nine calculated values is calculated. To do. Then, when the absolute value of the sum exceeds a predetermined threshold value L, the edge is determined. For example, when the value obtained by applying the filter of FIG. 4B exceeds the threshold value L, the edge determination target pixel is determined to be a horizontal edge, and the position and direction of the edge are recorded. On the other hand, when the value obtained by applying the filter of FIG. 4C exceeds the threshold value L, the edge determination target pixel is determined to be an edge in the vertical direction, and the position and direction of the edge are recorded.

  As an edge determination method, instead of using an edge detection filter as described above, wavelet transform may be used. The following is known about the wavelet transform.

  Each value of the multi-scale luminance gradient plane M (f (x, y)) has a certain K> 0 when the scale parameter j is sufficiently small, and is expressed by the following equation.

M _j (f (x, y)) = K × 2 ^jα
Each value of the horizontal wavelet transform W ¹ (f (x, y)) has a certain K ₁ > 0 when the scale parameter j is sufficiently small, and is as follows.

W ¹ _j (f (x, y)) = K ₁ × 2 ^jα
Each value of the wavelet transform W ² (f (x, y)) in the vertical direction has a certain K ₂ > 0 when the scale parameter j is sufficiently small, and is as follows.

W ² _j (f (x, y)) = K ₂ × 2 ^jα
Here, α is called a Lipschitz exponent, and f can be continuously differentiated a number of times equal to the maximum integer not exceeding α. If the Lipschitz exponent at the edge determination target pixel (x, y) is calculated, the number of continuously differentiable values can be estimated. In general, the smoother the change in luminance value, the larger the Lipschitz index. Therefore, when the Lipschitz index is less than or equal to a predetermined threshold value, that is, when the number of consecutive differentials is less than or equal to a predetermined threshold value, it can be determined that the pixel constitutes an edge, and whether it is a vertical edge or a horizontal edge It can be carried out.

  Such detection processing is performed for each pixel of the high-resolution image for edge determination shown in the example of FIG. 3, edge determination is performed for all the pixels, and edge detection and edge direction are stored to detect the edge. The process ends.

On the other hand, the N low resolution images selected by the storage unit 2 are input to the projective conversion unit 8, and the pixels on each low resolution image frame are mapped onto the reference frame. Consider mapping (projecting) the pixels in the low-resolution image frame sequence onto the coordinates of the reference frame and consolidating all the pixels on the reference frame. It is known that the following relationship holds between the coordinates (x, y) on the reference frame and the coordinates (x _k , y _k ) on the kth frame.

x = (cx _k + dy _k + e) / (ax _k + by _k +1)
y = (fx _k + gy _k + h) / (ax _k + by _k +1)
These equations are called projective transformations and are represented by eight parameters a to h. Parameters a to h represent the feature point sequence on the reference frame for each low resolution using the method of Tomsai et al. (Carlo Tomasi and Takeo Kanade, Detection and Tracking of Point Features, Carnegie Mellon University Technical Report CMU-CS, April 1991). It is obtained by tracking on the image and obtaining 8 or more correspondences. By obtaining eight parameters, a projective transformation formula is determined. All the pixels in each low-resolution image frame sequence are mapped onto the reference frame according to the determined projective transformation formula.

  Next, the image data mapped onto the reference frame and the image data existing on the reference frame from the beginning are enlarged using the enlargement magnification r input to the processing condition setting unit 3. Assuming that the size of the low resolution image frame is width w and height t, the width W and height T of the reconstructed image (super resolution image) are W = r × w and H = r × t. It becomes the same size as the edge determination high resolution image generated by the generation unit 5.

  FIG. 2 shows a state in which the image data of the low-resolution image frame sequence is mapped onto the reference frame and enlarged. The enlargement ratio r is a double enlargement ratio. White circles are pixels on the reference frame and are located on the intersections of the grid. Circles with diagonal lines indicate pixels on the low-resolution image frame other than the pixels on the reference frame. In the figure, only the mapping of the pixels of the low resolution image (N / 2) +1 and the low resolution image N is illustrated, but the low resolution image 1 to the low resolution image excluding the reference frame (low resolution image N / 2). All pixels contained up to N are mapped onto the reference frame. The white triangles indicate the constituent points of the reconstructed image in the image reconstruction area, and the luminance values (pixel values) of the constituent points are the image data on the reference frame coordinates, that is, the low resolution image 1 to the low resolution including the reference frame. Calculated from the pixels of the image N.

  The image data enlarged by the projection conversion unit 8 is sent to the image reconstruction unit 9. The image reconstruction unit 9 restores a super-resolution image by super-resolution processing. FIG. 6 shows a flow of super-resolution processing. In order to create a super-resolution image using pixels of a plurality of frames of a low-resolution image, a back projection method (back projection method) which is one of basic super-resolution methods is used.

  First, the number of iterations is initialized, the minimum value of the sum of errors is initialized (S1). Next, a back projection function (BPF) and a point spread function (PSF) are determined (S2). The backprojection function reconstructs the error between the observed low-resolution image sequence and the low-resolution image sequence obtained by simulation when generating the initial reconstructed image (high-resolution image) from the observed low-resolution image sequence. Used when back projecting onto an image. The point spread function is called a camera model. The characteristics of a complicated camera (imaging device), such as the MTF (Modulation Transfer Function) value of the camera, the behavior of a low-pass filter of a CCD, pixel density, AD conversion, etc. This is an abstraction of the electrical action and the like, and is used when generating a simulation low-resolution image frame sequence estimated from the reconstructed image.

  For example, a Mitchell filter having excellent high-frequency storage characteristics can be selected as the BPF, and a Gaussian filter can be selected as the PSF. Examples of these filters are shown in FIG. The Gaussian filter is displayed with a solid line, and the Mitchell filter is displayed with a broken line. In addition, the horizontal axis of FIG. 11 indicates the distance (pixel) from the filter center, and the vertical axis indicates the value returned by each filter. Looking at the range where each filter can return a value, that is, the range where the graph is drawn, the Gaussian filter is drawn from -1.25 pixels to 1.25 pixels, so the length is 2.5 pixels. On the other hand, in the Mitchell filter, the length is −2 pixels to 2 pixels, and the length is 4 pixels. The range in which each filter can return a value is called the size of the table, and the size of the table is S = 4 for the Mitchell filter and R = 2.5 for the Gaussian filter as described above.

  FIG. 12 and FIG. 13 show the concept of the two-dimensional stand of BPF and PSF. In FIG. 12, the circular range indicated as the back projection kernel base is a range where the constituent points (white circles) of the reconstructed image are affected, and the pixels of the low-resolution image existing within the range. A weight related to the constituent points of the reconstructed image is calculated from the circles with diagonal lines. The pixels of the low-resolution image existing outside the range of the table are not affected. The pixels of the low-resolution image that are referenced around the constituent point of the reconstructed image existing at the center of the circular platform are two-dimensionally distributed.

  A different value is returned from the BPF according to the distance between the pixel of the low-resolution image existing in the range of the table and the constituent point (white circle) of the reconstructed image, and this returned value becomes the weighting value. The product of the weight value and the pixel value of the low-resolution image used to calculate the weight contributes to the calculation of the luminance of the constituent points. As described above, the range of the platform is an area that refers to the pixel of the low-resolution image used for calculating the luminance value around the component point when calculating the luminance value of the component point.

  In addition, a range (reference region) represented as a point spread function platform shown in FIG. 13 is a range in which pixels of a low-resolution image (circles with diagonal lines) are affected, contrary to BPF. Then, the weight related to the pixel of the low-resolution image is calculated from the pixel (white circle) on the reconstructed image.

  Regarding the setting of the BPF platform, a circular platform may be set as in the example of FIG. 12, but in this embodiment, the length S1 = S in the X direction and the length S2 = S in the Y direction are set. A square base (reference area) is set.

  Regarding the setting of the BPF and PSF, for example, if the processing condition setting unit 3 can select the PSF and BPF functions as described above, the PSF and BPF can be appropriately changed to other functions. it can. In this way, PSF and BPF are set (S2). Next, the sum of the BPF weights for all the constituent points X of the reconstructed image is calculated (S3). FIG. 7 shows a flow of processing (S3) for calculating the sum of weights. First, the sum of weights is initialized (S31), and the enlarged map image data is acquired from the projective transformation unit 8 (S32).

On the other hand, the edge detection data in which the edge position and the edge direction are recorded is sent from the edge detection unit 6 to the reconstruction reference region setting unit 7, and the reference region is changed based on the edge detection data (S33). FIG. 10 shows the procedure for setting the reconstruction reference area (bases S1 and S2). As described above, since the initial value of the BPF reference area is set to a square base, it is initialized to that value (S21). If the coordinates of the point y on the low-resolution image in the reconstructed image area are Z _y , it is checked whether or not there are edges at the four constituent points surrounding the vicinity of Z _y (I, J) (S22).

  For this determination, edge region data from the edge detector 6 is used. Since the positions and sizes of the pixels of the edge determination high-resolution image created by the high-resolution image generation unit 5 and the constituent points of the reconstructed image are completely the same, it is determined as an edge in the edge determination high-resolution image. It is determined that an edge exists for the configuration point corresponding to the pixel. If there is an edge in S22, it is determined whether it is a vertical edge (S23) or a horizontal edge (S25).

If the edge is a vertical edge (S23 YES), the horizontal length of the table centering on the component point determined to be present among the four component points is reduced to half S / 2 (S24). If the edge is in the horizontal direction (S25 YES), the vertical length of the table is set to half S / 2 (S26). In the case of both the vertical edge and the horizontal edge, both the horizontal base length and the vertical base length are set to half S / 2. On the other hand, if there are no edges at the four composing points surrounding the vicinity of Z _y (I, J), the process is terminated without changing the size of the table (NO in S22).

FIGS. 14 and 15 specifically show the processing state of FIG. FIG. 14 shows the change of the reference area when an edge exists in the vertical direction, and FIG. 15 shows the change of the reference area when an edge exists in the horizontal direction. The black circles A, B, C, and D displayed in FIG. 14 indicate the constituent points of the reconstructed image, and the edge determination high-resolution image created by the high-resolution image generation unit 5 is the reconstructed image. The state of being superimposed on the constituent points is shown. The constituent points A, B, C, and D are constituent points that form a minimum area that encloses a point Z _y (I, J) obtained by mapping and enlarging the point y on the low-resolution image in the reconstructed image area. It is. After extracting the component points having such a relationship, it is confirmed based on the edge detection data from the edge detection unit 6 whether or not an edge is detected for each component point position.

From the high resolution image data for edge determination, if the luminance of each of the pixels A, B, C, and D is, for example, I _A = 100, I _B = 50, I _C = 100, and I _D = 50, The edge determination unit 6 determines that there is an edge in the (Y direction). Based on the edge detection data, the component points having the edge information are changed and set again in the direction of reducing the size of the reference area based on the edge direction.

For example, considering the configuration point B, since the configuration point B has edge information in the vertical direction, the base (reference region) of the BPF centering on the configuration point B is an S × S of the initial setting. The square area is changed to a rectangular area of (S / 2) × S. Although not shown, since all of the configuration points A, C, and D have edge information in the vertical direction, the horizontal length of the reference area is S / 2. In this way, at least the structural point B, since the point in each reference area D Z _{y (I,} J) from entering, constituting point B, the point when generating the luminance value of D Z _{y (I} , J) does not contribute, and the sharpness of the edge is preserved.

Similarly in FIG. 15, the black circles of A, B, C, and D indicate the constituent points of the reconstructed image, and the edge determination high-resolution image created by the high-resolution image generation unit 5 is used as the reconstructed image. The state of being superimposed on the constituent points is shown. Further, the configuration points A, B, C, and D are configuration points that form a minimum area surrounding the point Z _y (I, J) of the low-resolution image on the reconstructed image area in a grid pattern. If the brightness of each of the pixels A, B, C, and D is, for example, I _A = 100, I _B = 100, I _C = 50, and I _D = 50 from the edge determination high-resolution image data, The edge determination unit 6 determines that there is an edge in the (X direction).

  Based on the edge detection data, the component points having the edge information are changed and set again in the direction of reducing the size of the reference area based on the edge direction. Considering the configuration point B, since the configuration point B has edge information in the horizontal direction, the base of the BPF (reference region) centering on the configuration point B is an initially set S × S square region. To S × (S / 2) rectangular area.

Although not shown, since all of the configuration points A, C, and D have edge information in the horizontal direction, the length of the reference area in the vertical direction is S / 2. By this way, control points A, Point Z _{y (I,} J) to each reference area B becomes the the entering, while constituting point C, the point in each reference area D Z _y The luminance value at the point Z _y (I, J) does not contribute to the generation of the luminance value without (I, J) being entered, and the sharpness of the edge is preserved.

14 and 15, if the value of the size S of the default reference area is small and the point Z _y (I, J) does not enter the reference area of the constituent point B from the beginning, the configuration Even if the point B has edge information, it is not necessary to change the size of the initial reference area, and the initial setting is used. This applies for all configuration points.

In the above embodiment, when the constituent point near the point Z _y (I, J) has edge information, the size of the reference area is changed from the initial setting to half. Accordingly, the image may be reduced by another ratio instead of the half reduction ratio.

By the way, assuming that BPF is H _B , the sum of the BPF weights is S (X), and the coordinates of the point y on the low resolution image in the reconstructed image area are Z _y , S (X) = ΣH _B (X− Z _y ). For example, the point y = (i, j) on the low-resolution image is first converted into coordinates (i ′, j ′) on the reference frame using the projective transformation of the projective transformation unit 5. Thereafter, (i ′, j ′) is multiplied by r and moved to the point Z _y on the reconstructed image area.

When the constituent points of the reconstructed image forming the minimum area surrounding the point _Zy in a grid form have edge information, the size of the reference area is changed as described above, and the edge information is not included. remains the reference region of the initial setting, and calculates the weight by _{H B} with respect to configuration point X (p, q), is added to S (X) (S34). In this way, the weights for all the constituent points X on the reconstructed image are calculated using the pixel values of each low-resolution image frame and added to S (X) (S35).

Next, an initial high resolution image is calculated (S4). FIG. 8 shows the flow of the initial high-resolution image generation process. After the reconstructed image is initialized (S41), the enlarged map image data is acquired from the projective transformation unit 8 (S42). The luminance value (pixel value) F (X) of the point X on the initial high resolution image is F (y), where f (y) is the luminance value of the point y on the low resolution image and c is a constant for normalization. X) = Σ (f (y) × (H _B (X−Z _y )) ² / cS (X)). An initial high resolution image is generated by this arithmetic expression (S44). As described above, H _B (X−Z _y ) is used in the generation process, but the reference area (table) of this back projection function is changed according to the flow of FIG. 10 (S43).

10, 14, and 15 have already been described and will be omitted. However, when the constituent points of the reconstructed image forming the minimum area surrounding the point Z _y in a grid form have edge information, the size of the reference region change, remaining reference region of the default when no edge information, constituting point X (p, q) to calculate the weight by H _B respect, those obtained by squaring it with luminance values f ( The luminance value contributing to the component point X is calculated by multiplying by y) and normalizing by cS (X). When the luminance values contributing to the constituent point X are calculated for all the points Z _y existing in the reference region centered on the constituent point X, the sum of these values becomes the luminance value of the constituent point X. When the luminance values are calculated for all the constituent points of the reconstructed image, the process in FIG. 8 ends (S45).

Next, referring to FIG. 6 again, a low-resolution image frame sequence is estimated from the high-resolution image generated in S4 (S5). The simulation low resolution image frame sequence is calculated by the following equation. The PSF _{H P,} when the number n of the iterative operation, n + 1-th k-th low-resolution frame _f ^k during iterative operation of ^{(n) _{is, f k (n) (y}} ) = ΣF (n) (X) H _P (X- _Zy ).

  FIG. 5 shows a conceptual diagram of the processing of S4 and S5 (image reconstruction). As shown in FIGS. 5A and 5B, all the low-resolution image frames are mapped onto the reference frame and enlarged, and the constituent points constituting the reconstructed image are interpolated using the back projection method. Go. As shown in FIG. 2, some of the constituent points may overlap with pixels on the reference frame. In this case, the constituent points of the reconstructed image are not interpolated, but on the reference frame. It can be said that the correction processing of the luminance value of the pixel has been performed. On the other hand, as shown in FIG. 5B, after one initial high-resolution image is constructed, the high-resolution image is down-sampled using a point spread function, and a simulation as shown in FIG. 5C is performed. Low resolution images 1 to N are obtained.

The difference (error) between the low-resolution image frame sequence obtained by this down-sampling and the observed (captured) low-resolution image frame sequence is taken, weighted by the back projection function H _B , and displayed on the reconstructed image. Is calculated (S6). FIG. 9 shows a flow of error calculation (S6). After the error sum e (X) is initialized (S61), the difference between the luminance values of the corresponding pixels in each frame of the observed low resolution images 1 to N and the simulation low resolution images 1 to N is obtained, and these data are obtained. It transmits to the projective conversion unit 8. The projection conversion unit 8 maps the transmitted error data onto the reference frame using the projection conversion formula (S62), and expands the mapping data to r times (S63). The enlarged error data is sent to the image reconstruction unit 9 and a reference area is set (S64).

The error data subjected to the mapping expansion is aggregated on the reconstructed image area, and the error e (X) at the constituent point X on the reconstructed image is
e (X) = Σ ((f _k (y) −f _k ⁽ⁿ⁾ (y)) × (H _B (X−Z _y )) ² / cS (X))
It becomes. The portions (f _k (y) −f _k ⁽ⁿ⁾ (y)) will be described with reference to FIG. 5. The low resolution images 1 to N in FIG. 5A and the simulation low resolution image 1 in FIG. This corresponds to calculating an error between corresponding pixels of each corresponding frame of .about.N.

Further, as described above, H _B (X−Z _y ) is used in the process of calculating the error e (X), and the setting (S64) of the reference region (table) of this back projection function is as described above. As shown in FIGS. If the constituent points of the reconstructed image forming the minimum area surrounding the point Z _y in a grid form have edge information, the size of the reference area is changed, and if there is no edge information, the initial setting is changed. It remains the reference region, to calculate the weight by _{H B} with respect to configuration point X, multiplies it with those obtained by squaring the error ^{_{(f k (y) -f k}} (n) (y)), cS (X) The luminance value of the error contributing to the component point X is calculated by normalizing by (S65).

When the luminance values of errors that contribute to the constituent point X are calculated for all points Z _y existing in the reference region centered on the constituent point X, the sum of these values becomes the error of the constituent point X. When errors have been calculated for all the constituent points of the reconstructed image, the processing in FIG. 9 ends (S66).

  Referring to FIG. 6 again, the total error E = Σe (X) at all the constituent points on the reconstructed image is obtained (S7), which is once treated as the minimum error value Emin, and the error is displayed on the reconstructed image. Back projection is performed (S10). Back projection of the error onto the reconstructed image is performed by adding the error e (X) to the previous reconstructed image. The back projection of the error onto the reconstructed image is performed by adding the error e (X) to the initial high resolution image when the number of iterations n = 1.

  Next, a simulation low-resolution image sequence is generated again from the reconstructed image to which the error has been added (S5), and, similarly to the above, taking the difference (error) from the observed low-resolution image sequence, (S6), and the total error E on the reconstructed image is calculated (S7). When the total error E is smaller than Emin (S8 YES), the total error E at this time is set to Emin (S9). Then, the error is back-projected onto the reconstructed image (S10).

Back projection of the error onto the high resolution image is performed by adding the error e (X) to the previously reconstructed high resolution image, and the error e (X) is back onto the high resolution image. The nth iteration to project is
F ⁽ⁿ⁾ (X) = F ^(n-1) (X) + e (X).

S5 to S10 are repeated until the total error E becomes minimum. When the error sum E is minimized, the processing is terminated with F ⁽ⁿ⁻¹⁾ as the super-resolution image.

  In the above-described embodiment, generation of one super-resolution image from a plurality of low-resolution image frames has been described. However, when generating a plurality of super-resolution images from a plurality of low-resolution image frames. Can also be applied and edge blurring can be prevented.

In the above-described embodiment, the size of the reference area is reduced by the reconstruction reference area setting unit. However, the back projection function that defines the reference area may be switched. For example, a low-order polynomial may be set as a back projection function at a constituent point of a reconstructed image having edge information, and a high-order polynomial may be set at a constituent point not having edge information.

It is a figure which shows the structure of the image processing apparatus of this invention. It is a figure which shows a mode that each pixel of the low resolution image frame was mapped on the reference | standard frame. It is a figure which shows the state which produces the high-resolution image for edge determination by an interpolation process. It is a figure which shows the structure of the filter for edge detection. It is a figure which shows the concept of an image reconstruction. It is a flowchart figure which shows a super-resolution process. It is a flowchart figure which shows calculation of the sum total of the weight of BPF. It is a flowchart figure which shows calculation of an initial high-resolution image. It is a flowchart figure which shows calculation of the error of an observation low resolution image sequence and a simulation low resolution image sequence. It is a flowchart figure which shows the concept of showing a reconstruction reference area setting. It is a figure which shows an example of BPF and PSF. It is a figure which shows the concept of the stand of a back projection kernel. It is a figure which shows the concept of the stand of a point spread function. It is a figure which shows the change of a reference area | region when there exists an edge in a perpendicular direction. It is a figure which shows the change of a reference area when there exists an edge in a horizontal direction.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Image data input part 2 Storage part 3 Processing condition setting part 4 Reference frame selection part 5 High resolution image generation part 6 Edge detection part 7 Reconstruction reference area setting part 8 Projection conversion part 9 Image reconstruction part

Claims (4)

A low-resolution image frame sequence obtained by photographing the same scene is back-projected onto an image reconstruction area to estimate an initial high-resolution image, and a simulation low-resolution image frame sequence estimated from the initial high-resolution image and the low-resolution image In an image processing apparatus for reconstructing an ultra-high resolution image by back projecting an error from an image frame sequence to an image reconstruction area,
A reference frame selection unit that selects a reference image of the low-resolution image frame sequence as a reference frame;
A high-resolution image generation unit that generates a high-resolution image for edge determination using image data on the reference frame selected by the reference frame selection unit;
It is determined whether or not the edge determination target pixel is an edge pixel based on the edge determination target pixel in the edge determination high resolution image generated by the high resolution image generation unit and the degree of change in the luminance value in the neighboring pixels of the pixel. An edge detector for generating edge information relating to edge pixels;
A projective transformation unit that maps image data of a plurality of frames of the low-resolution image frame sequence and an error between the simulation low-resolution image frame sequence and the low-resolution image frame sequence onto reference frame coordinates to form the image reconstruction area; ,
When generating the luminance value of the component point in the reconstructed image set in the image reconstruction region using the image data mapped by the projective transformation unit, it exists in the reference region set around the component point An image reconstruction unit that generates a luminance value of the component point using the luminance value of the mapped pixel to be
When reconstructing an image with the image reconstruction unit, reference is made to the result of the edge detection unit, and edge information is included in the constituent points of the reconstructed image existing around the pixels mapped by the projective transformation unit. An image processing apparatus comprising: a reconfiguration reference area setting unit that changes a configuration point so that a size of a reference area centering on the configuration point is reduced.   The image processing apparatus according to claim 1, wherein the reconstruction reference area setting unit reduces the size of the reference area in the vertical direction when the edge information is a horizontal edge.   The image processing apparatus according to claim 1, wherein the reconstruction reference area setting unit reduces the size of the reference area in the horizontal direction when the edge information is an edge in the vertical direction. A low-resolution image frame sequence obtained by photographing the same scene is back-projected onto an image reconstruction area to estimate an initial high-resolution image, and a simulation low-resolution image frame sequence estimated from the initial high-resolution image and the low-resolution image In an image processing program for reconstructing an ultra-high resolution image by back projecting an error from an image frame sequence to an image reconstruction area,
A reference frame selecting means for selecting, as a reference frame, an image serving as a reference for the low-resolution image frame sequence;
High-resolution image generation means for generating a high-resolution image for edge determination using image data on the reference frame selected by the reference frame selection means;
It is determined whether or not the edge determination target pixel is an edge pixel based on the edge determination target pixel in the edge determination high resolution image generated by the high resolution image generation means and the degree of change in the luminance value in the neighboring pixels of the pixel. Edge detection means for generating edge information;
Projection conversion means for mapping the image data of a plurality of frames of the low resolution image frame sequence and an error between the simulation low resolution image frame sequence and the low resolution image frame sequence onto reference frame coordinates to form the image reconstruction area; ,
When the luminance value of the constituent point of the reconstructed image set in the image reconstruction area is generated using the image data mapped by the projective transformation means, it exists in the reference area set around the constituent point Image reconstructing means for generating the luminance value of the constituent point using the luminance value of the mapped pixel to be
When the image reconstruction unit reconstructs an image, it refers to the result of the edge detection unit, and has edge information among the constituent points of the reconstructed image existing around the pixels mapped by the projective transformation means. An image processing program for causing a component point to function as a reconstructed reference region setting unit that changes so as to reduce the size of a reference region centered on the component point.

Images