JP2017072954A - Image conversion device and image conversion method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image conversion technique for reducing the number of calculations required for convolution integration.SOLUTION: A super-resolution processing unit 30 reads an observation image captured by an optical system from a memory, and uses convolution integration with a point spread function representing a transfer function of the optical system to perform super-resolution processing on the observation image, thereby generating a high-resolution image with higher resolution than the observation image. When performing convolution integration of the point spread function and a target image which is a high-resolution image obtained by subjecting the observation image to reconfiguration super-resolution processing, the super-resolution processing unit 30 sequentially reads pixels of the target image by each line, buffers by each line in a FIFO form a current product-sum operation value of the target image obtained by delay adding, by each line, a pixel value of the target image obtained by multiplying a coefficient value of a coefficient matrix of the point spread function, and reuses the product-sum operation value as product-sum operation value of a future target image. The convolution integration of the point spread function and the target image is thus calculated.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image conversion technique, and more particularly to a technique for converting the resolution of an image.

  When performing resolution conversion processing for enlarging a captured image, it is difficult to restore components lost due to the characteristics of the optical system or image sensor that captured the image using existing magnification methods such as the bilinear method, resulting in blurring. Enlarged image. As a method for restoring the degradation component, there is a reconstruction type super-resolution process. In the reconstruction-type super-resolution processing, the initial high-resolution image is estimated from the observed low-resolution image, and the low-resolution image that is the observed image is estimated based on the degradation model from the estimated high-resolution image. Is generated. The high resolution image is updated to minimize the error between the simulation image and the actual observed low resolution image. By repeating this updating process until convergence, a final high-resolution image is obtained. The back projection method is widely known as a classic method of the reconstruction type super-resolution processing.

JP 2012-65187 A

  In the case of the reconstruction type super-resolution processing, blurring can be removed by sharpening the image by back-projecting the error between the simulation image based on the degradation model and the actually observed low-resolution image. However, if the assumed deterioration model does not accurately reflect actual imaging optical characteristics, a sufficient effect cannot be expected. The most important element in the degradation model is a “Point Spreading Function” that represents the transfer function of the optical system. A point spread function in an image indicates how much the point spreads when a certain point light source is photographed, and can be represented by a two-dimensional coefficient matrix.

  In general, the point spread function representing the transfer function of a low-cost optical system has a large spread of the image of the point light source, and the number of elements of the coefficient matrix representing it is also large. Convolution integration corresponds to product-sum operation for the number of elements. If the coefficient matrix of the point spread function increases, the number of multipliers for convolution integration increases, resulting in a problem that the processing scale increases. Furthermore, in the case of reconstruction-type super-resolution processing, it is necessary to perform convolution integration with a backprojection function of the same size as the point spread function or larger, so this processing is also for convolution integration. It is necessary to increase the number of multipliers.

  Further, when reading out the pixel to be calculated at the time of convolution integration from the storage memory, it is necessary to calculate an address according to the pixel position, and it is necessary to design in consideration of the reading time of all the target pixels.

  The present invention has been made in view of such a situation, and an object thereof is to provide an image conversion technique capable of reducing the number of multiplications necessary for convolution integration. Another object is to provide an image conversion technique capable of reducing address calculation and memory access waiting time.

  In order to solve the above problems, an image conversion apparatus according to an aspect of the present invention includes an image input unit that stores an observation image captured by an optical system in a memory, and a point spread function that indicates a transfer function of the optical system. A super-resolution processing unit that performs super-resolution processing on the observed image using convolution integration and generates a high-resolution image having a higher resolution than the observed image. When the super-resolution processing unit convolves and integrates the point spread function and a target image, which is a high-resolution image obtained by performing reconstruction-type super-resolution processing on the observed image, pixels of the target image Are sequentially read in line units, and the product-sum operation value of the current target image obtained by delay-adding the pixel values of the target image obtained by multiplying the coefficient values of the coefficient matrix of the point spread function for each line by FIFO. The convolution integral of the point spread function and the target image is calculated by buffering in the format and using it again as the product-sum operation value of the future target image.

  Another aspect of the present invention is an image conversion method. In this method, the observation image is super-resolved using a convolution integral between an image input step of storing an observation image taken by the optical system in a memory and a point spread function indicating a transfer function of the optical system. And a super-resolution processing step of generating a high-resolution image having a higher resolution than the observed image. In the super-resolution processing step, the pixel of the target image is convolved with the point spread function and the target image that is a high-resolution image obtained by performing a reconstruction-type super-resolution processing on the observed image. Are sequentially read in line units, and the product-sum operation value of the current target image obtained by delay-adding the pixel values of the target image obtained by multiplying the coefficient values of the coefficient matrix of the point spread function for each line by FIFO. The convolution integral of the point spread function and the target image is calculated by buffering in the format and using it again as the product-sum operation value of the future target image.

  It should be noted that any combination of the above-described constituent elements and a representation of the present invention converted between a method, an apparatus, a system, a recording medium, and the like are also effective as an aspect of the present invention.

  According to the present invention, the number of multiplications necessary for convolution integration can be reduced. In addition, address calculation and memory access waiting time can be reduced.

It is a block diagram of the image conversion apparatus which concerns on this Embodiment. It is a figure which shows the structural example of the general convolution integral by the coefficient matrix of a point spread function. FIG. 3A and FIG. 3B are diagrams for explaining the coefficient values of the coefficient matrix of the rotationally symmetric point spread function. FIG. 4A to FIG. 4C are diagrams for explaining the relationship between pixel input and convolution integral by line sequential scanning. It is a block diagram of the convolution integral calculating part on the assumption of the input of the pixel by line sequential scanning.

  In describing the embodiment of the present invention, first, the algorithm of the reconstruction super-resolution processing will be briefly described, and then a method of reducing the number of multiplications of the convolution integral of the reconstruction super-resolution processing will be described. . Then, the Example using the said method is described.

  An observation image g obtained by photographing through the optical system can be modeled as shown in Expression (1).

  Here, f is a correct high-resolution image, and the purpose of super-resolution processing is to restore f from g. h is an operator caused by deterioration of the optical system, and generally means a point spread function. σ is a downsampling operator indicating sampling by the sensor. n is an additive noise term.

Using this degradation model, as shown in Expression (2), a simulation image g ^{(n) in} which an observation image is estimated from the high-resolution image f ^{(n) obtained as a} result of the n-th reconstruction iteration is generated.

Here, ↓ s is an operator indicating downsampling to 1 / s, and * indicates a convolution integral with the point spread function h. Expression (2) is a result of down-sampling the high resolution image f ^{(n) with} the blur of the optical system at h and down to the sensor resolution, and a result of simulating the imaging optical system is obtained. Therefore, when the operator used in Expression (2) completely represents the photographing optical system, Expression (3) is established if f ⁽ⁿ⁾ is a correct high-resolution image.

  Next, the n-th iterative reconstruction process is to update the high-resolution image so as to minimize the error between the simulation image and the actual observation image, and is given by the recurrence formula of Equation (4).

Here, ↑ s is an operator indicating upsampling by s times, p is a back projection function, and can be generally calculated from a point spread function. As shown in equation (4), the error (g−g (n)) between the simulation image g ⁽ⁿ⁾ and the observed image g obtained from the high-resolution image f ⁽ⁿ⁾ as the nth iteration result is upsampled. Thus, a process for convolution integration with the backprojection function p is required.

  By repeating Expression (2) and Expression (4) until convergence, a final high-resolution image f is obtained. As described above, in the reconfiguration type super-resolution processing, the convolution integration with the point spread function h is necessary in the equation (2), and the convolution integration with the back projection function p is necessary in the equation (4).

  Here, the convolution integral of the point spread function and the image signal will be described. In order to simplify the description, a one-dimensional convolution integral will be described here. If the pixel value at the position x in the one-dimensional image f is represented by f (x), the convolution integral of the image f (x) and the point spread function h (n) is given by equation (5).

  Here, the point spread function h (n) having 5 elements is expressed by Expression (6).

  The convolution integral (f * h) of the image f (x) of the equation (6) and the point spread function h is given by the equation (7).

  Here, let us consider a case where the point spread function h having 5 elements is symmetric with respect to the center as shown in equation (8).

  In the case of the point spread function h in Expression (8), the convolution integral (f * h) in Expression (7) can be modified as in Expression (9).

  Here, for the sake of simplicity, one-dimensional convolution integration has been described, but the same applies to two-dimensional convolution integration.

  In general, a coefficient matrix expressing a point spread function indicating the blur characteristic of an optical system takes a Gaussian concentric coefficient pattern, and thus the coefficient matrix is rotationally symmetric. Here, “rotational symmetry” means that the coefficient values on concentric circles have the same value. In general, in an optical system, an image of a point light source is rotationally symmetric about the position of the light source, so that the coefficient value of a coefficient matrix expressing a point spread function is also rotationally symmetric.

  The convolution integral with the point spread function h has been described above, but the rotational symmetry of the coefficient matrix expressing the point spread function can be used similarly in the case of the convolution integral with the back projection function p.

  The backprojection function p is given as an inverse function of the point spread function h. In general, the inverse function of a rotationally symmetric function is also rotationally symmetric, and the rotational symmetry is maintained even in the case of a Wiener filter type inverse function to which a constant value is added in consideration of the influence of noise.

  Further, as shown in Expression (10), by approximating each coefficient value of the coefficient matrix expressing the point spread function h, the coefficient matrix of the backprojection function p that is an inverse function of the point spread function h is approximated. Is often used.

  Also in this case, if the point spread function h is rotationally symmetric, the approximate inverse function p is also rotationally symmetric.

  When it is assumed that the image is scanned line by line and the pixels are read sequentially, the number of multiplications is reduced by delaying and reusing the value previously multiplied by the coefficient value of the coefficient matrix of the point spread function for each pixel. In addition, address calculation and memory access waiting time can be reduced. Below, the Example using this method is described in detail.

  FIG. 1 is a configuration diagram of an image conversion apparatus 100 according to the present embodiment. The image conversion apparatus 100 includes an image input unit 10, a storage unit 20, a super-resolution processing unit 30, and a display unit 40. These configurations are realized by hardware, software, or a combination thereof.

  The image input unit 10 acquires a still image or a moving image and stores it in the storage unit 20. The input image is an image taken by, for example, a camera or the like, and is an observation image deteriorated due to the characteristics of the optical system or the image sensor.

  The super-resolution processing unit 30 performs super-resolution processing on the observation image stored in the storage unit 20, converts the observation image into a high-resolution image having a higher resolution than the observation image, and stores the high-resolution image in the storage unit 20. The display unit 40 displays the high resolution image stored in the storage unit 20.

  The super-resolution processing unit 30 performs reconstruction-type super-resolution processing on the input observation image and converts it into a high-resolution image. Specifically, a “simulation image” is calculated from the n-th high-resolution image using a point spread function, and an error between the simulation image and the observed image is back-projected to the n-th high-resolution image using a back-projection function. As a result, the image is updated to the (n + 1) -th high-resolution image. This process is repeated to reconstruct a high resolution image.

  Here, the super-resolution processing unit 30 limits the point spread function to a rotationally symmetric shape with respect to the center, thereby reducing the number of multipliers necessary for the convolution integration with the point spread function and the convolution integration with the back projection function. Reduce.

  Here, as an example, a configuration example of convolution integration according to the present embodiment will be described in reconstruction super-resolution processing using convolution integration with a point spread function and back projection function of three vertical pixels and three horizontal pixels. However, in general, the same configuration can be applied to vertical N pixels and horizontal N pixels (N is a natural number).

  FIG. 2 is a diagram illustrating a configuration example of a general convolution integral using a coefficient matrix of a point spread function for comparison.

  The coefficient matrix of the 3 × 3 point spread function A is given by nine coefficient values A0 to A8 (reference numeral 200). The 3 × 3 target image B is given by nine pixel values B0 to B8 (reference numeral 210).

  Here, the target image B is a high-resolution image as a result of the n-th reconstruction iteration of Equation (2).

  In order to acquire the target image B, the read address calculation unit 22 designates a read address for the storage unit 20 which is an image memory storing the entire image 24, and a 3 × 3 image area from the designated address. Must be read out as the target image B. For this reason, every time the target image B is read, it is necessary to calculate a read address, which takes a memory access standby time.

  When performing the multiplication of the convolution integral in parallel, the number of coefficients of the coefficient matrix of the point spread function, that is, nine multipliers are required (reference numeral 220). Each multiplier performs multiplication of the coefficient value An and the pixel value Bn, and outputs a multiplication result Xn = An × Bn (n = 0, 1, 2,... 8).

  The adder 230 calculates the sum ΣXn of the output values Xn (n = 0, 1, 2,... 8) of the nine multipliers, and outputs the result as a convolution integral A * B of the point spread function A and the target image B. .

FIGS. 3A and 3B are diagrams for explaining the coefficient values of the coefficient matrix of the rotationally symmetric point spread function F. FIG. When the 3 × 3 point spread function F is rotationally symmetric about the center, four coefficient values A0, A2, A6, A8 on the first concentric circle 301 are obtained for nine coefficient values A0 to A8 (reference numeral 300). Are the same value, the four coefficient values A1, A3, A5, A7 on the second concentric circle 302 are the same value, and the coefficient at the center 303 is A4 (FIG. 3A). Therefore, the coefficient value of the coefficient matrix of the 3 × 3 rotationally symmetric point spread function F has the following three types of values A, B, and C (FIG. 3B).
C = A0 = A2 = A6 = A8,
B = A1 = A3 = A5 = A7, and A = A4

  FIG. 4A to FIG. 4C are diagrams for explaining the relationship between pixel input and convolution integral by line sequential scanning. In this embodiment, the pixels are sequentially read in units of lines and convolution integration is executed. Here, it is assumed that the image input to the convolution integral is 4 × 4 and the pixel value is Pij (FIG. 4A). Here, i is a vertical pixel position (0 ≦ i ≦ 3), and j is a horizontal pixel position (0 ≦ j ≦ 3). The subscript i indicates the number of the line that scans the pixels of the image. In the case of a 4 × 4 input image, the number of lines is 4, and the number of pixels in each line is 4.

  In the case of line sequential scanning, pixels are read by sequentially scanning the 4 × 4 input image of FIG. That is, the pixels P00, P01, P02, and P03 are sequentially read by scanning the 0th line, the pixels P10, P11, P12, and P13 are sequentially read by scanning the first line, and the pixels P20, P20 are scanned by scanning the second line. P21, P22, and P23 are sequentially read. Finally, the third line is scanned to sequentially read pixels P30, P31, P32, and P33.

For example, the target image at the pixel position of the pixel P11 is a 3 × 3 image centered on the pixel P11 as shown in FIG. 4C, and the coefficient value of the coefficient matrix of the point spread function F multiplied by this is shown in FIG. 4 (b), the convolution integral of the target image and the point spread function F is as follows.
A × P11 + B × (P10 + P12 + P01 + P21) + C × (P00 + P02 + P20 + P22) (11)

When Equation (11) is rewritten into the calculation for each line of the target image, the following is obtained.
B × P01 + C × (P00 + P02) ···· 0th line + A × P11 + B × (P10 + P12) ······· First line + B × P21 + C × (P20 + P22) ······ Second line (12 )

  As can be seen from equation (12), since the point spread function has a rotationally symmetric shape, the product-sum operation of the pixel value and the coefficient value is such that the 0th line and the 2nd line have vertically symmetrical coefficient values across the 1st line. It will be used. Therefore, in the process of scanning the 4 × 4 input image line by line, the calculation result for each line of the current target image is buffered by a FIFO in units of lines, and again as the calculation result of the line of the future target image. It becomes possible to use.

  FIG. 5 is a configuration diagram of a convolution integral calculation unit on the premise of pixel input by line sequential scanning.

  Pixels of an image having a predetermined size are input to the convolution integral calculation unit in units of lines. Here, pixels are read one by one from the left end of each line in the order of the 0th line, the 1st line, the 2nd line, and the 3rd line of the image 400 of 4 × 4 size. The 0th line is read in the order of P00, P01, P02 and P03, the 1st line is read in the order of P10, P11, P12 and P13, and the 2nd line is read in the order of P20, P21, P22 and P23. The third line is read in the order of P30, P31, P32, and P33. FIG. 5 shows an input / output state when the pixel P22 is input for line sequential input (hereinafter referred to as “current time”) (S1).

  The three multipliers 50a, 50b, and 50c multiply the input pixels by coefficient values A, B, and C of the coefficient matrix of the point spread function F. The first multiplier 50a multiplies the input pixel P22 by the coefficient value A and outputs A × P22 (S2). The second multiplier 50b multiplies the input pixel P22 by the coefficient value B and outputs B × P22 (S4). The third multiplier 50c multiplies the input pixel P22 by the coefficient value C and outputs C × P22 (S7).

  The output value A × P22 from the first multiplier 50a is input to the delay unit 52a and output after being delayed by the input timing of one pixel. Therefore, at the present time, the delay unit 52a outputs the multiplication result A × P21 at the previous timing (S3).

  The output value B × P22 from the second multiplier 50b is input to the delay unit 52b and output after being delayed by one pixel. At this time, the delay unit 52b outputs the multiplication result B × P21 immediately before the input value B × P22 (S5). The multiplication result B × P21 is further input to the delay unit 54b and output after being delayed by one pixel. At the present time, the delay unit 54b outputs the multiplication result B × P20 immediately before the input value B × P21 (S6).

  The first adder 56 includes a multiplication result A × P21 of the coefficient value A delayed by one pixel by the delay device 52a, a multiplication result B × P22 of the coefficient value B, and two pixels by the two delay devices 52b and 54b. The multiplication result B × P20 of the coefficient value B delayed by the amount is added, and the addition result A × P21 + B × (P20 + P22) is output (S9).

  First, second, and third line FIFO (First In First Out) buffers 60, 62, and 64 hold product-sum operation results of pixel values in units of lines, and are delayed by line pixels (here, four pixels). This is a buffer that outputs in the FIFO manner.

  The product-sum operation result A × P21 + B × (P20 + P22) for the pixel P21 in the second line by the first adder 56 is input to the first line FIFO buffer 60. In the first line FIFO buffer 60, such a product-sum operation result is buffered for one line, that is, for four picture pixels. At this time, the first line FIFO buffer 60 has a pixel P11 four pixels before the pixel P21. The product-sum operation result A × P11 + B × (P10 + P12) in the first line is output (S11).

  The output value C × P22 from the third multiplier 50c is input to the delay unit 52c and output after being delayed by one pixel. At this time, the delay unit 52c outputs the multiplication result C × P21 immediately before the input value C × P22. The multiplication result C × P21 is further input to the delay unit 54c and output after being delayed by one pixel. At this time, the delay unit 54c outputs the multiplication result C × P20 immediately before the input value C × P21 (S8).

  The second adder 58 includes a multiplication result B × P21 of the coefficient value B delayed by one pixel by the delay device 52b, a multiplication result C × P22 of the coefficient value C, and two pixels by the two delay devices 52c and 54c. The multiplication result C × P20 of the coefficient value C delayed by the amount is added, and the addition result B × P21 + C × (P20 + P22) is output (S10).

  The product-sum operation result B × P21 + C × (P20 + P22) for the pixel P21 in the second line by the second adder 58 is input to the second line FIFO buffer 62. At this time, the second line FIFO buffer 62 outputs the product-sum operation result B × P11 + C × (P10 + P12) in the first line for the pixel P11 four pixels before the pixel P21. The product-sum operation result B × P11 + C × (P10 + P12) is further input to the third line FIFO buffer 64. At this time, the third line FIFO buffer 64 outputs the product-sum operation result B × P01 + C × (P00 + P02) in the 0th line for the pixel P01 four pixels before the pixel P11 (S12).

  The third adder 66 calculates the product-sum operation result A × P11 + B × (P10 + P12) in the first line from the first line FIFO buffer 60 and the product-sum operation result B × P21 + C in the second line from the second adder 58. X (P20 + P22) and the product-sum operation result B * P01 + C * (P00 + P02) in the 0th line from the third line FIFO buffer 64 are added, and the addition result B * P01 + C * (P00 + P02) + A * P11 + B * (P10 + P12) + B × P21 + C × (P20 + P22) is output. This addition result is a calculation result of the convolution integral of the 3 × 3 image centered on the pixel P11 and the point spread function F.

  As described above, the convolution integral calculation unit multiplies the pixel values Pij sequentially input in units of lines by the coefficient values A, B, and C of the coefficient matrix of the point spread function F, and the multiplication result A × Pij, B × Pij, and C × Pij are delayed by the horizontal size of the point spread function by a delay device such as a shift register. Further, the product-sum operation value for each line is obtained by adding the multiplication result with the delayed coefficient value, and the product-sum operation value is buffered by the line pixel in the line FIFO buffer, and the line by the required line delay is obtained. Cascade the FIFO buffers. Here, the coefficient values of the coefficient matrix of the 3 × 3 point spread function are vertically symmetrical about the center line, and the product-sum operation value of the second line can be used as the product-sum operation value of the second line. Therefore, the line FIFO buffer is cascade-connected for two lines. Finally, the convolution integral is calculated by adding the product-sum operation values of each line.

  When the 3 × 3 point spread function F is used, for the pixels other than the pixels P11, P12, P21, and P22 in the 4 × 4 input image 400, the pixels of the 3 × 3 target image necessary for the filter calculation are the same. Exception handling is performed because not all are available.

  With the above configuration, the multiplier can be reduced to the number of types of coefficient values of the coefficient matrix of the point spread function. In the present example, nine multipliers were originally required for the convolution integration of the 3 × 3 target image, but the number of multipliers can be reduced to three. In addition, by using the vertical symmetry of the coefficient values of the coefficient matrix of the point spread function, it is possible to manage the timing of the product-sum operation values necessary for the convolution integration by the delay in line units by FIFO.

  When the above convolution integration is realized by an electronic circuit, the circuit scale can be reduced by reducing the number of electronic circuits of the multiplier by the above method.

  In the above embodiment, an example of a 3 × 3 point spread function has been described. However, when a high-quality image is assumed, a point spread function having a large number of elements such as 4 × 4 to 10 × 10 is used. Therefore, the reduction effect of the multiplier is very large.

  Since the inverse projection function, which is the inverse function of the point spread function, is also rotationally symmetric, this configuration can also be used for convolution integration processing of the inverse projection function. In this case, the target image to be subjected to convolution integration with the backprojection image is an error between the simulation image and the observation image obtained from the high-resolution image of the nth reconstruction iteration result of Equation (4).

  In both the convolution integration with the point spread function and the convolution integration with the backprojection image, if the number of multipliers is reduced by the above method, the processing cost can be further reduced as the whole super-resolution processing. In this case, in order to execute the convolution integration with the point spread function and the convolution integration with the back projection function in parallel, three multipliers may be prepared for each convolution integration. Alternatively, the same three multipliers are shared by the convolution integral with the point spread function and the convolution integral with the backprojection function, so that the number of multipliers in the entire reconstruction type super-resolution processing can be reduced. Good.

  As described above, according to the embodiment of the present invention, the number of multipliers required for convolution integration with the point spread function is reduced by limiting the point spread function of the optical system to a rotationally symmetric shape with respect to the center. can do. Furthermore, since the inverse projection function, which is an inverse function of the point spread function, is also rotationally symmetric like the point spread function, the number of multipliers necessary for convolution integration with the inverse projection function can be reduced in the same manner. Furthermore, by delaying and using the pixel value that has been previously multiplied by the coefficient of the point spread function, the address calculation and memory access standby time can be reduced. As a result, even when modeling a low-cost optical system having a point spread function with a large number of elements, a good resolution-converted image having a sufficient blur recovery effect can be obtained while avoiding an increase in processing scale. be able to.

  In addition, as a method of approximately obtaining a back projection function that is an inverse function of the point spread function, each corresponding element of the coefficient matrix representing the point spread function is squared to obtain a corresponding element of the coefficient matrix representing the back projection function. There is a way to ask. Even in this approximation method, the backprojection function is rotationally symmetric like the point spread function, so that the number of multipliers necessary for convolution integration with the backprojection function can be reduced.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  10 image input unit, 20 storage unit, 30 super-resolution processing unit, 40 display unit, 50a, 50b, 50c multiplier, 52a, 52b, 52c, 54b, 54c delay unit, 56, 58, 66 adder, 60, 62, 64 line FIFO buffer.

Claims (4)

An image input unit that stores observation images captured by the optical system in a memory;
A super-resolution processing unit that performs super-resolution processing on the observed image using a convolution integral with a point spread function indicating a transfer function of the optical system, and generates a high-resolution image having a higher resolution than the observed image; Including
The super-resolution processing unit performs the convolution integration of the point spread function and a first target image that is a high-resolution image obtained by performing a reconstruction-type super-resolution process on the observed image. The pixels of one target image are sequentially read out line by line, and the pixel values of the first target image obtained by multiplying the coefficient values of the coefficient matrix of the point spread function are delayed and added for each line. Calculating the convolution integral of the point spread function and the first target image by buffering the product-sum calculation value in a FIFO format in units of lines and using it again as a product-sum calculation value of a future target image. A featured image conversion apparatus.   The super-resolution processing unit further performs super-resolution processing on the observed image using a convolution integral with a back projection function that is an inverse function of the point spread function, and the back projection function, When convolution integration is performed on a second target image, which is an error between the simulation image calculated by convolution integration of the point spread function and the first target image, and the observed image, pixels of the second target image are calculated. The product-sum operation value of the current target image obtained by delay-adding the pixel values of the second target image obtained by sequentially reading line by line and multiplying the coefficient values of the coefficient matrix of the point spread function by line. And calculating the convolution integral of the back projection function and the second target image by buffering in the FIFO format and reusing the result as a product-sum operation value of a future target image. Image converter according to 1. The super-resolution processor
An input unit that sequentially reads out pixels of the target image in line units;
A multiplication unit for obtaining a multiplication value of a pixel value of the target image and a coefficient value of a coefficient matrix of the point spread function;
A delay unit that outputs the multiplied value by delaying the input timing for a predetermined pixel; and
An addition unit that adds the multiplication value output by the multiplication unit and the multiplication value delayed by the delay unit and outputs a product-sum operation value;
A line FIFO buffer for buffering the product-sum operation value output by the adding unit in a line unit and outputting it in a FIFO format;
The output part which adds the product-sum operation value output by the said addition part, and the product-sum operation value output by the said line FIFO buffer, and outputs a convolution integration result, It is characterized by the above-mentioned. The image conversion apparatus according to 2. An image input step of storing an observation image captured by the optical system in a memory;
A super-resolution processing step of super-resolution processing the observed image using a convolution integral with a point spread function indicating a transfer function of the optical system, and generating a high-resolution image having a higher resolution than the observed image; Including
In the super-resolution processing step, the pixel of the target image is convolved with the point spread function and the target image that is a high-resolution image obtained by performing a reconstruction-type super-resolution processing on the observed image. Are sequentially read in line units, and the product-sum operation value of the current target image obtained by delay-adding the pixel values of the target image obtained by multiplying the coefficient values of the coefficient matrix of the point spread function for each line by FIFO. An image conversion method characterized by calculating a convolution integral between the point spread function and the target image by buffering in a format and reusing it as a product-sum operation value of a future target image.

Images