JP2017033287A - Image processing device, image processing method, and image processing system using the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing technique that makes it possible to stably obtain a latent image with a few blurs even when there exist a plurality of subjects differing in the manner of blurring within an observed image, and to obtain a latent image in a relatively short time.SOLUTION: Provided is an image processing method for achieving the above objective, the method being constituted with the steps of: inputting an image; dividing the image into a plurality of first areas, and finding a frequency component distribution for each of the first areas; calculating the similarity of frequency component distributions with each other, and integrating the first areas having a higher degree of similarity than a prescribed value as a second area; generating, for each second area, an image in which areas other than the second area are filled with a prescribed value; performing, for each second area, an image recovery process on the image obtained in the step of generating a filled image; and synthesizing the results of the image recovery step into one image and outputting the synthesized image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to image processing for recovering an image with less blur from an image including blur (due to out-of-focus blur or camera shake) generated in the process of taking an image, and more particularly, to a plurality of blurs with different blur methods in an image. The present invention relates to an image processing apparatus and method suitable for a case where a subject exists.

  When taking an image with a camera, the focal point of the lens may be blurred or the camera may be blurred, causing the image to be blurred, and the realization of an image processing system that recovers a low-blurred image from an image including the blur. It is desired.

  Such image processing is called image restoration, image restoration, etc., and these are hereinafter collectively referred to as image restoration processing. In general, as shown in the following (Equation 1), an image including blur (observed image) is a point spread function (PSF: Point Spread) that represents the blur characteristics with respect to an ideal image (latent image) that does not include blur. Function) is considered to be folded. In the image restoration process, as shown in the following (Equation 2), the latent image is estimated by performing a PSF deconvolution operation on the observed image.

Observation image = latent image * PSF (Equation 1)
(However, * indicates convolution operation (that is, multiplication in the frequency domain))
Latent image = Observation image / PSF (Equation 2)
(However, / indicates deconvolution (ie, division in the frequency domain))
Many techniques have been proposed so far for image restoration processing.

  For example, a non-blind method (Non-blind Devolution) for estimating a latent image from an observed image when PSF is known has been proposed, and a Wiener Filter is well known. In addition, since a PSF is unknown in a general image, a Blind method (Blin Devolution) for estimating both a PSF and a latent image from an observed image has been proposed. An AD method (Ayer's-Dainty method) or an RL method is proposed. (Richardson-Lucy method) is well known. In these Blind methods, the first generation latent image is estimated by the non-blind method using the observed image and the temporary PSF (first generation PSF), and the observed image and the first generation latent image are used. The second generation PSF is estimated by the non-blind method, and the third generation latent image is estimated by the non-blind method using the observation image and the second generation PSF. A potential latent image. In these techniques, image restoration processing is performed based on the premise that PSF is constant in an image.

  The blur included in the image is particularly perceived when a subject at a distant location is telephotographed or magnified. For this reason, a large image quality improvement effect by image restoration is expected in a surveillance camera for photographing a distant place. On the other hand, if the depression angle and elevation angle when shooting are close to horizontal, the distant view and the close view often appear in one image at the same time, and there are multiple subjects with different blurring methods (ie PSF) in the image. For this reason, there is a problem that the accuracy of the image recovery is lowered when the image recovery process of the Blind method based on the premise that the PSF is constant in the image as described above is used.

  Image restoration processing techniques that solve this problem using the Blind method are disclosed in, for example, Japanese Patent Application Laid-Open No. 2012-155456 (Patent Document 1) and WO 2010/098054 (Patent Document 2).

  In Patent Document 1, a small region dividing unit that divides an input image (observed image) into a plurality of small region images, a local PSF estimating unit that estimates a PSF of the small region image, and the estimated shape of the PSF are identified. The PSF shape identifying unit for classifying the small region images so that the highly similar PSF-shaped small region images belong to the same group, and the adjacent small region images classified into the same group for each group. A technique for obtaining subject images separated into groups by integration is disclosed. The subject image is degraded using the subject PSF estimation unit that estimates the PSF of each subject image using each subject image, the PSF estimated by the subject PSF estimation unit, and the subject image. A technique for obtaining a corrected restored image (latent image) is disclosed.

  In Patent Literature 2, an adaptive region dividing unit that divides an input image (observed image) into a plurality of adaptive regions in accordance with pixel values of pixels constituting the input image (observed image), and a plurality of divided adaptations. A PSF interpolation unit that interpolates PSFs of pixels located between representative pixels, which are pixels representing each of a plurality of adaptive regions, using a PSF indicating blur characteristics of an image calculated for each target region; A technique for generating a target image (latent image) by correcting an input image (observed image) using the interpolated PSF is disclosed. In addition, a technique for determining an area where blur (PSF) is common as one adaptive area is disclosed.

JP 2012-155456 A WO 2010/098054

  The techniques disclosed in Patent Documents 1 and 2 described above both divide an input image (observation image) into a plurality of small area images, estimate the PSF for each divided small area image, By collecting similar small region images into one adaptive region (group) and estimating the PSF again for each adaptive region (group), a more accurate final restored image (latent image) will be obtained. It is said.

  However, when the PSF is estimated for each divided small region image, when the number of vertical and horizontal pixels constituting the small region image is small, that is, when the area of the small region is small, or in the small region image, In the case where a characteristic pattern such as an edge is not included, that is, in the case of a flat pattern with a small change in luminance value in a small area image, the PSF estimation accuracy decreases, and an incorrect PSF is obtained. There is a problem that the PSF cannot be obtained because the result diverges in the process of iterative calculation.

  In addition, it is well known that the Blind method requires the iterative processing as described above and requires a long time to estimate the PSF. In the techniques disclosed in Patent Documents 1 and 2, after estimating the PSF for each divided small region image, the small region images having similar PSFs are combined into one adaptive region (group). Since it is necessary to estimate the PSF in two stages, such as estimating the PSF again for each region (group), there is a problem that it takes a very long time to obtain the final latent image. .

  The present invention has been made in view of such a situation, and even when there are a plurality of subjects having different blurring methods in an image, a latent image with less blur can be stably obtained, and is relatively short. An image processing apparatus and an image processing method for obtaining a latent image in time, and an image processing system using the image processing apparatus are provided.

  In order to solve the above problems, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above-described problems. To give an example, an image processing method includes an image input step, the image is divided into a plurality of first regions, and the first region Obtaining a frequency component distribution for each, calculating a similarity between the frequency component distributions, integrating a first region having a similarity higher than a predetermined value as a second region, and the second region for each second region The results of the step of generating an image in which the other areas are filled with a predetermined value, the step of performing the image restoration process for each second area on the image obtained in the step of creating the filled image, and the step of performing the image restoration process are as follows: And a step of combining and outputting the image.

  According to the present invention, even when there are a plurality of subjects with different blurring directions in an image, an image processing apparatus that can stably obtain a latent image with little blur and obtain a latent image in a relatively short time. And an image processing method and an image processing system using the same.

1 is a diagram illustrating a schematic configuration of an image processing apparatus in Embodiment 1. FIG. FIG. 3 is a diagram illustrating an example of detailed configurations of a frequency component distribution analysis unit and a similar region integration unit according to the first embodiment. FIG. 6 is a diagram for explaining the operation of a filling unit in the first embodiment. 6 is a schematic configuration diagram of an image processing system in Embodiment 2. FIG. FIG. 10 is a diagram illustrating details of an image processing unit in Embodiment 2. FIG. 10 is a diagram illustrating another example of an image processing unit according to the second embodiment. FIG. 10 is a diagram illustrating another example of an image processing unit according to the second embodiment. FIG. 10 is a diagram illustrating a schematic configuration of an image processing apparatus according to a third embodiment. FIG. 10 is a diagram illustrating a schematic configuration of an image processing apparatus according to a fourth embodiment. FIG. 10 is a diagram illustrating a configuration example of a PSF interpolation unit according to a fourth embodiment. 14 is a flowchart illustrating an example of a procedure of image restoration processing in the fifth embodiment.

  Embodiments of the present invention will be described below with reference to the accompanying drawings. In addition, the same reference number is attached | subjected about a common structure in each figure. The embodiment of the present invention may be implemented by software running on a general-purpose computer, or may be implemented by dedicated hardware or a combination of software and hardware.

  In this embodiment, even when there are a plurality of subjects with different blurring methods in an image, it is possible to stably obtain a latent image with less blur and to obtain a latent image in a relatively short time. The image restoration process will be described. In the present embodiment, similar to the techniques described in Patent Documents 1 and 2 described above, the input observation image is divided into small region images, and the small regions having similar features are grouped into similar regions, Image restoration is performed independently for each similar area, and the final latent image is output by combining the image restoration results into one image.

  Here, in the techniques described in Patent Documents 1 and 2 described above, the PSF shape estimated for each small region is used as a feature amount for determining whether the small regions are similar to each other. The problem is that the estimation of the PSF becomes unstable, diverges in the middle of the calculation, or requires a long time.

  Therefore, in this embodiment, instead of estimating the PSF for each divided small area, it is determined whether the small areas are similar by using the frequency component distribution for each small area of the observed image as a feature amount. The small regions having similar frequency component distributions are collectively defined as a similar region. That is, if the same subject at almost the same distance is shown in a plurality of small area images, the frequency component distributions of the observed images in each small area, the frequency component distributions of the latent images, and the PSFs are similar to each other. Introducing the assumption that it should be. This eliminates the need for image restoration processing (PSF estimation processing) for separating the latent image and the PSF for each of the divided small regions as in the techniques described in Patent Documents 1 and 2 described above. The problem described above can be solved.

However, the following two cases can be considered as examples in which this assumption includes an error.
(1) Although the original PSF is substantially the same between different small areas, the frequency component distributions corresponding to the patterns of the latent images in the small areas are greatly different from each other. In this case, the frequency component distributions of the observed images are greatly different from each other, so that they should be discriminated as different similar regions although they should originally be discriminated as the same similar region.
(2) Although the original PSF is not substantially the same between different small areas, the frequency component distribution corresponding to the pattern of the latent image for each small area is greatly different from each other. As described above, since the observed image is a product of the latent image and the PSF, the frequency component distribution of the observed image happens to be substantially the same, and it should be determined as a separate similar region, but it is determined as the same similar region. A case.

  In the case of (1), since different small areas are determined as different similar areas, the size of the similar area is smaller than the size of the similar image that should be originally. Since image restoration processing is performed separately for each area, no major problem occurs.

  In the case of (2), since the small areas where the original PSFs are not substantially identical are determined to be the same similar areas, if the similar areas are subjected to image restoration processing as they are, a correct PSF cannot be obtained, and it appears to be large. Seems to be a problem. However, it may be considered that the frequency component distribution corresponding to the pattern of the latent image and the PSF corresponding to the blur method are different from each other and the probability that the frequency component distribution of the observed image (see Equation 1) is similar to each other is considerably low. The case (2) is considered to occur rarely.

  Therefore, if the same subject at almost the same distance appears in a plurality of small area images, the frequency component distributions of the observed images in each small area, the frequency component distributions of the latent images, and the PSFs are similar to each other. Under the assumption that it should have been, the frequency component distribution for each small area of the observed image is used as a feature value to determine whether the small areas are similar, and the frequency component distribution is similar Even if the regions are grouped into similar regions and image restoration processing is performed for each similar region, it is considered that no major problem will occur.

  In accordance with this concept, in order to solve the above-described problems, the present embodiment proposes an image processing apparatus having a configuration as described below.

  FIG. 1 is a diagram illustrating a schematic configuration of an image processing apparatus according to the present embodiment. In FIG. 1, the input observation image (107) is divided into m (where m is an integer of 2 or more) small region images (108) by the region dividing unit (101). Subsequently, the frequency component distribution analysis unit (102) calculates a distribution (109) of a two-dimensional frequency component composed of a horizontal frequency and a vertical frequency for each divided small region image (108), and observes the image before division. A frequency distribution (110) for each small region with respect to the whole of (107) is obtained. At this time, m frequency component distribution analysis units (102) may be prepared as shown in FIG. 1 according to the number of divisions (m) of the small region, or one frequency component distribution analysis unit (102). May be repeated m times. Details of the frequency component distribution analysis unit (102) will be described later with reference to FIG.

  In the similar region integration unit (103) in FIG. 1, the similar regions are generated by collecting the small regions having similar frequency distributions obtained by the frequency component distribution analysis unit (102). Here, for example, in the observed image (107), a similar area (111-1) in which a distant ship is shown, a similar area (111-2) in which only the sea surface is shown, and a similar area in which a nearby ship is shown In the region (111-n), each frequency distribution is similar, and m (m = 12 in the example of FIG. 1) output from the region dividing unit (101) is n (however, n Is an integer greater than or equal to 1. In the example of FIG. Details of the similar region integration unit (103) will be described later with reference to FIG.

  In the filling unit (104) in FIG. 1, an image (112) is generated by filling a region other than the similar region with a predetermined value, for example, black (luminance value = 0) for each similar region (111). Details of the painting operation will be described later with reference to FIG.

  The image restoration unit (105) in FIG. 1 performs image restoration on the filled image (112) by a general non-blind method for each similar region.

  The combining unit (106) in FIG. 1 combines the unfilled areas of the image, which is not shown, into a single image and outputs the final latent image (113).

  FIG. 2 collectively shows an example of each detailed configuration of the frequency component distribution analysis unit (102) and the similar region integration unit (103) described above. In FIG. 2, m frequency component distribution analysis units (102-1 to m) prepared for each small region are respectively configured by a Fourier transform unit (201) and a level normalization unit (202). The Fourier transform unit (201) performs Fourier transform on the input small area image to obtain a power distribution (that is, power spectrum) for each frequency. In the level normalization unit (202), normalization is performed so that the peak (maximum value) of the power spectrum becomes constant so that the similarity between the power spectra of the small regions can be easily determined. In a general image, since the power of the DC component is larger than the power of the other frequency components, the power values of all the frequency components may be divided by the power values of the DC components. In addition, it is difficult to determine the similarity of power spectra between small areas because the power value of a specific frequency component (for example, DC component) is significantly higher than the power values of other frequency components. Therefore, the similarity of the power spectrum may be determined using the logarithm (log) of the power value of each frequency component.

  In the similar region integration unit (103) in FIG. 2, based on the result of the frequency component distribution analysis unit (102) described above, small regions having similar frequency component distributions are combined into one similar region (cluster). Integrate. As the similar region integration unit (103), for example, a similar region may be used when a correlation value for each frequency between power spectra for each small region is equal to or less than a predetermined threshold. In addition, a threshold value is dynamically obtained by using a generally known clustering technique (shortest distance method, longest distance method, average method, median method, ward method, k-means method, etc.). It is good. At this time, as a distance used when forming a cluster, for example, a correlation value for each frequency between power spectra for each small region may be used. Further, when the number of small regions (m) increases, the number of combinations (m (m + 1) / 2) for calculating correlation increases on the order of squares, so that the calculation cost becomes very high. Therefore, the processing is simplified in order to reduce the calculation cost, and as shown in FIG. 2, an average value of the power values of all frequency components is obtained for each small area using the average value calculation unit (203), and this average value is obtained. This difference may be used as a distance for forming a cluster. At this time, for example, in a small region where there are many clear edges, that is, a small region where the focus is relatively in focus, the power of each component is relatively large in all frequency bands from the direct current component to the high frequency component. The average value of the power values in all frequency bands after normalization so that the power of the DC component becomes a constant value also becomes a relatively large value. On the other hand, in a small region where there are few clear edges, that is, a small region where the focus is not relatively in focus, the power of the high frequency component is relatively small compared to the power of the DC component, so the power of the DC component is a constant value. The average value of the power values in all frequency bands after normalization is also a relatively small value.

  After that, the same cluster ID (206) is assigned to the small regions having similar average values in the clustering unit (204) that implements the clustering technique described above. Note that the number of clusters n (205) may be predetermined and input to the clustering unit (204) such as “2” (distant view and foreground) and “3” (distant view and foreground and intermediate). Alternatively, the number of clusters n (205) may be automatically determined using a widely known general clustering technique (such as the x-means method).

  The operation of the above-described filling unit (104) will be described with reference to FIG. As an example, only the region corresponding to the similar region (111-1) in which the distant ship is reflected is left, and the other regions (111-2) (111-n) are set to a predetermined value, for example, black (luminance value = The operation in the case of generating the image (112-1) filled with (0) will be described.

  When filling other areas (111-2) (111-n) simply with black (brightness value = 0), the brightness value (signal level) at the boundary (301) between the areas becomes steep. As a result, a high-frequency component having strong power is generated, and the PSF and the latent image are not correctly separated in the subsequent image restoration unit (105).

  Therefore, in order to prevent the signal level of the original image from changing sharply in the vicinity of the boundary (301) as shown in FIG. Using two (one set) signals (coefficients) having values that gradually change between 0 and 1.0, the signal in FIG. 3A is shown on the left side of the boundary (301). Is multiplied by the signal (coefficient) of FIG. 3 to obtain the signal of FIG. 3D, which is input to the subsequent image recovery unit (105-1). On the other hand, on the right side of the boundary (301), the signal of FIG. 3 (a) is multiplied by the signal (coefficient) of FIG. 3 (c) to obtain the signal of FIG. ). By performing the same processing for the boundary in the vertical direction, the luminance value (signal level) of the boundary can be prevented from changing sharply in both the horizontal direction and the vertical direction. Image separation can be performed correctly.

  As described above, the present embodiment is an image processing apparatus, which divides an input image into m (where m is an integer of 2 or more) first regions, and a frequency for each first region. A frequency component distribution analyzing unit for obtaining a component distribution, a similarity region calculating unit for calculating a similarity between frequency component distributions, and combining a first region having a similarity higher than a predetermined value as a second region, and a second region A filling unit that generates an image in which a region other than the second region is filled with a predetermined value, an image restoration unit that performs image restoration processing for each second region on the image obtained by the painting unit, and a result of the image restoration unit An image composition unit that composes and outputs a single image.

  This eliminates the need for image restoration processing (PSF estimation processing) for separating the latent image and the PSF for each divided small area, makes the PSF estimation unstable, diverges in the middle of the calculation, The problem that a long time is required can be solved. That is, even when there are a plurality of subjects with different blurring methods in the observed image, an image for stably obtaining a latent image with little blur and for obtaining a latent image in a relatively short time. A processing device and an image processing method can be realized.

  In this embodiment, an image enlargement processing system to which the image restoration processing of the first embodiment is applied will be described.

  FIG. 4 shows a schematic configuration of the image processing system according to the present embodiment. In FIG. 4, a mobile unit (401) such as an airplane includes a camera (404), an image processing unit-1 (405) described later, an encoding unit (406), a wireless transmission unit (not shown), and the like. 403) and an observation image (413) obtained by telephoto imaging a distant subject (402-1) is assumed to be wirelessly transmitted to a monitoring station (407) on the ground via a downlink (415). In the monitoring station (407), an image of the wireless signal received by the antenna (408) is decoded by a decoding unit (409) via a wireless receiving unit (not shown), and then via an image processing unit-2 (410) described later. The enlarged image (414) of the subject (402-1) to be monitored is displayed on the display unit (411). In addition, an operation unit (412) using a pointing system such as a general keyboard or mouse is provided, and the image processing unit-2 (410) or the image processing unit-1 (405) via the uplink (416) is provided. Assume that commands and parameters described later can be sent.

  At this time, if the subject is photographed by the camera (404) mounted on the moving body (401), the moving body (401) moves during image shooting, or the vibration of the moving body (401) propagates to the camera. In many cases, the image is blurred. In addition, when the depression angle and elevation angle at the time of shooting are close to the horizontal, the observation image (413) shot by the camera (404) is separated from the camera in addition to the subject (402-1) to be monitored. Subject (402-2) and (402-3) may be reflected, and it is difficult to focus the lens on all of these subjects at the same time, resulting in different blur for each subject. There are many.

  In the image processing unit-1 (405) or the image processing unit-2 (410), a partial region or the entire region of the observation image captured by the camera is enlarged, and an image with less blur is recovered by the above-described image recovery processing. Then, image processing for obtaining an enlarged image used for display is performed. These image processes may be performed only by the image processing unit-1 (405) for the image before encoding, or the image processing unit-2 (410) for the image after encoding and decoding. ), Or may be shared and processed by both the image processing unit-1 (405) and the image processing unit-2 (410).

  As described above, since image recovery involves repetitive processing, it is relatively difficult to process images (moving images) that are continuous in time series online (that is, in real time). Therefore, an image to be processed (observed image) is stored in a storage (semiconductor memory, hard disk drive (magnetic disk), optical disk, etc.) (not shown), and one (one frame) image (still image) is stored. ) Is taken out and the image can be recovered offline.

  At this time, if there is little deterioration in image quality caused by encoding and decoding, image recovery processing may be performed on either side of image processing unit-1 (405) or image processing unit-2 (410).

  On the other hand, in applications where images can only be transmitted at a low bit rate, such as when the transmission band of the downlink (415) shown in FIG. 4 is narrow, image quality degradation due to encoding and decoding becomes large, and image restoration becomes difficult. It may become. In such a case, a buffer memory (not shown) is provided immediately after the camera (404), and an image processing unit-1 is applied to the image stored in the buffer memory. Image recovery processing may be performed at (405). Alternatively, in addition to the normal image transmission mode (low image quality), an image transmission mode that transmits images with high image quality (low degradation) is provided, and image transmission that transmits images with high image quality (low degradation) when performing image recovery. The system may be configured to change to a mode.

  5 to 7 show configuration examples of the image processing unit-1 (405) or the image processing unit-2 (410) described above.

  In FIG. 5, in the image processing unit-1 (405) or the image processing unit-2 (410) described above, the observation image (413) is input through the terminal (501), and the image is enlarged by the image enlargement unit (502). After the enlargement, the image restoration unit (505) performs the image restoration process described above, and outputs the enlarged image (414) via the terminal (506). Further, the image enlargement unit (502) and the image restoration unit (505) are controlled based on the command and parameters input via the terminal (507). This command selects a control command for instructing each unit (502) (505) by selecting on (on) or off (not) of image enlargement processing and on (on) or off (no) of image restoration processing. Including. The parameters include the horizontal and vertical enlargement ratios (or the number of pixels in the target area and the enlarged image) and the coordinates of the target area (508) for image enlargement. The parameters of the horizontal and vertical enlargement ratios (or the number of pixels of the target area and the enlarged image) are passed through the enlargement ratio setting section (503), and the coordinate parameters of the target area (508) are the target area setting section (504). Are set in the image enlargement unit (502). As the image enlargement unit (502), an image enlargement process using a linear filter (polyphase filter) using a convolution operation such as generally known bilinear enlargement or bicubic enlargement is used as it is. Detailed illustration is omitted because it is possible. Further, as the image restoration unit (505), the configuration shown in FIG. 1 described above and the configurations shown in FIGS. 8 to 10 described later can be used as they are.

  The frequency characteristic diagram (509) is a diagram showing, as an example, the relationship between the frequency and gain in a general linear filter when the enlargement ratio is set to 4 times. When a general linear filter is used for the image enlargement unit (502), the passband characteristic from direct current (f = 0) to the Nyquist frequency (f = fN) does not become the ideal characteristic (510), but the actual filter characteristic. It is well known that the difference between (511) and the ideal characteristic (510) becomes an attenuation component (512), and the image is blurred. For this reason, by disposing the image restoration unit (505) in the subsequent stage of the image enlargement unit (502), the lens defocusing or camera shake generated in the image shooting process and the image blur generated in the image enlargement unit (502) are combined. Image recovery.

FIG. 6 shows a configuration in which the processing order of the image restoration unit (505) and the image enlargement unit (502) in the configuration shown in FIG. 5 is exchanged, and image enlargement is performed after image restoration. When a general linear filter is used for the image enlargement unit (502), the attenuation component (512) shown in FIG. 5 which is one of the causes of image blur is reduced by setting a wide pass band of the filter, or the enlargement ratio It is well known that when the frequency is increased, a component having a frequency higher than the Nyquist frequency (f _N ) becomes the aliasing component (513), and image distortion (aliasing) increases. Such a aliasing component cannot be removed by the image restoration process, but also becomes an obstacle to the image restoration process, and the accuracy of the image restoration often decreases. Therefore, when examining the frequency characteristics of the linear filter used for the image enlargement process and performing image enlargement in which the power of the aliasing component is larger than the power of the attenuation component, as shown in FIG. When the image is magnified with the method, a high-quality enlarged image with less image quality deterioration can be obtained.

  FIG. 7 integrates the processes shown in FIGS. 5 and 6 and includes an image enlargement-1 (701) before the image restoration unit (505), and an image enlargement-2 (after the image restoration unit (505). 702) is shown. As described above, the image blurring caused by the image enlargement-1 (701) in the preceding stage from the image restoration unit (505) may be recovered by the image restoration unit (505), while the image enlargement-1 (701). ) May be an obstacle to image recovery. Therefore, in order to perform a desired enlargement ratio (z), image enlargement is performed at an enlargement ratio (z1) that does not hinder image recovery in image enlargement-1 (701), and image restoration is performed by the image restoration unit (505). Then, the remaining enlargement ratio (z2 = z / z1) may be enlarged by the image enlargement unit-2 (702). Specifically, as described above, the frequency characteristic of the linear filter used for the image enlargement -1 (701) is examined, and the fixed magnification (z1, for example, 1) such that the power of the aliasing component becomes smaller than the power of the attenuation component. (Magnification (1.5 times), 1.5 times, 2 times, etc.) image enlargement-1 (701) is designed and prepared in advance, and is based on the command and parameters input via the terminal (507). The enlargement factor setting unit (703) selects the maximum fixed magnification (z1) that does not exceed the desired enlargement factor (z) and sets it in the image enlargement unit-1 (701). Further, the remaining enlargement factor (z2 = z / z1) is calculated by the enlargement factor setting unit (704) and is set in the image enlargement unit-2 (702), so that the image quality enlarged at a desired enlargement factor is obtained. Enlarged images can be obtained.

  As an example of the case where a lot of blurring occurs in the image, a moving body (for example, a manned airplane, an autopilot, an unmanned airplane that is manually operated remotely from a remote place, a ship, an automobile, etc.) is cited. The application example of the embodiment is not limited to this. For example, since a lot of blur occurs in an image taken by a fixed camera installed in a place receiving a strong wind, a camera attached to the end of a long stick, a hand-held camera, etc., the image processing system according to this embodiment is suitable. It can be an example.

  The encoding unit (406) and decoding unit (409) described above are generally known MPEG (Moving Picture Expert Group) -1, MPEG-2, MPEG-4, H.264, and H.264. H.264, VC-1, JPEG (Joint Photographic Experts Group), Motion JPEG, JPEG-2000, etc. may be standardized encoding and decoding, or non-standard encoding and decoding may be performed. Also good. Since these can be easily realized by a general known technique, illustration is omitted.

  As described above, the present embodiment is an image processing system using the image processing apparatus described in the first embodiment, and increases the number of pixels of the captured image to the first image size. The output of the image enlargement unit and the first image enlargement unit is input to the image processing apparatus described in the first embodiment to perform image restoration processing, and the number of pixels of the output image of the image processing apparatus is increased to increase the second number. And a second image enlargement unit that outputs the image with the image size enlarged.

  A configuration having an enlargement ratio setting unit for controlling the first image size so that the power of the aliasing distortion component generated by the first image enlargement unit is equal to or less than the power of the signal component attenuated by the first image enlargement unit; To do.

  Thereby, it is possible to configure an image processing system that appropriately performs image enlargement and image restoration with high image quality.

  In this embodiment, an image restoration process that approximates a difference in shooting distance will be described.

  In telephoto shooting as described with reference to FIG. 4 in the second embodiment, a distant view is generally shown in the upper part of the image, and a close view is shown in the lower part of the image. Therefore, when the image is divided into horizontally long strips, subjects with the same distance from the camera are shown in the strip, and the PSFs are considered to be substantially the same.

  Therefore, in this embodiment, as shown in FIG. 8, the observation image (801) is divided into horizontally long strips to form a small image area (802), and the PSF is considered to be constant in this small area. The image recovery units (804), (805), (806), (807), (808), and (809) prepared for each small region recover the respective regions, and the combining unit (810) combines one image. The latent image (811) is output. In FIG. 8, as an example, a configuration in which the image is divided into six small image areas in the vertical direction is shown. However, the number of divisions is not limited to (6), and the number of divisions can be flexibly increased or decreased. It is clear that it is good.

  In this processing, the frequency distribution analysis unit (102) and the similar region integration unit (103) in the configuration shown in FIG. 1 are omitted, and a similar region is created based on the vertical coordinates of the observation image, and the processing is simplified. As a result, the time required for processing can be shortened.

  Further, in the configuration shown in FIG. 1, the total pixels of the images (112-1), (112-2), and (112-n) to be processed by the image restoration (105-1) (105-2) (105-n). Since the number of pixels becomes n times the number of pixels of the observed image (107), the processing time required for image restoration becomes almost n times as compared with the processing of restoring the observed image (107) as it is. . On the other hand, in the configuration shown in FIG. 8, the total number of pixels of the image processed by the image restoration units (804) (805) (806) (807) (808) (809) is the same as the number of pixels of the observed image (107). Since they are almost equal, the processing time can be greatly shortened (about 1 / n) compared to the configuration shown in FIG.

  Note that as the image restoration units (804), (805), (806), (807), (808), and (809), the above-described iterative processing using the Blind method for estimating both the PSF and the latent image from the observed image is used as it is. it can.

  In addition, the boundary portion between the small regions (802) in FIG. 8 has two (one set) signals (FIGS. 3B and 3C) having values that change slowly as shown in FIG. By multiplying, it is possible to synthesize the boundary portion inconspicuously in the synthesis unit (810).

  As described above, the present embodiment is an image processing apparatus, which divides an input image into m (where m is an integer of 2 or more) horizontally long regions, and an image for each horizontally long region. The image recovery unit is configured to perform a recovery process, and an image composition unit that combines and outputs the result of the image recovery unit into a single image.

An image processing method includes: an image input step; an image division step into a plurality of horizontally long regions; an image recovery process for each horizontally long region; and an image recovery step. And the step of combining the result into one image and outputting it.
Thereby, it is possible to configure an image processing apparatus and an image processing method that perform image recovery of the entire image in a short time by simply creating a similar region based on the vertical coordinates of the observed image.

  In this embodiment, an image restoration process that approximates the difference in shooting distance and further shortens the processing time will be described.

  FIG. 9 shows the configuration of the image restoration process in this embodiment, and the processing time can be further shortened based on the image restoration process shown in FIG. As mentioned above, a distant view appears in the upper part of the image, and a close-up view appears in the lower part of the image. Therefore, if the image is divided into horizontally long strips, it is considered that subjects with approximately the same distance from the camera are shown. . At this time, it is considered that the blurring of the image in the region between the uppermost part and the lowermost part (intermediate part) is an intermediate blurring between the blurring of the uppermost image and the blurring of the lowermost image. Thus, the processing time can be shortened. In FIG. 9, as an example, a configuration in which the image is divided into six small image areas in the vertical direction is shown. However, the number of divisions is not limited to (6), and the number of divisions can be flexibly increased or decreased. Obviously, you may.

  As shown in FIG. 9, the observation image (801) is divided into horizontally long strips to form a small image area (802), and the PSF is considered to be constant in this small area. Subsequently, the latent image and the PSF (PSF-1 and PSF-6) are applied to the subregions at the top and bottom of the image by the image restoration unit (804) (809) by the iterative process using the Blind method described above. Ask.

  On the other hand, the intermediate part of the image is not subjected to the iterative process by the Blind method, but the uppermost PSF (PSF-1) and the lowermost PSF (PSF-) obtained by the PSF interpolation part (1002) described later. An interpolated PSF is generated from 6). That is, the processing time of the entire image can be shortened by not performing the iterative process for the intermediate part of the image.

  Thereafter, the final latent image is estimated by the final estimation unit (908) shown in FIG. 9 using the interpolated PSF and the observed image generated by the PSF interpolation unit (1002), which will be described later. The entire latent image (1007) is generated by combining the latent image at the bottom and the latent image at the bottom in the combining unit (810).

  FIG. 10 shows a configuration example of the PSF interpolation unit (1002) described above. The PSF interpolation unit (1002) interpolates PSFs (PSF-2 to PSF-5) corresponding to the small image areas from the input image top PSF (PSF-1) and image bottom (PSF-6). Is generated using weighted addition units (1101-2 to 1101-5) corresponding to the small image areas. The weighted adder (1101-2) includes multipliers (1102) (1103), a coefficient calculator (1104), and an adder (1105). Here, a coefficient k2 (1106-2) to be described later is set for the weighted addition unit (1101-2), and a coefficient (1-k2) generated by subtracting the coefficient k2 from 1.0 by the coefficient calculator (1104). ) And PSF-1 are multiplied by the multiplier (1102), and the result of multiplying the coefficient k2 (1106-2) and PSF-6 by the multiplier (1103) is added to the adder (1105). To obtain an interpolated PSF (PSF-2).

  That is, PSF-1 and PSF-6 are weighted and added according to the coefficient k2 to obtain an interpolated PSF (PSF-2). For the other small image areas, PSF-1 and PSF-6 are obtained by using weighted addition units (1101-3) (1101-4) (1101-5) having the same configuration as the weighted addition unit (1101-2). Weighted addition is performed according to each coefficient k3 (1106-3), coefficient k4 (1106-4), coefficient k5 (1106-5) to obtain an interpolated PSF (PSF-3 to PSF-5).

  Here, each of the coefficients k2 (1106-2) to k5 (1106-5) has a vertical coordinate of the center of gravity of the uppermost small region of 0.0 (reference) according to the vertical coordinate of the center of gravity of each small region of the image. And the internal ratio when the vertical coordinate of the center of gravity of the lowermost small area is 1.0. For example, when the observed image is divided into six small regions in the vertical direction and the number of vertical pixels in each small region is set to the same value as in the example shown in FIG. 9, the coefficient k2 = 0. 2 (= 1 / (6-1)), coefficient k3 = 0.4 (= 2 / (6-1)), coefficient k4 = 0.6 (= 3 / (6-1)), coefficient k4 = 0 .8 (= 4 / (6-1)).

  As described above, the present embodiment is an image processing apparatus, and an input image is input to a horizontal first region at the top of the image, a horizontal second region at the bottom of the image, a first region, and a second region. A region dividing unit that divides the central portion of the image into m (where m is an integer of 1 or more) horizontally long third regions, and a first image that performs image restoration processing on the first region A recovery unit, a second image recovery unit that performs image recovery processing on the second region, a first point spread function (PSF) output from the first image recovery unit, and a second image recovery unit A point spread function interpolation unit for obtaining a third point spread function using the second point spread function output from the second point spread function, and a third region for performing image restoration processing on the third region using the third point spread function. 3 image restoration units, and an image synthesis unit that synthesizes and outputs the results of the first, second, and third image restoration units into one image. To do.

  The image processing method may further include a step of inputting an image, an image having a horizontally long first area at the top of the image, a horizontally long second area at the bottom of the image, and a first area and a second area. Dividing the middle portion of the image into m (where m is an integer of 1 or more) horizontally long third regions, performing image restoration processing on the first region, and the second region The first point spread function (PSF) output from the step of performing the image restoration process, the step of performing the image restoration process on the first area, and the step of performing the image restoration process on the second area A step of obtaining a third point spread function using the second point spread function, a step of performing image restoration processing on the third region using the third point spread function, 2. The result of image restoration processing for the third area is displayed as one image. And a step of combining and outputting the image.

  Thus, in addition to simply creating a similar region based on the vertical coordinates of the observed image, it is possible to further improve the image recovery of the entire image by omitting the iterative process in the middle portion except the top and bottom of the image. An image processing apparatus and an image processing method that can be performed in a short time can be configured.

  In this embodiment, the procedure of image restoration processing will be described. In order to solve the above-described problem, in this embodiment, an image restoration process having the following procedure is used.

  FIG. 11 is a flowchart showing an example of the procedure of the image restoration process according to the present embodiment, in which the operation of each unit in the configuration of the image processing apparatus shown in FIG. 1 is associated with the processing step showing each procedure. . In FIG. 11, image restoration processing is started from step (1201), an image to be processed (observation image) is input at step (1202), and the observation image is divided into m small regions at step (1203). To do. Subsequently, in step (1204), frequency distribution analysis is performed for each small region, and in step (1205), small regions having similar frequency distribution analysis results are integrated to create n similar regions.

  In step (1206), an image is generated in which areas other than the similar area are filled with a predetermined value, for example, black (luminance value = 0) for each similar area. In step (1207), general image restoration processing is performed on each of the n similar regions for the filled image, and n images are combined into one latent image in step (1208). In step (1209), an image (latent image) is output, and in step (1210), the image restoration process is terminated. The detailed operation of each step described here is the same as the operation of each unit described with reference to FIGS.

  As described above, this embodiment is an image processing method, which includes a step of inputting an image, a step of dividing the image into a plurality of first regions, obtaining a frequency component distribution for each first region, and a frequency component. Calculating a similarity between distributions, integrating a first region having a similarity higher than a predetermined value as a second region, and generating an image in which a region other than the second region is filled with a predetermined value for each second region A step of performing image restoration processing for each second region on the image obtained in the step of generating a filled image, and a step of combining and outputting the result of the step of performing image restoration processing into a single image Consists of.

  By using the procedure as described above, the procedure of the image restoration process (PSF estimation process) for separating the latent image and the PSF for each divided small area becomes unnecessary, and the estimation of the PSF becomes unstable. Or the problem of divergence in the middle of a calculation or a long time required.

  The image processing apparatus described with reference to FIGS. 5 to 7 of the second embodiment, the image processing apparatus described with reference to FIG. 8 of the third embodiment, and the images described with reference to FIGS. 9 and 10 of the fourth embodiment. Similarly, in the processing apparatus, since the operation of each unit in each configuration can be easily associated with the processing step, the illustration and description of the processing step indicating each procedure are omitted.

  Although the embodiments of the present invention have been described above, the above embodiments can also be realized by software program codes that realize the functions. In this case, a storage medium in which the program code is recorded is provided to the system or apparatus, and the computer (or CPU or MPU) of the system or apparatus reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the program code itself and the storage medium storing it constitute the present invention. As a storage medium for supplying such program code, for example, a flexible disk, CD-ROM, DVD-ROM, hard disk, optical disk, magneto-optical disk, CD-R, magnetic tape, nonvolatile memory card, ROM Etc. are used.

  Also, based on the instruction of the program code, an OS (operating system) running on the computer performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing. Also good. Further, after the program code read from the storage medium is written in the memory on the computer, the computer CPU or the like performs part or all of the actual processing based on the instruction of the program code. Thus, the functions of the above-described embodiments may be realized.

  Furthermore, by distributing the program code of the software that realizes the functions of the embodiments via a network, it is stored in a storage circuit such as a hard disk or memory of a system or apparatus or a storage medium such as a CD-RW or CD-R. The program code stored in the storage circuit or the storage medium may be read out and executed by the computer (or CPU or MPU) of the system or apparatus during storage. The described software can be realized by a wide range of programs or script languages such as assembler, C / C ++, perl, Shell, PHP, Java (registered trademark), and the like.

  In the above embodiment, hardware, software, and firmware may be combined to realize the function.

  Further, the above-described embodiment shows control lines and information lines that are considered necessary for explanation, and does not necessarily show all control lines and information lines on the product.

  The present invention is not limited to the above-described embodiments, and includes various modifications. The above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Further, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

DESCRIPTION OF SYMBOLS 101 ... Area division part 102 ... Frequency component distribution analysis part 103 ... Similar area integration part 104 ... Filling part 105 ... Image restoration part 106 ... Composition part 107 ... Input image ( Observation image)
110 ... Frequency component distribution for each small area image 112 ... Image filled with areas other than small areas with similar frequency component distribution 113 ... Output image (latent image)

Claims (5)

An area dividing unit that divides an input image into m (where m is an integer of 2 or more) first areas;
A frequency component distribution analyzer for obtaining a frequency component distribution for each of the first regions;
Calculating a similarity between the frequency component distributions, and combining the first region with the similarity higher than a predetermined value as a second region;
A fill unit for generating an image in which a region other than the second region is filled with a predetermined value for each second region;
An image recovery unit that performs image recovery processing for each of the second regions on the image obtained by the filling unit;
An image processing apparatus comprising: an image synthesis unit that synthesizes and outputs the result of the image restoration unit into a single image. An image processing system using the image processing apparatus according to claim 1,
A first image enlarging unit that increases the number of pixels of the photographed image to enlarge to a first image size, and an output of the first image enlarging unit is input to the image processing device to perform image restoration processing; An image processing system, comprising: a second image enlargement unit that increases the number of pixels of the output image of the image processing device and enlarges the output image to a second image size. The image processing system according to claim 2,
An enlargement ratio setting unit that controls the first image size so that the power of the aliasing distortion component generated by the first image enlargement unit is equal to or less than the power of the signal component attenuated by the first image enlargement unit. Image processing system. An area dividing unit that divides an input image into m (where m is an integer of 2 or more) horizontally long areas;
An image recovery unit that performs image recovery processing for each of the horizontally long regions;
An image processing apparatus comprising: an image synthesis unit that synthesizes and outputs the result of the image restoration unit into a single image. Inputting an image;
The image is divided into a horizontal first region at the top of the image, a horizontal second region at the bottom of the image, and a middle portion of the image between the first region and the second region, where m is 1 or more. An integer) number of horizontally long third regions,
Performing image restoration processing on the first region;
Performing image restoration processing on the second region;
A first point spread function (PSF) output from the step of performing image restoration processing on the first region, and a second point spread output from the step of performing image recovery processing on the second region. Using the function to determine a third point spread function;
Performing image restoration processing on the third region using the third point spread function;
Combining and outputting the result of performing the image restoration processing on the first, second, and third areas into a single image;
An image processing method comprising:

Images