JP2011164967A - Image processor and image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device and circuit for performing two-dimensional super-resolution processing and three-dimensional super-resolution processing at low cost. <P>SOLUTION: The image processor includes: a first high resolution converting part for performing high resolution conversion of a low resolution input image to generate a high resolution input image; a low resolution converting part for making a high resolution input image or high resolution image to be an estimated high resolution image and performing low resolution conversion of the estimated high resolution image to generate a low resolution image; a difference image generating part for generating a difference image from the low resolution image and the low resolution input image; a second high resolution converting part for performing high resolution conversion of the difference image to generate a difference high resolution image; a high resolution image generating part for generating a high resolution image obtained by the difference high resolution image and the estimated high resolution image; and a control part for controlling whether to make the high resolution image generating part repeatedly generate a high resolution image on the basis of the estimated high resolution image or to output the estimated high resolution image as a super-resolution image to finish the processing. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing apparatus and an image processing method.

  Shooting devices such as a video camera and a digital still camera are spreading, and it is becoming possible to play back and display images shot with them on a display device such as a home TV, or to distribute images easily over the Internet. Here, since the resolution of video is generally different between the shooting device and the display device, when displaying the video shot with various shooting devices on various display devices, up-conversion and video conversion are performed. It is necessary to apply down-conversion. For example, when displaying a QVGA-size image taken with a mobile phone camera on a high-definition television, it is necessary to perform a significant up-conversion and increase the number of pixels in each frame in order to view the image in an easy-to-view size.

  First, as a general method for up-converting video, the nearest neighbor method that repeatedly copies the original pixel by the number of enlargement magnifications, the linear interpolation method that linearly interpolates between the surrounding pixels, or the bicubic interpolation method. There is. However, these up-conversion methods enlarge in the range of the spatial frequency of the original video, and thus have a detrimental effect that the enlarged video is blurred or the edges become thick.

  In contrast to such an up-conversion method, there is a process called super-resolution as a method for reproducing a band exceeding the spatial frequency of the original video and generating a high-resolution up-converted video. Super-resolution processing can be broadly divided into two-dimensional super-resolution and three-dimensional super-resolution. In 2D super-resolution, it is assumed that the original video is generated by reducing the up-converted video of the target size, this reduction process is defined, and the original video is inversely transformed. Estimate signals in a frequency band that exceeds the spatial frequency of the original video to obtain a sharp up-converted video. In 3D super-resolution, high resolution video that actually exceeds the spatial frequency of the original frame can be obtained by referring to and interpolating information that does not exist in the original frame from multiple frames adjacent to the original video frame. Generate (for example, Patent Document 1).

  However, in recent years, there has been a growing need for shooting high-definition still images with video cameras and shooting high-definition quality videos with digital still cameras. And high-definition still images are becoming mainstream.

JP2008-140012

  In conventional image processing devices, all of the optical system, image sensor, and image processing circuit of the imaging device have high resolution in order to capture both high-definition movies and high-definition still images using a single imaging device. However, there is a problem that the size of the apparatus increases and the cost increases.

  In addition, when a conventional image processing apparatus uses a low-resolution optical system and an image sensor and achieves high resolution by super-resolution processing, two-dimensional super-resolution is required to increase the resolution of both still images and moving images. When both of the three-dimensional super-resolutions are mounted on an image processing circuit, there is a problem that the image processing circuit becomes large and expensive.

  For this reason, super-resolution processing can be implemented with built-in software to reduce costs, but video equipment that must process 30 to 120 frames per second must meet the requirements for processing speed (frame rate). Was difficult.

  The present invention has been made to solve such problems, and an object thereof is to provide an apparatus and a circuit that perform two-dimensional super-resolution processing and three-dimensional super-resolution processing at low cost.

  An image processing apparatus according to the present invention includes a first high-resolution conversion unit that generates a high-resolution input image by converting a low-resolution input image into a high-resolution image, and a high-resolution input image generated by the first high-resolution conversion unit. Alternatively, a high-resolution image is used as an estimated high-resolution image, a low-resolution conversion unit that generates a low-resolution image by converting the estimated high-resolution image to low resolution, and the low-resolution image generated by the low-resolution conversion unit and the low-resolution input A difference image generation unit that generates a difference image from the image, a second high resolution conversion unit that generates a difference high resolution image by performing high resolution conversion on the difference image generated by the difference image generation unit, and the second A high-resolution image generation unit that generates the high-resolution image obtained from the differential high-resolution image generated by the high-resolution conversion unit and the estimated high-resolution image; and the high-resolution image based on the estimated high-resolution image. Or generate iteratively high resolution images to degrees image generation unit, or the estimated high resolution image and a control unit that controls whether to terminate with a super-resolution image.

  According to the image processing apparatus of the present invention, it is possible to provide an apparatus and a circuit that perform two-dimensional super-resolution processing and three-dimensional super-resolution processing at a low cost.

It is a flowchart which shows an example of the image processing method which concerns on Embodiment 1 of this invention. It is a block block diagram (two-dimensional super-resolution mode) which shows an example of the image processing apparatus which concerns on Embodiment 1 of this invention. It is a block block diagram (three-dimensional super-resolution mode) which shows an example of the image processing apparatus which concerns on Embodiment 1 of this invention. It is explanatory drawing which shows an example of the definition of high resolution conversion and low resolution conversion in the image processing method which concerns on Embodiment 1 of this invention. It is a flowchart (1st partial change example) which shows an example of the image processing method which concerns on Embodiment 1 of this invention. It is a flowchart (2nd partial modification) which shows an example of the image processing method which concerns on Embodiment 1 of this invention. It is a block block diagram which shows an example of the image processing apparatus which concerns on Embodiment 2 of this invention. It is a block block diagram which shows an example of the image processing apparatus which concerns on Embodiment 3 of this invention. It is a block block diagram which shows an example of the image processing apparatus which concerns on Embodiment 4 of this invention. It is explanatory drawing which shows an example of the motion area | region which the motion detection part of the image processing apparatus which concerns on Embodiment 5 of this invention detected. It is a block block diagram which shows an example of the image processing apparatus which concerns on Embodiment 5 of this invention. It is a flowchart (example of a partial change of motion detection) which shows an example of the three-dimensional super-resolution process in the image processing method which concerns on Embodiment 5 of this invention. It is a block block diagram which shows an example of the image processing apparatus which concerns on Embodiment 6 of this invention. It is a flowchart which shows an example of the three-dimensional super-resolution process in the image processing method which concerns on Embodiment 6 of this invention. It is explanatory drawing which shows the relationship from the optical axis center of the image input system of the image processing apparatus which concerns on Embodiment 7 of this invention, and MTF. It is explanatory drawing which shows the relationship between the distance from the optical axis center of the image input system of the image processing apparatus which concerns on Embodiment 7 of this invention, and the correction coefficient of a conversion parameter.

  The image processing apparatus and the image processing method according to the present invention generate a high-resolution image by performing super-resolution processing for increasing the resolution of a low-resolution image. In the image processing apparatus and the image processing method of the present invention, desired super-resolution processing can be performed by specifying and switching between the two-dimensional super-resolution processing and the three-dimensional super-resolution processing in the super-resolution mode. The super-resolution mode is designated by the two-dimensional super-resolution mode or the three-dimensional super-resolution mode.

  The super-resolution processing by the image processing apparatus and the image processing method of the present invention is a common hardware configuration, and switches between two-dimensional super-resolution processing and three-dimensional super-resolution processing, and controls to operate in a partly different procedure. By performing this, a desired super-resolution process is realized. Hereinafter, super-resolution processing by the image processing apparatus and image processing method of the present invention will be described. In the description of the image processing apparatus and the image processing method of the present invention, for example, a low-resolution input image with an image number n is represented as f (n), and a specific numerical value such as an image number n = 1. When a number is given in Fig. 1, it is described as f1 without parentheses.

Embodiment 1 FIG.
In Embodiment 1 of the present invention, an image processing apparatus and an image processing method for performing two-dimensional super-resolution processing and three-dimensional super-resolution processing will be described.

  FIG. 1 is a flowchart showing an example of an image processing method according to Embodiment 1 of the present invention. In the figure, steps ST201 to ST207 are common processing steps for performing two-dimensional super-resolution processing and three-dimensional super-resolution processing, and steps ST208 to ST214 branched at step ST207 are processing steps for performing two-dimensional super-resolution processing. Steps ST302 to ST314 indicate processing steps for performing a three-dimensional super-resolution process. The flow chart of FIG. 1 includes a loop for repeatedly generating super-resolution images corresponding to each low-resolution input image that is continuously input. Hereinafter, based on the flowchart of FIG. 1, an image processing method for performing two-dimensional super-resolution processing and three-dimensional super-resolution processing according to Embodiment 1 of the present invention will be described below. This is explained in association with

2 and 3 are block configuration diagrams showing an example of the image processing apparatus according to Embodiment 1 of the present invention. FIG. 2 shows the processing state and image name for performing the two-dimensional super-resolution processing when the image number n = 1 in the two-dimensional super-resolution mode. FIG. 3 has the same block configuration as FIG. 2, but shows the state and image name of the 3D super-resolution processing when the image number n = 1 in the 3D super-resolution mode. It is a thing. In the figure, a low resolution input image storage unit 11a stores an input image with a low resolution (hereinafter referred to as a low resolution input image) g (n). The first high resolution conversion unit 12a performs high resolution conversion (T ⁻¹ ) on the low resolution input image g (n) stored in the low resolution input image storage unit 11a to generate a high resolution input image. The estimated high resolution image storage unit 13a stores the high resolution input image or the high resolution image f ′ (n) generated by the first high resolution conversion unit 12a as the estimated high resolution image f (n). The low resolution conversion unit 14 performs low resolution conversion (T) on the estimated high resolution image f (n) stored in the estimated high resolution image storage unit 13a to generate a low resolution image g ′ (n). The low resolution image storage unit 15 stores the low resolution image g ′ (n) generated by the low resolution conversion unit 14. The difference image generation unit (subtraction unit) 16 calculates a difference from the low resolution input image g (n) stored in the low resolution input image storage unit 11a and the low resolution image g ′ (n) stored in the low resolution image storage unit 15. The image g ″ (n) is generated. The difference image storage unit 17a stores the difference image g ″ (n) generated by the difference image generation unit (subtraction unit) 16. The index value calculation unit 18 calculates the index value e from the low resolution input image g (n) stored in the low resolution input image storage unit 11a and the low resolution image g ′ (n) stored in the low resolution image storage unit 15. To do. The second high resolution conversion unit 19 performs high resolution conversion (T ⁻¹ ) on the difference image g ″ (n) stored in the difference image storage unit 17a to generate a difference high resolution image j (n). The resolution image storage unit 20 stores the differential high resolution image j (n) generated by the second high resolution conversion unit 19. The high resolution image generation unit (addition unit) 21 includes the estimated high resolution image storage unit 13a. A high resolution image f ′ (n) is generated from the stored estimated high resolution image f (n) and the differential high resolution image j (n) stored in the differential high resolution image storage unit 20. Control unit (not shown) Is to cause the high-resolution image generation unit (adding unit) 21 to repeatedly generate the high-resolution image f ′ (n) based on the estimated high-resolution image f (n) stored in the estimated high-resolution image storage unit 13a, Alternatively, the estimated high resolution image f (n) is output as a super resolution image. Control whether to exit.

Further, the control unit performs high resolution conversion (T ⁻¹ ) and low resolution of the first high resolution conversion unit 12 a and the second high resolution conversion unit 19 based on the index value e calculated by the index value calculation unit 18. The update of the conversion definition (conversion parameter) of the low resolution conversion (T) of the conversion unit 14 is determined.

  Here, the low resolution input image storage unit 11a includes a storage unit 111a for storing the low resolution input image, a first high resolution conversion unit 12a as an output destination of the low resolution input image stored by the storage unit 111a, and a differential image generation. Switch (SWa) 112a for switching to one of the units (subtraction unit) 16. The estimated high-resolution image storage unit 13a also stores a storage unit 132 that stores the estimated high-resolution image, and a first high-resolution conversion unit 12a or a high-resolution image generator as an input source of the estimated high-resolution image to be stored in the storage unit 132. Switch (SWb) 131a for switching to any one of the unit (adding unit) 21, the output unit for the estimated high resolution image stored in the storage unit 132, the low resolution conversion unit 14, the high resolution image generating unit (adding unit) 21, and the device Switch (SWc) 133a for switching to any one of the external outputs. The control unit executes a desired super-resolution process by controlling the processing path by switching the input source and output destination of the image by these switches.

  First, regarding the image processing apparatus and the image processing method for performing the two-dimensional super-resolution processing according to Embodiment 1 of the present invention, the flowchart of the image processing method shown in FIG. The three-dimensional super-resolution processing will be described in association with the block configuration diagram of the image processing apparatus shown in FIG.

(1) Common processing steps for performing two-dimensional super-resolution processing and three-dimensional super-resolution processing First, two-dimensional super-resolution processing and three-dimensional super-resolution processing in the flowchart of the image processing method shown in FIG. 1 are performed. Steps ST201 to ST207, which are common processing steps, will be described. In step ST201, high resolution conversion (T ⁻¹ ) performed by the image number n, the first high resolution conversion unit 12a and the second high resolution conversion unit 19, and low resolution conversion (T) performed by the low resolution conversion unit 14 are performed. Initialize conversion definition (conversion parameter). For example, the image number n is set to 1 as an initial value.

  In step ST202, the low resolution input image g (n) of the image number n is stored in the low resolution input image storage unit 11a.

In step ST203, the high resolution conversion (T ⁻¹ ) is applied to the low resolution input image g (n) by the first high resolution conversion unit 12a to generate a high resolution input image, and the generated high resolution input image is displayed. The estimated high resolution image storage unit 13a stores the estimated high resolution image f (n) as an initial image. At this time, the control unit controls the switch (SWa) 112a and the switch (SWb) 131a to secure and store the path of the image data.

The high-resolution conversion (T ⁻¹ ) in the first high-resolution conversion unit 12 a can use, for example, the back projection method described in Patent Document 1, and the formula for generating the estimated high-resolution image f (n) is as follows: It can be expressed as (Equation 1). Here, p represents a parameter called a backprojection kernel, and “x” in (Expression 1) represents a convolution operator.

  In the first embodiment of the present invention, the case of using the back projection method and the projection method as the forward conversion thereof as the super-resolution processing method has been described. However, the present invention is not limited thereto, and for example, the LR method (Lucy-Richardson method). Or, a general super-resolution processing method such as a Bayes method using conditional probabilities may be used.

In step ST204, the low resolution conversion unit 14 applies low resolution conversion (T) to the estimated high resolution image f (n) to generate a low resolution image g ′ (n) and stores it in the low resolution image storage unit 15. . At this time, the controller controls the switch (SWc) 133a to ensure and store the path of the image data. The low-resolution conversion (T) in the low-resolution conversion unit 14 is a positive conversion for the high-resolution conversion (T ⁻¹ ) in the first high-resolution conversion unit 12a. The formula for generating the low resolution image g ′ (n) can be expressed as the following (Formula 2). Here, h represents a blurring process in the optical system of the camera, and for example, a point spread function or a Gaussian function can be used.

  In step ST205, the low-resolution image g ′ (n) is subtracted from the low-resolution input image g (n) by the subtractor (difference image generation unit) 16 to generate a difference image g ″ (n), and the difference image storage unit 17a. At this time, the control unit controls the switch (SWa) 112a to secure and store the path of the image data. The formula for generating the difference image g ″ (n) is the following (formula 3). It can be expressed as

  In step ST206, a predetermined index value e is calculated for the low resolution input image g (n) and the low resolution image g ′ (n), and the similarity (correlation) between these images is quantified. As the index value e, for example, if it is a sum of absolute differences, it can be expressed as the following (Equation 4). In the following equation, the size of the low resolution image is U × V, u and v are coordinates, and the image number n is not shown.

  In the first embodiment of the present invention, since the case where the sum of absolute differences is used as the index value has been described, the block configuration diagram of FIG. 2 shows that the index value is calculated from the difference image g ″ (n). However, the index value calculation unit 18 may calculate an index value from the low-resolution input image g (n) and the low-resolution image g ′ (n). A general index value that can quantify the difference between two images, such as a sum, least squares method, or PSNR (Peak Signal-to-Noise Ratio), may be used.

  In step ST207, it is determined whether or not the designation of the super-resolution mode is the two-dimensional super-resolution mode. When the two-dimensional super-resolution mode is designated (Yes), the process proceeds from step ST208 to the two-dimensional super-resolution process in step ST214. On the other hand, when the two-dimensional super-resolution mode is not designated (the three-dimensional super-resolution mode is designated) (No), the process proceeds from step ST302 to the three-dimensional super-resolution process of step ST314. At this time, if the two-dimensional super-resolution mode is designated, the control unit proceeds to the two-dimensional super-resolution processing. If the three-dimensional super-resolution mode is designated, the control unit performs the three-dimensional super-resolution. Control to proceed to processing.

(2) Processing Step for Performing Two-Dimensional Super-Resolution Processing Next, from step ST201, which is a common processing step for performing two-dimensional super-resolution processing and three-dimensional super-resolution processing in the flowchart of the image processing method shown in FIG. Subsequent to step ST207, steps ST208 to ST214, which are processing steps for performing the two-dimensional super-resolution processing after step ST207, will be described.

In step ST208, the high-resolution conversion (T ⁻¹ ) is applied to the difference image g ″ (n) by the second high-resolution conversion unit 19 to generate the difference high-resolution image j (n) and the difference high-resolution image storage. This is stored in the unit 20. An equation for generating the differential high resolution image j (n) can be expressed as the following (Expression 5), where p is the high resolution conversion (T ⁻¹ ) of (Expression 1). This is the parameter described in the calculation.

  In step ST209, the difference high resolution image j (n) is added to the estimated high resolution image f (n) by the adder (high resolution image generating unit) 21 to generate a high resolution image f ′ (n), and the estimated high resolution image The image f (n) is re-stored and updated in the estimated high-resolution image storage unit 13a. At this time, the control unit controls the switch (SWc) 133a and the switch (SWb) 131a to secure and update the image data path. The estimated high resolution image f (n) stored previously is overwritten in the estimated high resolution image storage unit 13a with the high resolution image f '(n). An equation for generating the high resolution image f ′ (n) can be expressed as the following (Equation 6).

In step ST210, the similarity is determined based on whether or not the index value e calculated in step ST206 satisfies a predetermined convergence condition. Here, the predetermined convergence condition is, for example, whether or not the index value e is equal to or less than a predetermined threshold value. That is, if it is determined that the index value e is equal to or less than the predetermined threshold value and the predetermined convergence condition is satisfied (Yes), the process proceeds to step ST212, and the predetermined convergence condition is not satisfied (No). If determined to be, the process proceeds to step ST211. At this time, the control unit refers to the index value e, and determines whether the conversion definition (conversion parameter) of high resolution conversion (T ⁻¹ ) or low resolution conversion (T) is appropriate depending on whether or not a predetermined convergence condition is satisfied. Determine.

If it is determined in step ST210 that the index value e does not satisfy the predetermined convergence condition, it is assumed that a sufficient super-resolution effect has not yet been obtained, and in step ST211, the first high-resolution conversion unit 12a The conversion definition (conversion parameter) of the high resolution conversion (T ⁻¹ ) performed by the second high resolution conversion unit 19 and the low resolution conversion (T) performed by the low resolution conversion unit 14 is corrected. When the index value e is equal to or less than a predetermined threshold value and it is determined that the predetermined convergence condition is satisfied, the high resolution conversion performed by the first high resolution conversion unit 12a and the second high resolution conversion unit 19 is performed. (T ⁻¹ ), the low resolution conversion (T) conversion definition (conversion parameter) performed by the low resolution conversion unit 14 is not corrected. As described above, after updating the conversion definition (conversion parameter) of the high resolution conversion and the low resolution conversion in step ST211, the processing steps from step ST202 to step ST210 of the two-dimensional super-resolution processing are performed according to a predetermined convergence condition. The two-dimensional super-resolution process for the low-resolution input image g (n) is repeated until it is satisfied.

Here, definitions of high resolution conversion (T ⁻¹ ) and low resolution conversion (T) will be described. FIG. 4 is an explanatory diagram showing an example of definitions of high resolution conversion (T ⁻¹ ) and low resolution conversion (T) in the image processing method according to Embodiment 1 of the present invention. Here, description will be made assuming that the image number n = 1. As shown in the figure, for example, the low resolution conversion T can be defined by a point spread function. (A) shows a state in which an impulse signal exists in the estimated high-resolution image f1. Further, (b) shows an impulse response observed in the low resolution input image g1. For example, when the low-resolution conversion T is defined as this point spread function, the correction of the conversion definition in step ST208 has a wider distribution state with respect to ± 3σ as the standard deviation σ as the impulse response of (b). This is done by widening or narrowing the distribution to a narrower range. The standard deviation σ indicates a distribution range that covers 90% or more of the distribution. Therefore, by correcting the definition of the low resolution conversion T so as to extend to a wider distribution state, the definition of the corresponding high resolution conversion T- ¹ is corrected so as to further enhance the correction. Further, by correcting the definition of the low resolution conversion T so as to narrow the distribution state to a narrower range, the definition of the corresponding high resolution conversion T- ¹ is corrected so as to weaken the correction.

  If it is determined in step ST210 that the index value e satisfies a predetermined convergence condition, it is assumed that a sufficient super-resolution effect is obtained, and in step ST212, the estimated high-resolution image storage unit 13a is obtained. The estimated high-resolution image f (n) stored in is output as a super-resolution image (two-dimensional super-resolution image). By outputting the two-dimensional super-resolution image, the two-dimensional super-resolution processing for the low-resolution input image g (n) is completed. At this time, the control unit controls the switch (SWc) 133a to secure and output the path of the image data.

  In step ST213, it is determined whether there is an image after the low resolution input image g (n). If there is a subsequent image and does not end (No), the process proceeds to step ST214, and if there is no subsequent image and ends (Yes). Then, the two-dimensional super-resolution processing is finished. At this time, the control unit determines whether there is an image after the low-resolution input image g (n) and continues or ends the two-dimensional super-resolution processing.

  In step ST214, the image number n is updated. After updating the image number n, the two-dimensional super-resolution process for the new low-resolution input image g (n) is started from step ST202.

  As described above, by executing the processing steps from step ST201 to step ST214 in FIG. 1, two-dimensional super-resolution images f1, f2,... Can be generated from the low resolution input images g1, g2,.

In the first embodiment of the present invention, in the convergence condition determination of the two-dimensional super-resolution process in step ST210, it is determined whether or not the index value e has converged, and this determination is performed in the second-dimensional super-resolution process. Correction processing of conversion definition (conversion parameter) of high resolution conversion (T ⁻¹ ) performed by one high resolution conversion unit 12 a, second high resolution conversion unit 19, and low resolution conversion (T) performed by the low resolution conversion unit 14 However, the present invention is not limited to this. For example, FIGS. 5 and 6 are flowcharts (first and second partial modification examples) showing an example of the image processing method according to the first embodiment of the present invention. In step ST210 and step ST211 in FIG. 1, as shown in FIG. 5, the index value e is used only for determining whether or not the high resolution conversion and low resolution conversion conversion definition (conversion parameter) correction processing in step ST211 is necessary. In the added step ST215, as an end determination of the iterative process of the two-dimensional super-resolution processing of the low resolution input image g (n), it is determined whether the iterative process has been performed a predetermined number of times by counting the number of repetitions. May be. Further, as shown in FIG. 6, when the index value e converges (Yes) in FIG. 5 in the convergence condition determination of the two-dimensional super-resolution processing in step ST210 (Yes), the low-resolution input image g (n) The process may be advanced to step ST212 instead of step ST215 so that the iterative process of the two-dimensional super-resolution process can be completed even if it does not reach the predetermined number of times.

(3) Processing Steps for Performing 3D Super-Resolution Processing Steps ST201 to ST201 are common processing steps for performing 2D super-resolution processing and 3D super-resolution processing in the flowchart of the image processing method shown in FIG. Subsequent to ST207, steps ST302 to ST314, which are processing steps for performing the three-dimensional super-resolution processing after step ST207, will be described.

  In step ST302, similarly to step ST202 for the low resolution input image g (n) of image number n, the low resolution input image g (n + 1) of the image number (n + 1) is stored in the low resolution input image storage unit 11a. The low resolution input image g (n) previously stored in step ST202 is overwritten in the low resolution input image storage unit 11a with the low resolution input image g (n + 1).

In step ST303, similarly to step ST203 for the estimated high resolution image f (n) of the image number n, the first high resolution conversion unit 12a performs high resolution conversion (T ⁻¹ ) on the low resolution input image g (n + 1). The high-resolution input image is generated by application, and the generated high-resolution input image is stored in the estimated high-resolution image storage unit 13a as an initial image of the estimated high-resolution image f (n + 1). At this time, the control unit controls the switch (SWa) 112a and the switch (SWb) 131a to secure and store the path of the image data. The estimated high resolution image f (n) previously stored in step ST203 is overwritten in the estimated high resolution image storage unit 13a with the estimated high resolution image f (n + 1).

In step ST308, similarly to step ST208 for the differential high-resolution image j (n) with the image number n, the high-resolution conversion (T ⁻¹ ) is applied to the differential image g ″ (n) by the second high-resolution conversion unit 19. Then, a differential high resolution image j (n) is generated and stored in the differential high resolution image storage unit 20.

  In step ST309, the high-resolution image f ′ (n + 1) is generated by adding the difference high-resolution image j (n) to the estimated high-resolution image f (n + 1) by the adder (high-resolution image generation unit) 21 to generate the estimated high-resolution image. The image f (n + 1) is re-stored and updated in the estimated high-resolution image storage unit 13a. At this time, the control unit controls the switch (SWc) 133a and the switch (SWb) 131a to secure and update the image data path. The estimated high resolution image f (n + 1) previously stored in step ST303 is overwritten in the estimated high resolution image storage unit 13a with the high resolution image f '(n + 1). An equation for generating the high resolution image f ′ (n + 1) can be expressed as the following (Equation 7).

In step ST310, similar to step ST210, the degree of similarity is determined based on whether or not the index value e calculated in step ST206 satisfies a predetermined convergence condition. Here, the predetermined convergence condition is, for example, whether or not the index value e is equal to or less than a predetermined threshold value. That is, if it is determined that the index value e is equal to or less than the predetermined threshold value and the predetermined convergence condition is satisfied (Yes), the process proceeds to step ST312 and the predetermined convergence condition is not satisfied (No). If determined to be, the process proceeds to step ST311. At this time, the control unit refers to the index value e, and determines whether the conversion definition (conversion parameter) of high resolution conversion (T ⁻¹ ) or low resolution conversion (T) is appropriate depending on whether or not a predetermined convergence condition is satisfied. Determine.

If it is determined in step ST310 that the index value e does not satisfy the predetermined convergence condition, it is assumed that a sufficient super-resolution effect has not yet been obtained, and in step ST311, as in step ST211, the first The conversion definition (conversion parameter) of the high resolution conversion (T ⁻¹ ) performed by the high resolution conversion unit 12 a, the second high resolution conversion unit 19, and the low resolution conversion (T) performed by the low resolution conversion unit 14 is corrected. When the index value e is equal to or less than a predetermined threshold value and it is determined that the predetermined convergence condition is satisfied, the high resolution conversion performed by the first high resolution conversion unit 12a and the second high resolution conversion unit 19 is performed. (T ⁻¹ ), the low resolution conversion (T) conversion definition (conversion parameter) performed by the low resolution conversion unit 14 is not corrected. In the three-dimensional super-resolution process, the process is not repeated as in the two-dimensional super-resolution process, and the process proceeds to step ST312 regardless of whether or not the index value e satisfies a predetermined convergence condition.

  In step ST312, the estimated high resolution image f (n + 1) stored in the estimated high resolution image storage unit 13a is output as a super resolution image (three-dimensional super resolution image). By outputting the three-dimensional super-resolution image, the three-dimensional super-resolution processing for the low-resolution input image g (n) is completed. At this time, the control unit controls the switch (SWc) 133a to secure and output the path of the image data.

  In step ST313, it is determined whether there is an image after the low resolution input image g (n + 1). If there is a subsequent image and does not end (No), the process proceeds to step ST314, and if there is no subsequent image and ends (Yes). Then, the three-dimensional super-resolution processing is finished. At this time, the control unit determines whether there is an image after the low-resolution input image g (n + 1), and continues or ends the three-dimensional super-resolution processing.

  In step ST314, the image number n is updated. After updating the image number n, the three-dimensional super-resolution process for the new low-resolution input image g (n) is started from step ST202.

  Thus, by executing the processing steps from step ST201 to step ST207 and from step ST302 to step ST314 in FIG. 1, three-dimensional super-resolution images f2, f3,... Are generated from the low resolution input images g1, g2,. can do.

  In the first embodiment of the present invention, when the end of the two-dimensional super-resolution process or the three-dimensional super-resolution process is determined and not ended, the conversion definition modified so far by updating the image number n is updated. (Conversion parameter) is applied as it is, and the case where 2D super-resolution processing or 3D super-resolution processing is repeated has been shown. However, the present invention is not limited to this, and the conversion definition for each low-resolution input image g (n) is first set. It may be reinitialized.

  Further, in the first embodiment of the present invention, the index value e in step ST206 is calculated as a difference image g ″ () that calculates the difference between the low resolution input image g (n) and the low resolution image g ′ (n) in step ST205. n) is performed immediately after (or simultaneously with) the generation, but if the index value e is calculated from the difference image g ″ (n), as long as the difference image g ″ (n) is maintained. Even if the low-resolution input image g (n) and the low-resolution image g ′ (n) are updated, there is no effect, so that the convergence condition is determined from the index value e from the current processing position in FIG. You may calculate in any position before.

  In the first embodiment of the present invention, the convergence condition is determined from the index value e and the conversion definition is corrected immediately before the output of the two-dimensional super-resolution image f (n) or the three-dimensional super-resolution image f (n + 1). However, if the end determination is separately performed in the two-dimensional super-resolution processing, both the three-dimensional super-resolution processing and the current processing position in FIG. The conversion definition may be corrected by determining at any position.

  Further, in Embodiment 1 of the present invention, the accumulation of the low resolution input image g (n + 1) in step ST302 in the three-dimensional super-resolution processing is the difference image g in step ST205 referring to the low resolution input image g (n). If it is after the generation of “(n)”, the low-resolution image g ′ (step ST204) referring to the accumulation of the estimated high-resolution image f (n) restricted by the accumulation of the estimated high-resolution image f (n + 1) in step ST303. Since it is after the generation of n), it may be performed at any position up to the current processing position of FIG.

  Thus, in the image processing method according to the first embodiment of the present invention, as shown in FIG. 1, the two-dimensional super-resolution processing and the three-dimensional super-resolution processing are processing steps that are not common processing steps after step ST207. However, the high-resolution image f ′ (n) or f ′ (n + 1) in which the image input to the high-resolution image generation unit (addition unit) 21 of the image processing apparatus of FIGS. Although there is a difference in the presence or absence, it does not require a specially different processing unit, so that it can be realized by an image processing apparatus having the same configuration.

  Further, in the image processing apparatus according to the first embodiment of the present invention, each process of the image processing method shown in FIG. 1 can be processed in parallel by forming a pipeline as long as there is no restriction on reference or update of image data. Is clear.

  As described above, according to the image processing device and the image processing method according to Embodiment 1 of the present invention, the two-dimensional super-resolution processing and the three-dimensional super-resolution processing are configured in the same configuration, and the two-dimensional super-resolution processing is performed. Alternatively, by controlling to execute the three-dimensional super-resolution processing, a two-dimensional super-resolution image or a three-dimensional super-resolution image obtained by super-resolution processing of the low-resolution input image can be generated.

  Further, according to the image processing device and the image processing method according to the first embodiment of the present invention, it is not necessary to provide two-dimensional super-resolution processing and three-dimensional super-resolution processing circuits separately, so that the device scale and circuit Scale reduction and cost reduction can be achieved.

  Further, according to the image processing apparatus and the image processing method according to the first embodiment of the present invention, the processing of the next low-resolution input image is performed when the reference of the current low-resolution input image is finished in the three-dimensional super-resolution processing. Since it can be started, it is possible to efficiently construct a pipeline and perform three-dimensional super-resolution processing, minimize the time lag of inter-frame processing, and maintain the frame rate during moving image processing.

Embodiment 2. FIG.
In the second embodiment of the present invention, an image processing apparatus and an image processing method for performing two-dimensional super-resolution processing and three-dimensional super-resolution processing with an apparatus configuration different from that of the first embodiment of the present invention will be described.

  First, the case of operating as an image processing apparatus and an image processing method for performing three-dimensional super-resolution processing will be described.

  FIG. 7 is a block diagram showing an example of an image processing apparatus according to Embodiment 2 of the present invention. The block configuration diagram of FIG. 7 is a block configuration diagram showing an example of the image processing apparatus that performs the three-dimensional super-resolution processing according to the first embodiment of the present invention. FIG. 7 is a block diagram of the low-resolution input image storage unit 11a. The addition of the storage unit 114a, the storage unit 134 of the estimated high-resolution image storage unit 13a, the addition of the switch (SWd) 113 and the number of inputs / outputs of the switch (SWa) 112b, the switch (SWb) 131b, and the switch (SWc) 133b. This is a change. Other processing blocks are the same as the processing blocks denoted by the same reference numerals in FIG. 3, which is a block configuration diagram showing an example of the image processing apparatus according to Embodiment 1 of the present invention.

  In FIG. 7, the switch 113 of the low resolution input image storage unit 11a distributes the input low resolution input image to the storage units 111a and 114a. The accumulation unit 114a stores the low resolution input image g (n + 1). The switch 112b converts the low resolution input image g (n) accumulated in the accumulation unit 111a and the low resolution input image g (n + 1) accumulated in the accumulation unit 114a into the first high resolution conversion unit 12a or the difference image generation unit ( Output to the subtracting unit 16. Even if the switch 112b performs a single transfer operation with one input and one output, for example, a low-resolution input image g (n) is input to the difference image generation unit (subtraction unit) 16 in order to generate a difference image g ″ (n). The low-resolution input image g (n + 1) may be simultaneously transferred with two inputs and two outputs that are sent to the first high-resolution conversion unit 12a while the switch 131b of the estimated high-resolution image storage unit 13a is referred to. The high resolution input image and the high resolution image are distributed to the storage units 132 and 134. The switch 131b, for example, sends the high resolution image f ′ (n) to the storage unit 132 and simultaneously transmits the high resolution input image f (n + 1). The storage unit 134 can store the estimated high resolution image f (n + 1), and the switch 133b can store the estimated high resolution stored in the storage units 132 and 134. The images f (n) and f (n + 1) are output to the low resolution conversion unit 14, the high resolution image generation unit (addition unit) 21, or the difference image generation unit (subtraction unit) 16. This switch 133b has one input and one output. Even if a single transfer operation is performed, for example, while the estimated high resolution image f (n) is sent to the low resolution conversion unit 14, the estimated high resolution input image f (n + 1) is simultaneously sent to the outside and simultaneously transferred with two inputs and two outputs. At this time, the control unit controls the switch (SWd) 113, the switch (SWa) 112b, the switch (SWb) 131b, and the switch (SWc) 133b to secure and output the image data path. Let

  The operation as the image processing apparatus and the image processing method for performing the three-dimensional super-resolution processing corresponds to the case where the three-dimensional super-resolution mode is selected in the flowchart of FIG. 1, and relates to the first embodiment of the present invention. This can be explained in the same manner as the three-dimensional super-resolution processing in the image processing method.

  In addition, when the image is continuously subjected to three-dimensional super-resolution processing, the low-resolution input image g (n) input to the low-resolution input image storage unit 11a is alternately distributed to the storage units 111a and 113, for example. By updating the number n, the relative relationship between the low resolution input images g (n) and g (n + 1) of the image numbers n and n + 1 is kept.

  The difference between the three-dimensional super-resolution processing of the image processing apparatus according to the second embodiment of the present invention and the three-dimensional super-resolution processing of the image processing apparatus according to the first embodiment of the present invention is that low-resolution input image storage is performed. The low-resolution input image g (n + 1) and the estimated high-resolution image f (n + 1) are generated in the estimated high-resolution image storage unit 13a. Therefore, the low resolution input image g (n + 1) can be started without waiting for the generation of the difference image g ″ (n) referring to the low resolution input image g (n). By speeding up the start of g (n + 1), the subsequent processing can also be accelerated compared to the three-dimensional super-resolution processing of the image processing apparatus according to Embodiment 1 of the present invention, and the efficiency of the three-dimensional super-resolution processing is improved. improves.

  Next, a description will be given of the case of operating as an image processing apparatus and image processing method for performing two-dimensional super-resolution processing.

  The block configuration diagram of FIG. 7 is a block configuration diagram showing an example of the image processing apparatus that performs the two-dimensional super-resolution processing according to the first embodiment of the present invention. FIG. 7 is similar to FIG. The addition of the storage unit 114a of the low-resolution input image storage unit 11a, the storage unit 134 of the estimated high-resolution image storage unit 13a, the addition of the switch (SWd) 113 and the switch (SWa) 112b, the switch (SWb) 131b, The number of inputs and outputs of the switch (SWc) 133b is changed. Other processing blocks are the same as the processing blocks denoted by the same reference numerals in FIG. 2, which is a block configuration diagram showing an example of the image processing apparatus according to Embodiment 1 of the present invention.

  The operation as the image processing apparatus and the image processing method for performing the two-dimensional super-resolution processing corresponds to the case where the two-dimensional super-resolution mode is selected in the flowchart of FIG. 1, and relates to the first embodiment of the present invention. This can be explained in the same manner as the two-dimensional super-resolution processing in the image processing method.

  Here, in the configuration of FIG. 7 which is a block configuration diagram showing an example of the image processing apparatus according to the second embodiment of the present invention, the low resolution input image storage added to FIG. 2 of the first embodiment of the present invention. It goes without saying that the two-dimensional super-resolution processing similar to that of Embodiment 1 of the present invention can be explained by not using the storage unit 114a of the unit 11a and the storage unit 134 of the estimated high-resolution image storage unit 13a. Absent. Further, even if the storage units 111a and 114a of the low-resolution input image storage unit 11a and the storage units 132 and 134 of the estimated high-resolution image storage unit 13a are alternately used, for example, the two-dimensional super-resolution processing is executed. Good.

  In addition, the storage unit 114a of the low resolution input image storage unit 11a and the storage unit 134 of the estimated high resolution image storage unit 13a are used to perform two-dimensional super-resolution processing of independent low-resolution input images. In the same manner as described above, it is possible to configure a pipeline and execute it efficiently in parallel so that the use of processing blocks does not overlap.

  As described above, according to the image processing apparatus and the image processing method according to the second embodiment of the present invention, the two-dimensional super-resolution processing and the three-dimensional super-resolution processing are the same as in the first embodiment of the present invention. By controlling to execute 2D super-resolution processing or 3D super-resolution processing, the desired 2D super-resolution image or 3D super-resolution image obtained by super-resolution processing of the low-resolution input image is controlled. A resolution image can be generated.

  Further, according to the image processing apparatus and the image processing method according to the second embodiment of the present invention, as in the first embodiment of the present invention, the two-dimensional super-resolution processing and the three-dimensional super-resolution processing circuits are individually provided. Therefore, the device scale and circuit scale can be reduced and the cost can be reduced.

  Further, according to the image processing apparatus and the image processing method according to the second embodiment of the present invention, the processing of the next low-resolution input image is performed before the reference of the current low-resolution input image is finished in the three-dimensional super-resolution processing. Since it can be started, the pipeline can be more efficiently configured to perform the three-dimensional super-resolution processing, the time lag of the inter-frame processing can be minimized, and the frame rate during the moving image processing can be maintained.

  Further, according to the image processing apparatus and the image processing method according to the second embodiment of the present invention, the processing of the next low resolution input image is performed before the reference of the current low resolution input image is finished in the two-dimensional super-resolution processing. Since it can be started, as long as the use of processing blocks does not overlap, a pipeline can be configured more efficiently and two-dimensional super-resolution processing can be performed in parallel, thereby shortening the processing time. .

Embodiment 3 FIG.
Embodiment 3 of the present invention relates to an image processing apparatus and an image processing method for performing two-dimensional super-resolution processing and three-dimensional super-resolution processing with an apparatus configuration different from those of Embodiments 1 and 2 of the present invention. explain.

  FIG. 8 is a block diagram showing an example of an image processing apparatus according to Embodiment 3 of the present invention. The block configuration diagram of FIG. 8 is a block configuration diagram illustrating an example of the image processing apparatus according to the first embodiment of the present invention. FIG. 8 and FIG. A high resolution conversion unit 122 that is integrated and integrated with a high resolution conversion unit 12b that shares the resolution conversion unit 19, a switch (SWp) 121 that switches an input source of the high resolution conversion unit 122, and a switch that switches an output destination (SWq ) 123 is provided. Other processing blocks are the same as the processing blocks denoted by the same reference numerals in FIGS. 2 and 3, which are block configuration diagrams showing an example of the image processing apparatus according to the first embodiment of the present invention.

  In FIG. 8, the high resolution conversion unit 12 b includes a switch (SWp) 121, a high resolution conversion unit 122, and a switch (SWq) 123. The switch (SWp) 121 receives the low resolution input image g (n) stored in the low resolution input image storage unit 11a and the difference image g ″ (n) stored in the difference image storage unit 17a as inputs. (N) or the difference image g ″ (n) is output. The high resolution conversion unit 122 converts the low resolution input image g (n) or the difference image g ″ (n) output from the switch (SWp) 121 into a high resolution and outputs it. The switch (SWq) 123 outputs the high resolution conversion. The unit 122 outputs the high resolution input image obtained by converting the low resolution input image g (n) to the high resolution image storage unit 13a, and the high resolution conversion unit 122 converts the low resolution input image g (n) to the high resolution. The converted differential high-resolution image j (n) is output to the differential high-resolution image storage unit 20. At this time, the control unit controls the switch (SWp) 121 and the switch (SWq) 123 to ensure the path of the image data. And output.

  In the case of operating as an image processing apparatus and an image processing method for performing two-dimensional super-resolution processing or three-dimensional super-resolution processing, both of the two in the image processing method according to the first embodiment of the present invention in the flowchart of FIG. This can be explained in the same manner as the operation of the three-dimensional super-resolution processing or the three-dimensional super-resolution processing.

  The two-dimensional super-resolution process and the three-dimensional super-resolution process of the image processing apparatus according to the third embodiment of the present invention are the same as the two-dimensional super-resolution process and the three-dimensional super-resolution of the image processing apparatus according to the first embodiment of the present invention. The difference from the resolution processing is that the high resolution converter 12b in FIG. 8 is different from the first high resolution converter 12a and the second high resolution converter 19 in FIGS. 2 and 3 of the first embodiment of the present invention. Can be configured with a single high-resolution conversion unit. By sharing two high-resolution conversion units and integrating them into one high-resolution conversion unit, the apparatus scale and circuit scale can be reduced, and the cost can be reduced.

  Thus, according to the image processing apparatus and the image processing method according to the third embodiment of the present invention, as in the first and second embodiments of the present invention, the two-dimensional super-resolution processing and the three-dimensional super-resolution processing are performed. By controlling the resolution processing to execute two-dimensional super-resolution processing or three-dimensional super-resolution processing with the same configuration, the desired two-dimensional super-resolution obtained by super-resolution processing of the low-resolution input image An image or a three-dimensional super-resolution image can be generated.

  Further, according to the image processing apparatus and the image processing method according to Embodiment 3 of the present invention, as in Embodiments 1 and 2 of the present invention, two-dimensional super-resolution processing and three-dimensional super-resolution. Since it is not necessary to provide a processing circuit separately, it is possible to reduce the apparatus scale, circuit scale, and cost.

  Further, according to the image processing device and the image processing method according to the third embodiment of the present invention, by sharing two high resolution conversion units and integrating them into one high resolution conversion unit, the device scale and circuit scale can be reduced. It can be made smaller and the cost can be reduced.

  Further, according to the image processing apparatus and the image processing method according to the third embodiment of the present invention, as in the first embodiment of the present invention, the reference of the current low resolution input image is finished in the three-dimensional super-resolution processing. Since the processing of the next low-resolution input image can be started at that point, it is possible to efficiently construct a pipeline and perform 3D super-resolution processing, minimizing the time lag of inter-frame processing, The frame rate during processing can be maintained.

Embodiment 4 FIG.
Embodiment 4 of the present invention relates to an image processing apparatus and an image processing method for performing two-dimensional super-resolution processing and three-dimensional super-resolution processing with an apparatus configuration different from the first to third embodiments of the present invention. explain.

  FIG. 9 is a block diagram showing an example of an image processing apparatus according to Embodiment 4 of the present invention. 9 is a block diagram showing an example of an image processing apparatus according to Embodiment 3 of the present invention. The low-resolution input image storage unit 11a and the estimated high-resolution image storage unit 13a with respect to FIG. FIG. 7 is a block configuration diagram showing an example of the image processing apparatus according to the second embodiment, which is replaced with a low resolution input image storage unit 11b and an estimated high resolution image storage unit 13b with respect to FIG. Each processing block is the same as the processing block given the same number in FIGS. 7 and 8, which is a block configuration diagram showing an example of the image processing apparatus according to Embodiments 2 and 3 of the present invention.

  In the case of operating as an image processing apparatus and an image processing method for performing two-dimensional super-resolution processing or three-dimensional super-resolution processing, both of the two in the image processing method according to the first embodiment of the present invention in the flowchart of FIG. This can be explained in the same manner as the operation of the three-dimensional super-resolution processing or the three-dimensional super-resolution processing.

  As described above, according to the image processing apparatus and the image processing method according to the fourth embodiment of the present invention, as in the first to third embodiments of the present invention, the two-dimensional super-resolution processing and the three-dimensional super-resolution processing are performed. By controlling the resolution processing to execute two-dimensional super-resolution processing or three-dimensional super-resolution processing with the same configuration, the desired two-dimensional super-resolution obtained by super-resolution processing of the low-resolution input image An image or a three-dimensional super-resolution image can be generated.

  Further, according to the image processing apparatus and the image processing method according to the fourth embodiment of the present invention, as in the first to third embodiments of the present invention, the two-dimensional super-resolution processing and the three-dimensional super-resolution. Since it is not necessary to provide a processing circuit separately, it is possible to reduce the apparatus scale, circuit scale, and cost.

  Further, according to the image processing apparatus and the image processing method according to the fourth embodiment of the present invention, as in the third embodiment of the present invention, the two high resolution conversion units are shared to form one high resolution conversion unit. By integrating, the device scale and circuit scale can be further reduced, and the cost can be reduced.

  Further, according to the image processing apparatus and the image processing method according to the fourth embodiment of the present invention, as in the second embodiment of the present invention, the reference of the current low resolution input image is finished in the three-dimensional super-resolution processing. Since the processing of the next low-resolution input image can be started before, it is possible to configure the pipeline more efficiently and perform the three-dimensional super-resolution processing, minimizing the time lag of inter-frame processing, The frame rate during video processing can be maintained.

  Further, according to the image processing device and the image processing method according to the fourth embodiment of the present invention, as in the second embodiment of the present invention, the reference of the current low-resolution input image is finished in the two-dimensional super-resolution processing. Since the processing of the next low resolution input image can be started before, as long as the use of processing blocks does not overlap, it is possible to construct a pipeline more efficiently and perform two-dimensional super-resolution processing in parallel. The processing time can be shortened.

Embodiment 5 FIG.
In the fifth embodiment of the present invention, three-dimensional super-resolution processing is performed as shown in FIG. 7 showing the block configuration of the second embodiment of the present invention or FIG. 9 showing the configuration of the fourth embodiment of the present invention. In this case, two low-resolution input images g (n) and g (n + 1) of continuous video are stored in the low-resolution input image storage unit, motion is detected from the two low-resolution input images, and the motion is detected. An image processing apparatus and an image processing method for generating a differential high-resolution image j (n) from a differential image g ″ (n) that is referred to after motion compensation of a certain region and performing super-resolution processing will be described.

  FIG. 10 is an explanatory diagram showing an example of the motion region detected by the motion detection unit of the image processing apparatus according to the fifth embodiment of the present invention. In the figure, motion detection is performed using image information between the previous frame and the current frame. For example, a background area including a tree is stationary, a moving area is not detected, and an area around a moving person Is detected as a region with motion (motion detection region). Each of the non-moving region and the moving region is not limited to one region per image, and may be detected as a plurality of regions in a divided state even in continuous regions. Further, the region with movement may be extracted as the shape of the moving object, or may be extracted with a shape such as a rectangular block including the surrounding background. The size of the region used for motion detection may be fixed to one, or a plurality may be combined.

  FIG. 11 is a block diagram showing an example of an image processing apparatus according to Embodiment 5 of the present invention. In the figure, the part different from FIG. 7 showing the block configuration of the third embodiment of the present invention is that a motion detection unit (MV) 31a for detecting motion information is provided. Further, the difference image storage unit 17b outputs a difference image g ″ (n) based on the motion information detected by the motion detection unit (MV) 31a. Other processing blocks according to the third embodiment of the present invention. It is the same as the processing block which attached | subjected the same number to FIG. 7 which is a block block diagram which shows an example of an image processing apparatus.

  The motion detection unit (MV) 31a detects motion information in units of regions with reference to the low resolution input images g (n) and g (n + 1) stored in the storage units 111b and 114b of the low resolution input image storage unit 11c. (Motion detection) and output to the difference image storage unit 17b. The difference image storage unit 17b, based on the motion information detected by the motion detection unit (MV) 31a, the difference image g ″ corresponding to the region of the difference high resolution image j (n) to be high resolution converted by the high resolution conversion unit 19. The high resolution conversion unit 19 converts the reference image region of the differential image g ″ (n) output by the differential image storage unit 17b based on the motion information to high resolution and outputs a high differential image. A resolution image j (n) is generated. The differential high resolution image storage unit 20 stores the differential high resolution image j (n) generated by the high resolution conversion unit 19 in a corresponding area. At this time, the control unit outputs, from the difference image storage unit 17b, the reference image region of the difference image g ″ (n) corresponding to the region of the difference high resolution image j (n) subjected to high resolution conversion by the high resolution conversion unit 19. As described above, the motion detection unit (MV) 31a outputs motion information, and generates a motion-compensated differential high-resolution image j (n) for each region targeted for motion information detection.

  FIG. 12 is a flowchart (partial change example of motion detection) showing an example of the three-dimensional super-resolution processing in the image processing method according to Embodiment 5 of the present invention. In step ST308 of FIG. 1, as shown in FIG. 12, in step ST3081, the motion detection unit (MV) 31a detects motion information for each region. In step ST3082, based on the motion information detected by the motion detection unit (MV) 31a, the reference image of the differential image g ″ (n) corresponding to the region of the differential high-resolution image j (n) to be generated is the differential image. The output is output from the storage unit 17b, and the high resolution conversion unit 19 performs high resolution conversion, and the differential high resolution image j (n) is stored in the differential high resolution image storage unit 20. In step ST3083, the differential high resolution image j is output up to the final region. It is determined whether (n) has been generated. If an unprocessed area remains, the process repeats from step SY3081 for that area. If there is no unprocessed area, the generation of the differential high resolution image j (n) is terminated. Proceeding to step ST309, at this time, the control unit controls to perform these processes for each region.

  In the image processing apparatus according to Embodiment 5 of the present invention, for example, if the unit area for motion detection is a fixed size, the storage units 111b and 114b of the low resolution input image storage unit and the storage unit of the estimated high resolution image storage unit 132 and 134, the difference image storage unit 17b whose area is referred to by the motion information requires the capacity of one image to be stored, but the low resolution image storage unit 15 and the difference high resolution image storage unit 20 are area units. Then, the three-dimensional super-resolution processing may be advanced to implement a capacity of one area. However, the two-dimensional super-resolution processing performed with the same configuration as this image processing apparatus is also performed in units of regions.

  In the image processing apparatus according to the fifth embodiment of the present invention, the example in which the motion detection unit (MV) 31a is provided for the block configuration of FIG. 7 according to the third embodiment of the present invention has been described. It is needless to say that the three-dimensional super-resolution processing provided with the motion detection unit (MV) 31a can be similarly described for the block configuration of FIG.

  It should be noted that the first to fourth embodiments of the present invention can be realized by using the image processing apparatus and the image processing method according to the fifth embodiment of the present invention to perform processing without operating the motion detection unit (MV) 31a. It is obvious that two-dimensional super-resolution processing can be performed in the same manner as in FIG.

  Thus, according to the image processing apparatus and the image processing method according to the fifth embodiment of the present invention, as in the second and fourth embodiments of the present invention, the two-dimensional super-resolution processing and the three-dimensional super-resolution processing are performed. By controlling the resolution processing to execute two-dimensional super-resolution processing or three-dimensional super-resolution processing with the same configuration, the desired two-dimensional super-resolution obtained by super-resolution processing of the low-resolution input image An image or a three-dimensional super-resolution image can be generated.

  Further, according to the image processing apparatus and the image processing method according to the fifth embodiment of the present invention, as in the second and fourth embodiments of the present invention, the two-dimensional super-resolution processing and the three-dimensional super-resolution. Since it is not necessary to provide a processing circuit separately, it is possible to reduce the apparatus scale, circuit scale, and cost.

  Further, according to the image processing apparatus and the image processing method according to the fifth embodiment of the present invention, as in the fourth embodiment of the present invention, two high resolution conversion units are shared to form one high resolution conversion unit. By integrating, the device scale and circuit scale can be further reduced, and the cost can be reduced.

  Further, according to the image processing apparatus and the image processing method according to the fifth embodiment of the present invention, as in the second and fourth embodiments of the present invention, the three-dimensional super-resolution processing is performed on the current low resolution input image. Since the processing of the next low-resolution input image can be started before the reference of is finished, the pipeline can be configured more efficiently to perform the three-dimensional super-resolution processing, and the inter-frame processing time lag is minimized. And the frame rate at the time of moving image processing can be maintained.

  Further, according to the image processing device and the image processing method according to the fifth embodiment of the present invention, when performing the three-dimensional super-resolution processing, the motion information of the two low-resolution input images is detected and the motion compensation is performed. Since the resolution image is generated, the area correspondence between the estimated high resolution image added by the high resolution image generation unit (adding unit) and the difference high resolution image is improved, so that more accurate three-dimensional super-resolution processing can be realized. .

Embodiment 6 FIG.
In the sixth embodiment of the present invention, as in the fifth embodiment of the present invention, FIG. 7 showing the block configuration of the image processing apparatus of the second embodiment of the present invention, or the image processing of the fourth embodiment of the present invention. As shown in FIG. 9 showing the block configuration of the apparatus, when performing the three-dimensional super-resolution processing, two low-resolution input images g (n) and g (n + 1) in which a video is continuously stored in the low-resolution input image storage unit. , And motion detection is performed from the two low-resolution input images, and a differential high-resolution image j (n) is generated from the differential image g ″ (n) referred to by motion-compensating a motion region. An image processing apparatus and an image processing method for performing resolution processing will be described, except that in the image processing apparatus and the image processing method according to the sixth embodiment of the present invention, motion information is detected and an area with motion is detected in the present invention. The same processing as in the fifth embodiment is performed. The low-resolution input image is converted into a high-resolution conversion in a region where there is no motion by a conversion process different from that in Embodiment 5. Hereinafter, the high-resolution applied depending on the presence or absence of detected motion information in Embodiment 6 of the present invention. An image processing apparatus and an image processing method for switching between conversion and low resolution conversion will be described.

FIG. 13 is a block diagram showing an example of an image processing apparatus according to Embodiment 6 of the present invention. In the figure, the part different from FIG. 11 showing the block configuration of the fifth embodiment of the present invention is that a motion detection unit (MV) 31b is provided instead of the motion detection unit (MV) 31a for detecting motion information. . Further, the high resolution conversion unit 12c applies high resolution conversion (T ⁻¹ ) that is different between a region with motion and a region without motion based on the motion information. The difference image storage unit 17c sets the difference in the region to 0 in the region where there is no motion based on the motion information. Other processing blocks are the same as the processing blocks denoted by the same reference numerals in FIG. 11, which is a block configuration diagram showing an example of an image processing apparatus according to Embodiment 5 of the present invention.

  The motion detection unit 31b refers to the low resolution input images g (n) and g (n + 1) stored in the storage units 111b and 114b of the low resolution input image storage unit 11c to detect motion information in units of regions (motion detection). ) The detected motion information is notified to the high resolution conversion unit 12c and the difference image storage unit 17c.

Based on the motion information notified from the motion detection unit 31b, the high resolution conversion unit 12c can perform high resolution conversion (T ⁻¹ ) in a region where there is no motion even with a lower processing load than the reverse projection method. Image enlargement processing, for example, nearest neighbor method, linear interpolation method, bicubic interpolation, etc. are applied, and high-resolution conversion (T ⁻¹ ) by backprojection method is applied to a region with motion in the same manner as the high-resolution conversion unit 12a. To do. The estimated high-resolution image f (n) obtained as the high-resolution input image of the non-moving region is stored in the estimated high-resolution image storage unit 13b as a three-dimensional super-resolution image when this image enlargement process is performed.

  The difference image storage unit 17c sets the difference in the region to 0 based on the motion information notified from the motion detection unit 31b, and generates each differential high resolution image j (n) in the same manner as the motion detection unit 31a. The reference region of the difference image g ″ (n) corresponding to the region is output from the difference image storage unit 17b. That is, for the region having no motion, the low resolution conversion unit 14, the low resolution image storage unit 15, and the subtractor. (Difference image generation unit) The difference image g ″ (n) = 0 can be set without going through the region 16 and without converting or calculating the area. Here, the reason why the difference image g ″ (n) in the non-motion region is set to 0 is that it may be referred to when the difference high-resolution image j (n) is generated based on the motion information. If the entire difference image g ″ (n) is initialized in advance when starting the three-dimensional super-resolution processing of the low-resolution input image g (n), the difference image of each region without motion It is not necessary to set the difference image g ″ (n) = 0 at the storage processing stage.

  FIG. 14 is a flowchart showing an example of a three-dimensional super-resolution process in the image processing method according to the sixth embodiment of the present invention. In the figure, each processing step is the same as the processing step denoted by the same reference numeral in FIGS. 1 and 12 which is also the flowchart of the image processing method according to the fifth embodiment of the present invention.

  Although the flowchart of the image processing method shown in FIG. 1 illustrates both two-dimensional super-resolution processing and three-dimensional super-resolution processing, it is determined whether or not the designation of the super-resolution mode is the two-dimensional super-resolution mode. Branching in step ST207, a flowchart of the two-dimensional super-resolution mode two-dimensional super-resolution processing and the three-dimensional super-resolution mode three-dimensional super-resolution processing can be shown for each mode. The flowchart of the image processing method shown in FIG. 14 illustrates three-dimensional super-resolution processing in a three-dimensional super-resolution mode in which motion information is detected and super-resolution processing is performed using motion compensation. . Compared with the flowchart of the image processing method shown in FIG. 1, the processing procedure is changed for the three-dimensional super-resolution processing in the three-dimensional super-resolution mode, but the two-dimensional super-resolution in the two-dimensional super-resolution mode. The processing procedure is not changed.

  Here, the processing steps from step ST401 to step 406 and the processing steps from step ST407 to step ST410, which are changed as the three-dimensional super-resolution processing, are respectively performed in units of regions. At this time, the control unit controls to perform these processes for each region.

  Processing steps from the motion detection shown in step ST401 to the generation of the difference image g ″ (n) shown in step 406 will be described. In step ST401, motion information is detected for each region, as in step 3081 of FIG. Here, each piece of motion information to be detected is held as long as it is necessary until the three-dimensional super-resolution processing of the low-resolution input image g (n) is completed.

  In step ST402, the presence / absence of motion is determined based on the motion information detected in step ST401. When there is motion, the process proceeds to step ST303, and when there is no motion, the process proceeds to step ST403.

  When it is determined in step ST402 that the area to be processed is a moving area, as described in FIG. 1, the low resolution input image g (n) is converted to a high resolution by the reverse projection method in step ST303 and the high resolution input image is obtained. Is generated and stored as an estimated high resolution image f (n). In step ST204, the estimated high resolution image f (n) is converted to a low resolution to generate and store a low resolution image g ′ (n), and step ST205 is stored. The difference image g ″ (n) is generated and stored by taking the difference between the low resolution input image g (n) and the low resolution image g ′ (n).

  When it is determined in step ST402 that the area to be processed is a non-motion area, the low resolution input image g (n + 1) stored in the low resolution input image storage unit 11c (accumulation unit 114b) in step ST403 is converted to a high resolution. The unit 12c performs high resolution conversion by a general image enlargement process to generate a high resolution input image, and stores it as the estimated high resolution image f (n + 1) in the estimated high resolution image storage unit 13b (storage unit 134).

  In step ST405, a difference image g ″ (n) = 0 is generated and stored in the difference image storage unit 17b.

  In step ST406, it is determined whether the processing from step ST401 to step 405 has been performed to the final area. If there is an unprocessed area, the process returns to step ST401 to repeat the process, and if the process has been completed to the final area, the process proceeds to step ST206. .

  Next, processing steps from generation of the differential high resolution image j (n) to generation of the estimated high resolution image f (n + 1) shown in steps ST407 to ST410 will be described.

  In step ST407, the presence or absence of motion is determined based on the motion information detected and held in step ST401. When there is a motion, the process proceeds to step ST408, and when there is no motion, the process proceeds to step ST410.

  In step ST408, the reference area of the difference image g ″ (n) corresponding to the area of the difference high-resolution image j (n) to be generated based on the motion information detected and held in step ST401 is stored in the difference image storage unit. 17c, and the high resolution conversion unit 19 performs high resolution conversion to store the differential high resolution image j (n) in the differential high resolution image storage unit 20.

  In step ST409, the difference high-resolution image j (n) is added to the estimated high-resolution image f (n + 1) by the adder (high-resolution image generation unit) 21 to generate a high-resolution image f ′ (n + 1). The image f (n + 1) is re-stored and updated in the estimated high-resolution image storage unit 13a.

  In step ST410, it is determined whether the processing from step ST407 to step 409 has been performed up to the final region. If an unprocessed region remains, the process returns to step ST407 to repeat the processing. .

  As an image processing apparatus according to the sixth embodiment of the present invention, a motion detection section 31b is added to the high resolution conversion section 12b of FIG. 9 showing the block configuration of the image processing apparatus according to the fourth embodiment of the present invention. When applying, when the motion information detected by the motion detection unit 31b is notified to the shared high-resolution conversion unit 122 and the low-resolution input image g (n + 1) is converted to high resolution to generate a high-resolution input image, Use the presence / absence of motion information by switching to high-resolution conversion by backprojection for areas with motion based on motion information and high-resolution conversion by general image enlargement processing for areas without motion 3D super-resolution processing can be performed.

  Thus, according to the image processing apparatus and the image processing method according to the sixth embodiment of the present invention, as in the second, fourth and fifth embodiments of the present invention, two-dimensional super-resolution. Processing and 3D super-resolution processing with the same configuration and control to execute 2D super-resolution processing or 3D super-resolution processing. A two-dimensional super-resolution image or a three-dimensional super-resolution image can be generated.

  Further, according to the image processing apparatus and the image processing method according to the sixth embodiment of the present invention, as in the second embodiment, the fourth embodiment, and the fifth embodiment of the present invention, Since it is not necessary to separately provide a three-dimensional super-resolution processing circuit, the apparatus scale, the circuit scale, and the cost can be reduced.

  Further, according to the image processing apparatus and the image processing method according to the sixth embodiment of the present invention, as in the fourth and fifth embodiments of the present invention, two high-resolution conversion units are used in common. By integrating the high resolution conversion unit, the device scale and circuit scale can be further reduced, and the cost can be reduced.

  Further, according to the image processing apparatus and the image processing method according to the sixth embodiment of the present invention, the three-dimensional super-resolution processing is performed as in the second, fourth and fifth embodiments of the present invention. Since the processing of the next low-resolution input image can be started before the reference of the current low-resolution input image is completed, it is possible to construct a pipeline more efficiently and perform three-dimensional super-resolution processing. The time lag can be minimized and the frame rate at the time of moving image processing can be maintained.

  Further, according to the image processing apparatus and the image processing method according to the sixth embodiment of the present invention, as in the fifth embodiment of the present invention, two low-resolution input images are processed during the three-dimensional super-resolution processing. Since the difference high resolution image which detected the motion information and compensated the motion is generated, the area correspondence between the estimated high resolution image added by the high resolution image generation unit (adding unit) and the difference high resolution image is improved. Accurate 3D super-resolution processing can be realized.

  Further, according to the image processing device and the image processing method according to the sixth embodiment of the present invention, high-resolution conversion is performed on the moving area by the back projection method based on the detected motion information, and the non-motion area is obtained. Since switching is performed for each region so as to perform high resolution conversion by general image enlargement processing, the calculation load for performing high resolution conversion can be reduced.

Embodiment 7 FIG.
In Embodiment 7 of the present invention, in the image processing apparatus and image processing method of Embodiments 1 to 6 of the present invention, an image processing apparatus and image processing in which optical correction of a photographing device such as a digital still camera is applied to resolution conversion. A method will be described.

  FIG. 15 is an explanatory diagram showing the relationship between the distance from the optical axis center of the image input system and the MTF (Modulation Transfer Function) of the image processing apparatus according to Embodiment 7 of the present invention. An apparatus having an optical system such as a digital camera has a characteristic (optical characteristic) in which the resolution of the peripheral part is lower than the central part of the image due to lens distortion. In other words, it appears as a characteristic of the lens of the image input system in which the MTF decreases as the processing target pixel moves away from the optical axis center.

In order to correct such lens characteristics (MTF characteristics), in the high-resolution conversion process, the conversion definition (conversion parameter) is adjusted according to the distance r from the optical axis center of the processing target pixel. The calculation for correcting the conversion definition according to the distance r from the optical axis center can be expressed as the following (Expression 8) with respect to (Expression 1) that does not consider the distance r from the optical axis center. Here, α _r is a coefficient (correction coefficient) for correcting the width of ± 3σ in FIG. 4 at the distance r from the center of the optical axis according to the MTF characteristic in FIG.

FIG. 16 is an explanatory diagram showing the relationship between the distance from the optical axis center of the image input system of the image processing apparatus according to Embodiment 7 of the present invention and the correction coefficient of the conversion parameter. The processing target pixel located near the center of the optical axis has an MTF close to 1.0, and the processing target pixel located at the periphery of the image (distance r from the center of the optical axis) decreases the MTF as the distance r increases. Since there is the MTF characteristic shown in FIG. 15, the correction coefficient α _r is set as follows, for example.

When the process object pixel is located near the center optical axis is close to the MTF is 1.0 in FIG. 15, selects a correction coefficient alpha ₀ to be converted near the normal inverse transform T ^-1. On the other hand, when the pixel to be processed is located in the peripheral portion of the image (distance R from the center of the optical axis) and a greater blur reduction effect is required, the width based on the MTF characteristics in FIG. Is selected, the correction coefficient α _R that increases the degree of correction by the inverse transformation T ⁻¹ is selected. Here, the correction coefficient α _r may be defined as a function of the distance r from the center of the optical axis, or the correction coefficient α _r is obtained for a representative distance r from the center of the optical axis, and the actual coefficient It may be set by interpolation, for example, linear interpolation, from two correction coefficients close to the distance r from the center of the optical axis.

The high resolution conversion (T ⁻¹ ) corrected with the correction coefficient α _r based on the distance r from the optical axis center where each processing target pixel is located in accordance with such MTF characteristics is the first to sixth embodiments of the present invention. Needless to say, the present invention can be applied to the image processing apparatus and the image processing method according to the present invention. At this time, the control unit of the image processing apparatus manages the distance r from the center of the optical axis where each processing target pixel is located, and applies it to the high resolution conversion (T ⁻¹ ) corrected with the correction coefficient α _r .

For the conversion definition (conversion parameter), for example, the initialization of the conversion definition in step ST201 in the flowchart of the image processing method shown in FIG. 1 and the correction of the conversion definition in steps ST211 and ST311 are performed on the processing target pixel at the center of the optical axis. Embodiments of the present invention are described in step ST203 and step ST303 in which conversion definition (conversion parameters) of high resolution conversion (T ^-1 ) is targeted, and each pixel to be processed is actually subjected to high resolution conversion (T ^-1 ). The conversion definition (conversion parameter) corrected using the correction coefficient α _r corresponding to the distance r from the optical axis center described in 7 is applied.

In the two-dimensional super-resolution processing and the three-dimensional super-resolution processing performed by the image processing apparatus and the image processing method according to the first to sixth embodiments of the present invention, for example, steps in the flowchart of the image processing method shown in FIG. In ST203 and step ST303, when the low resolution input image g (n) is subjected to high resolution conversion (T ⁻¹ ) to obtain a high resolution input image (estimated high resolution image f (n)), each process is performed according to the MTF characteristics. In addition to correcting with the correction coefficient α _r based on the distance r from the optical axis center where the target pixel is located, in step ST204, the estimated high resolution image f (n) is converted to a low resolution (T) and the low resolution image g 'when you get a (n), and obtained in step ST208 and step ST 308, a difference image g "(n) of high-resolution conversion of ^{(T -1)} to the differential high-resolution image j (n) In some cases, the correction coefficient needs to be corrected with the correction coefficient, and the calculation for correcting the conversion definition with the correction coefficient based on the distance r from the optical axis center does not consider the distance r from the optical axis center (Formula 2). ) And (Expression 5) can be expressed as the following (Expression 9) and (Expression 10), where α _r is the correction coefficient shown in (Expression 8) and β _r is the high resolution conversion This is a correction coefficient for low resolution conversion (T) with respect to the correction coefficient α _r of (T ⁻¹ ).

By configuring such a correction coefficient according to the MTF characteristics for resolution conversion, the MTF characteristics of the image can be obtained when the image is captured by a photographing apparatus equipped with an optical system having a complicated group configuration. Even in the case of dynamic fluctuation due to the zoom operation, it is preferable to sequentially correct the high resolution conversion (T ⁻¹ ) and the low resolution conversion (T) according to the distance from the center of the optical axis to the processing target pixel. MTF correction can be realized.

  Thus, according to the image processing device and the image processing method according to Embodiment 7 of the present invention, the two-dimensional super-resolution processing performed by the image processing device and the image processing method according to Embodiments 1 to 6 of the present invention. In addition, for the three-dimensional super-resolution processing, it is possible to obtain the same effects as those of the respective embodiments in which optical correction is applied with a correction coefficient corresponding to the MTF characteristics.

  Further, according to the image processing device and the image processing method according to Embodiment 7 of the present invention, the two-dimensional super-resolution processing and the tertiary performed by the image processing device and the image processing method according to Embodiments 1 to 6 of the present invention. By applying optical correction to the resolution conversion of the original super-resolution processing with a correction coefficient corresponding to the MTF characteristics, it is possible to obtain a super-resolution image that does not deteriorate even in a processing target pixel far from the optical axis center.

  As described above, according to the image processing apparatus and the image processing method according to the present invention, the following effects can be obtained.

  According to the image processing apparatus and the image processing method according to the present invention, the two-dimensional super-resolution processing or the three-dimensional super-resolution processing is executed with the same configuration in the two-dimensional super-resolution processing and the three-dimensional super-resolution processing. By controlling in this way, it is possible to generate a two-dimensional super-resolution image or a three-dimensional super-resolution image obtained by super-resolution processing of a low-resolution input image.

  In addition, according to the image processing apparatus and the image processing method according to the present invention, it is not necessary to provide two-dimensional super-resolution processing and three-dimensional super-resolution processing circuits separately. Cost can be reduced.

  Further, according to the image processing apparatus and the image processing method according to the present invention, by sharing the two high resolution conversion units and integrating them into one high resolution conversion unit, the apparatus scale and the circuit scale can be further reduced. Cost can be reduced.

  Further, according to the image processing apparatus and the image processing method according to the present invention, since the processing of the next low-resolution input image can be started when the reference of the current low-resolution input image is finished in the three-dimensional super-resolution processing, the efficiency Thus, a pipeline can be configured to perform 3D super-resolution processing, the time lag of inter-frame processing can be minimized, and the frame rate during moving image processing can be maintained.

  In addition, according to the image processing apparatus and the image processing method according to the present invention, the processing of the next low-resolution input image can be started before the reference of the current low-resolution input image is completed in the three-dimensional super-resolution processing. It is possible to efficiently construct a pipeline and perform 3D super-resolution processing, minimize the time lag of inter-frame processing, and maintain the frame rate during moving image processing.

  Further, according to the image processing apparatus and the image processing method according to the present invention, the processing of the next low-resolution input image can be started before the reference of the current low-resolution input image is finished in the two-dimensional super-resolution processing. As long as the blocks are not overlapped, the pipeline can be configured more efficiently and the two-dimensional super-resolution processing can be performed in parallel, and the processing time can be shortened.

  Further, according to the image processing apparatus and the image processing method according to the present invention, when performing the three-dimensional super-resolution processing, the motion information of the two low-resolution input images is detected and the motion-compensated differential high-resolution image is generated. Therefore, the area correspondence between the estimated high-resolution image and the differential high-resolution image added by the high-resolution image generation unit (adding unit) is improved, so that more accurate three-dimensional super-resolution processing can be realized.

  Further, according to the image processing apparatus and the image processing method according to the present invention, based on the detected motion information, high-resolution conversion is performed on a region with motion by a reverse projection method, and general image enlargement is performed on a region without motion. Since each region is switched so as to perform high resolution conversion by processing, the calculation load for performing high resolution conversion can be reduced.

  Further, according to the image processing apparatus and the image processing method according to the present invention, the optical correction is applied with the correction coefficient corresponding to the characteristics of the MTF for the resolution conversion of the two-dimensional super-resolution processing and the three-dimensional super-resolution processing. Thus, it is possible to obtain a super-resolution image that does not deteriorate even in the pixel to be processed away from the optical axis center.

11a, 11b, 11c Low-resolution input image storage units 12a, 12b, 12c High-resolution conversion units 13a, 13b, 13c Estimated high-resolution image storage unit 14 Low-resolution conversion unit 15 Low-resolution image storage unit 16 Difference image generation unit (subtraction unit) )
17a, 17b, 17c Difference image storage unit 18 Index value calculation unit 19 High resolution conversion unit 20 Difference high resolution image storage unit 21 High resolution image generation unit (addition unit)
31a, 31b Motion detectors 111a, 111b, 114a, 114b Accumulator 112a, 112b, 113 Switch 121, 123 Switch 122 Shared high resolution converter 131a, 131b, 133a, 133b Switch 132, 134 Accumulator

Claims (12)

A first high-resolution conversion unit that generates a high-resolution input image by converting a low-resolution input image into a high-resolution image;
A low-resolution conversion unit that generates a low-resolution image by converting the estimated high-resolution image into a low-resolution image using the high-resolution input image or high-resolution image generated by the first high-resolution conversion unit as an estimated high-resolution image;
A difference image generation unit that generates a difference image from the low resolution image generated by the low resolution conversion unit and the low resolution input image;
A second high-resolution conversion unit that generates a differential high-resolution image by performing high-resolution conversion on the differential image generated by the differential image generation unit;
A high-resolution image generation unit that generates the high-resolution image obtained from the differential high-resolution image generated by the second high-resolution conversion unit and the estimated high-resolution image;
A control unit that controls whether the high resolution image generation unit repeatedly generates the high resolution image based on the estimated high resolution image or outputs the estimated high resolution image as a super-resolution image and ends the output. An image processing apparatus. The high-resolution image generation unit generates the high-resolution image from a differential high-resolution image obtained for the low-resolution input image and an estimated high-resolution image obtained for the second low-resolution input image. The image processing apparatus according to claim 1, wherein: A motion detector for detecting motion information from the low resolution input image and the second low resolution input image;
The high-resolution image generation unit refers to the differential image obtained for the low-resolution input image based on the motion information detected by the motion detection unit, and the differential high-resolution image obtained by high-resolution conversion The image processing apparatus according to claim 2, wherein the high-resolution image is generated from an estimated high-resolution image obtained for the second low-resolution input image. The motion detection unit detects motion information for each region,
Based on the motion information detected by the motion detection unit, the first high-resolution conversion unit applies an enlargement process that has a lower processing load than the high-resolution conversion for the region in which the motion is detected to the region in which no motion is detected. The image processing apparatus according to claim 3, wherein the high-resolution input image is generated. The control unit counts the number of times that the high-resolution image generation unit has repeatedly generated the high-resolution image with respect to the low-resolution input image, and based on the counted number of repeated generations, the high-resolution image 5. The method according to claim 1, wherein the image generation unit determines whether to generate the high-resolution image repeatedly or to output and output the estimated high-resolution image as a super-resolution image. The image processing apparatus according to any one of the above. An index value calculation unit that calculates an index value from the low resolution input image and the low resolution image;
The control unit repeatedly outputs the estimated high resolution image stored in the estimated high resolution image storage unit to the low resolution conversion unit and the high resolution image generation unit based on the index value calculated by the index value calculation unit. 5. The image processing apparatus according to claim 1, wherein the image processing apparatus determines whether or not to output as a super-resolution image and to end. An index value calculation unit that calculates an index value from the low resolution input image and the low resolution image;
The control unit is a conversion parameter that defines high-resolution conversion performed by the first and second high-resolution conversion units and low-resolution conversion performed by the low-resolution conversion unit based on the index value calculated by the index value calculation unit. The image processing apparatus according to claim 1, wherein the image processing apparatus is modified. The control unit is configured to perform high resolution conversion performed by the first and second high resolution conversion units and low resolution performed by the low resolution conversion unit based on a distance from the optical axis center of the optical system to the position of the conversion target pixel. The image processing apparatus according to claim 1, wherein a conversion parameter that defines the conversion is modified. A high-resolution conversion unit that generates a high-resolution input image obtained by converting a low-resolution input image or a differential image into a high-resolution image, a high-resolution input image obtained by converting the low-resolution input image into a high-resolution image, or a high-resolution converted image obtained by converting the differential image into a high-resolution image;
A high resolution input image or a high resolution image generated by the high resolution converter as an estimated high resolution image, a low resolution converter that generates a low resolution image by converting the estimated high resolution image to a low resolution;
A difference image generation unit for generating the difference image obtained from the low resolution image generated by the low resolution conversion unit and the low resolution input image;
A high-resolution image generation unit that generates the high-resolution image obtained from the difference high-resolution image generated by the high-resolution conversion unit and the estimated high-resolution image from the difference image generated by the difference image generation unit;
A control unit that controls whether the high resolution image generation unit repeatedly generates the high resolution image based on the estimated high resolution image or outputs the estimated high resolution image as a super-resolution image and ends the output. An image processing apparatus. A first high-resolution conversion step of converting a low-resolution input image to high-resolution to generate a high-resolution input image;
A low-resolution conversion step for generating a low-resolution image by converting the estimated high-resolution image into a low-resolution image using the high-resolution input image or the high-resolution image generated in the first high-resolution conversion step as an estimated high-resolution image;
A difference image generation step for generating a difference image from the low resolution image generated in the low resolution conversion step and the low resolution input image;
A second high resolution conversion step of generating a differential high resolution image by converting the differential image generated in the differential image generation step to high resolution;
A high-resolution image generation step for generating the high-resolution image obtained from the differential high-resolution image generated in the second high-resolution conversion step and the estimated high-resolution image;
Based on the estimated high-resolution image, the high-resolution image generation step is repeated from the low-resolution conversion step to generate the high-resolution image, or the estimated high-resolution image is output as a super-resolution image and the processing is terminated. And a control step for controlling the image processing method. The high-resolution image generation step includes generating the high-resolution image from a differential high-resolution image obtained for the low-resolution input image and an estimated high-resolution image obtained for the second low-resolution input image. The image processing method according to claim 10. An index value calculating step of calculating an index value from the low resolution input image and the low resolution image;
The control step defines a conversion parameter that defines high-resolution conversion performed in the first and second high-resolution conversion steps and low-resolution conversion performed in the low-resolution conversion step based on the index value calculated in the index value calculation step. The image processing method according to claim 10, wherein the image processing method is modified.

Images