JP2010004386A - Super resolution image capturing system, image capturing apparatus, super resolution image capturing method, super resolution image capturing program, and recording medium recorded with super resolution image capturing program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system for capturing a suitable number of position deviation images and generating a super resolution image. <P>SOLUTION: In the super resolution image capturing system, an image sensor and an optical system are relatively shifted and a plurality of position deviation image signals are captured and supplied to a super resolution processing section for creating a super resolution image. The super resolution image capturing system is further constituted of: an image feature analyzing section for analyzing an image signal from the image sensor to analyze features of the image; a shift data setting section which sets shift data for relatively shifting the image sensor and the optical system on the basis of a feature analysis signal from the image feature analyzing section; and a shift control section for controlling a shift section on the basis of a shift data signal from the shift data setting section. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present application belongs to a technical field of a super-resolution image acquisition system, an image acquisition device, a super-resolution image acquisition method, a super-resolution image acquisition program, and a recording medium on which the super-resolution image acquisition program is recorded. More specifically, the present application relates to a super-resolution image acquisition system, an image acquisition device, a super-resolution image acquisition method, a super-resolution image acquisition program, and a super-resolution image acquisition system that acquire a super-resolution image based on a plurality of misregistration images. The present invention belongs to the technical field of recording media on which an image image acquisition program is recorded.

  When generating a high-definition image by super-resolution, a plurality of misaligned images are acquired based on continuous shooting or moving image frames, etc., and the amount of misalignment (or amount of motion) of the plurality of low-resolution misaligned images And a high-definition image is generated by grid alignment from a plurality of low-resolution positional shift images.

  When photographing the plurality of misalignment images, there is a method of using camera shake of a handheld video camera. When there is a subtle movement at the time of shooting, such as hand-held shooting of a camera or an in-vehicle camera, it can be used as a misaligned image because of pixel movement between images.

  However, in the case of a fixed camera (such as a surveillance camera), since there is no movement of the camera itself, a misaligned image cannot be obtained for a still image, and as a result, super-resolution processing is performed. A high resolution image cannot be generated.

  In view of this, it is conceivable to obtain a plurality of misaligned images by moving the camera for a still image. According to such a method, it is possible to acquire a plurality of misaligned images for a still image and generate a high-resolution image by super-resolution processing.

For example, in the image input device described in Patent Document 1, the relative positional relationship between the imaging optical system and the imaging element is changed, and by using such a technique, a plurality of still images are displayed. A misalignment image can be acquired.
Japanese Patent Laid-Open No. 10-023321

  When using the image input device described in Patent Literature 1 and changing the relative positional relationship between the imaging optical system and the imaging element for a still image and acquiring a plurality of misaligned images, misaligned images If the number of images is small, there is a problem that the quality of the super-resolution processed image is deteriorated. On the other hand, if the number of misaligned images is more than necessary, there is a problem that the amount of processing, the amount of memory, and the like for generating a super-resolution image increase.

  The present application has been made in view of the above problems, and an example of the purpose thereof is to acquire an appropriate number of misaligned images, thereby improving the quality of the super-resolution processed image, Super-resolution image acquisition system, image acquisition device, super-resolution image acquisition method, super-resolution image acquisition program, and super-resolution The object is to provide a recording medium in which an image acquisition program is recorded.

  In order to solve the above-mentioned problem, the invention according to claim 1 is an image sensor that acquires an image of an object and outputs an image signal indicating the image, and an optical path between the object and the image sensor. A shift unit that relatively shifts the optical system disposed in the image sensor and the optical system so that a plurality of misaligned image signals are output from the image sensor, and a plurality of positions from the image sensor. A super-resolution image acquisition system including: a super-resolution processing unit that receives a misaligned image signal and creates a super-resolution image based on the plurality of misaligned image signals, and analyzes the image signal from the image sensor And shift data for setting shift data for relatively shifting the image sensor and the optical system based on a feature analysis signal from the image feature analysis unit. Setting And parts, based on the shift data signal from the shift data setting unit, a super-resolution image acquisition system, which comprises a shift control unit that controls the shift unit.

  In order to solve the above problems, an invention according to claim 8 is directed to an image sensor that acquires an image of an object and outputs an image signal indicating the image, and an optical path between the object and the image sensor. A shift unit that relatively shifts the optical system disposed in the image sensor and the optical system so that a plurality of misaligned image signals for generating a super-resolution image is output from the image sensor. And an image feature analysis unit that analyzes an image signal from the image sensor and analyzes a feature of the image, and based on a feature analysis signal from the image feature analysis unit, A shift data setting unit that sets shift data for relatively shifting the sensor and the optical system, and a shift control unit that controls the shift unit based on a shift data signal from the shift data setting unit. An image acquisition apparatus according to claim.

  In order to solve the above-mentioned problem, an invention according to claim 9 includes a step of acquiring an image of an object by an image sensor, an optical system disposed on an optical path between the object and the image sensor, and A step of relatively shifting the image sensor so that a plurality of misregistration image signals are output from the image sensor, and a step of creating a super-resolution image based on the plurality of misregistration image signals. A method for acquiring a super-resolution image, the step of analyzing an image signal from the image sensor and analyzing the characteristics of the image, and relatively shifting the image sensor and the optical system based on the characteristics of the image A method for acquiring a super-resolution image, comprising: setting shift data for controlling the image data; and controlling a relative shift between the optical system and the image sensor based on the shift data signal. .

  In order to solve the above-mentioned problem, a tenth aspect of the invention is a super-resolution image acquisition program that causes a computer to function as the system according to any one of the first to seventh aspects. It is.

  In order to solve the above problem, an invention according to claim 11 is a recording medium in which the program according to claim 10 is recorded so as to be readable by the computer.

  Next, the best mode for carrying out the present application will be described with reference to the drawings.

  FIG. 1 shows a block circuit of a super-resolution image acquisition system according to an embodiment of the present application.

  In FIG. 1, reference numeral 10 denotes an image sensor. The image sensor 10 acquires an image of an object (not shown) through a lens 12 as an optical system, and outputs an image signal 14 indicating the image. To do. An image sensor shift unit 16 is provided to shift the image sensor 10 with respect to the lens 12. When the image sensor shift unit 16 relatively shifts the image sensor 10 and the lens 12, an image of the object is sampled and acquired, and a plurality of misregistration image signals 14 are output from the image sensor 10.

  A plurality of misaligned image signals 14 from the image sensor 10 are supplied to a super-resolution processor 18. The super-resolution processing unit 18 creates a super-resolution image based on the plurality of misaligned image signals 14 and outputs a super-resolution image signal 20 indicating a high-definition super-resolution image. The super-resolution image signal 20 is supplied and displayed on an image display unit (not shown) such as an LCD (Liquid Crystal Display), or is supplied to an image recording unit (not shown) and recorded.

  The image signal 14 from the image sensor 10 is supplied to the image feature analysis unit 22. The image feature analysis unit 22 analyzes the feature of the image indicated by the image signal 14, that is, acquires the frequency component (edge component) for each of the horizontal direction, the vertical direction, and the oblique direction for the image indicated by the image signal 14. . The feature analysis signal 24 from the image feature analysis unit 22 is supplied to a shift data setting unit 26, which sets the shift direction and the shift amount of the image sensor 10 based on the feature analysis signal 24. That is, the shift amount for each of the horizontal direction, the vertical direction, and the oblique direction is set. The shift amount is M / N (M, N: integer) of the pixel interval of the image sensor 10.

  The setting signal 28 from the shift data setting unit 26 is supplied to an image sensor shift control unit 30. The image sensor shift control unit 30 controls the image sensor shift unit 16 based on the setting signal 28, and the image sensor 10 Shift in a predetermined direction by a predetermined shift amount. In other words, the image sensor shift control unit 30 reduces the shift amount in the direction in which the high frequency component is large in the image, and increases the shift amount in the direction in which the high frequency component is small. 10 is shifted in a predetermined direction by a predetermined shift amount.

  The operation of the super-resolution image acquisition system according to the embodiment of the present application will be described below with reference to FIGS. 1 and 2A to 2D.

  In the super-resolution image acquisition system according to the embodiment of the present application, the shift amount of the image sensor 10 is changed according to the characteristics of the acquired image, that is, the frequency component (edge component) in each of the horizontal direction and the vertical direction. To do. Note that the shift amount = (pixel interval of the image sensor 10) × (M / N) (M, N: integer), and N is changed according to the characteristics of the image, thereby shifting at the sub-pixel interval. Achieved.

  As described above, the shift amount of the image sensor 10 is changed according to the frequency component (edge component) for each of the horizontal direction, the vertical direction, and the oblique direction of the acquired image. The shift amount is reduced in the direction in which the high frequency component is large in the image, while the shift amount is increased in the direction in which the high frequency component is small. Hereinafter, the change of the shift amount will be described with reference to FIGS.

  In FIG. 2A, an image 32 of an object (not shown) contains many high frequency components in the horizontal direction (that is, many edge components in the vertical direction). Therefore, the horizontal shift amount 34 is set smaller than the vertical shift amount 36.

  In FIG. 2B, the image 32 of the object contains many high frequency components in the vertical direction (that is, many edge components in the horizontal direction). Therefore, the vertical shift amount 36 is set smaller than the horizontal shift amount 34.

  In FIG. 2C, the image 32 of the object contains many high-frequency components in both the horizontal direction and the vertical direction (that is, there are many edge components in both the horizontal direction and the vertical direction). Accordingly, both the horizontal shift amount 34 and the vertical shift amount 36 are set to be small.

  In FIG. 2D, the image 32 of the object includes a small amount of high frequency components in both the horizontal direction and the vertical direction (that is, the edge component is small in both the horizontal direction and the vertical direction). Accordingly, both the horizontal shift amount 34 and the vertical shift amount 36 are set large.

  As described above, according to the super-resolution image acquisition system according to the embodiment of the present application, the image sensor according to the frequency component (edge component) in each of the horizontal direction, the vertical direction, and the diagonal direction of the acquired image. Therefore, an appropriate number of misaligned images can be acquired.

  Therefore, when generating a super-resolution image based on a plurality of misaligned low-resolution images, the quality of the super-resolution image can be improved, and further, the processing amount and memory for generating the super-resolution image The amount and the like can be reduced.

  Next, FIG. 3 shows an example in which the number of misaligned images is fixed.

  In FIG. 3, the image 32 of the object contains a lot of high frequency components in the horizontal direction (that is, there are many edge components in the vertical direction), and therefore the horizontal shift amount 34 is larger than the vertical shift amount 36. In other words, the number of images taken in the horizontal direction is set larger than the number of images taken in the vertical direction. That is, if the number of low-resolution images 38 when shifted in the horizontal direction is L, and the number of low-resolution images 40 when shifted in the vertical direction is M, L is larger than M. Is set to Based on the low-resolution images 38 and 40 thus obtained, a super-resolution high-definition image 42 is obtained.

  As described above, when the number of low-resolution images used to obtain a super-resolution image is constant, by assigning a large number of images to be captured in the horizontal direction containing a large amount of high-frequency components, An appropriate number of misaligned low resolution images can be acquired according to the feature.

  Therefore, when generating a super-resolution image based on a plurality of misaligned low-resolution images, a high-resolution image by super-resolution can be generated efficiently and with high quality, and further, a super-resolution image can be generated. It is possible to reduce the amount of processing, the amount of memory, and the like that are necessary for this.

  In the present application, when acquiring a plurality of misaligned image signals, for example, a camera shake correction function used in a digital video camera or the like may be used. In this case, a shift amount by image stabilization and an image Therefore, a configuration for calculating the optimum shift amount is required based on the shift amount based on the above feature (that is, the high frequency component in each direction of the image).

  Further, although the image sensor 10 is shifted with respect to the lens 12 in the embodiment of the present application, the lens 12 may be shifted with respect to the image sensor 10. In short, any configuration that shifts either an optical system such as a lens or an image sensor may be used.

  Further, when the image sensor 10 includes three sensor elements in charge of RGB (Red, Green, and Blue), the three sensor elements may be shift-controlled independently.

  There are several super-resolution processing methods for generating one clear high-resolution image using a plurality of low-resolution positional shift images acquired from an image sensor such as a camera. Here, An example of the super-resolution processing method will be described with reference to FIGS.

  In FIG. 4A, a plurality of low-resolution positional deviation images 44 are acquired by continuously shooting an object (subject) or shooting a moving image.

  In FIG. 4B, sub-pixel accuracy alignment (position shift detection and alignment) is performed. That is, the correspondence between images obtained by photographing the same part of the object among a plurality of images is obtained, and one high-resolution image 46 is generated.

  The image generated here includes a blur component, a noise component, and pixels that are insufficient for alignment.

  Then, a high-resolution image is reconstructed by MAP estimation or the like, that is, a pattern on the high-resolution image is estimated based on a certain condition (such as a statistical method) from corresponding pixel groups of a plurality of images. This process is performed for all pixels on the image.

  In FIG. 4C, a high-resolution image 48 is generated, which makes it possible to generate a clear image for an unclear image or a noisy image.

  In addition, this application is applicable about a surveillance camera, a vehicle-mounted camera (drive recorder), etc., for example.

1 is a block circuit diagram of a super-resolution image acquisition system according to an embodiment of the present application. It is a figure which shows the effect | action of the super-resolution image acquisition system by embodiment of this application. It is a figure which shows the example at the time of fixing the number of position shift images. It is a figure which shows an example of the method of a super-resolution process.

Explanation of symbols

10: Image sensor 12: Lens 16: Image sensor shift unit 18: Super-resolution processing unit 22: Image feature analysis unit 26: Shift data setting unit 30: Image sensor shift control unit

Claims (11)

An image sensor that acquires an image of the object and outputs an image signal indicating the image;
An optical system disposed on an optical path between the object and the image sensor;
A shift unit that relatively shifts the image sensor and the optical system so that a plurality of misaligned image signals are output from the image sensor;
A super-resolution processor that receives a plurality of misaligned image signals from the image sensor and creates a super-resolution image based on the misaligned image signals;
A super-resolution image acquisition system including:
An image feature analysis unit for analyzing an image signal from the image sensor and analyzing the feature of the image;
A shift data setting unit for setting shift data for relatively shifting the image sensor and the optical system based on a feature analysis signal from the image feature analysis unit;
A shift control unit for controlling the shift unit based on the shift data signal from the shift data setting unit;
A super-resolution image acquisition system comprising: The system of claim 1, wherein
The shift data setting unit sets a shift amount for each shift direction. The system of claim 2, wherein
The shift data setting unit sets a shift amount in each shift direction based on a frequency component of the image in each shift direction. The system of claim 3, wherein
The shift data setting unit obtains a super-resolution image characterized by setting a small shift amount for a shift direction with a high frequency component of the image and setting a large shift amount for a shift direction with a low frequency component of the image system. The system according to any one of claims 2 to 4,
The super-resolution image acquisition system, wherein the shift data setting unit sets a shift amount for each of a horizontal direction, a vertical direction, and an oblique direction. The system according to any one of claims 1 to 5,
The super-resolution image acquisition system, wherein the shift unit shifts one of the image sensor and the optical system. The system according to any one of claims 1 to 6,
The image sensor is composed of three sensor elements in charge of RGB,
The super-resolution image acquisition system, wherein the shift control unit controls the shift unit so that the three sensor elements shift independently. An image sensor that acquires an image of the object and outputs an image signal indicating the image;
An optical system disposed on an optical path between the object and the image sensor;
A shift unit that relatively shifts the image sensor and the optical system so that a plurality of misaligned image signals for generating a super-resolution image are output from the image sensor;
An image acquisition device comprising:
An image feature analysis unit for analyzing an image signal from the image sensor and analyzing the feature of the image;
A shift data setting unit for setting shift data for relatively shifting the image sensor and the optical system based on a feature analysis signal from the image feature analysis unit;
A shift control unit for controlling the shift unit based on the shift data signal from the shift data setting unit;
An image acquisition apparatus comprising: Acquiring an image of the object by an image sensor;
Relatively shifting an optical system arranged on an optical path between the object and the image sensor and the image sensor so that a plurality of misaligned image signals are output from the image sensor;
A super-resolution image acquisition method including a step of creating a super-resolution image based on the plurality of misalignment image signals,
Analyzing the image signal from the image sensor and analyzing the characteristics of the image;
Setting shift data for relatively shifting the image sensor and the optical system based on the characteristics of the image;
Controlling a relative shift between the optical system and the image sensor based on the shift data signal;
A super-resolution image acquisition method comprising:   A super-resolution image acquisition program for causing a computer to function as the system according to any one of claims 1 to 7. 11. A recording medium on which the program according to claim 10 is recorded so as to be readable by the computer.

Images