JP4529010B1 - Imaging device - Google Patents

Abstract

An imaging apparatus capable of generating a high-quality, high-definition two-dimensional image regardless of the parallax of a stereo image, that is, regardless of a shooting distance.
A plurality of imaging elements; a plurality of solid lenses that form an image on each of the imaging elements; a plurality of optical axis controllers that control directions of optical axes of light incident on the imaging elements; A photoelectric conversion signal output from each of the plurality of image sensors is input, a plurality of video processing units that convert and output the video signal, and stereo matching processing based on the plurality of video signals are performed. The shift amount is calculated, and the shift amount exceeding the pixel pitch of the image sensor inputs a stereo image processing unit that generates a composite parameter normalized by the pixel pitch, and a video signal output from each of a plurality of video processing units and a composite parameter. And a video synthesis processing unit that generates a high-definition video by synthesizing the plurality of video signals based on a synthesis parameter.
[Selection] Figure 10

Description

  The present invention relates to an imaging apparatus.

  In recent years, high-quality digital still cameras and digital video cameras (hereinafter referred to as digital cameras) are rapidly spreading. At the same time, the development of miniaturization and thinning of digital cameras is also underway, and small and high-quality digital cameras have been mounted on mobile phone terminals and the like. An image pickup apparatus represented by a digital camera includes an image pickup element, an imaging optical system (lens optical system), an image processor, a buffer memory, a flash memory (card type memory), an image monitor, an electronic circuit and a mechanical mechanism for controlling these, and the like. It is composed of A solid-state electronic device such as a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor is usually used for the image sensor. The light quantity distribution imaged on the image sensor is photoelectrically converted, and the obtained electric signal is processed by an image processor and a buffer memory. A DSP (Digital Signal Processor) or the like is used as the image processor, and a DRAM (Dynamic Random Access Memory) or the like is used as the buffer memory. The captured image is recorded and accumulated in a card type flash memory or the like, and the recorded and accumulated image can be displayed on a monitor.

  An optical system for forming an image on an image sensor is usually composed of several aspheric lenses in order to eliminate aberrations. In addition, when an optical zoom function is provided, a driving mechanism (actuator) that changes the focal length of the combination lens and the distance between the lens and the image sensor is necessary. In response to demands for higher image quality and higher functionality of imaging devices, the imaging element has increased in number and resolution, the imaging optical system has lower aberration and higher precision, and has a zoom function, autofocus function, and camera shake. Advanced functions such as correction functions are advancing. Along with this, there is a problem that the imaging device becomes large and it is difficult to reduce the size and thickness.

  In order to solve such problems, it has been proposed to reduce the size and thickness of the imaging device by adopting a compound eye structure in the imaging optical system or combining non-solid lenses such as liquid crystal lenses and liquid lenses. ing. For example, an imaging lens device that includes a solid lens array, a liquid crystal lens array, and a single imaging device arranged in a planar shape has been proposed (for example, Patent Document 1). As shown in FIG. 36, a lens system having a fixed focal length lens array 2001 and the same number of variable focus type liquid crystal lens arrays 2002, and a single image sensor for imaging an optical image formed through this lens system. 2003. With this configuration, the same number of images as the number of lens arrays 2001 are divided and imaged on a single image sensor 2003. A plurality of images obtained from the image sensor 2003 are subjected to image processing by the arithmetic unit 2004 to reconstruct the entire image. Further, focus information is detected from the arithmetic unit 2004, and each liquid crystal lens of the liquid crystal lens array 2002 is driven via the liquid crystal drive unit 2005 to perform auto focus. As described above, the imaging lens device of Patent Document 1 has an autofocus function and a zoom function and can be downsized by combining a liquid crystal lens and a solid lens.

  There is also an image pickup apparatus that includes one non-solid lens (liquid lens, liquid crystal lens), a solid lens array, and one image pickup device (for example, Patent Document 2). As shown in FIG. 37, it is composed of a liquid crystal lens 2131, a compound eye optical system 2120, an image synthesizer 2115, and a drive voltage calculation unit 2142. Similar to Patent Document 1, the same number of images as the number of lens arrays are formed on a single image sensor 2105, and the image is reconstructed by image processing. As described above, in the imaging apparatus of Patent Document 2, a focus adjustment function can be realized with a small size and a thin shape by combining one non-solid lens (liquid lens, liquid crystal lens) and a solid lens array.

  In addition, in a thin camera with subpixel resolution composed of a detector array, which is an image sensor, and an imaging lens array, the relative displacement of the images on the two subcameras is changed to increase the resolution of the composite image. The method of making it known is known (for example, patent document 3). This method solves the problem that the resolution cannot be improved depending on the subject distance by providing a diaphragm in one of the sub-cameras and blocking light for half a pixel by this diaphragm. Japanese Patent Laid-Open No. 2004-228561 also changes the image formation position and the pixel phase simultaneously by changing the focal length by combining a liquid lens capable of controlling the focal length by applying a voltage from the outside. Thus, the resolution of the composite image is increased. As described above, the thin camera disclosed in Patent Document 3 achieves high definition of the composite image by combining the imaging lens array and the imaging element having the light shielding unit. Further, by combining a liquid lens with the imaging lens array and the imaging device, it is possible to realize a high definition of the composite image.

  Also, there is known an image generation method and apparatus for mapping an image to a spatial model by performing super-resolution interpolation processing on a specific region where the parallax of the stereo image is small among image information of a plurality of imaging means (for example, patents) Reference 4). In this apparatus, it is possible to solve the problem that the definition of image data to be pasted on a distant spatial model is lacking in spatial model generation performed in the process of generating a viewpoint conversion image from images captured by a plurality of imaging means.

JP 2006-251613 A JP 2006-217131 A Special table 2007-520166 Special table 2006-119843 gazette

  However, the imaging lens devices disclosed in Patent Documents 1 to 3 have a problem in that the accuracy of adjustment of the relative position between the optical system and the imaging element affects the image quality, and thus it is necessary to make an accurate adjustment during assembly. In addition, when the relative position is adjusted only by mechanical accuracy, a highly accurate non-solid lens or the like is required, which increases the cost. Even if the device is accurately adjusted at the time of assembling the apparatus, the relative position between the optical system and the image sensor may change due to changes over time, and image quality may be deteriorated. If the position is adjusted again, the image quality is improved. However, there is a problem that the same adjustment as that during assembly is required. Furthermore, in an apparatus provided with a large number of optical systems and image sensors, there are a number of locations to be adjusted, and there is a problem that a great amount of work time is required.

  In addition, in the image generation method and apparatus of Patent Document 4, it is necessary to generate an accurate space model in order to generate a viewpoint conversion image. However, three-dimensional information called a space model is accurately acquired as a stereo image. There is a problem that it is difficult. In particular, in a distant image with a small parallax of a stereo image, it is difficult to obtain three-dimensional information called a spatial model with a stereo image with high accuracy due to the change in luminance of the image and the influence of noise. Therefore, even if a super-resolved image can be generated in a specific region with a small parallax of a stereo image, it is difficult to map to a spatial model with high accuracy.

  The present invention has been made in view of such circumstances, and in order to realize a high-quality image pickup apparatus, the relative position of the optical system and the image pickup element is easily adjusted without requiring manual work. An object of the present invention is to provide an imaging device capable of performing the above. It is another object of the present invention to provide an imaging apparatus capable of generating a high-quality, high-definition two-dimensional image regardless of the parallax of a stereo image, that is, regardless of the shooting distance.

  The present invention includes a plurality of image sensors, a plurality of solid lenses that form images on each of the image sensors, and a plurality of optical axis controllers that control the directions of the optical axes of light incident on the image sensors. And by inputting a photoelectric conversion signal output from each of the plurality of image sensors, converting the video signal into a video signal, and performing stereo matching processing based on the video signal, A shift amount for each pixel is obtained, and a shift amount exceeding the pixel pitch of the image sensor generates a composite parameter normalized by the pixel pitch, and the video output from each of the plurality of video processing units A video synthesis processing unit that inputs a signal and the synthesis parameter and generates a high-definition video by synthesizing the plurality of video signals based on the synthesis parameter; And wherein the door.

  The present invention further includes a stereo image noise reduction processing unit that reduces noise of a parallax image used for the stereo matching processing based on the synthesis parameter generated by the stereo image processing unit.

  The present invention is characterized in that the video composition processing unit increases the definition only in a predetermined area based on the parallax image generated by the stereo image processing unit.

  According to the present invention, a plurality of image sensors, a plurality of solid lenses that form images on the image sensors, and a plurality of optical axis controllers that control the optical axes of light incident on the image sensors, respectively. Accordingly, adjustment of the relative position of the optical system and the image sensor can be easily performed without requiring manual work, and an effect that an image pickup apparatus with high image quality can be realized is obtained. In particular, since it is possible to control the optical axis of incident light to be set at an arbitrary position on the image sensor surface, the position between the optical system and the image sensor can be easily adjusted. An imaging device can be realized. In addition, since the direction of the optical axis is controlled based on the relative position between the imaging target and the plurality of optical axis controllers, it is possible to set the optical axis at an arbitrary position on the imaging element surface, and focus adjustment An imaging device with a wide range can be realized.

A plurality of imaging elements; a plurality of solid lenses that form images on each of the imaging elements; a plurality of optical axis control units that control directions of optical axes of light incident on the imaging elements; The pixel conversion is performed by performing a stereo matching process based on a plurality of video processing units that input photoelectric conversion signals output from each of the image pickup devices, convert them into video signals, and output the video signals. The amount of shift exceeding the pixel pitch of the imaging element is determined, and a stereo image processing unit that generates a synthesis parameter normalized by the pixel pitch, and the video signal output from each of the plurality of video processing units and the synthesis And a video composition processing unit that generates a high-definition video by synthesizing the plurality of video signals based on the synthesis parameters. Regardless parallax Oh image, i.e. regardless of the photographing distance, high-quality, it is possible to generate a high-resolution two-dimensional image.

  In addition, according to the present invention, since the stereo image noise reduction processing unit for reducing the noise of the parallax image used for the stereo matching processing is further provided based on the synthesis parameter generated by the stereo image processing unit, Noise can be removed.

  Further, according to the present invention, the video composition processing unit performs high-definition only in a predetermined area based on the parallax image generated by the stereo image processing unit, so that the high-definition processing can be speeded up.

1 is a block diagram illustrating a configuration of an imaging apparatus according to a first embodiment of the present invention. It is a detailed block diagram of the unit imaging part of the imaging device by 1st Embodiment shown in FIG. It is a block diagram of a liquid crystal lens. It is a schematic diagram explaining the function of the liquid crystal lens used for the imaging device by 1st Embodiment. It is a schematic diagram explaining the liquid crystal lens of the imaging device by 1st Embodiment. It is a schematic diagram explaining the image pick-up element of the image pick-up device by 1st Embodiment shown in FIG. It is a detailed schematic diagram of an image sensor. It is a block diagram which shows the whole structure of the imaging device 1 shown in FIG. 2 is a detailed block diagram of a video processing unit of the imaging apparatus according to the first embodiment. FIG. It is a detailed block diagram of a video composition processing unit of video processing of the imaging device according to the first embodiment. 2 is a detailed block diagram of a video processing control unit of the imaging apparatus according to the first embodiment. FIG. It is a flowchart explaining an example of operation | movement of a control part. It is explanatory drawing which shows the operation | movement of the sub pixel image | video synthetic | combination high definition process shown in FIG. It is a flowchart explaining an example of high definition determination. It is a flowchart explaining an example of a control voltage change process. It is a flowchart explaining an example of camera calibration. It is a schematic diagram explaining the camera calibration of a unit imaging part. It is a schematic diagram explaining the camera calibration of a several unit imaging part. It is another schematic diagram explaining the camera calibration of a some unit imaging part. 3 is a schematic diagram illustrating a state of imaging by the imaging apparatus 1. FIG. It is a schematic diagram explaining a high-definition subpixel. It is another schematic diagram explaining a high-definition subpixel. It is explanatory drawing which shows the relationship between an imaging target (object) and image formation. 6 is a schematic diagram illustrating the operation of the imaging apparatus 1. FIG. It is a schematic diagram when an image sensor is displaced and attached due to an attachment error. It is a schematic diagram which shows the operation | movement of optical axis shift control. It is explanatory drawing which shows the relationship between an imaging distance and an optical axis shift. It is explanatory drawing which shows the relationship between an imaging distance and an optical axis shift. It is explanatory drawing which shows the effect of the image shift by depth and an optical axis shift. It is a flowchart explaining an example which produces | generates the translation parameter for every pixel. It is explanatory drawing which shows an example of the epipolar line in the case of a parallel stereo structure. It is explanatory drawing which shows an example of the area | region base matching in the case of a parallel stereo structure. It is explanatory drawing which shows an example of a parallax image. It is a detailed block diagram of a video composition processing unit of video processing of an imaging device according to another embodiment. It is a flowchart explaining an example of noise removal. It is a block diagram which shows the structure of the conventional imaging device. It is a block diagram which shows the structure of another conventional imaging device.

  Embodiments of the present invention will be described below in detail with reference to the drawings. FIG. 1 is a functional block diagram showing the overall configuration of the imaging apparatus according to the first embodiment of the present invention. The imaging apparatus 1 illustrated in FIG. 1 includes six unit imaging units 2 to 7. The unit imaging unit 2 includes an imaging lens 8 and an imaging element 14. Similarly, the unit imaging unit 3 is an imaging lens 9 and an imaging element 15, the unit imaging unit 4 is an imaging lens 10 and an imaging element 16, the unit imaging unit 5 is an imaging lens 11 and an imaging element 17, and the unit imaging unit 6 is an imaging lens 12. The image pickup device 18 and the unit image pickup unit 7 include an image pickup lens 13 and an image pickup device 19. Each of the imaging lenses 8 to 13 forms an image of light from the subject on the corresponding imaging elements 14 to 19, respectively. Reference numerals 20 to 25 illustrated in FIG. 1 indicate optical axes of light incident on the imaging elements 14 to 19.

  Hereinafter, the flow of signals will be described using the unit imaging unit 3 as an example. The image formed by the imaging lens 9 is photoelectrically converted by the imaging element 15 to convert the optical signal into an electrical signal. The electrical signal converted by the image sensor 15 is converted into a video signal by a parameter set in advance by the video processing unit 27. The video processing unit 27 outputs the converted video signal to the video composition processing unit 38. The video composition processing unit 38 inputs video signals converted by the video processing units 26 and 28 to 31 corresponding to the electrical signals output from the other unit imaging units 2 and 4 to 7. The video composition processing unit 38 synthesizes the six video signals picked up by the unit image pickup units 2 to 7 into one video signal while synchronizing them, and outputs it as a high-definition video. Here, the video composition processing unit 38 synthesizes a high-definition video based on the result of stereo image processing described later. In addition, when the synthesized high-resolution video is deteriorated from a predetermined determination value, the video composition processing unit 38 generates a control signal based on the determination result and outputs the control signal to the six control units 32 to 37. . Each control unit 32 to 37 performs optical axis control of the corresponding imaging lens 8 to 13 based on the input control signal. Then, the video composition processing unit 38 determines the high-definition video again. If this determination result is good, a high-definition video is output, and if it is bad, the operation of controlling the imaging lens is repeated.

  Next, a detailed configuration of the imaging lens 9 of the unit imaging unit 3 and the control unit 33 that controls the imaging lens 9 illustrated in FIG. 1 will be described with reference to FIG. The unit imaging unit 3 includes a liquid crystal lens (non-solid lens) 301 and an optical lens (solid lens) 302. The control unit 33 includes four voltage control units 33a, 33b, 33c, and 33d that control the voltage applied to the liquid crystal lens 301. The voltage control units 33a, 33b, 33c, and 33d determine the voltage to be applied to the liquid crystal lens 301 based on the control signal generated by the video composition processing unit 38, and control the liquid crystal lens 301. Since the imaging lenses and control units of the other unit imaging units 2 and 4 to 7 shown in FIG. 1 have the same configuration, detailed description thereof is omitted here.

  Next, the configuration of the liquid crystal lens 301 shown in FIG. 2 will be described with reference to FIG. In the present embodiment, the liquid crystal lens 301 includes a transparent first electrode 303, a second electrode 304, a transparent third electrode 305, and a liquid crystal disposed between the second electrode 304 and the third electrode 305. A first insulating layer 307 disposed between the layer 306, the first electrode 303 and the second electrode 304, and a second insulating layer disposed between the second electrode 304 and the third electrode 305. 308, a third insulating layer 311 disposed outside the first electrode 303, and a fourth insulating layer 312 disposed outside the third electrode 305. Here, the second electrode 304 has a circular hole, and is composed of four electrodes 304a, 304b, 304c, and 304d divided vertically and horizontally as shown in the front view of FIG. The voltage can be applied independently to each electrode. In addition, the liquid crystal layer 306 has liquid crystal molecules aligned in one direction so as to face the third electrode 305, and by applying a voltage between the electrodes 303, 304, and 305 that sandwich the liquid crystal layer 306, Alignment control is performed. Further, the insulating layer 308 is made of, for example, a transparent glass having a thickness of about several hundreds μm in order to increase the diameter.

  As an example, the dimensions of the liquid crystal lens 301 are shown below. The size of the circular hole of the second electrode 304 is about φ2 mm, the distance from the first electrode 303 is 70 μm, and the thickness of the second insulating layer 308 is 700 μm. The thickness of the liquid crystal layer 306 is 60 μm. Although the first electrode 303 and the second electrode 304 are different layers in this embodiment mode, they may be formed on the same surface. In that case, the shape of the first electrode 303 is smaller than the circular hole of the second electrode 304 and is arranged at the hole position of the second electrode 304, and the electrode is taken out at the divided portion of the second electrode 304. It is set as the structure which provided the part. At this time, the electrodes 304a, 304b, 304c, and 304d constituting the first electrode 303 and the second electrode can be independently voltage controlled. With this configuration, the overall thickness can be reduced.

  Next, the operation of the liquid crystal lens 301 shown in FIG. 3 will be described. In the liquid crystal lens 301 shown in FIG. 3, a voltage is applied between the transparent third electrode 305 and the second electrode 304 made of an aluminum thin film or the like, and at the same time, the first electrode 303 and the second electrode 304 are applied. By applying a voltage between them, an electric field gradient targeted for the axis can be formed on the central axis 309 of the second electrode 304 having a circular hole. The liquid crystal molecules of the liquid crystal layer 306 are aligned in the direction of the electric field gradient due to the axial target electric field gradient around the edge of the circular electrode formed in this way. As a result, the refractive index distribution of extraordinary light changes from the center to the periphery of the circular electrode due to the change in the orientation distribution of the liquid crystal layer 306, and thus the lens can function. The refractive index distribution of the liquid crystal layer 306 can be freely changed by applying a voltage to the first electrode 303 and the second electrode 304, and optical characteristics such as a convex lens and a concave lens can be freely controlled. Is possible.

  In this embodiment, an effective voltage of 20 Vrms is applied between the first electrode 303 and the second electrode 304, and an effective voltage of 70 Vrms is applied between the second electrode 304 and the third electrode 305. Thus, an effective voltage of 90 Vrms is applied between the first electrode 303 and the third electrode 305 to function as a convex lens. Here, the liquid crystal driving voltage (voltage applied between the electrodes) is a sine wave or a rectangular wave AC waveform with a duty ratio of 50%. The voltage value to be applied is represented by an effective voltage (rms: root mean square value). For example, an AC sine wave of 100 Vrms has a voltage waveform having a peak value of ± 144V. Further, for example, 1 kHz is used as the frequency of the AC voltage. Further, by applying different voltages between the electrodes 304a, 304b, 304c, and 304d constituting the second electrode 304 and the third electrode 305, the refractive index was axially symmetric when the same voltage was applied. The distribution is an asymmetric distribution with the axis shifted with respect to the second electrode central axis 309 having a circular hole, and the effect of deflecting the incident light from the straight direction is obtained. In this case, the direction of incident light deflection can be changed by appropriately changing the voltage applied between the divided second electrode 304 and third electrode 305. For example, by applying 70 Vrms between the electrode 304a and the electrode 305, between the electrode 304c and the electrode 305, 71Vrms between the electrode 304b and the electrode 305, and between the electrode 304d and the electrode 305, respectively, the optical axis denoted by reference numeral 309 The position is shifted to the position indicated by reference numeral 310. The shift amount is 3 μm, for example.

  FIG. 4 is a schematic diagram for explaining the optical axis shift function of the liquid crystal lens 301. As described above, by controlling the voltage applied between the electrodes 304a, 304b, 304c, and 304d constituting the second electrode and the third electrode 305 for each of the electrodes 304a, 304b, 304c, and 304d, the imaging device The center axis of the liquid crystal lens and the center axis of the refractive index distribution of the liquid crystal lens can be shifted. Since this corresponds to the lens being displaced in the xy plane with respect to the image pickup device surface, the light beam input to the image pickup device can be deflected in the u and v planes.

  FIG. 5 shows a detailed configuration of the unit imaging unit 3 shown in FIG. The optical lens 302 in the unit imaging unit 3 includes two optical lenses 302a and 302b, and the liquid crystal lens 301 is disposed between the optical lenses 302a and 302b. Each of the optical lenses 302a and 302b is composed of one or a plurality of lenses. Light rays incident from the object plane are collected by an optical lens 302a disposed on the object plane side of the liquid crystal lens 301, and are incident on the liquid crystal lens 301 in a state where the spot is reduced. At this time, the incident angle of the light beam to the liquid crystal lens 301 is almost parallel to the optical axis. The light beam emitted from the liquid crystal lens 301 is imaged on the surface of the image sensor 15 by the optical lens 302b disposed on the image sensor 15 side of the liquid crystal lens 301. With such a structure, the diameter of the liquid crystal lens 301 can be reduced, the voltage applied to the liquid crystal lens 301 can be reduced, the lens effect can be increased, and the thickness of the second insulating layer 308 can be reduced. The lens thickness can be reduced.

  In the image pickup apparatus 1 shown in FIG. 1, one image pickup lens is arranged for one image pickup device. However, in the liquid crystal lens 301, a plurality of second electrodes 304 are formed on the same substrate, and a plurality of second electrode 304 is formed. A configuration in which a liquid crystal lens is integrated may be used. That is, in the liquid crystal lens 301, since the hole portion of the second electrode 304 corresponds to the lens, a plurality of patterns of the second electrode 304 are arranged on one substrate, whereby each of the second electrodes 304 is arranged. The hole portion has a lens effect. Therefore, by arranging the plurality of second electrodes 304 on the same substrate in accordance with the arrangement of the plurality of image sensors, it is possible to deal with all the image sensors with a single liquid crystal lens unit.

  In the above description, the number of liquid crystal layers is one. However, by reducing the thickness of one layer and forming it with a plurality of layers, the responsiveness is improved while maintaining the same light collecting property. It is also possible to do. This is because the response speed deteriorates as the thickness of the liquid crystal layer increases. When the liquid crystal layer is composed of a plurality of layers, the lens effect can be obtained in all the polarization directions with respect to the light incident on the liquid crystal lens by changing the direction of polarization between the liquid crystal layers. Furthermore, the number of electrode divisions is exemplified as a four-division type as an example, but the number of electrode divisions can be changed according to the direction in which the electrode is desired to move.

  Next, the configuration of the image sensor 15 shown in FIG. 1 will be described with reference to FIGS. 6 and 7. As an example, a CMOS imaging device can be used as the imaging device of the imaging apparatus according to the present embodiment. In FIG. 6, the image sensor 15 is composed of pixels 501 in a two-dimensional array. The pixel size of the CMOS image sensor of this embodiment is 5.6 μm × 5.6 μm, the pixel pitch is 6 μm × 6 μm, and the effective pixel number is 640 (horizontal) × 480 (vertical). Here, the pixel is a minimum unit of an imaging operation performed by the imaging device. Usually, one pixel corresponds to one photoelectric conversion element (for example, a photodiode). There is a light receiving part with a certain area (spatial spread) in each pixel size of 5.6 μm, and the pixel averages and integrates the light incident on the light receiving part to convert it into an electric signal and convert it into an electric signal To do. The averaging time is controlled by an electronic or mechanical shutter or the like, and its operating frequency generally matches the frame frequency of the video signal output from the imaging device, for example 60 Hz.

  FIG. 7 shows a detailed configuration of the image sensor 15. The pixel 501 of the CMOS image sensor 15 amplifies the signal charge photoelectrically converted by the photodiode 515 by the amplifier 516. The signal of each pixel is selected by the vertical scanning circuit 511 and the horizontal scanning circuit 512 by the vertical horizontal address method, and is taken out as a voltage or a current. A CDS (Correlated Double Sampling) 518 is a circuit that performs correlated double sampling, and can suppress 1 / f noise among random noises generated by the amplifier 516 and the like. The pixels other than the pixel 501 have the same configuration and function. In addition, it can be mass-produced by applying CMOS logic LSI manufacturing processes, so it is cheaper than CCD image sensors with high-voltage analog circuits, consumes less power because of its smaller elements, and in principle smears and blooming There is also an advantage that it does not occur. In this embodiment, the monochrome CMOS image sensor 15 is used, but a color-compatible CMOS image sensor in which R, G, and B color filters are individually attached to each pixel can also be used. Using a Bayer structure in which the repetition of R, G, G, and B is arranged in a checkered pattern, colorization can be easily achieved with a single image sensor.

  Next, the overall configuration of the imaging apparatus 1 will be described with reference to FIG. In FIG. 8, the same parts as those shown in FIG. In FIG. 8, P001 is a CPU (Central Processing Unit) that controls the overall processing operation of the imaging apparatus 1 and may be called a microcontroller (microcomputer). P002 is a ROM (Read Only Memory) composed of a non-volatile memory, and stores setting values necessary for the CPU and P001 programs and each processing unit. P003 is a RAM (Random Access Memory), which stores temporary data of the CPU. P004 is a VideoRAM, which mainly stores video signals and image signals in the middle of calculation, and is composed of SDRAM (Synchronous Dynamic RAM) or the like.

  FIG. 8 shows a configuration in which the RAM P003 is stored for program storage of the CPU P001 and the VideoRAM P004 is stored for image storage. However, for example, two RAM blocks may be unified to the VideoRAM P004. P005 is a system bus to which a CPU / P001, a ROM / P002, a RAM / P003, a VideoRAM / P004, a video processing unit 27, a video composition processing unit 38, and a control unit 33 are connected. The system bus P005 is also connected to internal blocks of the video processing unit 27, the video composition processing unit 38, and the control unit 33, which will be described later. The CPU P001 controls the system bus P005 as a host, and setting data necessary for video processing, image processing, and optical axis control flows bidirectionally. Further, for example, the system bus P005 is used when an image being processed by the video composition processing unit 38 is stored in the VideoRAM · P004. A bus for an image signal that requires a high transfer speed and a low-speed data bus may be different bus lines. The system bus P005 is connected to an external interface such as a USB or flash memory card (not shown) and a display drive controller of a liquid crystal display as a viewfinder.

  Next, processing operations of the video processing unit 27 and the video composition processing unit 38 will be described with reference to FIGS. 9 and 10. FIG. 9 is a block diagram illustrating a configuration of the video processing unit 27. In FIG. 9, reference numeral 601 denotes a video input processing unit, 602 denotes a correction processing unit, and 603 denotes a calibration parameter storage unit. The video input processing unit 601 inputs the video signal captured from the unit imaging unit 3, performs signal processing such as knee processing and gamma processing, and also performs white balance control. The output of the video input processing unit 601 is passed to the correction processing unit 602, and distortion correction processing based on calibration parameters obtained by a calibration procedure described later is performed. For example, distortion caused by the mounting error of the image sensor 15 is calibrated. The calibration parameter storage unit 603 is a RAM (Random Access Memory) and stores a calibration value (calibration value). The corrected video signal that is output from the correction processing unit 602 is output to the video composition processing unit 38. The data stored in the calibration parameter storage unit 603 is updated by the CPU P001 when the imaging apparatus is powered on, for example. Alternatively, the calibration parameter storage unit 603 may be a ROM (Read Only Memory), and the stored data may be determined and stored in the ROM by a calibration procedure at the time of factory shipment.

  The video input processing unit 601, the correction processing unit 602, and the calibration parameter storage unit 603 are each connected to the system bus P005. For example, the aforementioned gamma processing characteristics of the video input processing unit 601 are stored in the ROM P002. The video input processing unit 601 receives the data stored in the ROM P002 via the system bus P005 by the program of the CPU P001. Further, the correction processing unit 602 writes image data in the middle of calculation to VideoRAM · P004 via the system bus P005 or reads out from the VideoRAM · P004. In this embodiment, a monochrome CMOS image sensor 15 is used. However, when a color CMOS image sensor is used, for example, when the image sensor has a Bayer structure, the image processing unit 601 performs Bayer interpolation processing. It will be.

  FIG. 10 is a block diagram showing a configuration of the video composition processing unit 38. The composition processing unit 701 performs composition processing on the imaging results of the plurality of unit imaging units 2 to 7. The composition processing can improve the resolution of the image as will be described later. The synthesis parameter storage unit 702 stores image shift amount data obtained from, for example, three-dimensional coordinates between unit imaging units derived by calibration described later. The determination unit 703 generates a signal to the control unit based on the video composition result. The stereo image processing unit 704 obtains a shift amount for each pixel (shift parameter for each pixel) from each captured image of the plurality of unit imaging units 2 to 7. Also, data normalized by the pixel pitch of the image sensor is obtained according to the imaging condition (distance).

  The composition processing unit 701 shifts the image based on this shift amount and composes it. The determination unit 703 detects the power of the high-band component of the video signal by, for example, Fourier transforming the result of the synthesis process. Here, it is assumed that, for example, a combination process of four unit imaging units is performed. It is assumed that the image sensor is a wide VGA (854 pixels × 480 pixels). Further, it is assumed that the video output that is the output of the video composition processing unit 38 is a high-vision signal (1920 pixels × 1080 pixels). In this case, the frequency band determined by the determination unit 703 is approximately 20 MHz to 30 MHz. The upper limit of the video frequency band at which a wide VGA video signal can be reproduced is approximately 10 to 15 MHz. By using the wide VGA signal, the synthesis processing unit 701 performs synthesis processing to restore a component of 20 MHz to 30 MHz. Here, although the image pickup device is a wide VGA, the image pickup optical system mainly including the image pickup lenses 8 to 13 is required to have characteristics that do not deteriorate the band of the high vision signal.

  The control unit 32 to the control unit 37 are controlled so that the power of the frequency band (20 MHz to 30 MHz component in the above example) of the video signal after synthesis is maximized. In order to make a determination on the frequency axis, the determination unit 703 performs a Fourier transform process, and determines the magnitude of energy above a specific frequency (for example, 20 MHz). The effect of restoring the video signal band that exceeds the band of the image sensor changes depending on the phase when the image formed on the image sensor is sampled within a range determined by the size of the pixel. In order to make this phase into an optimal state, the imaging lenses 8 to 13 are controlled using the control units 32 to 37. Specifically, the control unit 33 controls the liquid crystal lens 301 in the imaging lens 9. By controlling the balance of the voltages applied to the divided electrodes 304a, 304b, 304c, and 304d of the liquid crystal lens 301, the image on the image sensor surface moves as shown in FIG. The ideal state of the control result is a state in which the sampling phase of the imaging result of each unit imaging unit is shifted in the horizontal, vertical, and diagonal directions by ½ of the pixel size. In such an ideal state, the energy of the high band component as a result of the Fourier transform is maximized. That is, control is performed so that the energy of the result of the Fourier transform is maximized by a feedback loop that controls the liquid crystal lens and determines the resultant synthesis process.

  In this control method, the imaging lens 2 and the imaging lenses 4 to 7 are controlled via the control units 32 and 34 to 37 other than the control unit 33 with the video signal from the video processing unit 27 as a reference. In this case, the optical axis phase of the imaging lens 2 is controlled by the control unit 32. The optical axis phase is similarly controlled for the other imaging lenses 4 to 7. By controlling the phase with a size smaller than the pixel of each image sensor, the phase offset averaged by the image sensor is optimized. In other words, when sampling an image formed on the image sensor with pixels, the sampling phase is controlled to an ideal state for high definition by controlling the optical axis phase. As a result, it becomes possible to synthesize high-definition and high-quality video signals. The determination unit 703 determines the synthesis processing result, and if a high-definition and high-quality video signal is synthesized, maintains the control value and outputs the high-definition and high-quality video signal as video. On the other hand, if a high-definition and high-quality video signal cannot be synthesized, the imaging lens is controlled again.

  Here, since the phase of the imaging element 1 and the imaging target image is equal to or smaller than the size of the pixel, the subpixel is defined with a name, but the structure of the subpixel that divides the pixel is on the imaging element. It does n’t really exist. The output of the video composition processing unit 38 is, for example, a video signal, which is output to a display (not shown), or passed to an image recording unit (not shown) and recorded on a magnetic tape or an IC card. The synthesis processing unit 701, the synthesis parameter storage unit 702, the determination unit 703, and the stereo image processing unit 704 are each connected to the system bus P005. The synthesis parameter storage unit 702 is composed of a RAM. For example, it is updated via the system bus P005 by the CPU P001 when the imaging apparatus is powered on. Further, the composition processing unit 701 writes the image data in the middle of the calculation to the VideoRAM / P004 via the system bus P005, or reads it from the VideoRAM / P004.

  The stereo image processing unit 704 obtains data normalized by the shift amount for each pixel (shift parameter for each pixel) and the pixel pitch of the image sensor. This means that when a video is synthesized with multiple image shift amounts (shift amounts for each pixel) within one screen of the captured video, specifically, a focused video is shot from a subject with a short shooting distance to a subject with a long shooting distance. Effective when you want to. That is, an image with a deep depth of field can be taken. Conversely, when one image shift amount is applied to one screen instead of the shift amount for each pixel, a video with a shallow depth of field can be captured.

  Next, the configuration of the control unit 33 will be described with reference to FIG. In FIG. 11, reference numeral 801 denotes a voltage control unit, and 802 denotes a liquid crystal lens parameter storage unit. The voltage control unit 801 controls the voltage of each electrode of the liquid crystal lens 301 included in the imaging lens 9 in accordance with a control signal from the determination unit 703 of the video composition processing unit 38. The voltage to be controlled is determined based on the parameter value read from the liquid crystal lens parameter storage unit 802. By such processing, the electric field distribution of the liquid crystal lens 301 is ideally controlled, and the optical axis is controlled as shown in FIG. . Such control ideally controls the pixel phase, resulting in improved resolution of the video output signal. If the control result of the control unit 33 is in an ideal state, the energy detection of the result of the Fourier transform, which is the process of the determination unit 703, is maximized. In order to achieve such a state, the control unit 33 forms a feedback loop by the imaging lens 9, the video processing unit 27, and the video synthesis processing unit 38 so that high-frequency energy can be greatly obtained. To control. The voltage control unit 801 and the liquid crystal lens parameter storage unit 802 are each connected to the system bus P005. The liquid crystal lens parameter storage unit 802 is constituted by a RAM, for example, and is updated by the CPU P001 via the system bus P005 when the image pickup apparatus 1 is turned on.

  Note that the calibration parameter storage unit 603, the composite parameter storage unit 702, and the liquid crystal lens parameter storage unit 802 shown in FIGS. 9 to 11 may be configured to be selectively used according to the stored addresses using the same RAM or ROM. Further, a configuration may be used in which some addresses of ROM • P002 and RAM • P003 are used.

  Next, the control operation of the imaging device 1 will be described. FIG. 12 is a flowchart showing the operation of the imaging apparatus 1. Here, an example in which the spatial frequency information of the video is used in the video synthesis process is shown. First, when the CPU P001 instructs the start of control processing, the correction processing unit 602 reads calibration parameters from the calibration parameter storage unit 603 (step S901). The correction processing unit 602 performs correction for each of the unit imaging units 2 to 7 based on the read calibration parameters (step S902). This correction is to remove distortion for each of the unit imaging units 2 to 7 described later. Next, the synthesis processing unit 701 reads a synthesis parameter from the synthesis parameter storage unit 702 (step S903). Further, the stereo image processing unit 704 obtains data normalized by the shift amount for each pixel (shift parameter for each pixel) and the pixel pitch of the image sensor (step S911). Then, the synthesis processing unit 701 executes the sub-pixel video synthesis high-definition processing based on the read synthesis parameters, the shift amount for each pixel (shift parameter for each pixel), and data normalized by the pixel pitch of the image sensor. (Step S904). As will be described later, a high-definition image is constructed based on information having different phases in units of subpixels.

  Next, the determination unit 703 executes high definition determination (step S905), and determines whether or not the definition is high (step S906). The determination unit 703 holds a determination threshold value inside, determines a high-definition degree, and passes information of the determination result to each of the control units 32 to 37. Each control unit 32 to 37 maintains the same value of the liquid crystal lens parameters without changing the control voltage when high definition is achieved (step S907). On the other hand, when it determines with it not being high definition, the control parts 32-37 change the control voltage of the liquid crystal lens 301 (step S908). The CPU P001 manages the control end condition, for example, determines whether or not the device power-off condition is satisfied (step S909). If the control end condition is not satisfied, the process returns to step S903. Repeat the process. On the other hand, if the control end condition is satisfied, the process ends. Note that the control end condition may be set such that the number of high-definition determinations is 10 in advance when the apparatus is turned on, and the processing in steps S903 to S909 may be repeated for the specified number of times.

Next, with reference to FIG. 13, the operation of the sub-pixel video composition high-definition processing (step S904) shown in FIG. 12 will be described. The image size, the magnification, the rotation amount, and the shift amount are synthesis parameters, and are parameters read from the synthesis parameter storage unit 702 in the synthesis parameter reading process (step S903). Here, it is assumed that one high-definition image is obtained from four unit imaging units. From four images picked up by the individual unit image pickup units, they are superimposed on one coordinate system using parameters of the rotation amount and the shift amount. Then, the filter calculation is performed using the four images and the weighting coefficient based on the distance. For example, the filter uses cubic (third order approximation). The weight w acquired from the pixel at the distance d is as follows.
w = 1-2 × d2 + d3 (0 ≦ d <1)
= 4-8 * d + 5 * d2-d3 (1≤d <2)
= 0 (2 ≦ d)

  Next, with reference to FIG. 14, the detailed operation of the high definition determination process (step S905) performed by the determination unit 703 illustrated in FIG. 12 will be described. First, the determination unit 703 extracts a signal within the defined range (step S1001). For example, when one frame in a frame is defined as a definition range, a frame memory block (not shown) is separately provided, and signals for one screen are stored in advance. For example, in the case of VGA resolution, one screen is two-dimensional information composed of 640 × 480 pixels. The determination unit 703 performs Fourier transform on the two-dimensional information to convert time-axis information into frequency-axis information (step S1002). Next, a high-pass signal is extracted by a high pass filter (HPF) (step S1003). For example, the image sensor 9 is a VGA signal (640 pixels × 480 pixels) with an aspect ratio of 4: 3 and 60 fps (Frame Per Second) (progressive), and a video output signal that is an output of the video composition processing unit is Quad−. Assume the case of VGA. Assume that the limit resolution of the VGA signal is about 8 MHz and a signal of 10 MHz to 16 MHz is reproduced by the synthesis process. In this case, the HPF has a characteristic of passing a component of 10 MHz or more, for example. The determination unit 703 performs determination by comparing the signal of 10 MHz or higher with a threshold value (step S1004). For example, when the DC (direct current) component resulting from Fourier transform is set to 1, the threshold value of energy of 10 MHz or higher is set to 0.5, and the threshold value is compared with the threshold value.

  In the above description, the case where the Fourier transform is performed on the basis of the image of one frame of the imaging result of a certain resolution has been described. However, the definition range is the line unit (the unit of horizontal synchronization repetition, the number of effective pixels in the case of a high-definition signal) If it is defined in units of 1920 pixels, the frame memory block becomes unnecessary, and the circuit scale can be reduced. In this case, for example, in the case of a high-definition signal, the Fourier transform is repeatedly executed, for example, 1080 times of the number of lines, and the threshold comparison judgment for 1080 times for each line is integrated to determine the high-definition degree of one screen. Good. Further, the determination may be made using several frames of threshold comparison determination results in units of screens. In this way, by making a comprehensive determination based on a plurality of determination results, it is possible to remove the influence of sudden noise and the like. The threshold determination may use a fixed threshold, but may adaptively change the threshold. The feature of the image being judged may be extracted separately, and the threshold value may be switched based on the result. For example, image features may be extracted by histogram detection. Further, the current threshold value may be changed in conjunction with the past determination result.

  Next, with reference to FIG. 15, the detailed operation of the control voltage changing process (step S908) executed by the control units 32 to 37 shown in FIG. 12 will be described. Here, the processing operation of the control unit 33 will be described as an example, but the processing operations of the control units 32 and 34 to 37 are the same. First, the voltage control unit 801 reads out the current parameter value of the liquid crystal lens from the liquid crystal lens parameter storage unit 802 (step S1101). Then, the voltage control unit 801 updates the parameter value of the liquid crystal lens (step S1102). The liquid crystal lens parameters have a past history. For example, for the current four voltage control units 33a, 33b, 33c, and 33d, the voltage of the voltage control unit 33a is 40V, 45V, 50V, and 5V in the past history. If it is in the middle of being raised, it is determined that the voltage should be further increased from the history and the determination that the current high definition is not obtained, and the voltage values of the voltage control unit 33b, the voltage control unit 33c, and the voltage control unit 33d are determined. The voltage of the voltage control unit 33a is updated to 55V. In this manner, the voltage values applied to the electrodes 304a, 304b, 304c, and 304d of the four liquid crystal lenses are sequentially updated. Further, the updated value updates the value of the liquid crystal lens parameter as a history.

  Through the above processing operation, the captured images of the plurality of unit imaging units 2 to 7 are synthesized in sub-pixel units, the degree of high definition is determined, and the control voltage is changed so as to maintain high definition performance. Thus, an image pickup apparatus with high image quality can be realized. Specimens for sampling images formed on the image sensor by the imaging lenses 8 to 13 with the pixels of the image sensor by applying different voltages to the divided electrodes 304a, 304b, 304c, and 304d. Change the conversion phase. The ideal state of the control is a state in which the sampling phase of the imaging result of each unit imaging unit is shifted in the horizontal, vertical, and diagonal directions by ½ of the pixel size. The determination unit 703 determines whether the state is ideal.

  Next, the camera calibration processing operation will be described with reference to FIG. This processing operation is, for example, processing performed at the time of factory production of the imaging apparatus 1, and this camera calibration is performed by performing a specific operation such as simultaneously pressing a plurality of operation buttons when the imaging apparatus is powered on. This camera calibration process is executed by the CPU P001. First, an operator who adjusts the imaging device 1 prepares a checker pattern or checkered test chart with a known pattern pitch, and acquires images by capturing images in 30 postures of the checker pattern while changing the posture and angle. (Step S1201). Subsequently, the CPU P001 analyzes the captured image for each of the unit imaging units 2 to 7, and derives an external parameter value and an internal parameter value for each of the unit imaging units 2 to 7 (step S1202). For example, in the case of a general camera model referred to as a pinhole camera model, six external parameter values are external information including three-dimensional rotation information and translation information of the posture of the camera. Similarly, there are five internal parameters. Deriving such parameters is calibration. In a general camera model, there are a total of six external parameters including a three-axis vector of yaw, pitch, and roll indicating the camera attitude with respect to world coordinates, and a three-axis component of a translation vector indicating a translation component. . The internal parameters are the image center (u0, v0) where the optical axis of the camera intersects the image sensor, the angle and aspect ratio of the coordinates assumed on the image sensor, and the focal length.

  Next, the CPU P001 stores the obtained parameters in the calibration parameter storage unit 603 (step S1203). As described above, by using this parameter in the correction processing (step S902 shown in FIG. 12) of the unit imaging units 2 to 7, the individual camera distortion of the unit imaging units 2 to 7 is corrected. That is, since a checker pattern that was originally a straight line may be imaged as a curve due to camera distortion, parameters for returning the checker pattern to a straight line may be derived by this camera calibration process, 7 is corrected.

  Next, the CPU P001 derives the parameters between the unit imaging units 2 to 7 and derives the external parameters between the unit imaging units 2 to 7 (step S1204), and stores them in the composite parameter storage unit 702 and the liquid crystal lens parameter storage unit 802. The stored parameters are updated (steps S1205 and S1206). This value is used in the sub-pixel video composition high-definition processing S904 and the control voltage change S908.

  In addition, although the case where the CPU or microcomputer in the imaging device has a camera calibration function is shown here, for example, a separate personal computer is prepared and the same processing is executed on this personal computer. The configuration may be such that only the parameters are downloaded to the imaging apparatus.

  Next, the principle of camera calibration of the unit imaging units 2 to 7 will be described with reference to FIG. Here, the state of projection by the camera is considered using a pinhole camera model as shown in FIG. In the pinhole camera model, all the light reaching the image plane passes through a pinhole, which is one point at the center of the lens, and forms an image at a position intersecting the image plane. The coordinate system that takes the intersection of the optical axis and the image plane as the origin and takes the x-axis and y-axis to match the camera element placement axis is called the image coordinate system. The camera lens center is the origin and the optical axis is the Z-axis. A coordinate system taking the X axis and the Y axis in parallel with the x axis and the y axis is called a camera coordinate system. Here, the three-dimensional coordinates M = [X, Y, Z] T in the world coordinate system (Xw, Yw, Zw) which is a coordinate system representing the space, and the image coordinate system (x, y) which is the projection thereof. The point m = [u, v] T is associated with the equation (1).

ここで、Ａは内部パラメータ行列といい、次の（２）式のような行列である。

Here, A is called an internal parameter matrix, and is a matrix like the following equation (2).

α，βは画素の大きさと焦点距離との積からなるスケール係数、（ｕ０，ｖ０）は画像中心、γは画像の座標軸の歪みを表すパラメータである。また、［Ｒ ｔ］は外部パラメータ行列といい、３×３の回転行列Ｒと平行移動ベクトルｔを並べた４×３行列である。

α and β are scale factors formed by the product of the pixel size and the focal length, (u0, v0) is the image center, and γ is a parameter representing the distortion of the coordinate axes of the image. [R t] is an external parameter matrix, which is a 4 × 3 matrix in which a 3 × 3 rotation matrix R and a translation vector t are arranged.

平面上の点と画像上の点との関係は３×３のホモグラフィ行列Ｈで、（４）式のように記述できる。 In the Zhang calibration method, internal parameters, external parameters, and lens distortion parameters can be obtained simply by taking an image (three or more times) while moving a flat plate on which a known pattern is pasted. In this method, the calibration plane is calibrated as a plane of Zw = 0 in the world coordinate system. The relationship between the point M on the calibration plane represented by the equation (1) and the corresponding point m on the image obtained by photographing the plane can be rewritten as the following equation (3).

The relationship between the points on the plane and the points on the image is a 3 × 3 homography matrix H, which can be described as in equation (4).

キャリブレーション平面の画像が１つ与えられると、ホモグラフィ行列Ｈが１つ得られる。このホモグラフィＨ＝［ｈ１ ｈ２ ｈ３］が得られると、（４）式より次の（５）式が得られる。

Given one image of the calibration plane, one homography matrix H is obtained. When this homography H = [h1 h2 h3] is obtained, the following expression (5) is obtained from the expression (4).

Ｒが回転行列なのでｒ１とｒ２は直交であることから、次に示す内部パラメータに関する２つの拘束式である（６）、（７）式が得られる。

Since R1 is a rotation matrix and r1 and r2 are orthogonal, the following two expressions (6) and (7) relating to the internal parameters are obtained.

とするＢの要素を並べたベクトルを

と定義する。ホモグラフィＨのｉ番目の列ベクトルをｈｉ＝［ｈｉ１ ｈｉ２ ｈｉ３］Ｔ，（ｉ＝１，２，３）とすると、ｈｉＴＢｈｊは

Since A-TA-1 includes 6 unknowns in a 3 × 3 target matrix as shown in Equation (8) and two equations can be established for each H, three or more H can be obtained. The internal parameter A can be determined. Here, A-TA-1 has a target

A vector with B elements

It is defined as Assuming that the i-th column vector of homography H is hi = [hi1 hi2 hi3] T, (i = 1, 2, 3), hiTBhj is

It can be expressed.

As a result, Equations (6) and (7) become as follows:

を得る。ここで、Ｖは２ｎ×６の行列である。これより、ｂはＶＴＶの最小固有値に対応する固有ベクトルとして求められる。この場合、ｎ≧３であれば直接ｂに関する解を得ることができるが、ｎ＝２の場合は内部パラメータ中のγ＝０とすることで、式［０ １ ０ ０ ０ ０］ｂ＝０を（１３）式に加えることで解を得る。また、ｎ＝１であれば２つの内部パラメータしか求めることができないため、例えばαとβのみを未知とし、残りの内部パラメータを既知とすることで解を得る。ｂを求めることでＢが求まれば、Ｂ＝μＡ−ＴＡからカメラの内部パラメータは（１４）式で計算される。

If n images are obtained, by stacking the above n equations,

Get. Here, V is a 2n × 6 matrix. Thus, b is obtained as an eigenvector corresponding to the minimum eigenvalue of VTV. In this case, if n ≧ 3, a solution for b can be obtained directly. However, when n = 2, by setting γ = 0 in the internal parameters, the expression [0 1 0 0 0 0] b = 0 Is obtained by adding to the equation (13). Further, if n = 1, only two internal parameters can be obtained, so that, for example, only α and β are unknown and the remaining internal parameters are known to obtain a solution. If B is obtained by obtaining b, the internal parameters of the camera are calculated by the equation (14) from B = μA−TA.

として求めることができ、ここまでで得られたパラメータを初期値とする非線形最小二乗法によってパラメータを最適化することで、最適な外部パラメータを得ることができる。 Moreover, if the internal parameter A is obtained from this, the external parameter is also obtained from the equation (5),

Optimum external parameters can be obtained by optimizing the parameters by the non-linear least square method using the parameters obtained so far as initial values.

  As described above, when all the internal parameters are unknown, camera calibration can be performed by using three or more images taken with the internal parameters fixed from different viewpoints. At this time, generally, the larger the number of images, the higher the parameter estimation accuracy. Also, the error increases when the rotation between images used for calibration is small.

と表せる。 Next, referring to FIG. 18 and FIG. 19, from the camera parameters representing the position / orientation of the camera (imaging device) obtained by camera calibration, the area where the same area is shown in each image is sub-pixel accuracy. The method of associating with will be described. FIG. 18 shows a point M on the target object plane by using the above-mentioned liquid crystal lens with a basic image sensor 15 (referred to as a basic camera) and an adjacent image sensor 16 adjacent thereto (referred to as an adjacent camera). In this case, the image is projected (photographed) onto a point m1 or m2 on each image sensor. FIG. 19 shows FIG. 18 using the pinhole camera model shown in FIG. The relationship between the point M on the world coordinate system and the point m on the image coordinate system can be expressed by using the central projection matrix P from the viewpoint of the mobility of the camera, etc.

It can be expressed.

（２）算出された三次元位置より、隣接カメラの中心射影行列Ｐ２を用いて、隣接画像の対応点ｍ２を求める。

By using the calculated P, the correspondence between the points in the three-dimensional space and the points on the two-dimensional image plane can be described. In the configuration shown in FIG. 19, the center projection matrix of the basic camera is P1, and the center projection matrix of the adjacent camera is P2. In order to obtain the point m2 on the image plane 2 corresponding to the point m1 on the image plane 1, the following method is used.
(1) The point M in the three-dimensional space is obtained from m1 from the equation (16). Since the central projection matrix P is a 3 × 4 matrix, it is obtained using a pseudo inverse matrix of P.

(2) The corresponding point m2 of the adjacent image is obtained from the calculated three-dimensional position using the center projection matrix P2 of the adjacent camera.

  Since the camera parameter P has an analog value, the corresponding point m2 between the calculated basic image and the adjacent image is obtained in sub-pixel units. Corresponding point matching using camera parameters has an advantage that the corresponding points can be instantaneously calculated only by matrix calculation because the camera parameters have already been obtained.

  Next, lens distortion and camera calibration will be described. Up to this point, the description has been made with a pinhole model in which the lens is regarded as one point. However, since the lens actually has a finite size, it may not be explained with the pinhole model. Correction of distortion in such a case will be described below. When a convex lens is used, distortion occurs due to refraction of incident light. Correction coefficients for such radial distortion are set as k1, k2, and k5. In addition, when the lens and the image sensor are not arranged in parallel, distortion in the tangential direction occurs. Correction coefficients for distortion in the normal direction are set as k3 and k4. These distortions are called distortion aberrations. Here, the distortion correction formula is as follows.

  Here, (xu, yu) is an image coordinate of an imaging result of an ideal lens without distortion, and (xd, yd) is an image coordinate of a lens having distortion. The coordinate systems of these coordinates are both the above-described image coordinate system x-axis and y-axis. R is the distance from the image center to (xu, yu). The image center is determined by the aforementioned internal parameters u0 and v0. Assuming the above model, if the coefficients k1 to k5 and internal parameters are derived by calibration, the difference in imaging coordinates due to the presence or absence of distortion can be obtained, and distortion caused by the real lens can be corrected.

  FIG. 20 is a schematic diagram illustrating an imaging state of the imaging apparatus 1. The unit imaging unit 3 including the imaging element 15 and the imaging lens 9 images the imaging range a. The unit imaging unit 4 including the imaging element 16 and the imaging lens 10 images the imaging range b. The two unit imaging units 3 and 4 image substantially the same imaging range. For example, when the arrangement interval of the imaging elements 15 and 16 is 12 mm, the focal length of the unit imaging units 3 and 4 is 5 mm, the distance to the imaging range is 600 mm, and the optical axes of the unit imaging units 3 and 4 are parallel, the imaging range The area in which a and b are different is about 3%. In this way, the same part is imaged, and the composition processing unit 38 performs high definition processing.

  Next, referring to FIG. 21 and FIG. 22, high definition of the imaging device 1 will be described. The horizontal axis of FIG. 21 shows the expansion of the space. The expansion of the space indicates both the case of the actual space and the expansion of the virtual space on the image sensor. These are synonymous because they can be mutually converted and converted by using external parameters and internal parameters. When the video signals sequentially read from the image sensor are considered, the horizontal axis in FIG. 21 is the time axis. In this case as well, when displayed on the display, it is recognized that the space is widened by the observer's eyes. Therefore, the time axis of the video signal is synonymous with the expansion of space. The vertical axis in FIG. 21 represents amplitude and intensity. Since the intensity of the object reflected light is photoelectrically converted by a pixel of the image sensor and output as a voltage level, it may be regarded as an amplitude.

  FIG. 21A shows an outline of an object in the real space. In order to integrate this contour, that is, the intensity of reflected light of the object with the spread of the pixels of the image sensor, the image is captured by the unit imaging units 2 to 7 as shown in FIG. For example, the integration is performed using an LPF (Low Pass Filter). An arrow in FIG. 21B indicates the spread of pixels of the image sensor. FIG. 21C shows the result of imaging with different unit imaging units 2 to 7, and the light is integrated by the spread of the pixels indicated by the arrows in FIG. As shown in FIGS. 21B and 21C, the contour (profile) of reflected light that is less than the spread determined by the resolution (pixel size) of the image sensor cannot be reproduced by the image sensor.

  However, a feature of the present invention is that both phase relationships have an offset in FIGS. 21B and 21C. The contour shown in FIG. 21D can be reproduced by capturing light with such an offset and optimally combining the light by the combining processing unit. As is clear from FIGS. 21A to 21D, FIG. 21D can reproduce the outline of FIG. 21A most, and corresponds to the width of the arrow in FIG. This is equivalent to the performance of the pixel size of the image sensor. In this embodiment, a non-solid lens typified by a liquid crystal lens and a plurality of unit imaging units including imaging elements are used to obtain a video output exceeding the resolution limit by the above-described averaging (integration using LPF). Is possible.

  FIG. 22 is a schematic diagram illustrating a relative phase relationship between two unit imaging units. When high definition is performed in the subsequent image processing, it is desirable that the relative relationship of the sampling phase by the image sensor is equal. Here, sampling is synonymous with sampling, and refers to processing for extracting analog signals at discrete positions. Since FIG. 22 assumes the case where two unit imaging units are used, the phase relationship of 0.5 pixel size is ideal as shown in (a). However, there may be cases as shown in FIGS. 22B and 22C depending on the imaging distance and the assembly of the apparatus. In this case, even if the image processing calculation is performed using only the averaged video signal, the signal that has already been averaged in the phase relationship as shown in FIGS. 22B and 22C cannot be restored. It is. Therefore, it is essential to control the phase relationship shown in FIGS. 22B and 22C to that shown in FIG. In the present invention, this control is realized by an optical axis shift by the liquid crystal lens shown in FIG. By the above processing, an ideal phase relationship is always maintained, so that an optimal image can be provided to the observer.

  Here, the one-dimensional phase relationship has been described with reference to FIG. For example, by using four unit imaging units and performing one-dimensional shifts in respective directions of horizontal, vertical, and oblique 45 degrees, the phase control of the two-dimensional space can be performed by the operation shown in FIG. In addition, for example, by using two unit imaging units, two-dimensional phase control is realized by performing phase control of the unit imaging unit on one side with respect to the reference one in two dimensions (horizontal, vertical, horizontal + vertical). May be.

  For example, a case is assumed in which four unit imaging units are used to capture substantially the same imaging target (subject) and four images are obtained. Using an image as a reference, individual images are Fourier transformed to determine feature points on the frequency axis, calculate the rotation amount and shift amount relative to the reference image, and use the rotation amount and shift amount to perform interpolation filtering processing By doing so, it becomes possible to obtain a high-definition image. For example, if the number of pixels of the image sensor is VGA (640 × 480 pixels), a quad-VGA (1280 × 960 pixels) high-definition image can be obtained by four VGA unit imaging units. The interpolation filtering process described above uses, for example, a cubic (third order approximation) method. This is a weighting process based on the distance to the interpolation point. Although the resolution limit of the image sensor is VGA, the imaging lens has the ability to pass the Quad-VGA band, and the Quad-VGA band component equal to or higher than VGA is imaged at the VGA resolution as aliasing. By using this aliasing distortion, the high-band component of Quad-VGA is restored by the video composition process.

  FIG. 23 is a diagram illustrating a relationship between an imaging target (subject) and image formation. This figure is based on a pinhole model that ignores lens distortion. An image pickup apparatus with a small lens distortion can be explained by this model, and can be explained only by geometric optics. In FIG. 23 (a), P1 is an imaging target and is an imaging distance H apart. The pinholes O and O ′ correspond to the imaging lenses of the two unit imaging units, and are schematic diagrams in which one image is captured by the two unit imaging units of the imaging elements M and N. FIG. FIG. 23B shows a state where an image P1 is formed on the pixels of the image sensor. In this way, the phase of the image formed with the pixel is determined. This phase is determined by the positional relationship (baseline length B) of the imaging elements, the focal length f, and the imaging distance H.

  That is, it may differ from the design value depending on the mounting accuracy of the image sensor, and the relationship changes depending on the imaging distance. In this case, depending on a certain combination, there is a case where the phases of each other coincide with each other as shown in FIG. The light intensity distribution image in FIG. 23B schematically shows the light intensity with respect to a certain spread. With respect to such light input, the image sensor averages within the range of pixel expansion. As shown in FIG. 23B, when the two unit imaging units capture at different phases, the same light intensity distribution is averaged at different phases. If the element is VGA resolution, a higher band than VGA resolution) can be reproduced. Since there are two unit imaging units, a phase shift of 0.5 pixels is ideal.

  However, if the phases match as shown in FIG. 23C, the information captured by each image sensor becomes the same, and high resolution is impossible. Therefore, as shown in FIG. 23C, high resolution is achieved by controlling the phase to an optimum state by optical axis shift. The optimum state is realized by the processing in FIG. As for the phase relationship, it is desirable that the phase of the unit imaging unit to be used is equally spaced. Since the present invention has an optical axis shift function, such an optimal state can be achieved by voltage control from the outside.

  FIG. 24 is a schematic diagram for explaining the operation of the imaging apparatus 1. A state in which an image is picked up by an image pickup apparatus including two unit image pickup units is illustrated. Each image sensor is shown enlarged in pixel units for convenience of explanation. The plane of the image sensor is defined in two dimensions u and v, and FIG. 24 corresponds to a cross section of the u axis. The imaging targets P0 and P1 are at the same imaging distance H. Images of P0 are formed on u0 and u'0, respectively. u0 and u'0 are distances on the image sensor with respect to the respective optical axes. In FIG. 24, P0 is on the optical axis of the image sensor M, and therefore u0 = 0. Further, the distance from the optical axis of each image of P1 is u1, u'1. Here, the relative phase with respect to the pixels of the image sensors M and N at the positions where P0 and P1 are imaged on the image sensors M and N determines the image shift performance. This relationship is determined by the imaging distance H, the focal length f, and the baseline length B that is the distance between the optical axes of the imaging elements.

  In FIG. 24, the positions where the images are formed, that is, u0 and u'0 are shifted by half the size of the pixel. u0 (= 0) is located at the center of the pixel of the image sensor M. On the other hand, u'0 forms an image around the pixel of the image sensor N. That is, the pixel size is shifted by a half pixel. Similarly, u1 and u′1 are shifted by the size of a half pixel. FIG. 24B is a schematic diagram of an operation for restoring and generating one image by calculating the same images of the captured images. Pu indicates the pixel size in the u direction, and Pv indicates the pixel size in the v direction. FIG. 24B shows a relationship where the pixels are shifted by half of each other, which is an ideal state for generating a high-definition image by performing image shift.

  FIG. 25 is a schematic diagram of the case where the image sensor N is attached with a deviation of half the pixel size from the design due to attachment errors, for example. In this case, the relationship between u1 and u′1 is the same phase for the pixels of each image sensor. In FIG. 25A, both images are formed at positions on the left side of the pixel. The relationship between u0 (= 0) and u′0 is the same. Therefore, as shown in FIG. 25 (b), the phases are substantially the same.

  FIG. 26 is a schematic diagram when the optical axis shift of the present invention is operated with respect to FIG. The movement in the right direction called optical axis shift in FIG. Thus, by shifting the pinhole O ′ using the optical axis shift means, the position at which the imaging target forms an image can be controlled with respect to the pixels of the imaging device. As shown in FIG. 26B, an ideal phase relationship can be achieved.

  Next, the relationship between the imaging distance and the optical axis shift will be described with reference to FIG. FIG. 27 is a schematic diagram for explaining a case where the subject is switched from the state in which P0 is imaged at the imaging distance H0 to the object P1 at the distance H1. In FIG. 27, since it is assumed that P0 and P1 are on the optical axis on the image sensor M, u0 = 0 and u1 = 0. Attention is paid to the relationship between the pixels of the image sensor B and the images of P0 and P1 when P0 and P1 form an image on the image sensor N. P0 forms an image at the center of the pixel of the image sensor M. On the other hand, the image sensor N forms an image around the pixel. Therefore, it can be said that the phase relationship was optimal when P0 was imaged. FIG. 27B is a schematic diagram showing the phase relationship between the image sensors when the subject is P1. After changing the subject to P1 as shown in FIG. 27 (b), the phases of each other substantially coincide.

  Therefore, as shown in FIG. 28A, by moving the optical axis by the optical axis shift means at the time of imaging the subject P1, it becomes possible to control the ideal phase relationship as shown in FIG. Therefore, high definition by image shift can be achieved. Here, as a method for obtaining information on the imaging distance, it is sufficient to have a distance measuring unit for measuring the distance. Alternatively, the distance may be measured with the imaging device of the present invention. An example of measuring distance using a plurality of cameras (unit imaging units) is common in surveying and the like. The distance measurement performance is in inverse proportion to the distance to the distance measurement object in proportion to the base line length which is the distance between the cameras and the focal length of the camera.

  The imaging apparatus of the present invention has, for example, an eight-eye configuration, that is, a configuration including eight unit imaging units. When the measurement distance, that is, the distance to the subject is 500 mm, four cameras with short distances between the optical axes (baseline lengths) among the eight-eye cameras are assigned to imaging and image shift processing, and the remaining baselines are long with respect to each other. Measure the distance to the subject with four cameras. In addition, when the distance to the subject is as long as 2000 mm, high resolution processing of image shift is performed using eight eyes, and ranging is performed by determining the amount of blur by analyzing the resolution of the captured image, for example. You may make it the structure performed by the process which estimates a distance. Even in the case of the above-mentioned 500 mm, the accuracy of distance measurement may be improved by using other distance measuring means such as TOF (Time of Flight) together.

Next, with reference to FIG. 29, the effect of image shift by depth and optical axis shift will be described. FIG. 29A is a schematic diagram of imaging P1 and P2 considering the depth Δr. The distance difference (u1-u2) from each optical axis is expressed by equation (22).
(U1-u2) = Δr × u1 / H (22)

Here, u1-u2 is a value determined by the baseline length B, the imaging distance H, and the focal length f. Here, these conditions B, H, and f are fixed and regarded as constants. Further, it is assumed that an ideal optical axis relationship is obtained by the optical axis shift means. The relationship between Δr and the position of the pixel (the distance from the optical axis of the image formed on the image sensor) is expressed by equation (23).
Δr = (u1−u2) × H / u1 (23)

  That is, Δr is inversely proportional to u1. FIG. 29B shows a condition in which the influence of depth falls within the range of one pixel, assuming a pixel size of 6 μm, an imaging distance of 600 mm, and a focal length of 5 mm as an example. Since the effect of image shift is sufficient under the condition that the influence of depth falls within the range of one pixel, it is possible to avoid image shift performance deterioration due to depth if used properly depending on the application, such as narrowing the angle of view. It becomes.

As shown in FIG. 29, when Δr is small (the depth of field is shallow), high definition processing may be performed by applying the same image shift amount on one screen. The case where Δr is large (the depth of field is deep) will be described with reference to FIGS. FIG. 30 is a flowchart showing the processing operation of the stereo image processing unit 704 shown in FIG. In FIG. 27, since the phase shift of sampling by pixels of a plurality of imaging elements having a certain baseline length varies depending on the imaging distance, in order to achieve high definition at any imaging distance, an image shift is performed according to the imaging distance. It is necessary to change the amount. For example, if the subject has a large depth, even if the phase difference is optimal at a certain distance, the phase difference is not optimal at other distances. That is, it is necessary to change the shift amount for each pixel. Here, the imaging distance and the amount of movement of the point imaged on the imaging device are expressed by equation (24).
u0−u1 = f × B × ((1 / H0) − (1 / H1)) (24)

  The stereo image processing unit 704 (see FIG. 10) obtains data normalized by the shift amount for each pixel (shift parameter for each pixel) and the pixel pitch of the image sensor. The stereo image processing unit 704 performs stereo matching using two captured images corrected based on camera parameters obtained in advance (step S3001). Corresponding feature points in the image are obtained by stereo matching, and a shift amount for each pixel (shift parameter for each pixel) is calculated therefrom (step S3002). Next, the stereo image processing unit 704 compares the shift amount for each pixel (shift parameter for each pixel) with the pixel pitch of the image sensor (step S3003). As a result of this comparison, if the shift amount for each pixel is smaller than the pixel pitch of the image sensor, the shift amount for each pixel is used as a synthesis parameter (step S3004). On the other hand, if the shift amount for each pixel is larger than the pixel pitch of the image sensor, data normalized by the pixel pitch of the image sensor is obtained, and the data is used as a synthesis parameter (step S3005). By performing video synthesis based on the synthesis parameters obtained here, a high-definition image can be obtained regardless of the imaging distance.

  Here, stereo matching will be described. Stereo matching is a process of searching for a projected point of the same spatial point from another image with respect to a pixel at a position (u, v) in the image on the basis of one image. Since the camera parameters necessary for the camera projection model are obtained in advance by camera calibration, the search for corresponding points can be limited to a straight line (epipolar line). In particular, when the optical axes of the unit imaging units are set in parallel as in the present embodiment, the epipolar line is a straight line on the same horizontal line as shown in FIG. Thus, since the corresponding points on the other image with respect to the reference image are limited to the epipolar line, in stereo matching, it is only necessary to search on that line. This is important for reducing the matching error and speeding up the processing.

  Specific search methods include area-based matching and feature-based matching. In the area-based matching, as shown in FIG. 32, corresponding points are obtained using a template. On the other hand, feature-based matching extracts feature points such as edges and corners of each image and obtains correspondence between the feature points.

  There is a method called multi-baseline stereo as a method for obtaining a more accurate corresponding point. This is a method that uses not only stereo matching by a set of cameras but also a plurality of stereo image pairs by more cameras. A stereo image is obtained by using a pair of stereo cameras having a base line (baseline) in various lengths and directions with respect to a reference camera. For example, in the case of parallel stereo, the parallax in a plurality of image pairs is a value corresponding to the distance in the depth direction by dividing each parallax by the baseline length. Therefore, stereo matching information obtained from each stereo image pair, specifically, an evaluation function such as SSD (Sum of Squared Differences) representing the likelihood of correspondence to each parallax / baseline length is added, and the most from there Determine the corresponding location. In other words, if the change in SSDS (Sum of SSD), which is the sum of SSD for each parallax / baseline length, is examined, a clearer minimum value appears, so that the stereo matching error can be reduced and the estimation accuracy is improved. Can be made. In addition, multi-baseline stereo can reduce the problem of occlusion that a part that can be seen by one camera is hidden behind another object and cannot be seen by another camera.

  FIG. 33 shows an example of a parallax image. (A) is an original image (reference image), and (b) is a parallax image obtained as a result of obtaining parallax for each pixel of the image of (a). The parallax image indicates that the higher the luminance of the image, the larger the parallax, that is, the imaged object is closer to the camera, and the lower the luminance is, the smaller the parallax is, ie, the imaged object is far from the camera.

  Next, noise removal in stereo image processing will be described with reference to FIG. FIG. 34 is a block diagram showing the configuration of the video composition processing unit 38 when noise removal is performed in stereo image processing. The video composition processing unit 38 shown in FIG. 34 is different from the video composition processing unit 38 shown in FIG. 10 in that a stereo image noise reduction processing unit 705 is provided. The operation of the video composition processing unit 38 shown in FIG. 34 will be described with reference to the flowchart of the noise removal processing operation in the stereo image processing shown in FIG. 35, the processing operations in steps S3001 to S3005 are the same as those in steps S3001 to S3005 performed by the stereo image processing unit 704 shown in FIG. The stereo image noise reduction processing unit 705 determines the shift amount of the adjacent pixel when the shift amount of the synthesis parameter for each pixel obtained in step S3105 is significantly different from the shift amount of the adjacent surrounding synthesis parameter. The noise is removed by substituting with the most frequent value (step S3106).

  Next, the processing amount reduction operation will be described with reference to FIG. 33 again. Usually, the entire image is made high-definition using the synthesis parameters obtained by the stereo image processing unit 704. For example, only the face part (part where the luminance of the parallax image is high) in FIG. This part (part where the luminance of the parallax image is low) is not made high definition, so that the processing amount can be reduced. In this process, as described above, a part of an image with a face (a part where the distance is close and the brightness of the parallax image is high) is extracted from the parallax image, and the image data of the image part and the synthesis obtained by the stereo image processing unit High definition can be similarly achieved using parameters. As a result, power consumption can be reduced, which is effective in a portable device that operates on a battery or the like.

  As described above, it is possible to synthesize video signals obtained by individual imaging devices into high-definition video by controlling the optical axis of the liquid crystal lens. Conventionally, the image quality deteriorates due to the crosstalk on the image sensor and it is difficult to improve the image quality. However, according to the image pickup apparatus of the present invention, the crosstalk is controlled by controlling the optical axis of the light incident on the image sensor. Therefore, it is possible to realize an image pickup apparatus that can eliminate high image quality and obtain high image quality. In addition, in the conventional imaging device, the image formed on the imaging device is captured by image processing. Therefore, the resolution of the imaging device needs to be larger than the required imaging resolution. In the imaging device of the present invention, the liquid crystal Since it is possible to control not only the optical axis direction of the lens but also the optical axis of light incident on the image sensor at an arbitrary position, the size of the image sensor can be reduced, and a light, thin and small size is required. It can be mounted on a portable terminal or the like. In addition, a high-definition, high-definition two-dimensional image can be generated regardless of the shooting distance. Furthermore, it is possible to remove noise due to stereo matching and speed up the high definition processing.

DESCRIPTION OF SYMBOLS 1 Imaging device 2-7 Unit imaging part 8-13 Imaging lens 14-19 Imaging element 20-25 Optical axis 26-31 Image processing part 32-37 Control part 38 Image composition processing part

Claims (3)

A plurality of image sensors;
A plurality of solid lenses that form an image on each of the imaging elements;
A plurality of optical axis controllers that control the direction of the optical axis of light incident on each of the image sensors;
A plurality of video processing units that input photoelectric conversion signals output from each of the plurality of image sensors, convert the signals into video signals, and output the video signals;
A stereo image processing unit that obtains a shift amount for each pixel by performing stereo matching processing based on a plurality of video signals, and generates a composite parameter that is normalized by the pixel pitch for a shift amount that exceeds the pixel pitch of the image sensor When,
A video composition processing unit that inputs the video signal output from each of the plurality of video processing units and the synthesis parameter and generates a high-definition video by synthesizing the plurality of video signals based on the synthesis parameter; An imaging apparatus comprising: The imaging apparatus according to claim 1, further comprising: a stereo image noise reduction processing unit that reduces noise of a parallax image used for the stereo matching processing based on the synthesis parameter generated by the stereo image processing unit.   The imaging apparatus according to claim 1, wherein the video composition processing unit increases the definition only in a predetermined region based on the parallax image generated by the stereo image processing unit.

Images