WO2012014695A1

WO2012014695A1 - Three-dimensional imaging device and imaging method for same

Info

Publication number: WO2012014695A1
Application number: PCT/JP2011/066089
Authority: WO
Inventors: 田中　誠一
Original assignee: シャープ株式会社
Priority date: 2010-07-27
Filing date: 2011-07-14
Publication date: 2012-02-02
Also published as: JP5088973B2; JP2012029199A

Abstract

Disclosed is a three-dimensional imaging device comprising: two imaging units that capture images of the same subject; a parallax calculation unit that detects points of correspondence between videos captured by the two imaging units and calculates parallax information for the captured videos from the two imaging units; and a compositing processor that, using the viewpoint of each of the two imaging units for reference and based on parallax information and the videos captured by the two imaging units, combines videos with more pixels than the captured videos and generates a two-system video with many pixels.

Description

Stereo imaging device and imaging method thereof

The present invention relates to a stereoscopic imaging device and an imaging method thereof.
This application claims priority on July 27, 2010 based on Japanese Patent Application No. 2010-168145 filed in Japan, the contents of which are incorporated herein by reference.

In recent years, stereoscopic image devices have been actively developed in order to enhance the power and presence of images. As a technique for generating a stereoscopic image, there is known a technique in which two imaging devices for the left channel (L) and the right channel (R) are arranged on the left and right, and the subject is simultaneously photographed by the two imaging devices. Yes. Further, as a technique for displaying a stereoscopic image, a left channel (L) image and a right (R) channel image are alternately displayed on a single display screen for each pixel, and a kamaboko-shaped lens is provided in a predetermined manner. The viewer's left and right eyes use a special optical system such as a lenticular lens arranged at intervals, a parallax barrier with fine slits arranged at a predetermined interval, and a patterning phase difference plate with regularly arranged fine polarizing elements. A technique is known in which only the left channel (L) image is visible to the viewer's left eye and only the right channel (R) image is visible to the right eye by adjusting the viewing area.

In recent years, technology for generating high-definition video using a plurality of cameras has also been developed. For example, a thin color camera having a sub-pixel resolution by combining four sub-cameras including an imaging lens, a color filter, and a detector array has also been proposed (for example, see Patent Document 1). As shown in FIG. 18, the thin color camera includes four lenses 22a to 22d, four color filters 25a to 25d, and a detector array 24. The color filter 25 includes a filter 25a that transmits red light (R),

filters

25b and 25c that transmit green light (G), and a filter 25d that transmits blue light (B). Take green and blue images. With this configuration, a high-resolution composite image is formed from two green images having high sensitivity in the human visual system, and a full-color image can be obtained by combining red and blue.

Special table 2007-520166

However, the conventional technology for capturing a stereoscopic image requires two imaging devices for the left channel (L) and the right channel (R) in order to obtain one stereoscopic image. There is a problem of becoming a scale. For example, when the imaging devices described in Patent Document 1 are used for the left channel and the right channel, the two imaging devices are arranged side by side. Since the imaging apparatus described in Patent Literature 1 includes four sub cameras, eight sub cameras that are twice that number are required.

The present invention has been made in view of such circumstances, and an object thereof is to provide a stereoscopic imaging apparatus and an imaging method thereof in which an increase in apparatus scale is suppressed.

(1) The present invention has been made to solve the above-described problem. A stereoscopic imaging apparatus according to an embodiment of the present invention includes two imaging units that capture the same subject, and imaging of the two imaging units. A disparity calculating unit that detects corresponding points between the images and calculates disparity information of the captured images of the two image capturing units; and the disparity information and the two image capturing units on the basis of the viewpoints of the two image capturing units, respectively. And a synthesis processing unit that synthesizes an image having a larger number of pixels than the captured image and generates two systems of images having the larger number of pixels.

(2) In the stereoscopic imaging device described above, the imaging unit includes an optical system that forms an image of a subject on an imaging surface, and an imaging element that generates a signal of a captured video of the subject formed on the imaging surface, One of the imaging units may be displaced up or down by half of the imaging pixel of the imaging device with respect to the optical system as compared with the other imaging unit.

(3) The above-described stereoscopic imaging device includes three or more imaging units, and among the imaging units, between the side-by-side imaging units, the position of the imaging element with respect to the optical system is the imaging pixel of the imaging element. The position of the image sensor with respect to the optical system is either half left or right of the shooting pixel of the image sensor. It may be shifted.

(4) In the above-described stereoscopic imaging device, the four imaging units are arranged at the vertices of a square along which each side is either horizontal or vertical, and the parallax calculation unit includes the four imaging units, The parallax information of the captured video of the two imaging units arranged at the adjacent vertices of the square is calculated, and the synthesis processing unit performs parallax in the horizontal direction and the vertical direction of the captured video performed when the video is synthesized. The parallax information may be used for correction.

(5) The stereoscopic imaging device described above includes at least three of the imaging units, and the synthesis processing unit captures the parallax information and at least two of the imaging units with reference to the viewpoints of the at least three imaging units. Based on the video, a video having a larger number of pixels than the captured video may be synthesized to generate a video having a larger number of pixels of at least three systems.

(6) In an imaging method according to another embodiment of the present invention, a corresponding point is detected between captured images of two imaging units that capture the same subject, and parallax information of the captured images of the two imaging units is calculated. And synthesizing an image having a larger number of pixels than the captured image based on the parallax information and the captured images of the two image capturing units on the basis of the viewpoints of the two image capturing units. Generating a large number of images.

(7) The step of generating a video having a large number of pixels may include a step of performing parallax correction of the captured video using the parallax information when the video is synthesized.

(8) The step of generating the video with a large number of pixels is based on the parallax information and at least two captured images of the imaging units based on the viewpoints of the at least three imaging units that capture the same subject. The method may include a step of synthesizing an image having a larger number of pixels than the captured image and generating at least three systems of the image having the larger number of pixels.

According to this invention, an increase in the scale of the apparatus can be suppressed.

1 is an overview diagram showing an overview of a stereoscopic imaging apparatus 10 according to a first embodiment of the present invention. It is a schematic block diagram which shows the structure of the three-dimensional imaging device 10 in the embodiment. It is a figure which shows the example of arrangement | positioning of the imaging lens and imaging device in the embodiment. It is a figure which shows another example of arrangement | positioning of the imaging lens and imaging device in the embodiment. It is a schematic block diagram which shows the structure of the parallax calculation part 21 in the embodiment. It is a figure showing reference picture RG in the embodiment. It is a figure which shows the reference | standard image BG in the embodiment. It is a figure which shows the structure of the reference attention block RB in the same embodiment. It is a figure which shows the structure of the reference attention block BB in the embodiment. It is a flowchart explaining the calculation process of the parallax data in the embodiment. It is a schematic block diagram which shows the function structure of the high resolution synthetic | combination process part 20 in the embodiment. It is a figure explaining operation | movement of the synthetic | combination process part 906 in the embodiment. It is a general-view figure which shows the general appearance of the three-dimensional imaging device 111 in 2nd embodiment of this invention. It is a schematic block diagram which shows the function structure of the three-dimensional imaging device 111 in the embodiment. It is a figure explaining arrangement of an image pick-up lens and an image sensor in the embodiment. It is a schematic block diagram which shows the function structure of the multi-view high resolution synthetic | combination process part 121 in the embodiment. It is a figure explaining operation | movement of the multi-view synthesis process part 131 in the embodiment. It is a schematic block diagram which shows the structure of the camera which produces | generates the conventional high definition image | video.

(First embodiment)
A first embodiment of the present invention will be described in detail with reference to the drawings. An overview of a stereoscopic imaging apparatus 10 according to an embodiment of the present invention is shown in FIG. 1, and a schematic block diagram showing its functional configuration is shown in FIG. As shown in FIGS. 1 and 2, the stereoscopic imaging device 10 includes an imaging unit 101, an imaging unit 102, a parallax calculation unit 21, and a high-resolution composition processing unit 20 arranged in the x-axis direction in the drawings. The imaging unit 101 includes an imaging lens 11-1 and an imaging element 12-1. The imaging unit 102 includes an imaging lens 11-2 and an imaging element 12-2. Note that the imaging unit 101 and the imaging unit 102 are arranged so that their optical axes are parallel so as to capture the same subject.

Since the imaging lens 11-1 and the imaging lens 11-2 have the same configuration except for the installed position, the description of the common part will be described as the imaging lens 11. In addition, since the image pickup device 12-1 and the image pickup device 12-2 have the same configuration except for the installed position, the description of the common part will be described as the image pickup device 12. The imaging lens 11 forms an image of light from the subject on the imaging element 12. As shown in FIG. 2, the imaging element 12 (optical system) may be composed of a single convex lens or an optical system composed of a plurality of lenses. The image sensor 12 is a CMOS image sensor or the like, and photoelectrically converts the image formed and outputs it as a video signal. Hereinafter, the video signal output from the imaging device 12-1 of the imaging unit 101 is referred to as a video signal R, and the video signal output from the imaging device 12-2 of the imaging unit 102 is referred to as a video signal L.

Two systems of video signals (video signal R and video signal L) output by the imaging unit 101 and the imaging unit 102 are input to the parallax calculation unit 21 and the high-resolution composition processing unit 20. The parallax calculation unit 21 searches for corresponding points between the two input video signals, and R reference parallax data RS based on the viewpoint of the imaging unit 101 and the viewpoint of the imaging unit 102 based on the search result. The L reference parallax data LS is calculated and output to the high resolution composition processing unit 20. The high-resolution composition processing unit 20 synthesizes the two input video signals based on the parallax data (disparity information), and outputs a right-eye video signal RC and a left-eye video signal LC.

FIG. 3 is a diagram illustrating an arrangement example of the imaging lens and the imaging element. In FIG. 3, the x axis is taken in the horizontal direction (lateral direction), the y axis is taken in the vertical direction (up and down direction), and the z axis is taken in the depth direction. That is, FIG. 3 shows the arrangement of the imaging lens and the imaging element when the stereoscopic imaging device 10 is viewed from the front. As shown in FIG. 3, the imaging lens 11-1 and the imaging lens 11-2 are arranged at the same position in the y-axis direction. On the other hand, the image pickup device 12-1 is shifted from the image pickup device 12-2 by py / 2 in the y-axis direction (vertical direction). Here, py is the length of the pixel in the image sensor 12 in the y-axis direction. That is, the image pickup device 12-1 and the image pickup device 12-2 are arranged so as to be shifted in the y-axis direction (vertical direction) by half the pixel height of the image pickup device 12.

Thereby, in the imaging unit 101, the position of the imaging device with respect to the imaging lens is shifted upward by half of the imaging pixel of the imaging device, as compared with the imaging unit 102. Conversely, the image sensor 12-1 may be arranged to be shifted by py / 2 below the image sensor 12-2 in the y-axis direction (vertical direction). In that case, the arrangement order of the pixels when the composition processing is performed in the high-resolution composition processing unit 20 described later is reversed.

FIG. 4 is a diagram illustrating another arrangement example of the imaging lens and the imaging element. In FIG. 4, the x axis is taken in the horizontal direction (lateral direction), the y axis is taken in the vertical direction (up and down direction), and the z axis is taken in the depth direction. In the example shown in FIG. 4, the imaging lens 11-1 is arranged to be shifted downward by py / 2 in the y-axis direction (vertical direction) from the imaging lens 11-2. On the other hand, the image sensor 12-1 and the image sensor 12-2 are disposed at the same position in the y-axis direction. That is, the image pickup lens 11-1 and the image pickup lens 11-2 are arranged so as to be shifted in the y-axis direction (vertical direction) by half the pixel height of the image pickup element 12.

Thereby, in the imaging unit 101, the position of the imaging element 12 with respect to the imaging lens 11 is shifted by half of the imaging pixel of the imaging element 12 as compared with the imaging unit 102. Conversely, the image sensor 12-1 may be arranged to be shifted by py / 2 below the image sensor 12-2 in the y-axis direction (vertical direction). In that case, the arrangement order of the pixels when the composition processing is performed in the high-resolution composition processing unit 20 described later is reversed.

Next, the parallax calculation unit 21 will be described in detail with reference to FIG. FIG. 5 is a schematic block diagram illustrating the configuration of the parallax calculation unit 21. The parallax calculation unit 21 calculates parallax data from the video signal R output from the imaging unit 101 in FIG. 1 and the video signal L output from the imaging unit L102. The parallax calculation unit 21 includes coordinate

conversion units

31 and 32, a right camera parameter storage unit 30R, a left camera parameter storage unit 30L, and a corresponding point search unit 33.

The right camera parameter storage unit 30R holds camera parameters including internal parameters such as a focal length and lens distortion parameters specific to the imaging unit 101, and external parameters representing the positional relationship between the two

imaging units

101 and 102. . Similarly, the left camera parameter storage unit 30L holds camera parameters specific to the imaging unit 102.
The coordinate conversion unit 31 performs geometric conversion (coordinate conversion) on the video represented by the video signal R output from the imaging unit 101 by a known method for the purpose of placing the video of the imaging unit 101 and the imaging unit 102 on the same plane. To make the epipolar line parallel. At this time, the coordinate conversion unit 31 uses the camera parameters stored in the right camera parameter storage unit 30R. The coordinate conversion unit 32 geometrically converts the video represented by the video signal L output from the imaging unit 102 by an known method for the purpose of placing the video images of the imaging unit 101 and the imaging unit 102 on the same plane. Is parallelized. At this time, the coordinate conversion unit 32 uses the camera parameters stored in the left camera parameter storage unit 30L.

The corresponding point search unit 33 searches for the corresponding pixel between the images in which the epipolar lines are parallelized by the coordinate conversion unit 31 and the coordinate conversion unit 32, and represents the parallax between the viewpoint of the imaging unit 101 and the viewpoint of the imaging unit 102 Ask for data. The corresponding point search unit 33 is composed of blocks that calculate two types of parallax data. One is an R reference parallax calculation unit 34, and the other is an L reference parallax calculation unit 35. The R reference parallax calculation unit 34 corresponds to each pixel of the reference image, using the image of the imaging unit 101 with the epipolar line parallelized as a reference image and the image of the imaging unit 102 with the epipolar line parallelized as a reference image. The reference video pixel is searched to calculate R-standard parallax data RS. The L reference parallax calculation unit 35 uses the video of the imaging unit 102 with the epipolar line parallelized as a reference video and the video of the imaging unit 101 with the epipolar line parallelized as a reference video, and references corresponding to each pixel of the standard video The pixel of the video is searched to calculate L reference parallax data. The R-standard parallax calculation block 34 and the L-standard parallax calculation block 35 have the same corresponding point search operation except that the standard video and the reference video are reversed.

Next, a processing operation for searching for a corresponding pixel by the R reference parallax calculation unit 34 will be described with reference to FIGS. FIG. 6 is a diagram illustrating the reference image RG. FIG. 7 is a diagram illustrating the reference image BG. As described above, the epipolar lines are parallelized in both the reference image RG and the standard image BG. In order to obtain a pixel (corresponding pixel) on the reference image RG corresponding to a pixel on the standard image BG, a method of moving the target pixel on the standard image BG will be described with reference to FIG. The R reference parallax calculation unit 34 sets a block centered on the target pixel on the reference image BG (hereinafter referred to as a reference target block BB) from the upper left end (reference start block BS) of the reference image BG to the right along the line. If the moved reference attention block BB reaches the right end of the line, the reference attention block BB is moved pixel by pixel from the left end under one line to the right along the line. This is repeated until the block at the lower right corner of the reference image BG (reference end block BE) is reached.

Next, the processing operation for the R-standard parallax calculation unit 34 to search for a block on the reference image RG similar to a certain target block (standard target block BB) on the standard image BG shown in FIG. 7 will be described with reference to FIG. The description will be given with reference. The R reference parallax calculation unit 34 first refers to a block (reference start block RS) on the reference image RG having the same coordinates as the coordinates (x, y) of the reference target block BB on the reference image BG shown in FIG. The target block RB is set, and thereafter, the reference target block is moved pixel by pixel along the line to the right. Then, as shown in FIG. 6, it is moved from the reference start block RS to a reference end block RE that is separated to the right by the search range. The search range (search range) here is a value corresponding to the maximum parallax of the photographed subject, and the shortest distance to the subject for which parallax data can be calculated is determined by the set search range. The R reference parallax calculation unit 34 searches the reference attention block BB on the reference image BG illustrated in FIG. 7 for the reference attention block described above.

Next, a method of determining a reference block of interest similar to the reference block of interest by the R reference parallax calculation unit 34 will be described with reference to FIGS. FIG. 9 is a diagram illustrating the configuration of the reference block of interest BB. The reference attention block BB is a block having a size of horizontal M × vertical N around the target pixel on the reference image BG. FIG. 8 is a diagram illustrating a configuration of the reference attention block RB. The reference attention block RB is a block having a size of horizontal M × vertical N around the target pixel on the reference image RG. In order to represent an arbitrary pixel on the reference target block RB shown in FIG. 8 and the reference target block BB shown in FIG. 9, pixel values of coordinates (i, j) where the horizontal direction is i and the vertical direction is j are respectively R Let (i, j), T (i, j).

The R reference parallax calculation unit 34 calculates a similarity for each combination of the reference attention block BB and the reference attention block RB, and determines a reference attention block RB similar to each reference attention block BB. As a method of obtaining the similarity, generally used SAD (Sum of Absolute Difference) is used. SAD calculates the absolute value of the difference between R (i, j) and T (i, j) for all the pixels of the block as in the similarity determination formula shown in equation (1), and sums them (SSAD) It is. The R reference parallax calculation unit 34 selects a reference attention block having the smallest SSAD value in the expression (1) among the reference attention blocks RG in the search range on the reference image RG corresponding to a certain reference attention block BB. It is determined that the reference attention block is similar. Then, the R reference parallax calculation unit 34 sets a pixel at the center of the reference attention block RB similar to the reference attention block BB as a pixel corresponding to the attention pixel at the center of the reference attention block BB.

The processing operation for searching for a corresponding pixel by the L reference parallax calculation unit 35 is substantially the same as that of the R reference parallax calculation unit 34, but the search range is different. In the case of the L reference parallax calculation unit 35, the reference start block RS is a block that is separated from the same coordinates as the reference target block BB by a search range to the left side, and the reference end block RE is a block at the same coordinates as the reference target block BB. It is.

The above description describes the method for calculating parallax data for each processing unit. Next, description will be given along the sequence of input pixels with reference to the processing flows shown in FIGS. 6 to 9 and FIG. The R reference parallax calculation unit 34 first sets the reference attention block at the head (reference start block BS) of the reference image BG (FIG. 7) (step S900). Then, all pixel values of the reference target block BB are read from the reference image BG (step S901). Next, the R standard parallax calculation unit 34 sets the block in the reference image RG (FIG. 6) having the same coordinates as the standard target block BB, that is, the head of the reference image RG (reference start block RS) as the reference target block RB. (Step S902). Then, all pixel values of the reference attention block RB are read from the reference image RG (step S903). The R reference parallax calculation unit 34 calculates and stores the SSAD values of the pixel values of the read reference attention block BB and reference attention block RB according to the equation (1) (step S904).

Next, the R reference parallax calculation unit 34 determines whether or not the search range has ended (step S905). If the search range has not ended, the R reference parallax calculation unit 34 moves the reference block of interest to the right along the line direction by one pixel (step S906). ), Step S903 and Step S904 are performed again. These steps S903 to S906 are repeated while the reference block of interest RB is within the search range, and all SSAD values within the search range are calculated. Based on these calculation results, the R reference parallax calculation unit 34 detects the reference block of interest RB having the smallest SSAD value (step S907). Here, the minimum SSAD value calculated in step S907 is not necessarily a correct similar block. For example, the parallax cannot be detected correctly when the reference target block BB has no pattern (texture) or contour as a feature point, or when the search area on the reference image RG is an occlusion area. Whether the parallax has been detected correctly can be determined from how small the minimum SSAD value is.

Therefore, the R reference parallax calculation unit 34 compares the minimum SSAD value with the threshold (step 908), and when the SSAD value is equal to or lower than the threshold (that is, when the similarity is high), the center of the reference block of interest BB (reference The difference between the x coordinate of the target pixel on the image BG) and the x coordinate of the center of the detected reference target block RB (the corresponding pixel on the reference image RG) is output as parallax data of the target pixel (step S909). ). On the other hand, when the SSAD value exceeds the threshold (that is, when the degree of similarity is low), the R reference parallax calculation unit 34 determines that the parallax has not been detected, and sets the parallax data to 0 or a unique value as the error flag. And output (step 910).

Finally, the R reference parallax calculation unit 34 determines whether or not the reference block of interest BB has reached the reference end block BE, that is, whether or not the processing has ended (step S911). The block BB is moved to the right by one pixel along the line direction (step S912), and steps S901 to S910 are performed again. If it is determined in step S911 that the process has ended, the process ends. In this way, the steps S901 to S910 are repeated until the reference target block BB becomes the search end block of the reference image BG, and the parallax data of each pixel on the reference image BG is obtained.

In the above description, as an example of the processing method of the corresponding point search unit 33 in FIG. 5, the pixel on the reference image similar to the target pixel on the base image is searched using the SAD similarity evaluation function. The parallax data may be obtained using any technique as long as it is a technique for searching for similar pixels on the standard image and the reference image.

Next, the detailed configuration and operation of the high resolution composition processing unit 20 shown in FIG. 2 will be described with reference to FIGS. FIG. 11 is a schematic block diagram showing a functional configuration of the high resolution composition processing unit 20. As shown in FIG. 11, the high-resolution composition processing unit 20 includes a left-eye composition unit 908 that generates a left-eye video signal LC, a right-eye composition unit 909 that generates a right-eye video signal RC, and a right camera parameter storage unit 902R. And a left camera parameter storage unit 902L. Each of the left-eye composition unit 908 and the right-eye composition block unit 909 includes an alignment correction processing unit 901, a correction processing unit 903, and a composition processing unit 906. Since the basic operation is the same for the left-eye synthesis unit 908 and the right-eye synthesis unit 909 except that the combination of input parallax data is different, the operation of the left-eye synthesis unit 908 will be described here. Description is omitted.

In the left eye combining unit 908, the video signal R of the imaging unit 101 is input to the alignment correction processing unit 901, and the video signal L of the imaging unit 102 is input to the correction processing unit 903. The alignment correction processing unit 901 corrects the lens distortion of the video represented by the video signal R based on the camera parameters indicating the lens distortion state of the imaging unit 101 stored in the right camera parameter storage unit 902R, and then the parallax calculation unit 21 Based on the L reference parallax data LS input from, and the camera parameters indicating the orientation and orientation of the image capturing unit 101 stored in the right camera parameter storage unit 902R, each pixel of the image with the lens distortion corrected is an image of the image capturing unit 102 ( Alignment is performed so as to capture the same subject position as a pixel having the same coordinates in the image corrected by the correction processing unit 903.

However, in this alignment, when the image pickup unit 101 described in FIGS. 3 and 4 is compared with the image pickup unit 102, the position of the image pickup device 12-1 with respect to the image pickup lens 11-1 is determined by the image pickup device 12-1. Correction is not performed for a state of being shifted upward by half of the pixel. That is, the pixel at the coordinate (x, y) as the processing result of the alignment correction processing unit 901 is the pixel at the coordinate (x, y) as the processing result of the correction processing unit 903 and the pixel at the coordinate (x, y−1). The subject position between is captured. On the other hand, the correction processing unit 903 corrects the lens distortion of the video represented by the video signal L based on the camera parameters indicating the lens distortion state stored in the left camera parameter storage unit 902L.

That is, the alignment correction processing unit 901 and the correction processing unit 903 perform parallelization of epipolar lines using camera parameters and correction of parallax using L reference parallax data LS. Here, the parallelization of the epipolar line can be performed by a known method. In addition, as the parallax correction, the alignment correction processing unit 901 shifts the pixels of the video obtained by parallelizing the epipolar line to the video represented by the video signal R by the amount corresponding to the parallax indicated by the L reference parallax data LS. Move to. For example, if the L reference parallax data LS at the coordinates (x, y) is d, the pixel at the coordinates (x + d, y) is moved to the coordinates (x, y). Note that the video signal R and the camera parameters of the imaging unit 101 are input to the correction processing unit 903 of the right-eye synthesis unit 909, and the video signal L and the camera of the imaging unit 102 are input to the alignment correction processing unit 901. A parameter and R reference parallax data RS are input. Further, in the parallax correction in the alignment correction processing unit 901 of the right-eye synthesis unit 909, the pixel is moved to the right by the amount of parallax indicated by the parallax data RS.

Next, the operation of the composition processing unit 906 of the left-eye composition unit 908 will be described with reference to FIG. In FIG. 12, the horizontal axis indicates the spread (size) of the space in the y-axis direction, and the vertical axis indicates the light amplitude (light intensity). In the graph 40a, an image of a subject is formed by the imaging lenses 11-1 and 11-2 among the imaging elements 12-1 and 12-2 of the

imaging units

101 and 102, and is incident on a certain vertical row of pixels. Shows the distribution of light. The graph denoted by reference numeral 40e indicates the distribution of the output of the alignment correction processing unit 901 corresponding to the pixel on which the light of the graph 40a is incident among the pixels of the imaging element 12-1 of the imaging unit 101. A graph denoted by reference numeral 40f indicates an output distribution of the correction processing unit 903 corresponding to the pixel on which the light of the graph 40a is incident among the pixels of the imaging element 12-2 of the imaging unit 102. The graph of reference numeral 40g shows the distribution of the output of the synthesis processing unit 906 with respect to the distribution of

reference numerals

40e and 40f. Here, for simplicity of explanation, the relationship between the graphs will be described ignoring the effects of correction by the alignment correction processing unit 901 and the correction processing unit 903.

Of the vertical lines in the figure, the solid line is the boundary line of the pixel of the imaging element 12-1 of the imaging unit 101, and the broken line is the boundary line of the pixel of the imaging element 12-2 of the imaging unit 102. That is, reference numerals 40b and 40c in the figure are pixels of the imaging unit 101 and the imaging unit 102, respectively, and the relative positional relationship is shifted by an offset indicated by an arrow 40d. Note that the offset is preferably set to be half the size of the pixels (reference numerals 40b and 40c) of the imaging elements 12-1 and 12-2. A half-pixel offset makes it possible to generate the highest definition image. Since the image sensors 12-1 and 12-2 integrate the light intensity in units of pixels, when an image of a subject indicated by reference numeral 40a is taken by the image pickup element 12-1, a video signal having a light intensity distribution indicated by reference numeral 40e. When a picture is taken with the image sensor 12-2, a video signal having a light intensity distribution indicated by reference numeral 40f is obtained.

The composition processing unit 906 synthesizes an image by alternately arranging the output of the alignment correction processing unit 901 and the output of the correction processing unit 903 in the y-axis direction, and compared with the outputs of the

imaging units

101 and 102. A left-eye video signal LC in which the resolution in the y-axis direction is doubled is generated. At this time, the imaging unit 101 considers that the position of the imaging device 12-1 with respect to the imaging lens 11-1 is shifted upward by half of the imaging pixel of the imaging device 12-1, as compared with the imaging unit 102. The pixel b of the correction processing unit 903 derived from the imaging unit 102 and having the same coordinates as the pixel a of the alignment correction processing unit 901 derived from the imaging unit 101 is disposed below the pixel a. That is, when the origin is the upper right of the image, the x-axis is the right direction, and the y-axis is the lower direction, the pixel at the coordinates (x, y) being output from the alignment correction processing unit 901 is The pixel at the coordinates (2x, 2y) being outputted, and the pixel at the coordinates (x, y) being outputted by the correction processing unit 903 is a pixel at the coordinates (2x, 2y + 1) being outputted by the synthesis processing unit 906.

The composition processing unit 906 can reproduce a high-definition image close to the graph 40a indicated by reference numeral 40g by combining the two images in this way.
Note that the synthesis processing unit 906 of the right-eye synthesis unit 909 operates in the same manner as the synthesis processing unit 906 of the left-eye synthesis unit 908, but is a pixel having the same coordinates as the pixel c of the correction processing unit 903 derived from the imaging unit 101. Therefore, the pixel d of the alignment correction processing unit 901 derived from the imaging unit 102 is arranged below the pixel c.

The above combining operation is executed by the left eye combining unit 908 and the right eye combining unit 909. As a result, the left-eye compositing unit 908 outputs a left-eye video signal LC, which is a high-definition video signal taken from the position of the imaging unit 102 (ie, viewed from the left eye), and the right-eye compositing unit 909 captures an image. A right-eye video signal RC that is a high-definition video signal shot from the position of the unit 102 (that is, viewed from the right eye) is output. By displaying the left-eye and right-eye images on the stereoscopic display device, it is possible to display a stereoscopic image having a resolution twice that of the

imaging units

101 and 102.

As described above, the left-eye video signal LC and the right-eye video signal RC output from the stereoscopic imaging device 10 have twice the resolution as compared with the video signals R and L output from the

imaging units

101 and 102. ing. Therefore, the stereoscopic imaging device 10 can generate a stereoscopic video image having a resolution twice that of a stereoscopic imaging device having a resolution equivalent to that of the video signal output by the imaging unit 101.

(Second embodiment)
A second embodiment of the present invention will be described in detail with reference to the drawings. In addition, about the part of the same function as 1st embodiment, description is abbreviate | omitted using the same code | symbol. FIG. 13 is an overview diagram illustrating an overview of the stereoscopic imaging apparatus according to the present embodiment. As shown in FIG. 13, the stereoscopic imaging apparatus 111 according to the present embodiment differs from the first embodiment in the number of imaging units, from two eyes (reference numerals 101, 102) to four eyes (

reference numerals

101, 102, 103, 104). ) And increasing. That is, the stereoscopic imaging device 111 includes an imaging unit 101, an imaging unit 102, an imaging unit 103, and an imaging unit 104. The imaging unit 101 includes an imaging lens 11-1 and an imaging element 12-1. Similarly, the imaging unit 102 includes an imaging lens 11-2 and an imaging element 12-2, the imaging unit 103 includes an imaging lens 11-3 and an imaging element 12-3, and the imaging unit 104 includes an imaging lens. 11-4 and an image sensor 12-4.

FIG. 14 is a schematic block diagram illustrating a functional configuration of the stereoscopic imaging device 111 according to the present embodiment. The stereoscopic imaging device 111 includes

imaging units

101, 102, 103, 104, a parallax calculation unit 21, and a multi-view high-resolution composition processing unit 121. The video signals R and L output from the imaging unit 101 and the imaging unit 102 are input to the multi-view high resolution synthesis processing unit 121 and the parallax calculation unit 21. The video signal R ′ output from the imaging unit 103 and the video signal L ′ output from the imaging unit 104 are input to the multi-view high-resolution composition processing unit 121. The processing of the parallax calculation unit 21 is the same as that of the first embodiment, and calculates the R reference parallax data and the L reference parallax data, and outputs them to the multi-view high-resolution synthesis processing unit 121. The high-resolution synthesis processing unit 121 performs synthesis processing on the input four video signals based on the two parallax data, and outputs a right-eye video signal RC and a left-eye video signal LC.

FIG. 15 is a diagram illustrating the arrangement of the imaging lens and the imaging element in the present embodiment. In FIG. 15, the x axis is taken in the horizontal direction (lateral direction), the y axis is taken in the vertical direction (up and down direction), and the z axis is taken in the depth direction. That is, FIG. 15 shows the arrangement of the imaging lens and the imaging element when the stereoscopic imaging device 111 is viewed from the front. As shown in FIG. 15, the imaging lens 11-1 and the imaging lens 11-2 are arranged at the same position in the y-axis direction. The imaging lens 11-3 and the imaging lens 11-4 are disposed at the same position in the y-axis direction. The imaging lens 11-1 and the imaging lens 11-3 are disposed at the same position in the x-axis direction. The imaging lens 11-2 and the imaging lens 11-4 are disposed at the same position in the x-axis direction. The distance Dx from the center of the imaging lens 11-1 to the center of the imaging lens 11-2 is equal to the distance Dy from the center of the imaging lens 11-1 to the center of the imaging lens 11-3. As described above, in the present embodiment, the imaging units 101 to 104 are arranged at the vertices of a square in which each side is along either the horizontal or vertical direction.

On the other hand, the image sensor 12-1 is arranged to be shifted by py / 2 in the y-axis direction (vertical direction) from the image sensor 12-2. Further, the image sensor 12-3 is arranged so as to be shifted by py / 2 in the y-axis direction (vertical direction) from the image sensor 12-4. Here, py is the length of the pixel in the image sensor 12 in the y-axis direction. The image sensor 12-1 is arranged to be shifted to the left by px / 2 in the x-axis direction (lateral direction) from the image sensor 12-3. The image sensor 12-2 is arranged to be shifted to the left by px / 2 in the x-axis direction (lateral direction) from the image sensor 12-4. Here, px is the length of the pixel in the x-axis direction in the image sensor.

As a result, in the imaging unit 101, the position of the imaging element with respect to the imaging lens is shifted upward by half of the imaging pixel of the imaging element, as compared with the imaging unit 102. Similarly, in the imaging unit 103, as compared with the imaging unit 104, the position of the imaging element with respect to the imaging lens is shifted upward by half of the imaging pixel of the imaging element. Further, in the imaging unit 101, the position of the imaging element with respect to the imaging lens is shifted to the left by half of the imaging pixel of the imaging element, as compared with the imaging unit 103. Similarly, in the imaging unit 102, the position of the imaging element with respect to the imaging lens is shifted to the left by half of the imaging pixel of the imaging element, as compared with the imaging unit 104.
Here, an example in which the imaging elements are displaced is shown, but the imaging lens may be displaced as shown in FIG. 4 described in the first embodiment.

Next, the detailed configuration and operation of the multi-view high resolution composition processing unit 121 will be described with reference to FIGS. The multi-view high-resolution composition processing unit 121 includes a left-eye multi-view composition unit 130 that generates a left-eye high-definition image, a right-eye multi-view composition unit 132 that generates a right-eye high-definition image, and the imaging unit 101. A camera parameter storage unit 902 for storing 104 camera parameters. The left-eye composition unit 130 and the right-eye composition block 130 include an alignment correction processing unit 901, a correction processing unit 903, a vertical / horizontal alignment correction processing unit 904, a vertical alignment correction processing unit 905, and a multi-view synthesis processing unit 131, respectively. It comprises. Since the left eye synthesis unit 130 and the right eye synthesis unit 132 have the same basic operation except that the combination of the input video signal and the parallax data is different, the operation of the left eye synthesis block 131 will be described here.

The video signal R output from the imaging unit 101 is input to the alignment correction processing unit 901. Similar to the first embodiment, the alignment correction processing unit 901 performs the correction processing and alignment of the video represented by the video signal R, the camera parameters of the imaging unit 101 stored in the camera parameter storage unit 902, and the L reference parallax. Based on the data LS, an image from the viewpoint of the imaging unit 102 is generated. The video signal L output from the imaging unit 102 is input to the correction processing unit 903. Similar to the first embodiment, the correction processing unit 903 performs correction processing of the video represented by the video signal L based on the camera parameters of the imaging unit 102 stored in the camera parameter storage unit 902.

The video signal R ′ output from the imaging unit 103 is input to the vertical / horizontal alignment correction processing unit 904. The vertical / horizontal alignment correction processing unit 904 performs correction processing and alignment of the video represented by the video signal R ′ based on the camera parameters of the imaging unit 103 stored in the camera parameter storage unit 902 and the L reference parallax data LS. An image from the viewpoint of the imaging unit 102 is generated. At this time, the distance Dx between the center of the imaging lens 11-1 of the imaging unit 101 and the center of the imaging lens 12-1 of the imaging unit 102 is equal to the center of the imaging lens 11-1 of the imaging unit 101 and the imaging unit 103. Since it is equal to the distance Dy from the center of the imaging lens 12-3, L reference parallax data LS is used as vertical parallax data. That is, the vertical / horizontal alignment correction processing unit 904 performs alignment by applying the L reference parallax data LS in the vertical direction and the horizontal direction. For example, the pixel value of the coordinate (x, y) in the image aligned by the vertical / horizontal alignment correction processing unit 904 is the video signal output by the imaging unit 103 when the L reference parallax data LS at the coordinate is d. Is a pixel value of coordinates (x + d, yd) in the image obtained by correcting the image represented by the camera parameter.

The video signal L ′ output from the imaging unit 104 is input to the vertical alignment correction processing unit 905. The vertical alignment correction processing unit 905 performs correction processing and alignment of the video represented by the video signal L ′ based on the camera parameters of the imaging unit 104 stored in the camera parameter storage unit 902 and the L reference parallax data LS. An image from the viewpoint of the imaging unit 102 is generated. That is, the vertical alignment correction processing unit 905 performs alignment by applying the L reference parallax data LS in the vertical direction. For example, the pixel value of the coordinates (x, y) in the image aligned by the vertical alignment correction processing unit 905 is the video signal output by the imaging unit 104 when the L reference parallax data LS at the coordinates is d. Is a pixel value at coordinates (x, yd) in the image obtained by correcting the image represented by the camera parameter.

Next, the operation of the multi-eye synthesis processing unit 131 in the left-eye synthesis unit 130 and the right-eye synthesis unit 132 will be described with reference to FIG. In the first embodiment, high-resolution composition processing using two images is performed, but in this embodiment, the multi-eye composition processing unit 131 includes four systems obtained by the four

imaging units

101, 102, 103, and 104. High-resolution synthesis using the video signal. The multi-view synthesis processing unit 131 of the left-eye synthesis unit 130 generates and outputs a left-eye video signal LC ′ that is a signal representing video from the viewpoint of the imaging unit 102. In addition, the multi-view synthesis processing unit 131 of the right-eye synthesis unit 132 generates and outputs a right-eye video signal RC ′ that is a signal representing video from the viewpoint of the imaging unit 101. The four-line high-resolution synthesis is the same as the principle described in the light intensity distribution of FIG. 12, but more specifically, the resolution of the four

imaging units

101, 102, 103, and 104 is VGA (640 × 480). In the following description, a high-resolution composition process is performed on a quad-VGA pixel (1280 × 960 pixels) that is four times the number of pixels.

As shown in FIG. 17, it is possible to obtain a high-resolution image by allocating four adjacent pixels of Quad-VGA pixels (1280 × 960 pixels) by allocating pixels captured by different imaging units. It is. Here, the image pickup device 11-1 of the image pickup unit 101 is arranged so as to be shifted upward by a half pixel with respect to the image pickup device 11-2 of the image pickup unit 102. On the other hand, the image sensor 11-4 of the image capturing unit 104 is shifted to the left by half a pixel with respect to the image sensor 11-2 of the image capture unit 102. ing. Therefore, in accordance with the direction in which the image sensor is displaced, the corrected image MR derived from the imaging unit 101, the corrected image ML derived from the imaging unit 102, and the imaging unit 103, as shown in FIG. The multi-eye synthesis processing unit 131 uses pixels G11, G21, G31, and G41, which are pixels having the same coordinates in each of the corrected image MR ′ derived from and the corrected image ML ′ derived from the imaging unit 104. Arrange so that. That is, the pixel G31 is arranged on the right side of the pixel G11, the pixel G21 is arranged on the lower side of the pixel G11, and the pixel G41 is arranged on the right side of the pixel G21.

In the left-eye composition unit 131, the corrected video MR derived from the imaging unit 101 is an image generated by the alignment correction processing unit 901, and the corrected video ML derived from the imaging unit 102 is the correction processing unit. The corrected image MR ′ derived from the image capturing unit 103 is an image generated by the image capturing unit 103, and the corrected image ML ′ derived from the image capturing unit 104 is the vertical position. This is an image generated by the alignment correction processing unit 905. In the right-eye composition unit 132, the corrected video MR derived from the imaging unit 101 is an image generated by the correction processing unit 903, and the corrected video ML derived from the imaging unit 102 is processed by the alignment correction processing unit 901. The corrected image MR ′ derived from the imaging unit 103, which is the generated image, is the image generated by the vertical alignment correction processing unit 905, and the corrected video ML ′ derived from the imaging unit 104 is the vertical / horizontal alignment correction. It is an image generated by the processing unit 904.

The above combining operation is executed by the left eye combining unit 130 and the right eye combining unit 132. As a result, the left-eye synthesizing unit 130 outputs a high-definition image obtained by synthesizing four images captured from the position of the imaging unit 102 (that is, viewed from the left eye). Further, the right-eye combining unit 132 outputs a high-definition image obtained by combining four images captured from the position of the imaging unit 101 (that is, viewed from the right eye). That is, an image having a resolution four times that of the output of the imaging unit is output. By displaying the high-definition video for the left eye and the right eye on the stereoscopic display device, it is possible to display the high-definition stereoscopic video.

As described above, the left-eye video signal LC ′ output from the stereoscopic imaging device 111 and the right-eye video signal RC ′ have four times higher resolution than the video signals output from the

imaging units

101, 102, 103, and 104. Have. That is, the left-eye video signal LC ′ output from the stereoscopic imaging device 111 and the right-eye video signal RC ′ are a video signal obtained by combining the video signal output from the imaging unit 101 and the video signal output from the imaging unit 103. In comparison, it has four times the resolution. Therefore, the stereoscopic imaging device 111 has a device scale equivalent to that of a stereoscopic imaging device having a resolution equivalent to a video signal obtained by combining the video signal output from the imaging unit 101 and the video signal output from the imaging unit 103, and twice that size. 3D images can be generated.

In the present embodiment, the example in which the video signals input to the parallax calculation unit 21 are two video signals output by the

imaging units

101 and 102 has been described. However, the number of input video signals can be increased. For example, it is also possible to input four systems of video output from the

imaging units

101, 102, 103, and 104, calculate four systems of parallax with each as a reference, and perform multi-view high-resolution synthesis with each parallax data. is there. In this case, a high-definition video of a plurality of viewpoints other than those for the right eye and the left eye is output, and as a result, a multi-view stereoscopic video can be generated without resolution deterioration.

In addition, FIG. 2 shows a program for realizing the functions of the parallax calculation unit 21 and the high-resolution synthesis processing unit 20 or the parallax calculation unit 21 and the multi-view high-resolution synthesis processing unit 121 in FIG. 14 on a computer-readable recording medium. The processing of each unit may be performed by recording, reading the program recorded on the recording medium into a computer system, and executing the program. Here, the “computer system” includes an OS and hardware such as peripheral devices.

Further, the “computer-readable recording medium” means a storage device such as a flexible disk, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design changes and the like without departing from the gist of the present invention.

The present invention can be applied to a thin color camera that generates a high-definition image using a plurality of cameras.

DESCRIPTION OF

SYMBOLS

10, 111 Three-dimensional imaging device 11-1, 11-2, 11-3, 11-4 Imaging lens 12-1, 12-2, 12-3, 12-4 Image sensor 20 High-resolution composition processing part 21 Parallax calculation part 30R Right camera parameter storage unit 30L Left camera

parameter storage unit

31, 32 Coordinate conversion unit 33 Corresponding point search unit 34 R reference parallax calculation unit 35 L reference

parallax calculation unit

101, 102, 103, 104 Imaging unit 121 Multi-lens high-resolution synthesis Processing unit 131 Multi-lens synthesis processing unit 901 Position correction processing unit 902 Camera parameter storage unit 902R Right camera parameter storage unit 902L Left camera parameter storage unit 903 Correction processing unit 904 Vertical / horizontal alignment correction processing unit 905 Vertical alignment correction processing unit 906

Compositing processing unit

908, 130 Combining unit for left eye 909, 13 The right-eye synthesis unit

Claims

Two imagers that image the same subject;
A parallax calculation unit that detects corresponding points between the captured images of the two imaging units and calculates parallax information of the captured images of the two imaging units;
Based on the viewpoints of each of the two image capturing units, based on the parallax information and the captured images of the two image capturing units, an image having a larger number of pixels than the captured image is synthesized, and two systems of images having the larger number of pixels A stereoscopic imaging device comprising: a synthesis processing unit that generates
The imaging unit
An optical system that forms an image of a subject on the imaging surface;
An image sensor that generates a signal of a photographic image of a subject imaged on the imaging surface,
2. The one imaging unit according to claim 1, wherein the position of the imaging element with respect to the optical system is shifted up or down by half of the imaging pixel of the imaging element as compared to the other imaging unit. Stereo imaging device.
Including three or more imaging units,
Among the imaging units, between the side-by-side imaging units, the position of the imaging element with respect to the optical system is shifted up or down by half of the imaging pixel of the imaging element,
3. The stereoscopic imaging according to claim 2, wherein a position of the imaging element with respect to the optical system is shifted to either the left or right by half of a photographing pixel of the imaging element between the imaging units arranged vertically in the imaging unit. apparatus.
Four of the imaging units are arranged at the vertices of a square along which each side is either horizontal or vertical,
The parallax calculation unit calculates parallax information of captured images of two imaging units arranged at adjacent vertices of the square among the four imaging units,
The stereoscopic imaging apparatus according to claim 1, wherein the synthesis processing unit uses the parallax information for horizontal and vertical parallax correction of the captured video performed when the video is synthesized.
Comprising at least three imaging units;
The synthesis processing unit synthesizes a video having a larger number of pixels than the captured video based on the parallax information and the captured video of the at least two imaging units based on the viewpoints of each of the at least three imaging units, and at least The stereoscopic imaging device according to claim 1, wherein the stereoscopic imaging device generates three images with a large number of pixels.
Detecting corresponding points between the captured images of the two imaging units that capture the same subject, and calculating parallax information of the captured images of the two imaging units;
Based on the viewpoints of each of the two image capturing units, based on the parallax information and the captured images of the two image capturing units, an image having a larger number of pixels than the captured image is synthesized, and two systems of images having the larger number of pixels A method of generating an image.
The imaging method according to claim 6, wherein the step of generating the video having a large number of pixels includes a step of performing parallax correction of the captured video using the parallax information when the video is synthesized.
The step of generating an image with a large number of pixels is based on the parallax information and at least two captured images of the imaging units based on the viewpoints of the at least three imaging units that capture the same subject. The imaging method according to claim 6, further comprising: synthesizing an image with a large number of pixels to generate an image with a large number of pixels of at least three systems.