JP2012060512A - Multi-eye imaging apparatus and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To suppress increase in throughput and amount of use memory in generation of parallax data by stereo matching processing.SOLUTION: A multi-eye imaging apparatus is provided with: a plurality of imaging units; and a parallax calculation unit which calculates the parallax data by stereo matching processing of searching videos taken by at least two imaging units among the plurality of imaging units, for corresponding pixels. The parallax calculation unit is provided with: a small range corresponding pixel search unit which calculates the parallax data by searching for the corresponding pixels in a range smaller than the whole parallax range of the two imaging units; and a search range control unit which controls the range searched by the small range corresponding pixel search unit.

Description

  The present invention relates to a multi-eye imaging apparatus and a program.

  In recent years, stereoscopic image devices have been actively developed in order to enhance the power and presence of images. As a technique for generating a stereoscopic image, there is known a technique in which two cameras for a left channel (L) and a right channel (R) are arranged on the left and right to simultaneously photograph a subject as in Patent Document 1. . Further, as a technique for displaying a stereoscopic image, a left channel (L) image and a right (R) channel image are alternately displayed on a single display screen for each pixel, and a kamaboko-shaped lens is provided in a predetermined manner. The left and right eyes of the viewer using a special optical system such as a lenticular lens arranged at intervals, a parallax barrier with fine slits arranged at a predetermined interval, and a patterning phase difference plate with regularly arranged fine polarizing elements A technique is known in which only the left channel (L) image is visible to the viewer's left eye and only the right channel (R) image is visible to the right eye.

  In addition to being able to display as a stereoscopic image, an image obtained by simultaneously photographing an object with two cameras, it is also possible to measure the distance to the object. The distance to the subject is obtained by a method called stereo matching. Stereo matching is a technique for obtaining corresponding pixels of a plurality of captured images and calculating parallax (distance) information, which is a shift amount. By performing stereo matching for each captured pixel, parallax (distance) information of each captured pixel can be calculated. A representation of this parallax (distance) data for each screen is called a “parallax map” or “depth map”.

  This parallax map shows three-dimensional information of a subject that could not be acquired by shooting with a single camera until now, and various uses are being studied. For example, when the stereoscopic image is displayed, the stereoscopic effect can be adjusted by performing image processing based on the parallax map. Furthermore, in the technology for generating a high-definition video using a plurality of cameras as in Patent Document 2, when a high-definition synthesis process is performed on a plurality of camera videos, which pixel of the left video and which pixel of the right video are combined. A parallax map is indispensable for determining whether to do this.

JP 2006-251613 A Special table 2007-520166

  However, since the above-described stereo matching processing searches for corresponding pixels in units of pixels, there is a problem that a very large amount of processing and memory usage are required. In recent years, this problem has become prominent because the number of pixels of the imaging device has increased remarkably and real-time processing is necessary for moving image shooting.

  The present invention has been made in view of such problems, and an object of the present invention is to provide a multi-view imaging apparatus and program capable of suppressing an increase in processing amount and memory usage when generating parallax data by stereo matching processing. It is to provide.

(1) The present invention has been made to solve the above-described problems, and the multi-lens imaging device of the present invention has taken a plurality of imaging units and at least two imaging units among the plurality of imaging units. A multi-view imaging device including a parallax calculation unit that calculates parallax data by a stereo matching process for searching for a corresponding pixel from a video, wherein the parallax calculation unit has a smaller range than a total parallax range of the two imaging units. A small-range corresponding pixel search unit that searches for corresponding pixels and calculates parallax data, and a search range control unit that controls a range searched by the small-range corresponding pixel search unit.

(2) The multi-view imaging apparatus of the present invention is the multi-view imaging apparatus described above, wherein the small range corresponding pixel search unit searches for a preset number of pixels, and the search range control unit includes: The small range corresponding pixel search unit is executed as many times as the total parallax range can be searched.

(3) Moreover, the multi-view imaging apparatus of the present invention is the multi-view imaging apparatus described above, and the search range control unit is configured to search a range to be processed in real time based on a frame rate of the imaging unit. It is determined, and the small range corresponding pixel search unit is controlled according to the determination.

(4) The multi-view imaging apparatus of the present invention is the multi-view imaging apparatus described above, wherein the search range control unit determines a search range based on focal position information of the imaging unit, and the determination The small range corresponding pixel search unit is controlled according to the above.

(5) Moreover, the multi-view imaging apparatus of the present invention is the multi-view imaging apparatus described above, wherein the search range control unit determines the depth of field of the imaging unit based on the focal position information of the imaging unit. A corresponding range is determined as a search range, and the small range corresponding pixel search unit is controlled according to the determination.

(6) The multi-view imaging apparatus of the present invention is the multi-view imaging apparatus described above, wherein the search range control unit determines a search range based on a shooting mode set by a user, and follows the determination. The small range corresponding pixel search unit is controlled.

(7) Further, the program of the present invention is a stereo matching that searches a computer of a multi-view imaging apparatus including a plurality of imaging units for corresponding pixels from images captured by at least two imaging units among the plurality of imaging units. A program that functions as a parallax calculation unit that calculates parallax data by processing, wherein the parallax calculation unit searches for corresponding pixels in a range smaller than the total parallax range of the two imaging units and calculates a parallax data The image processing apparatus includes a corresponding pixel search unit and a search range control unit that controls a range searched by the small range corresponding pixel search unit.

  According to the present invention, it is possible to suppress an increase in the amount of processing and the amount of memory used when generating parallax data by stereo matching processing.

1 is an external view showing an overview of a multi-eye imaging device 10 according to a first embodiment of the present invention. It is a schematic block diagram which shows the function structure of the multiview imaging device 10 in the embodiment. It is a schematic block diagram which shows the function structure of the parallax calculation part 21 in the embodiment. It is a figure which shows the reference image RG after the epipolar line parallelization in the same embodiment. It is a figure which shows the reference | standard image BG after the epipolar line parallelization in the same embodiment. It is a figure which shows the reference attention block RB in the same embodiment. It is a figure showing standard attention block BB in the embodiment. It is a figure showing an example of control by search range control part 34 in the embodiment. It is a figure which shows the other example of control by the search range control part 34 in the embodiment. It is a figure which shows the other example of control by the search range control part 34 in the embodiment. It is a figure which shows the other example of control by the search range control part 34 in the embodiment. It is a flowchart explaining the process of the search range control part 34 and the small range corresponding point search part 33 in the embodiment. It is a flowchart explaining another example of the process of the search range control part 34 and the small range corresponding point search part 33 in the embodiment. It is a schematic block diagram which shows the function structure of the image | video synthetic | combination process part 20 in the embodiment. It is a schematic block diagram which shows the function structure of the high resolution synthetic | combination process part 403 in the embodiment. It is a schematic block diagram which shows the function structure of the multiview imaging device 10a which concerns on the 2nd Embodiment of this invention. It is a schematic block diagram which shows the function structure of the parallax calculation part 21a in the embodiment. It is a figure explaining the example of control by the search range control part 34a in the embodiment. It is a figure explaining other examples of control by search range control part 34a in the embodiment. It is a schematic block diagram which shows the function structure of the multiview imaging device 10b which concerns on the 3rd Embodiment of this invention. It is a schematic block diagram which shows the function structure of the parallax calculation part 21b in the embodiment. It is a figure explaining the example of control by the search range control part 34b in the embodiment. It is a figure explaining other examples of control by search range control part 34b in the embodiment. It is a figure explaining other examples of control by search range control part 34b in the embodiment. It is a figure explaining other examples of control by search range control part 34b in the embodiment. It is a schematic block diagram which shows the function structure of the parallax calculation part 21c which concerns on the 4th Embodiment of this invention. It is a figure explaining the example of control by the search range control part 34c in the embodiment.

[First Embodiment]
A first embodiment of the present invention will be described in detail with reference to the drawings. FIG. 1 is an external view showing an overview of a multi-view imaging apparatus 10 according to the first embodiment of the present invention. As shown in FIG. 1, the multi-lens imaging device 10 according to the present embodiment includes four systems of imaging units 101, 102, 103, and 104. The imaging unit 101 is arranged on the upper left side toward the front of the multi-eye imaging device 10. The imaging unit 102 is arranged on the upper right side. The imaging unit 103 is arranged on the lower left side. The imaging unit 104 is arranged on the lower right side. Note that the horizontal and vertical distances between the imaging units 101, 102, 103, and 104 are the same, that is, the imaging units 101, 102, 103, and 104 are arranged at the vertices of a square.

  FIG. 2 is a schematic block diagram illustrating a functional configuration of the multi-eye imaging apparatus 10. The multi-view imaging apparatus 10 includes imaging units 101, 102, 103, 104, a video composition processing unit 20, a parallax calculation unit 21, an image sensor control unit 22, an AF (Auto Focus) control unit 23, and a shooting mode setting unit 24. Consists of. Each of the imaging units 101, 102, 103, and 104 has an imaging lens group 11 having a focus function, and an image sensor 12 (photographing the image formed on the imaging surface by the imaging lens group 11 and outputting it as a video signal. For example, a complementary metal oxide semiconductor (CMOS) image sensor is provided. The imaging unit 101 outputs a video signal RU, the imaging unit 102 outputs a video signal LU, the imaging unit 103 outputs a video signal RD, and the imaging unit 104 outputs a video signal LD.

The imaging lens group 11 is arranged so as to be movable in the optical axis direction. When the imaging lens group 11 moves in the optical axis direction, the focus of the imaging unit is adjusted and the light from the subject is condensed on the image sensor 12. Image. Each of the imaging units 101, 102, 103, and 104 is equipped with a motor (not shown), and moves the imaging lens group 11 with its power.
The shooting mode setting unit 24 sets shooting modes such as landscape shooting, portrait portrait shooting, and close-up shooting (macro shooting) in accordance with user setting input. Based on the setting of the shooting mode from the shooting mode setting unit 24, the image sensor control unit 22 sets the number of shooting pixels and the frame rate of the image sensor 12 of each of the imaging units 101, 102, 103, and 104, and sets the set pixels. The image sensor control information such as the number and the frame rate is output to the parallax calculation unit 21.

The AF control unit 23 calculates a contrast value from the luminance information of the input video signal RU of the imaging unit 101, moves the imaging lens group 11 by a predetermined amount in a direction in which the contrast value increases, and the contrast value increases. The imaging lens group 11 is automatically adjusted to the position where the contrast value disappears, that is, the position where the contrast value is maximized. The AF control unit 23 determines a base point for moving the imaging lens group 11 and a moving range based on the setting of the shooting mode from the shooting mode setting unit 24. In addition, since the same drive signal is input from the AF control unit 23 to each of the imaging units 101, 102, 103, and 104, control is performed so that the focus positions of the imaging units 101, 102, 103, and 104 are all the same. Is done.
Note that the AF control unit 23 of the present embodiment has been described as performing AF control using the contrast value of the captured video signal as an evaluation value, but in addition to this, the time until the wave returns after irradiation with infrared rays or ultrasonic waves Alternatively, the AF control may be performed by the so-called active AF method in which the distance is detected by the irradiation angle.

  All of the four video signals RU and LU output from the four imaging units 101, 102, 103, and 104 are input to the video composition processing unit 20. Furthermore, the video signals RU and LU photographed by the two imaging units 101 and 102 are also input to the parallax calculation unit 21. The parallax calculation unit 21 performs a corresponding pixel search between the two input video signals, calculates parallax data D (equivalent to distance data) between the two imaging units from the search result, and outputs the parallax data D to the video composition processing unit 20 To do. The video composition processing unit 20 synthesizes the video signals RU, LU, RD, and LD based on the input parallax data D to produce a high-definition video whose vertical and horizontal resolution is twice that of the video signal RU or the like. The high-definition video signal G that is the video signal is generated.

  FIG. 3 is a schematic block diagram illustrating a functional configuration of the parallax calculation unit 21. The parallax calculation unit 21 includes an image represented by the video signal RU output from the imaging unit 101 illustrated in FIG. 2 (hereinafter referred to as a reference image) and an image represented by the video signal LU output from the imaging unit 102 (hereinafter referred to as a reference image). From this, the parallax data is calculated. The parallax calculation unit 21 includes a coordinate conversion unit 31, a coordinate conversion unit 32, a small range corresponding point search unit 33, and a search range control unit 34. The two coordinate conversion units 31 and 32 parallelize the epipolar lines by performing geometric conversion (coordinate conversion) on both images for the purpose of placing the image plane of the base image and the reference image on the same plane. The coordinate conversion unit 31 converts the standard image, that is, the image of the video signal RU, and the coordinate conversion unit 32 converts the reference image, that is, the image of the video signal LU. The coordinate conversion unit 31 includes a camera parameter storage unit 30-1. The camera parameter storage unit 30-1 holds in advance internal parameters such as a focal length and lens distortion parameters specific to the imaging unit 101, and external parameters representing the positional relationship between the four imaging units 101, 102, 103, and 104. Yes.

  Similarly, the coordinate conversion unit 32 includes a camera parameter storage unit 30-2. The camera parameter storage unit 30-2 holds camera parameters (internal parameters, external parameters) unique to the imaging unit 102. The coordinate conversion units 31 and 32 output the video signal of the reference image and the reference image in which the epipolar lines are parallel to the small range corresponding pixel search unit 33. The small range corresponding pixel search unit 33 searches for pixels in the reference image corresponding to each pixel of the base image in which the epipolar lines are parallel from the pixels designated by the search range control unit 34. A predetermined range (search range) determined by the number of pixels α set in advance is searched for and detected. The small range corresponding pixel search unit 33 outputs the distance in the image from each pixel of the reference image to the detected corresponding pixel as the parallax data D in the pixel of the reference image. The search range control unit 34 searches by the search processing in the small range corresponding point search unit 33 based on the image sensor control information C output by the image sensor control unit 22 and the shooting mode M output by the shooting mode setting unit 24. The range to be determined is determined and output to the small range corresponding point search unit 33. Details of the search range control unit 34 and the small range corresponding point search unit 33 will be described later.

  A corresponding pixel search process by the small range corresponding pixel search unit 33, that is, a stereo matching process will be described with reference to FIGS. FIG. 4 is a diagram illustrating the reference image RG after the epipolar line is parallelized. FIG. 5 is a diagram showing the reference image BG after parallelizing the epipolar lines. A method of moving a pixel of interest on the reference image BG for obtaining a pixel (corresponding pixel) on the reference image RG corresponding to a pixel on the reference image BG will be described with reference to FIG. The small range corresponding pixel search unit 33 moves a block centering on the target pixel on the reference image BG (hereinafter referred to as the reference target block BB) from the upper left corner of the image (start block BS) to the right in the line direction for each pixel. When the moved reference attention block BB reaches the right end of the line, the reference attention block BB is moved to the right in the line direction from the left end one line below. This is repeated until the reference block of interest BB comes to the block at the lower right end (end block BE) of the reference image BG.

  Next, the processing operation in which the small range corresponding pixel search unit 33 searches for a block on the reference image RG similar to one reference target block BB in the reference image BG shown in FIG. 5 will be described with reference to FIG. To do. The small range corresponding pixel search unit 33 moves the reference attention block RB to the right in the line direction from the block of coordinates (search start block) designated by the search range control unit 34 for each pixel. It repeats until the search end block RE which is a block separated from the RS in the line direction by the search range. The small range corresponding pixel search unit 33 searches this reference attention block for each of the reference attention blocks BB shown in FIG. In FIG. 4, the search start block RS is a block (minimum parallax block) on the reference image RG having the same coordinates as the coordinates (x, y) of the reference block BB on the reference image BG shown in FIG. ing.

  The maximum parallax search range illustrated in FIG. 4 is a range of distances that the imaging units 101 to 104 can capture (focus on). The minimum parallax block at the same coordinates as the reference target block in FIG. 5 corresponds to the farthest position (infinity), and the maximum parallax block in FIG. 4 corresponds to the closest distance. As shown in FIG. 4, the search range of the small range corresponding pixel search unit 33 is smaller than the entire parallax search range. The search range is greatly related to the processing amount and the memory amount, and the processing amount and the memory amount can be suppressed small by setting the search range to be small like the small range corresponding pixel search unit 33.

Next, a reference attention block determination method similar to the reference attention block will be described with reference to FIGS. FIG. 7 shows the reference target block BB, which is a block of size of horizontal M × vertical N with the target pixel on the reference image BG as the center. FIG. 6 shows the reference attention block RB, which is a block having a size of horizontal M × vertical N with the target pixel on the reference image RG as the center. In order to represent an arbitrary pixel on the reference target block RB shown in FIG. 6 and the reference target block BB shown in FIG. 7, the pixel value of the coordinates (i, j) when the horizontal direction is i and the vertical direction is j is Let R (i, j) and T (i, j) respectively. In this embodiment, SAD (Sum of Absolute Difference), which is generally used, is used as a method for obtaining the similarity. SAD calculates the absolute value of the difference between R (i, j) and T (i, j) for all the pixels of the block as in the similarity determination formula shown in equation (1), and sums them (S _SAD ).

The small range corresponding pixel search unit 33 sets the reference attention block having the smallest value of S _SAD as the reference attention block BB among the reference attention blocks RB in the search range on the reference image RG shown in FIG. 4 described above. Determined to be similar. Then, the small range corresponding pixel search unit 33 sets the center pixel of the reference target block RB determined to be similar to the pixel corresponding to the center pixel (target pixel) of the reference target block BB. In the present embodiment, a pixel on the reference image RG similar to the target pixel on the standard image BG is searched using the SAD similarity evaluation function. However, the present invention is not limited to this method. Any technique may be used to obtain the parallax data as long as it is a technique for searching for similar pixels on the image.

  The coordinates of the search start block are set in the small range corresponding pixel search unit 33 by the search range control unit 34. The example of the control method by the search range control part 34 is demonstrated using FIG.3 and FIG.8. As shown in FIG. 3, image sensor control information C is input to the search range control unit 34. The search range control unit 34 calculates the number of pixels in the entire parallax search range from the number of pixels of the captured video obtained from the input image sensor control information C. The calculation of the total parallax search range is to calculate how many pixels the shooting parallax corresponds to. The maximum parallax is determined from the focal length (FL) of the camera, the closest distance (MD) that can be photographed, and the distance (CI) between the two cameras, and the maximum parallax = FL × CI / MD. Therefore, when the horizontal width of the image sensor is W and the shooting pixel is 640 × 480 pixels, the total parallax search range = maximum parallax × 640 pixels / W. In order to search the entire parallax search range (from the minimum parallax block SD to the maximum parallax block MD) by dividing the number of pixels by the search range (preset pixel number α) of the small range correspondence pixel search unit 33, the small range correspondence pixel It calculates how many times the search unit 33 needs to be executed.

FIG. 8 shows an example in which the number of executions of the small range corresponding pixel search unit 33 necessary for the full parallax search is calculated to be four times. As shown in FIG. 8, the search range control unit 34 controls the small range corresponding pixel search unit 33 so that the search range 1 to 4 is executed four times while shifting the search range. The small range corresponding pixel search unit 33 determines that the reference attention block having the smallest value of S _SAD among the four executions is similar to the reference attention block BB. In this control method, since the search by the small range corresponding pixel search unit 33 is executed a plurality of times, the processing time becomes long. However, since the memory usage per execution is small, the processing distance is less than the conventional distance from the farthest distance. It is possible to search all the parallaxes up to the nearest distance.

  It is also possible to perform control so as to search a part of the entire parallax search range according to the shooting mode information output from the shooting mode setting unit 24 instead of searching the entire parallax search range. For example, when the shooting mode is a landscape mode for shooting a distant view, the search range control unit 34 sets the minimum parallax block SD as a search start block as shown in FIG. Accordingly, the search range control unit 34 sets the small range corresponding pixel search unit 33 so as to search only once from the position of the minimum parallax block SD. For this reason, the small range corresponding pixel search unit 33 searches only a small parallax range (that is, a subject at a far shooting distance).

  For example, when the shooting mode is a close-up shooting mode in which a close-up shooting is performed such as close-up shooting, the search range control unit 34 sets the number of pixels set in advance from the maximum parallax block as shown in FIG. The block on the left side by α is set as the search start block RS. Accordingly, the search range control unit 34 sets the small range corresponding pixel search unit 33 so as to search only once up to the position of the maximum parallax block in the entire parallax search range. For this reason, the small range corresponding pixel search unit 33 searches only a large parallax range (that is, a subject at a close shooting distance).

  Further, for example, if the shooting mode is a mode in which the subject is at an intermediate position, such as a portrait mode for shooting a person, the search range control unit 34 starts from the center of the total parallax search range as shown in FIG. The block on the left side by a half of the preset pixel number α is set as the search start block RS. Accordingly, the search range control unit 34 sets the small range corresponding pixel search unit 33 so as to search the central portion of the entire parallax search range only once. For this reason, the small range corresponding pixel search unit 33 searches only the subject at the intermediate position.

  A flow in which the search range control unit 34 and the small range corresponding point search unit 33 generate parallax data for one screen will be described with reference to a processing flow shown in FIG. First, the search range control unit 34 determines the search range and the number of executions of the small range corresponding point search unit 33 (step S900). Here, as described with reference to FIGS. 8 to 11, the search range and the number of executions are determined using the number of pixels and the shooting mode of the input image sensor control information. Subsequently, based on the determined search range and the position of the reference block of interest, a search start block is set and the small range corresponding point search unit 33 is executed (step S901). After this execution, the small range corresponding point search unit 33 obtains and stores the SAD minimum value and the parallax data (step S902). Next, the search range control unit 34 determines whether or not the small range corresponding point search unit 33 has been executed the number of times determined in step S900 (step S903). Next, the next search start block is set (step S904), and the process returns to step S902.

  After repeating these steps S902 to S904 for the number of executions, the small range corresponding point search unit 33 selects the minimum SAD value from among the SAD minimum values stored for the number of executions, and outputs it as the parallax data D (step). S905). Finally, the search range control unit 34 determines whether or not the processing for one screen has been completed (step S906). As a result of this determination, if not completed, the search range control unit 34 moves the reference target block by one pixel (step S907), and returns to step S901. If the result of determination in step S906 is that the processing has been completed, that is, if the reference block of interest is the end block of the reference image, the processing is boosted. Thereby, parallax data of each pixel on the reference image is obtained.

  In the flowchart of FIG. 12, the process of calculating the parallax value by executing the small range corresponding point search unit 33 a plurality of times while changing the search range in units of pixels has been described. However, the search range is changed in units of screens. Also good. The processing in this case will be described using the flowchart of FIG. The same reference numerals are used for the same processing steps as in FIG. Since the processing from S900 to S902 is the same, the description thereof is omitted. Next, the search range control unit 34 determines whether or not the processing for one screen has been completed (step S906). If not, the search range control unit 34 moves the reference target block and the search range by one pixel (step S101). ), The process returns to step S902. After repeating these steps S902, S906, and S101 until the processing for one screen is completed, the search range control unit 34 determines whether or not the small range corresponding point search unit 33 has been executed the number of times determined in step S900. (Step S903). As a result of this determination, if the determined number of times has not been executed, the search range control unit 34 returns the target block of the reference image to the start position (step S102), and sets the next search range to the small range corresponding point search unit 33. After setting (step S904), the processing of S902, S906, and S101 is performed for one screen again.

  After repeating for the number of executions, the small range corresponding point search unit 33 selects the minimum SAD value from the SAD minimum values for the number of executions for each pixel and outputs it as the parallax data D (step S103). In the present processing flow, the number of times of step S904 for setting the search range is reduced as compared with the processing flow of FIG. Step S904 requires a large amount of processing because it is necessary to read all pixel values within the search range. The amount of processing can be reduced by reducing the number of times of S904. Although the example in which the search range is updated in units of screens has been described with reference to FIG. 20, a processing flow in which the search range is updated in units of lines or memory units installed in the apparatus is also possible in addition to screen units.

  FIG. 14 is a schematic block diagram illustrating a functional configuration of the video composition processing unit 20. The video composition processing unit 20 includes an alignment correction processing unit 401-1, an alignment correction processing unit 401-2, an alignment correction processing unit 401-3, an alignment correction processing unit 401-4, and a high resolution composition processing unit 403. Consists of including. The alignment correction processing unit 401-1 includes a camera parameter storage unit 402-1. The alignment correction processing unit 401-2 includes a camera parameter storage unit 402-2. The alignment correction processing unit 401-3 includes a camera parameter storage unit 402-3. The alignment correction processing unit 401-4 includes a camera parameter storage unit 402-4.

  The alignment correction processing units 401-1, 401-2, 401-3, and 401-4 are the parallax data D output from the parallax calculation unit 21 and the camera parameter storage units 402-1, 402-2, and 402 included in each. -3 and 402-4 indicate the orientation and orientation of each imaging unit and the state of lens distortion, and the pixels at the same coordinate position are the same subject position between the images of the video signals RU, LU, RD, and LD. So that the images of the video signals RU, LU, RD, and LD are converted. The high-resolution composition processing unit 403 performs high-definition composition using images that have been aligned by the four alignment correction processing units 401-1, 401-2, 401-3, and 401-4. A high-definition video signal G representing an image is output.

  The operation of the alignment correction processing units 401-1 to 401-4 will be described. Camera parameters that are stored in the camera parameter storage units 402-1 to 402-4 and indicate the orientation, orientation, and lens distortion state of each imaging unit are numerical values while changing the posture and angle of a checkered checker pattern whose pattern shape is known. It can be obtained by camera calibration calculated from the captured image after multiple imaging. The camera parameters are composed of external parameters and internal parameters, and the external parameters are composed of a total of six parameters including a three-axis vector of yaw, pitch, and roll indicating the posture of the camera and a three-axis translation vector indicating a translation component. The internal parameters are composed of five parameters: the image center position, the angle and aspect ratio of the coordinates assumed on the image sensor, and the focal length.

  Here, the image capturing unit 101 is used as a reference image capturing unit, and the images of the other image capturing units 102, 103, and 104 are matched with the images of the image capturing unit 101. The shift amount calculated from the parallax data D representing the distance to the subject, the interval (camera baseline length) between the reference imaging unit 101 and the other imaging units 102, 103, and 104 is a translation amount that is an external parameter of the camera parameter. By performing the geometric correction process using the parameters added to the above, the alignment correction processing units 401-1 to 401-4 can detect the same point of the subject in the image represented by the video signals RU, LU, RD, and LD at the same position ( The position can be adjusted so as to be captured by (pixel). As shown in FIG. 14, the registration correction processing unit 401-1 processes the video signal RU, and the registration correction processing unit 401-2 processes the video signal LU, and the registration correction processing unit 401-. 3 processes the video signal RD, and the alignment correction processing unit 401-4 processes the video signal LD.

  Next, an example of the synthesis process by the high resolution synthesis processing unit 403 is shown in FIG. Here, the resolution of the four imaging units 101, 102, 103, and 104 is VGA (640 × 480 pixels), and the resolution is increased to Quad-VGA pixels (1280 × 960 pixels), which is four times the number of pixels. This will be described in the case of performing the combining process. As shown in FIG. 15, pixels having the same coordinates of the images RU1, LU1, RD1, and LD1 based on video signals output from different imaging units are added to four adjacent pixels of Quad-VGA pixels (1280 × 960 pixels). A high-resolution image is obtained by assigning and compositing. Here, the image RU1 is an image generated by processing the video signal RU by the alignment correction processing unit 401-1. Similarly, the image LU1 is an image generated by processing the video signal LU by the alignment correction processing unit 401-2. The image RD1 is an image generated by the alignment correction processing unit 401-3 processing the video signal RD. The image LD1 is an image generated by the alignment correction processing unit 401-4 processing the video signal LD.

  Thus, the search range of the small range corresponding pixel search unit 33 is smaller than the entire parallax search range. The search range is largely related to the amount of processing and the amount of memory used, and since the search range is set small, the amount of processing and the amount of memory used for stereo matching processing can be kept small.

[Second Embodiment]
A second embodiment of the present invention will be described in detail with reference to the drawings. FIG. 16 is a schematic block diagram illustrating a functional configuration of the multi-lens imaging device 10a according to the present embodiment. In the figure, the same reference numerals (101 to 104, 11, 12, 20, 23) are assigned to portions corresponding to the respective portions in FIG. 2, and the description thereof is omitted. The multi-eye imaging device 10a is a device that captures moving images. The multi-eye imaging device 10a includes imaging units 101, 102, 103, and 104, a video composition processing unit 20, a parallax calculation unit 21a, an image sensor control unit 22a, and an AF control unit 23. The image sensor control unit 22a adds image sensor control information C ′ obtained by adding information indicating the frame rate of the image sensor 12 of the imaging units 101 to 104 to the image sensor control information C output by the image sensor control unit 22 of FIG. It outputs to the parallax calculation part 21a. The parallax calculation unit 21a calculates parallax data using the image sensor control information C ′.

  FIG. 17 is a schematic block diagram illustrating a functional configuration of the parallax calculation unit 21a. In the figure, the same reference numerals (30-1, 30-2, 31 to 33) are assigned to portions corresponding to the respective portions in FIG. The parallax calculation unit 21a includes coordinate conversion units 31 and 32, a small range corresponding point search unit 33, and a search range control unit 34a. The search range control unit 34 a controls the small range corresponding point search unit 33 based on the image sensor control information C ′.

  FIG. 18 is a diagram illustrating an example of a control method of the small range corresponding point search unit 33 by the search range control unit 34a. The search range control unit 34a obtains a frame rate from the input image sensor control information C ', and calculates a processing time per frame. Thereafter, the search range control unit 34a divides the processing time per frame by the processing time per time of the small range corresponding pixel search unit 33 held in advance, and can perform processing within one frame. The number of processing times of the unit 33 is calculated. Further, the search range control unit 34a divides the number of processes that can be processed in one frame by the number of pixels in one frame obtained from the image sensor control information C ′, and can process a small range corresponding pixel search unit for each pixel. The number of executions 33 is calculated.

  FIG. 18 shows an example in which the number of processes per pixel is calculated as two. In this example, the search range control unit 34a sets a search range (search range 1, search range 2) at the center of the entire parallax search range. Specifically, the search range control unit 34a multiplies the preset number of pixels α by the number of processings and further divides by two. The search range control unit 34a uses the position on the left side from the center of the total parallax search range by the number of pixels as a result of dividing by 2 as the search start block RS1 of the search range 1, and the position of the search end block of the search range 1 Is a search start block RS2 of the search range 2. Thereby, the search range control unit 34a controls the small range corresponding point search unit 33 to search the central portion of the entire parallax search range twice. This control method cannot search the entire disparity search range, but can output a moving image disparity map having the same frame rate of the captured video in real time.

  FIG. 19 is a diagram for explaining another example of the control method of the small range corresponding point search unit 33 by the search range control unit 34a. In FIG. 18, the processing count (two times) of the small range corresponding pixel search unit 33 that can be processed in one frame is set to a continuous search range. However, as shown in FIG. 19, the entire parallax search range is equally spaced. In such a manner, it may be set in a distributed manner. In FIG. 19, the range (search range 1) searched by the small range corresponding pixel search unit 33 for the first time is indicated by a thin line, and the range searched for the second time (search range 2) by a bold line. In this example, the search range control unit 34a equally divides the entire parallax search range into the calculated number of processes + 1 region, and positions the left side by a half of the preset number of pixels α from each boundary of the region. Are the search start blocks RS1 ′ and RS2 ′ of each search range (search range 1, search range 2). Thus, by setting the search range to be distributed at equal intervals, it is possible to output a moving image parallax map roughly searched for the entire parallax search range.

  When video processing using parallax data performs real-time processing of a moving image, real-time processing is also required for parallax data calculation. For example, in the case of shooting while adjusting the stereoscopic effect of a stereoscopic image, it is necessary to calculate parallax data of the captured image in real time and perform stereoscopic effect adjustment image processing, which is possible in this embodiment. .

[Third Embodiment]
A third embodiment of the present invention will be described in detail with reference to the drawings. FIG. 20 is a schematic block diagram illustrating a functional configuration of the multi-lens imaging device 10b according to the present embodiment. In the figure, the same reference numerals (101 to 104, 11, 12, 20, 22) are assigned to portions corresponding to the respective portions in FIG. 2, and the description thereof is omitted. The multi-eye imaging device 10b includes imaging units 101, 102, 103, and 104, a video composition processing unit 20, a parallax calculation unit 21b, an image sensor control unit 22, and an AF control unit 23b. The AF control unit 23b outputs, to the parallax calculation unit 21b, the focus position information F indicating the focus position of the imaging lens group 11, that is, the distance between the lens and the image sensor. The parallax calculation unit 21b calculates the parallax data D using the focal position information F.

  FIG. 21 is a schematic block diagram illustrating a functional configuration of the parallax calculation unit 21b. In the figure, the same reference numerals (30-1, 30-2, 31 to 33) are assigned to portions corresponding to the respective portions in FIG. The parallax calculation unit 21b includes coordinate conversion units 31 and 32, a small range corresponding point search unit 33, and a search range control unit 34b. The search range control unit 34 b controls the small range corresponding point search unit 33 based on the image sensor control information C and the focal position information F.

  As shown in FIG. 21, the focus position information F from the AF control unit 23b is input to the search range control unit 34b. The search range control unit 34 calculates the parallax value of the input focal position information F (distance of the focused subject focused). Specifically, the search range controller 34 first uses the focal length f (mm) of the imaging lens group 11 and the distance l (mm) between the lens and the image sensor 12 as the focal position information F to reach the subject. Is calculated using the following equation (1). Here, the distance between the lens represented by the focal position information F and the image sensor 12 is the distance between the fictitious convex lens optically equal to the lens group 11 and the image sensor 12. Next, the search range control unit 34 uses the calculated shooting distance L, the camera interval t (mm) between the imaging units 101 and 102, and the focal length f (mm) to set the parallax value d (mm) as follows: (2) Calculate using the formula.

  Then, the search range control unit 34 obtains a parallax value dp (pixel) obtained by converting the parallax value d (mm) into the number of pixels from the pixel pitch of the image sensor 12. Then, as shown in FIG. 22, the search range of the small range corresponding pixel search unit 33 is set so that the parallax value dp of the focal position is centered. That is, the search range control unit 34 sets the number of pixels obtained by subtracting half of the preset number of pixels α (search range) from the parallax value dp to the right side from the left end (position of the minimum parallax block SD) of the entire parallax search range. Let the position be the position of the search start block RS.

  With the control method according to this embodiment, the parallax data near the focused subject, which is the most important parallax data for high-definition video processing, can be calculated with a small amount of processing and the amount of memory used.

  Other control method examples using the parallax value obtained from the focal position information will be described with reference to FIGS. 23, 24, and 25. FIG. 23 shows an example in which the search range is set twice so that the parallax value dp at the focal position is at the center of the search range. For example, when the number of processing times of the small range corresponding pixel search unit 33 that can be processed in one frame is calculated as 2, the control shown in FIG. 23 is effective. FIG. 24 shows an example in which the parallax value dp at the focal position is set to be a range for searching for the minimum parallax block SD side. The example of FIG. 24 searches for a parallax for a subject far from the focal position, and is an effective control method when, for example, shooting is performed against a landscape. On the other hand, FIG. 25 shows an example in which the maximum parallax block MD side is searched from the parallax value dp at the focal position. The example of FIG. 25 searches for a parallax for a subject close to the focal position, and is an effective control method when, for example, an object is placed in front and a person is photographed.

Note that which of these control methods is to be selected may be determined according to the shooting mode set by the user, as in the first embodiment.
The AF control unit 22b has been described as adjusting the focal position by autofocus and outputting the focal position information F indicating the focal position to the parallax calculation unit 21b. However, the focus indicating the focal position adjusted by manual focus is described. The position information may be output.

[Fourth Embodiment]
A fourth embodiment of the present invention will be described in detail with reference to the drawings. The multi-eye imaging device 10c in the present embodiment has the same configuration as the multi-eye imaging device 10b shown in FIG. 20, except that it has a parallax calculation unit 21c instead of the parallax calculation unit 21b. FIG. 26 is a schematic block diagram illustrating a functional configuration of the parallax calculation unit 21c according to the present embodiment. In the figure, the same reference numerals (30-1, 30-2, 31 to 33) are assigned to portions corresponding to the respective portions in FIG. 21, and the description thereof is omitted. The parallax calculation unit 21c includes coordinate conversion units 31 and 32, a small range corresponding point search unit 33, and a search range control unit 34c.

  The search range control unit 34c holds a value of the depth of field obtained from the optical specifications of the imaging lens group 11 in a table for each shooting distance. The search range control unit 34c calculates the shooting distance to the subject using the input focal position information F (the distance of the focused subject of interest), and refers to the table to determine the parallax according to the depth of field. Get the range. Then, the search range of the small range corresponding pixel search unit 33 is set so as to fill the parallax range of the depth of field as shown in FIG. This control method can search for the parallax of all subjects in focus.

In each of the above-described embodiments, the high-definition synthesis process (video synthesis processing unit 20) is described as the video process using the generated parallax data. However, the present invention is not limited thereto, and other video processes may be performed. Good.
In addition, the multi-lens imaging device in each of the first, third, and fourth embodiments may be a device that captures a still image or may be a device that captures a moving image.
Further, in each of the above-described embodiments, the small range corresponding pixel search unit 33 has been described on the assumption that the preset number of pixels α is the width of the search range, and the search start position is set by the search range control unit. The search range control unit may control the start position and the search width that the small range corresponding pixel search unit 33 searches.

  Also, a program for realizing the function of the parallax calculation unit in each embodiment is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed to calculate the parallax. May be processed. Here, the “computer system” includes an OS and hardware such as peripheral devices.

  The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes design changes and the like within a scope not departing from the gist of the present invention.

  Although it is suitable for use in a digital still camera and a digital video camera having a plurality of imaging units, it is not limited to these.

10, 10a, 10b, 10c ... multi-lens imaging devices 101, 102, 103, 104 ... imaging unit 11 ... imaging lens group 12 ... image sensor 20 ... video composition processing unit 21, 21a, 21b, 21c ... parallax calculation unit 22, 22a, ... Image sensor control unit 23, 23b ... AF control 24 ... Shooting mode control unit 30-1, 30-2 ... Camera parameter storage unit 31, 32 ... Coordinate conversion unit 33 ... Small range corresponding point search unit 34, 34a, 34b, 34c ... Search range control unit

Claims (7)

A multi-lens imaging device comprising: a plurality of imaging units; and a parallax calculation unit that calculates parallax data by a stereo matching process that searches for corresponding pixels from videos captured by at least two imaging units among the plurality of imaging units. There,
The parallax calculation unit
A small range corresponding pixel search unit that searches for corresponding pixels in a range smaller than the total parallax range of the two imaging units and calculates parallax data;
A multi-view imaging apparatus comprising: a search range control unit that controls a range to be searched by the small range corresponding pixel search unit. The small range corresponding pixel search unit searches for a preset number of pixels,
The multi-view imaging apparatus according to claim 1, wherein the search range control unit causes the small range corresponding pixel search unit to be executed as many times as the total parallax range can be searched. The search range control unit determines a search range so as to be processed in real time based on a frame rate of the imaging unit, and controls the small range corresponding pixel search unit according to the determination. The multi-lens imaging device according to 1. 2. The multi-range search unit according to claim 1, wherein the search range control unit determines a search range based on focal position information of the imaging unit, and controls the small range corresponding pixel search unit according to the determination. Eye imaging device. The search range control unit determines a range corresponding to the depth of field of the imaging unit as a search range based on the focal position information of the imaging unit, and determines the small range corresponding pixel search unit according to the determination. The multi-lens imaging device according to claim 4, wherein the multi-eye imaging device is controlled. The multi-view according to claim 1, wherein the search range control unit determines a search range based on a shooting mode set by a user, and controls the small range corresponding pixel search unit according to the determination. Imaging device. A parallax calculation unit that calculates parallax data by stereo matching processing that searches for a corresponding pixel from videos captured by at least two imaging units among the plurality of imaging units. A functioning program,
The parallax calculation unit
A small range corresponding pixel search unit that searches for corresponding pixels in a range smaller than the total parallax range of the two imaging units and calculates parallax data;
A search range control unit that controls a range to be searched by the small range corresponding pixel search unit.

Images