JP2011048295A - Compound eye photographing device and method for detecting posture of the same - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To inexpensively and easily determine whether or not a photographed image has a failure in stereoscopic view. <P>SOLUTION: A feature point 74 and a corresponding point 77 are extracted from standard image data and reference image data obtained by stereo photography respectively. The position of the feature point 74 and the position of the corresponding point 77 are compared with each other to obtain the posture of a camera body 11. A deviation amount (posture deviation amount) between the obtained posture and the preset standard posture of the camera body 11 is obtained. An image not having a problem in stereoscopic view is sorted into an OK image group and an image having a failure in stereoscopic view is sorted into an NG image group, based on the posture deviation amount. The standard image data and reference image data of the OK image group are stored in the OK image storage part 20a of a memory card 20, and the standard image data and reference image data of the NG image group is stored in the NG image storage part 20b of the memory card 20 or is discarded. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a compound eye photographing apparatus that performs compound eye photographing such as stereo photographing and a posture detection method thereof.

  There is known a compound eye camera that includes a pair of imaging units arranged symmetrically and that stereo-shoots left and right viewpoint images with parallax (see Patent Document 1). A stereoscopic image can be observed by separately displaying the left viewpoint image and the right viewpoint image obtained by stereo shooting separately from the left and right eyes of the observer.

  When performing stereo shooting with such a compound-eye camera, it is necessary to perform stereo shooting while holding the compound-eye camera in a reference posture such that the arrangement direction of each imaging unit is substantially parallel to the horizontal line. When stereo shooting is performed in such an orientation that the arrangement direction has an inclination angle with respect to the horizontal line, the vertical displacement of the subject in the left and right viewpoint images becomes large, and stereoscopic image observation (stereoscopic view) (See FIG. 9).

  In Patent Literature 2, it is determined whether or not a person's posture is appropriate by performing face detection for detecting a person's face included in the through image. An imaging system for informing that effect is described. In the imaging system of Patent Document 1, when the camera body is held in the above-described tilted posture, the posture of the person in the through image is tilted, so that a notification that the posture of the person is inappropriate is performed. For this reason, the photographer can be prompted to hold the camera body in the reference posture.

  Patent Document 3 describes a display angle adjustment device that automatically adjusts the angle of a display in a direction in which a person can easily view based on the result of detecting a human face from a captured image.

  In Patent Documents 4 and 5, a posture detection sensor for detecting the posture is provided in the camera body, and when the posture of the camera body matches the reference posture based on the detection result of the posture detection sensor, photographing is automatically performed. The camera to be executed is described.

JP 2005-157721 A JP 2005-148245 A Utility Model Registration No. 3144172 JP-A-10-133308 JP 2002-271654 A

  By the way, in the imaging system of the above-mentioned patent document 2, there is a problem that it is impossible to deal with a case where the subject is other than a person. Furthermore, in this imaging system, it is determined whether or not the person's posture is appropriate by detecting the face. However, it is difficult to distinguish between an appropriate posture and an inappropriate posture. When taking a picture of a person who is, there is a risk that the posture is determined to be inappropriate.

  In addition, when taking a picture with a compound eye camera, the photographer holds the compound eye camera, and as described in Patent Document 3, the posture of the camera body is automatically adjusted based on the face detection result. I can't.

  In the cameras of Patent Documents 4 and 5, the posture of the camera body can be detected. However, since a posture detection sensor needs to be provided separately, there arises a problem that the manufacturing cost of the camera increases.

  The present invention has been made to solve the above problems, and an object of the present invention is to provide a compound eye photographing apparatus capable of detecting the posture of a camera body at the time of photographing at low cost and a posture detecting method thereof.

  In order to achieve the above object, a compound eye photographing apparatus of the present invention is integrated into a camera body, and includes a plurality of imaging means for photographing a subject, and each of the photographing images obtained from the plurality of imaging means. By comparing corresponding point extraction means for extracting corresponding points indicating common characteristic points existing in the images, and the positions of the corresponding points corresponding to each other in each of the captured images, the respective captured images Posture detecting means for detecting the posture of the camera body at the time of shooting.

  Based on the detection result of the posture detection means, a deviation amount between a predetermined reference posture of the camera body and the posture of the camera body at the time of shooting is obtained, and each captured image is determined according to the magnitude of the deviation amount. , Distribution means for allocating to any of a plurality of photographed image groups divided according to the magnitude of the shift amount, and each of the photographic images allocated to a preselected photographed image group from among the plurality of photographed image groups It is preferable to include image storage means for storing an image.

  The reference posture is a posture of the camera body in which the arrangement direction of the plurality of imaging units is substantially parallel to a horizontal line, and the posture detection unit is configured such that the camera body has an inclination angle with respect to the horizontal line. It is preferable to detect the posture of the camera body when tilted.

  It is preferable that the posture detection unit detects the posture of the camera body by comparing the vertical positions of the corresponding points corresponding to each other in the captured images.

  The corresponding point extraction unit extracts a plurality of the corresponding points from each of the captured images, and the posture detection unit detects the posture of the camera body by comparing the positions of the corresponding points corresponding to each other. It is preferable.

  Based on the extraction result of the corresponding point extracting unit, the apparatus includes a parallax calculating unit that calculates parallax between the corresponding points corresponding to each other, and the posture detecting unit compares the positions of the corresponding points corresponding to each other. It is preferable that weighting is performed according to the magnitude of the parallax, and the posture of the camera body is detected based on the respective comparison results after weighting.

  The corresponding point extracting unit extracts a plurality of the corresponding points from each of the captured images, and calculates a parallax between a plurality of corresponding points corresponding to each other based on an extraction result of the corresponding point extracting unit. Preferably, the posture detecting means detects the posture of the camera body by comparing the positions of the corresponding points having the largest parallax among the corresponding points corresponding to each other.

  It is preferable that the image storage unit stores the captured images allocated to the captured image group such that the magnitude of the shift amount is equal to or less than a predetermined first threshold value.

  Each of the plurality of captured image groups is assigned with each of the captured images in which the shift amount is greater than the first threshold and less than or equal to a second threshold greater than the first threshold. Image processing means that includes a captured image group, and that performs image processing on each captured image distributed to the first captured image group so that the shift amount is equal to or less than the first threshold value. It is preferable to provide.

  It is preferable that the image processing unit performs a trimming process for cutting out an image range that is captured from each captured image.

  A continuous shooting mode in which a plurality of times of shooting are continuously performed, and the plurality of captured image groups have the smallest deviation amount in each of the captured images obtained in the continuous shooting mode. It is preferable that a second photographed image group to which each photographed image is distributed is included, and the image storage unit stores each photographed image distributed to the second photographed image group.

  Preferably, when the subject is a person, the corresponding point extraction unit detects a person's face from each of the captured images, and extracts the corresponding point from each detected face image.

  When the subject is a building, the corresponding point extraction unit preferably extracts a building from each captured image, and extracts the corresponding point from each detected building image.

  Further, the posture detection method of the compound eye photographing device of the present invention includes a photographing step of photographing a subject by a plurality of imaging units integrated in the camera body, and each of the photographed images obtained in the photographing step. A corresponding point extracting step for extracting corresponding points indicating common characteristic points respectively present in the photographed images, and comparing the positions of the corresponding points in the respective photographed images with each other, And a posture detecting step for detecting the posture of the camera body at the time of taking an image.

  The compound eye photographing apparatus and the posture detection method of the present invention extract the corresponding points from each captured image obtained by the plurality of imaging means, and based on the result of comparing the positions of the extracted corresponding points, the posture of the camera body is determined. Since the detection is made, the posture of the camera body can be detected at low cost without providing a posture detection sensor or the like in the camera body.

  Determining the amount of deviation between the camera body's predetermined reference posture and the posture of the camera body at the time of shooting, and assigning each shot image to one of multiple shot image groups according to the magnitude of the obtained deviation amount Therefore, it is possible to easily determine whether or not each captured image stored in the storage medium is a defective image captured when the camera body is in the tilted posture.

  A plurality of corresponding points are extracted from each captured image, and the positions of the corresponding corresponding points are compared to detect the posture of the camera body. Compared with the case where the amount of deviation is determined, the amount of deviation can be obtained with higher accuracy.

  Since the comparison results of the positions of corresponding points corresponding to each other are weighted according to the magnitude of the parallax, the comparison results of the positions of corresponding points extracted from the closest subject, that is, the camera Since the comparison result of the position of the corresponding point that more accurately reflects the posture of the main body is emphasized, the deviation amount can be obtained more accurately.

  Based on the comparison result of the positions of the corresponding points having the largest parallax among the corresponding points corresponding to each other, that is, the comparison result of the position of the corresponding point that most accurately reflects the posture of the camera body, By detecting the posture, the amount of deviation can be obtained easily and accurately.

  By performing image processing so that the deviation amount is equal to or less than the first threshold value for each photographed image whose deviation amount is equal to or greater than the first threshold value and less than the second threshold value, Even if a picture is taken when is in a tilted position, an image having no malfunction can be obtained.

  In the continuous shooting mode, each captured image with the smallest deviation is selected and stored in the storage medium, so that the data amount of the image stored in the storage medium can be reduced.

It is a front perspective view of the compound eye camera of the present invention. It is a rear perspective view of a compound eye camera. It is a block diagram which shows the electrical constitution of a compound eye camera. FIG. 4 is a functional block diagram of a storage control unit in FIG. 3. It is explanatory drawing for demonstrating the extraction method of the feature point and corresponding point from a face image. It is explanatory drawing for demonstrating the conditions of the distribution process by a distribution part. It is a flowchart for demonstrating the flow of the process at the time of performing stereo imaging | photography. It is explanatory drawing for demonstrating the process when stereo imaging | photography is performed with the reference | standard attitude | position. It is explanatory drawing for demonstrating the process when stereo imaging | photography is performed by the inclination attitude | position. It is explanatory drawing for demonstrating a trimming process. It is a flowchart for demonstrating the flow of a process at the time of continuous shooting mode. It is explanatory drawing for demonstrating the flow of a process at the time of performing a special distribution process in the continuous shooting imaging | photography mode. It is explanatory drawing for demonstrating the extraction method of the feature point and corresponding point from a building image. It is explanatory drawing for demonstrating the guide display for framing.

  As shown in FIG. 1, a pair of left and right first and second imaging units 12 and 13, a strobe light emitting unit 14, and the like are provided on the front surface of the camera body 11 of the compound-eye camera 10. Both the imaging units 12 and 13 are integrally incorporated in the camera body 11 and are provided at a predetermined interval so that their optical axes are substantially parallel to each other or have a letter C shape. The camera body 11 in the figure is in a reference posture in which the arrangement direction of the imaging units 12 and 13 is substantially parallel to the horizontal line. A shutter button 15 and a power switch 16 are provided on the upper surface of the camera body 11.

  As shown in FIG. 2, a liquid crystal display (hereinafter referred to as LCD) 18 and an operation unit 19 are provided on the back surface of the camera body 11. Although not shown, the bottom surface of the camera body 11 has a card slot into which a memory card (photographed image storage means, storage medium) 20 is detachably loaded, and a loading lid that opens and closes the opening of the card slot. Is provided.

  The LCD 18 functions as an electronic viewfinder in a shooting standby state, and displays a through image (also referred to as a live view image). The through image display means that an image captured by each of the imaging units 12 and 13 at a predetermined frame rate is immediately displayed on the LCD 18. Further, at the time of image reproduction, the image is reproduced and displayed on the LCD 18 based on the image data recorded on the memory card 20.

  The operation unit 19 includes a mode switch 22, a menu button 23, a cross key 24, an execution key 25, and the like. The mode switch 22 is operated when switching the operation mode of the compound eye camera 10. The operation modes include a still image shooting mode in which still image shooting is performed, a moving image shooting mode in which moving image shooting is performed, and a playback mode in which a captured image obtained by shooting is played back and displayed on the LCD 18. In the still image shooting mode, for example, a left viewpoint image for stereoscopic viewing (hereinafter referred to as a reference image) and a right viewpoint image (hereinafter referred to as a reference image) are captured. Note that the right viewpoint image may be a reference image, and the left viewpoint image may be a reference image.

  The menu button 23 is operated when a menu screen or a setting screen is displayed on the LCD 18. The cross key 24 is operated when moving a cursor displayed on the menu screen or the setting screen. The execution key 25 is operated when confirming the setting of the camera.

  As shown in FIG. 3, the CPU 33 executes various programs and data read from a ROM (not shown) sequentially based on input signals from the shutter button 15 and the operation unit 19, thereby centralizing each unit of the compound eye camera 10. Control.

  The SDRAM 34 functions as a work memory for the CPU 33 to execute processing. The VRAM 35 has a through-image memory area for storing continuous two-field fractions, and temporarily stores image data for display.

  The first imaging unit 12 includes a lens unit 38 in which a photographing lens 37 is incorporated, a CCD image sensor (hereinafter referred to as CCD) 39, an AFE (analog front end) 40, and the like. A MOS type image sensor may be used instead of the CCD.

  The lens unit 38 incorporates a zoom mechanism, a focus mechanism, and a diaphragm device (not shown). The zoom mechanism moves the photographing lens 37 to perform zooming. The focus mechanism moves the focus lens incorporated in the photographing lens 37 to perform focusing. The diaphragm device adjusts the intensity of subject light incident on the CCD 39 by adjusting a diaphragm (not shown). The zoom mechanism, the focus mechanism, and the aperture device are controlled by the CPU 33 via the lens driver 41.

  Behind the photographic lens 37 is a CCD 39 in which a large number of photodiodes are arranged on the light receiving surface, and converts the subject light from the photographic lens 37 into an electrical imaging signal and outputs it. A CCD driver 42 controlled by the CPU 33 is connected to the CCD 39. The CCD driver 42 is driven by a synchronization pulse from a TG (Timing Generator) 43, and controls the charge accumulation time and charge read transfer timing of the CCD 39.

  The imaging signal output from the CCD 39 is input to the AFE 40. The AFE 40 is composed of a CDS (correlated double sampling) circuit, an AGC (automatic gain adjustment amplifier), and an A / D converter. By receiving a synchronization pulse from the TG 43, the AFE 40 is synchronized with the charge readout transfer operation of the CCD 39. Works. The CDS circuit performs correlated double sampling to remove noise from the imaging signal. The AGC circuit amplifies the imaging signal with a gain corresponding to the imaging sensitivity set by the CPU 33. The A / D converter converts the analog imaging signal from the AGC circuit into a digital reference image signal, and sends the converted reference image signal to the image input controller 45.

  The second imaging unit 13 has the same configuration as the first imaging unit 12 and includes a lens unit 47, a CCD 48, an AFE 49, a lens driver 50, a CCD driver 51, a TG 52, and the like, and sends a reference image signal to the image input controller 45. .

  The CPU 33 is connected to the SDRAM 34, VRAM 35, image input controller 45, signal processing circuit 56, AF detection circuit 57, AE / AWB detection circuit 58, trimming processing circuit (image processing means) 59, compression / decompression processing circuit via the bus 54. 60, a media controller 61, a display circuit 62, and the like are connected.

  The image input controller 45 has a buffer of a predetermined capacity, accumulates the standard and reference image signals output from the first and second imaging units 12 and 13, respectively, and each of the standard and reference image signals for one frame. Is stored, a standard and reference image signal corresponding to an image for one frame is sent to the signal processing circuit 56.

  The signal processing circuit 56 performs various processing such as gradation conversion, white balance correction, γ correction processing, and YC conversion processing on the standard and reference image signals from the image input controller 45, thereby converting each of the images into one frame image. Corresponding standard and reference image data are generated and stored in the VRAM 35.

  The AF detection circuit 57 calculates an AF evaluation value obtained by evaluating the contrast for each standard and reference image based on the standard and reference image signals from the image input controller 45. The CPU 33 controls the lens drivers 41 and 50 based on the AF evaluation value from the AF detection circuit 57 to adjust the focus of the photographing lens 37.

  The AE / AWB detection circuit 58 detects subject luminance and calculates a WB evaluation value used for white balance correction based on the standard and reference image signals. The CPU 33 controls the lens drivers 41 and 50 and the CCD drivers 42 and 51 based on the subject luminance information from the AE / AWB detection circuit 58 to perform exposure control. Further, the CPU 33 controls the signal processing circuit 56 so that the white balance of the subject image is appropriate based on the WB evaluation value from the AE / AWB detection circuit 58.

  The trimming processing circuit 59 reads the standard and reference image data from the VRAM 35 under the control of the CPU 33, and cuts out the image range (area) designated by the CPU 33 from the standard and reference image data.

  The compression / decompression processing circuit 60 performs compression processing on the non-compressed standard and reference image data stored in the VRAM 35 when the shutter button 15 is pressed, and generates standard and reference compressed image data of a predetermined file format. The compression / decompression processing circuit 60 performs decompression processing on the standard and reference compressed image data recorded in the memory card 20 during image reproduction, and generates uncompressed standard and reference image data. The media controller 61 performs recording and reading of image data with respect to the memory card 20.

  The display circuit 62 performs predetermined signal processing on the standard image data read from the VRAM 35 or the uncompressed standard and reference image data expanded by the compression / decompression processing circuit 60 to generate a signal for image display. Is output to the LCD 18 at a fixed timing. Thus, the reference image is displayed on the LCD 18 as a through image in each shooting mode. In the normal playback mode, the image data read from the memory card 20 is displayed on the LCD 18. Note that the standard image and the reference image may be displayed on the LCD 18 as a through image on two screens.

  The CPU 33 functions as a storage control unit 64 that controls storage of captured image data in the memory card 20 by sequentially executing various programs read from the ROM. The storage control unit 64 analyzes the reference and reference image data newly stored in the memory card 20 when a shooting instruction is given, detects the posture of the camera body 11 during stereo shooting, and the detection result Accordingly, the reference and reference image data are separately distributed and stored in the memory card 20.

  As shown in FIG. 4, the storage control unit 64 includes a feature point extraction unit 66, a corresponding point extraction unit 67, a position information acquisition unit 68, a posture calculation unit 69, a parallax calculation unit 70, a posture deviation amount determination unit 71, and a vibration It consists of a minute part 72.

  The feature point extraction unit 66 and the corresponding point extraction unit 67 correspond to the corresponding point extraction unit of the present invention. The feature point extraction unit 66 analyzes the reference image data and detects a feature point 74 (see FIG. 5) indicating a characteristic point from the reference image data. The feature point extraction unit 66 has a face area detection function.

  As shown in FIG. 5, the feature point extraction unit 66 detects the faces of the persons A and B from the reference image data 75 </ b> L, and sets the peripheral area of the detected face as the face area 76. As the face detection method, for example, various known methods such as a face area detection method by hue detection or skin color detection can be used. Next, the feature point extraction unit 66 extracts a plurality of feature points 74 from the face region 76. The feature point 74 is a point (pixel) having a characteristic change in the pixel value in the face region 76, and is particularly preferably a corner (corner) or end point having a change in the pixel value in the horizontal and vertical directions. Examples of the feature point extraction method include a Harris method, a Moravec method, and a Shi-Tomasi method, and any of these may be used.

  The corresponding point extraction unit 67 analyzes the reference and reference image data 75L and 75R based on the feature point extraction result by the feature point extraction unit 66, and detects the position of the corresponding point 77 corresponding to each feature point 74. As a corresponding point detection method, for example, there are a block matching method, a KLT-Tracker method, and the like, and any of these may be used. The feature points 74 and the corresponding points 77 correspond to the corresponding points of the present invention.

  Returning to FIG. 4, the position information acquisition unit 68, based on the feature point extraction result and the corresponding point extraction result, the position coordinates (x, y) of each feature point 74 in the standard image and each correspondence in the reference image. A corresponding coordinate list 78 indicating the position coordinates (x, y) of the point 77 and the corresponding relationship between the feature point 74 and the corresponding point 77 is formed. Note that the x and y coordinates are coordinates when the origin is the pixel at the lower left corner of the standard or reference image, for example.

  Based on the corresponding coordinate list 78, the posture calculation unit 69 obtains the difference between the y-coordinates of the feature points 74 and the corresponding points 77 (hereinafter simply referred to as “feature / corresponding points 74 and 77”) corresponding to each other. The difference value of the y coordinate is a value representing the posture of the camera body 11. For example, when stereo shooting is performed in a posture in which the camera body 11 is tilted (hereinafter simply referred to as a tilt posture) so that the arrangement direction of the imaging units 12 and 13 has an inclination angle with respect to the horizontal line, In the standard and reference image data 75L and 75R, the y-coordinate values of the feature / corresponding points 74 and 77 are shifted. The deviation (difference value) of the y coordinate value increases as the tilt of the camera body 11 increases.

  The parallax calculation unit 70 obtains the parallax magnitudes of the features / corresponding points 74 and 77 based on the corresponding coordinate list 78. Specifically, the difference between the x coordinates of the features / corresponding points 74 and 77 is obtained as the magnitude of parallax (parallax amount).

  The posture deviation amount determination unit 71 determines the deviation amount between the posture of the camera body 11 during stereo shooting and the above-described reference posture based on the difference value of the y coordinate of each feature / corresponding point 74 and 77 and the amount of parallax (hereinafter referred to as the reference posture). , Referred to as the amount of posture deviation). The posture of the camera body 11 is represented by the difference value of the y-coordinate of each feature / corresponding point 74, 77. If there are a plurality of subjects from which each feature / corresponding point 74, 77 is extracted, the y-coordinate Variation occurs in the difference value.

  Specifically, even if the tilt angle of the camera body 11 is the same, it is extracted from the subject on the near side rather than the y coordinate difference value of the feature / corresponding points 74 and 77 extracted from the subject on the far side. The difference value between the y coordinates of the feature / corresponding points 74 and 77 thus made becomes larger. This is because the latter is a value that more accurately reflects the inclination of the posture of the camera body 11. Regarding which of the features / corresponding points 74 and 77 is extracted from the subject on the closer distance side, the feature / corresponding points 74 and 77 extracted from the subject on the closer distance side are more parallax. Since the value of becomes larger, it can be determined by comparing the amount of parallax.

  Therefore, the posture deviation amount determination unit 71 performs weighting processing according to the magnitude of the parallax amount on the difference value of the y-coordinates of the individual feature / corresponding points 74 and 77. Specifically, in order to give more importance to the y-coordinate difference value of the feature / corresponding points 74 and 77 having a large amount of parallax, the weighting coefficient having a larger value as the parallax amount of the feature / corresponding points 74 and 77 becomes larger. To weight the difference value of the y coordinate. Next, the posture deviation amount determination unit 71 obtains a posture deviation amount based on the weighted difference value of each y coordinate.

For example, when the amount of parallax is not taken into account, when the position of the base image of the feature point i is (x1i, y1i), the position of the reference image is (x2i, y2i), and the weighting coefficient is Ai, the posture deviation amount W is represented by the following formula (1).
Expression (1): Posture displacement amount W = {ΣAi × | y1i−y2i |} / ΣAi
This formula (1) is W = Σ | y1i−y2i | / n when weighting is not performed, and is a simple average of the difference values of the respective y coordinates.

Thus, since the amount of posture deviation when the amount of parallax is not taken into consideration is expressed as in equation (1), when the amount of parallax is taken into account, the amount of posture deviation W is calculated from the amount of parallax ∝ | x1i−x2i |. Is represented by the following formula (2). Here, Bi in Expression (2) is a weighting coefficient when the amount of parallax is considered.
Expression (2): Posture displacement amount W = {ΣBix × | x1i−x2i | × | y1i−y2i |} / {ΣBix × | x1i−x2i |}

  The allocating unit 72, based on the posture deviation amounts obtained from the reference and reference image data 75L and 75R, divides the reference and reference image data 75L and 75R into a plurality of images that are divided according to the magnitude of the posture deviation amount. Assign to one of the image groups. The photographed image group includes an OK image group, a trimming process group (first photographed image group), and an NG image group. In the distribution unit 72, a threshold value 1 and a threshold value 2 that is larger than the threshold value 1 are registered in advance in order to perform the distribution process.

  The threshold value 1 is a maximum value of the amount of posture deviation from which the reference image data 75L and 75R with which there is no problem in stereoscopic vision is obtained. The threshold value 2 is the maximum value of the posture deviation amount from which the reference and reference image data 75L and 75R can be obtained by which the stereoscopic view can be obtained by the trimming process even though the stereoscopic view is not suitable as it is. When the amount of posture deviation exceeds the threshold value 2, the camera body 11 has a large inclination, and even if image processing such as trimming processing is performed on the reference and reference image data 75L and 75R, there is a problem with stereoscopic viewing. Arise.

  As shown in FIG. 6, the distribution unit 72 distributes the reference and reference image data 75 </ b> L and 75 </ b> R to the OK image group when the posture deviation amount is equal to or less than the threshold value 1. Further, the distribution unit 72 distributes the reference and reference image data 75L and 75R to the trimming process group when the amount of posture deviation exceeds the threshold value 1 but is equal to or less than the threshold value 2. Further, the distribution unit 72 distributes the standard and reference image data 75L and 75R to the NG image group when the posture deviation amount exceeds the threshold value 2.

  Returning to FIG. 4, the reference and reference image data 75 </ b> L and 75 </ b> R assigned to the OK image group are stored in the OK image storage unit 20 a in the memory card 20. The reference and reference image data 75L and 75R distributed to the trimming processing group are sent to the trimming processing circuit 59. Further, the standard and reference image data 75L and 75R (hereinafter referred to as “NG image data” as appropriate) distributed to the NG image group are stored or discarded in the NG image storage unit 20b in the memory card 20. Whether or not to discard the NG image data is set by the operation unit 19.

  The reference and reference image data 75L and 75R stored in the memory card 20 are reproduced by a stereoscopic image display device (not shown).

  Next, the flow of processing when performing stereo shooting with the compound-eye camera 10 having the above-described configuration will be described using the flowchart shown in FIG. When the power of the compound eye camera 10 is turned on, the CPU 33 loads a control program from the ROM and starts operation control of the compound eye camera 10. Next, when performing stereo shooting, the mode switch 22 is set to the still image shooting mode.

  When the still image shooting mode is set, the subject light incident through the shooting lenses 37 of the imaging units 12 and 13 is photoelectrically converted by the CCDs 39 and 48, and further converted into digital image signals by the AFEs 40 and 49. . The reference and reference image signals output from the imaging units 12 and 13 are sent to the signal processing circuit 56 via the image input controller 45 and subjected to various image processing. Thus, the standard and reference image data for one frame are sequentially stored in the VRAM 35 at a predetermined frame rate. The CPU 33 issues a through image display command to the display circuit 62.

  The display circuit 62 receives the through image display command, reads the reference image data 75L from the VRAM 35, and causes the LCD 18 to display the reference image as a through image. When the shutter button 15 is half-pressed after the framing of the subject, the AF detection circuit 57 and the AE / AWB detection circuit 58 are operated to perform shooting preparation processing such as focus control and exposure control, and the camera is ready for shooting.

  After the preparation for photographing is completed, stereo photographing is executed in response to the full pressing operation of the shutter button 15. Image signals for one frame are read out from the CCDs 39 and 48, respectively, are subjected to predetermined signal processing by the AFEs 40 and 49 and the signal processing circuit 56, and the reference and reference image data 75L and 75R are stored in the VRAM 35.

  The storage control unit 64 of the CPU 33 reads the standard and reference image data 75L and 75R from the VRAM 35 every time new standard and reference image data 75L and 75R are stored in the VRAM 35 by the signal processing circuit 56. The feature point extraction unit 66 analyzes the reference image data 75L to detect the face region 76, and then extracts a plurality of feature points 74 from the face region 76. Further, the corresponding point extraction unit 67 extracts corresponding points 77 corresponding to the feature points 74 from the reference image data 75R. The feature point extraction unit 66 and the corresponding point extraction unit 67 send the extraction results to the position information acquisition unit 68, respectively.

  The position information acquisition unit 68 obtains the position coordinates of each feature point 74 and each corresponding point 77 in the reference and reference images based on the extraction results input from the feature point extraction unit 66 and the corresponding point extraction unit 67, respectively. A corresponding coordinate list 78 is created. The position information acquisition unit 68 sends the created corresponding coordinate list 78 to the posture calculation unit 69 and the parallax calculation unit 70, respectively.

  The posture calculation unit 69 obtains the difference between the y coordinate values of the features / corresponding points 74 and 77 in the standard and reference images based on the corresponding coordinate list 78. As shown in FIG. 8, in the reference and reference image data 75L and 75R taken when the camera body 11 is in the reference posture, the x coordinate values (x1, x2) of the features / corresponding points 74 and 77 due to the influence of parallax. While the deviation occurs in x2), the y coordinate values (y1, y2) are almost the same value.

  On the other hand, as shown in FIG. 9, in the reference and reference image data 75L and 75R taken when the camera body 11 is in the tilted posture, the y-coordinate values of the feature / corresponding points 74 and 77 are also shifted ( Δy) occurs. Therefore, it is possible to determine whether the posture of the camera body is the reference posture or the tilt posture based on the difference value of the y coordinate without providing the posture detection sensor in the camera body 11. Further, in the latter case, the magnitude of the inclination can also be determined based on the difference value of the y coordinate. Since the posture detection sensor becomes unnecessary and the number of parts can be reduced, the manufacturing cost of the compound eye camera 10 can be reduced. The posture calculation unit 69 sends the difference calculation result of the y coordinate values of the features / corresponding points 74 and 77 to the posture deviation amount determination unit 71.

  Returning to FIG. 7, the parallax calculation unit 70 obtains the parallax amount (difference value of the x coordinate) of each feature / corresponding point 74 and 77 in the standard and reference images based on the corresponding coordinate list 78, and the obtained parallax The amount calculation result is sent to the posture deviation amount determination unit 71.

  The posture deviation amount determination unit 71 first compares the parallax amount of each feature / corresponding point 77 based on the parallax calculation result from the parallax calculation unit 70. Then, the weighting coefficient to be assigned to each feature / corresponding point 74, 77 is determined so that the weighting coefficient having a larger value is assigned to the feature / corresponding point 74, 77 having a larger amount of parallax.

  Next, the posture deviation amount determination unit 71 weights the difference value of the y coordinate of each feature / corresponding point 74, 77 with the weighting coefficient determined in advance, and then the difference of each weighted y coordinate. The amount of posture deviation is obtained based on the value. When the posture amount is determined based on the difference value of the y coordinates of one set of features / corresponding points by determining the amount of posture deviation based on the difference value of the y coordinates of the plurality of sets of features / corresponding points 74 and 77; In comparison, the amount of posture deviation can be obtained with high accuracy.

  Further, by weighting the difference value of each y coordinate according to the magnitude of the parallax amount, the difference value of the y coordinate of the feature / corresponding points 74 and 77 extracted from the subject on the short distance side, that is, the camera body Since the difference value of the y coordinate that more accurately reflects the inclination of 11 postures is emphasized, the amount of posture deviation can be obtained more accurately. The posture deviation amount determination unit 71 sends the obtained posture deviation amount to the distribution unit 72.

  The allocating unit 72 compares the posture deviation amount input from the posture deviation amount determining unit 71 with the threshold value 1 and the threshold value 2. The allocating unit 72 allocates the reference and reference image data 75L and 75R to the OK image group when the posture deviation amount is equal to or less than the threshold value 1. The standard and reference image data 75L and 75R assigned to the OK image group are compressed in a predetermined file format by the compression / decompression processing circuit 60 and then stored in the OK image storage unit 20a by the media controller 61.

  The distribution unit 72 distributes the reference and reference image data 75L and 75R to the trimming processing group when the amount of posture deviation exceeds the threshold value 1 but is equal to or less than the threshold value 2. The reference and reference image data 75L and 75R distributed to the trimming processing group are sent to the trimming processing circuit 59.

  As shown in FIG. 10A, the CPU 33 shoots the reference and reference images overlapping each other based on the difference value of the y coordinate of each feature / corresponding point 74 and 77 previously calculated by the posture calculation unit 69. The displayed range (the range not shaded) is determined as the trimming range. Next, the CPU 33 sends the determined trimming range information to the trimming processing circuit 59 and issues a trimming command to the trimming processing circuit 59.

  As shown in (B), the trimming processing circuit 59 receives the trimming command and cuts out the trimming ranges designated by the CPU 33 from the reference and reference image data 75L and 75R, respectively, and the reference and reference cut-out image data 80L and 80R. Is generated. In the reference and reference cut-out image data 80L and 80R, the difference value between the y-coordinates of the feature / corresponding points 74 and 77 is greatly reduced as compared to before the trimming process. This is the same as that corrected so that the posture deviation amount is equal to or less than the threshold value 1, so that stereoscopic viewing is possible. As a result, even when stereo shooting is performed when the camera body 11 is in the tilted posture, an image having no problem in stereoscopic vision can be obtained.

  Returning to FIG. 7, the reference and reference cut-out image data 80 </ b> L and 80 </ b> R are compressed by the compression / decompression processing circuit 60 and then stored in the OK image storage unit 20 a by the media controller 61.

  The distribution unit 72 distributes the reference and reference image data 75L and 75R to the NG image group when the posture deviation amount exceeds the threshold value 2. When setting is made to leave NG image data in the operation unit 19, the NG image data is stored in the NG image storage unit 20b after being subjected to compression processing. Further, when the operation unit 19 is set to discard the NG image data, the NG image data is discarded.

  Hereinafter, while the still image shooting mode is set, the above-described series of processing is repeatedly executed every time the shutter button 15 is fully pressed. Of the standard and reference image data obtained by stereo shooting, those that are not defective in stereoscopic vision are sequentially stored in the OK image storage unit 20a, and those that are defective are stored or discarded in the NG image storage unit 20b. . Since it is possible to sort and store images that are not defective in stereoscopic vision and images that are defective, it is easy to select standards and reference images that are not defective in stereoscopic vision from the standards and reference image data stored in the memory card 20. Can be determined.

  In addition, since NG image data having a defect in stereoscopic vision can be easily identified, additional information indicating that the image is an NG image is attached to the tag of the NG image data, and another stereoscopic image display device or the like is used. A warning indicating that the image data is NG image data may be made when performing stereoscopic display.

  Next, the compound eye camera of 2nd Embodiment of this invention is demonstrated. The compound eye camera of the second embodiment is basically the same as the compound eye camera 10 of the first embodiment except that the operation mode has a continuous shooting mode in which a plurality of times of shooting are continuously performed. Components having the same configuration and the same function and configuration as those of the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  Further, in the compound eye camera of the second embodiment, in the continuous shooting mode, the normal distribution processing for distributing the individual standards and the reference image data 75L and 75R obtained by the continuous shooting to each of the above-described groups, respectively. A special distribution process for distributing the reference image data 75L and 75R with the smallest amount of posture deviation to the OK image group (corresponding to the second captured image group) and the other to the NG image group is performed selectively. Can do. Note that which of the normal distribution process and the special distribution process is performed is set by the operation unit 19.

  As shown in FIG. 11, when the mode switch 22 is set to the continuous shooting mode, when the shutter button 15 is fully pressed, the above-described shooting operation is continuously performed at regular intervals, and the obtained reference is obtained. The reference image data 75L and 75R are sequentially stored in the VRAM 35. At this time, when the normal distribution process is set in the operation unit 19, the reference and reference image data 75L and 75R newly stored in the VRAM 35 are sequentially read out by the storage control unit 64, and the first implementation described above. As described in the embodiment, determination processing for the amount of posture deviation (feature point extraction processing and the like are not shown), sorting processing, trimming processing, storage or discarding processing are sequentially executed.

  Also, as shown in FIG. 12, when the special distribution processing is set, the reference and new reference image data 75L and 75R newly stored in the VRAM 35 are sequentially read out, and the posture deviation amount determination processing is performed. After being performed, the value of the posture deviation amount is added to the tags of the standard and reference image data 75L and 75R as supplementary information. The added standard and reference image data 75L and 75R are temporarily stored in the SDRAM 34 (or the VRAM 35). Thereafter, while the shutter button 15 is fully pressed, the above-described processing is repeatedly executed, and the standard and reference image data 75L and 75R are sequentially stored in the SDRAM 34.

  When the shutter button 15 is fully pressed, the allocating unit 72 refers to the reference and reference image data 75L and 75R tags stored in the SDRAM 34, and the reference and reference image with the smallest amount of posture deviation. Data 75L and 75R are searched. Next, the distribution unit 72 distributes the retrieved standard and reference image data 75L and 75R into OK image groups. The standard and reference image data 75L and 75R are compressed and stored in the OK image storage unit 20a.

  Further, the distribution unit 72 distributes the remaining reference and reference image data 75L and 75R stored in the SDRAM 34 to NG image groups. These standard and reference image data 75L and 75R are all discarded. As a result, the standard and reference image data 75L and 75R having the smallest posture deviation amount, that is, the standard and reference image data 75L and 75R that are most suitable for stereoscopic viewing can be selected and stored. As a result, the amount of image data stored in the memory card 20 can be reduced. In addition, instead of discarding the standard and reference image data 75L and 75R distributed to the NG image group, it may be stored in the NG image storage unit 20b as in the first embodiment.

  In the above embodiment, the case where the subject is a person has been described as an example. However, for example, it may be a building. In particular, an image of a building (including a building or an artificial object) has an advantage that feature points 74 can be easily extracted because there are many corners and end points that have pixel value changes in the horizontal and vertical directions. In this case, the feature point extraction unit 66 described above has a building area detection function in addition to the face area detection function.

  As illustrated in FIG. 13, the feature point extraction unit 66 detects a building C from the reference image data 75 </ b> L, and sets a peripheral region of the detected building C as a building region 83. The feature point extraction unit 66 detects the building C, for example, by performing pattern matching processing using template data of various building patterns registered in advance (including patterns of each part of the building). Various known methods other than pattern matching may be used. As a method of extracting the feature points 74 from the building area 83, various feature point extraction methods described in the first embodiment are used. The method for extracting the corresponding points 77 from the reference image data 75R is also the same as that in the first embodiment, and the description thereof is omitted.

  In the above embodiment, a person or a building has been described as an example of the subject. However, the present invention can also be applied to cases where various objects other than these are subjects.

  In the above embodiment, framing is performed while confirming the through image displayed on the LCD 18, but at this time, a framing guide display may be performed on the through image. For example, as shown in FIG. 14, a horizontal reference line 86 and a vertical reference line 87 perpendicular to the horizontal reference line 86 are superimposed on the through image. When the compound eye camera 10 is in an inclined posture, the horizontal reference line 86 and the vertical reference line 87 have an inclination angle on the screen of the LCD 18 with respect to the horizontal edge and the vertical edge of the subject, respectively. For this reason, it is possible to prompt the photographer to adjust the posture of the camera body 11.

  In addition, the method of the guide display for framing is not specifically limited, For example, you may display a grid line. Further, for example, the value of the posture shift amount obtained by the posture shift amount determination unit 71 described above may be superimposed and displayed on the through image.

  In the above embodiment, the posture deviation amount is obtained after performing the weighting process corresponding to the magnitude of the parallax amount with respect to the difference value of the y coordinate of each feature / corresponding point 74, 77. The feature / corresponding points 74 and 77 having the largest amount, that is, the difference value between the y coordinates of the feature / corresponding points 74 and 77 extracted from the closest subject may be used as the posture deviation amount. In this case, the determination process of the amount of posture deviation can be performed more easily (short time) than when the weighting process is performed.

  In the above embodiment, the feature points 74 and corresponding points 77 of all the persons or buildings in the reference image and the reference image are extracted. For example, the person or building in the center area of the image is determined as the main subject, and its peripheral area And the feature point 74 and the corresponding point 77 may be extracted from the main subject region.

  The position information acquisition unit 68 of the above embodiment acquires the position coordinates (x, y) of each feature point 74 and each corresponding point 77 in the standard and reference images. For example, the feature point 74 in the CCDs 39 and 48 is obtained. In addition, the pixel position of the light receiving pixel that receives the light corresponding to each of the corresponding points 77 may be acquired as the position coordinates (x, y).

  The posture calculation unit 69 of the above embodiment calculates the difference value of the y coordinates of the feature / corresponding points 74 and 77 as a value representing the posture of the camera body 11. The method is not particularly limited as long as the posture of the camera body 11 can be obtained based on the above.

  In the above embodiment, when the posture deviation amount exceeds the threshold value 1 but is equal to or less than the threshold value 2, the reference and reference image data 75L and 75R are trimmed, but various image processing methods other than the trimming process are performed. May be used to generate the reference and reference image data in which the amount of posture deviation is equal to or less than the threshold value 1. Further, when the posture deviation amount exceeds the threshold value 1, the posture of the subject in the standard and reference images is also tilted (see FIG. 9). For this reason, an inclination correction process (rotation correction process) may be performed on the reference and reference images (the subject in the reference and reference images) based on the amount of posture deviation.

  The compound eye camera of the above embodiment does not have a stereoscopic image table function, but may include a stereoscopic image display device of various display methods such as a lenticular method, a parallax barrier method, a parallax barrier method, and an anaglyph method.

  In the above embodiment, the processing flow in the still image shooting mode or the continuous shooting mode has been described. However, for example, the processing in the moving image shooting mode is basically the processing in the continuous shooting mode (normal distribution processing). The flow is the same.

  In the above embodiment, a compound eye camera having a pair of imaging units 12 and 13 has been described as an example. However, the present invention can also be applied to a compound eye camera having three or more imaging units.

  In the above embodiment, the compound eye camera 10 that performs stereo shooting has been described as an example. However, the present invention can also be applied to a compound eye camera that has a panorama shooting mode that generates a wide-angle panorama image. In this case, the compound eye camera may be provided with an image synthesis circuit that synthesizes the captured images obtained by the imaging units to generate a panoramic image. Furthermore, in this case, a posture in which the arrangement direction of the imaging units is substantially perpendicular to the horizontal line may be defined as the reference posture.

  In the panoramic shooting mode, the threshold value 1 in the above embodiment is the maximum value of the posture deviation amount that can obtain a panoramic image with a smooth joint when the reference and reference image data 75L and 75 are combined. Become. Further, the threshold value 2 is a maximum value of the amount of posture deviation so that a panoramic image has a problem if it is left as it is, but a panoramic image having a smooth joint can be obtained by the trimming process.

DESCRIPTION OF SYMBOLS 10 Compound eye camera 11 Camera main body 12,13 1st, 2nd imaging part 20 Memory card 33 CPU
59 Trimming processing circuit 64 Storage control unit 66 Feature point extraction unit 67 Corresponding point extraction unit 68 Position information extraction unit 69 Posture calculation unit 70 Parallax calculation unit 72 Distribution unit 74 Feature point 77 Corresponding point

Claims (14)

A plurality of imaging means that are integrated into the camera body and shoot a subject;
Corresponding point extraction means for extracting corresponding points indicating common characteristic points existing in the respective photographed images from the photographed images obtained by the plurality of imaging means;
Posture detection means for detecting the posture of the camera body at the time of shooting each of the captured images by comparing the positions of the corresponding points corresponding to each other in each of the captured images;
A compound eye photographing apparatus comprising: Based on the detection result of the posture detection means, a deviation amount between a predetermined reference posture of the camera body and the posture of the camera body at the time of shooting is obtained, and each captured image is determined according to the magnitude of the deviation amount. Allocating means for allocating to any of a plurality of photographed image groups divided according to the magnitude of the deviation amount;
2. The compound eye photographing apparatus according to claim 1, further comprising image storage means for storing each of the photographed images allocated to a pre-selected photographed image group from the plurality of photographed image groups. The reference posture is a posture of the camera body in which an arrangement direction of the plurality of imaging units is substantially parallel to a horizontal line,
The compound eye photographing apparatus according to claim 2, wherein the posture detecting means detects a posture of the camera body when the camera body is tilted so that the arrangement direction has an inclination angle with respect to a horizontal line.   The compound eye according to claim 3, wherein the posture detection unit detects the posture of the camera body by comparing the vertical positions of the corresponding points corresponding to each other in the captured images. Shooting device. The corresponding point extracting means extracts a plurality of the corresponding points from each captured image,
5. The compound eye photographing apparatus according to claim 1, wherein the posture detection unit detects the posture of the camera body by comparing positions of the corresponding points corresponding to each other. 6. Based on the extraction result of the corresponding point extracting means, comprising a parallax calculating means for calculating the parallax between the corresponding points corresponding to each other;
The posture detection unit weights the comparison results of the positions of the corresponding points corresponding to each other according to the magnitude of the parallax, and based on the respective comparison results after weighting, the posture of the camera body The compound eye photographing apparatus according to claim 5, wherein: The corresponding point extracting means extracts a plurality of the corresponding points from each captured image,
Based on the extraction result of the corresponding point extracting means, comprising a parallax calculating means for calculating the parallax between the corresponding points corresponding to each other;
The posture detecting means detects the posture of the camera body by comparing the positions of the corresponding points having the largest parallax among the corresponding points corresponding to each other. 5. The compound eye photographing apparatus according to any one of claims 4 to 4.   2. The image storage unit stores each of the captured images allocated to the captured image group such that the magnitude of the shift amount is equal to or less than a predetermined first threshold value. The compound eye photographing device according to any one of Items 7 to 7. Each of the plurality of captured image groups is assigned with each of the captured images in which the shift amount is greater than the first threshold and less than or equal to a second threshold greater than the first threshold. Contains a group of captured images,
9. The image processing unit according to claim 8, further comprising an image processing unit configured to perform image processing on each of the captured images distributed to the first captured image group so that the shift amount is equal to or less than the first threshold value. The compound eye photographing apparatus described.   10. The compound eye photographing apparatus according to claim 9, wherein the image processing means performs a trimming process for cutting out an image range that is photographed overlapping each other from the photographed images. It has a continuous shooting mode that performs multiple shots continuously,
The plurality of photographed image groups include a second photographed image group to which the photographed images having the smallest deviation amount are distributed among the photographed images obtained in the continuous photographing mode. ,
8. The compound eye photographing apparatus according to claim 1, wherein the image storage unit stores each of the photographed images allocated to the second photographed image group. 9.   12. The corresponding point extraction unit, when the subject is a person, detects a person's face from each captured image, and extracts the corresponding point from each detected face image. A compound eye photographing apparatus according to claim 1.   12. The corresponding point extracting unit, when the subject is a building, extracts a building from each captured image, and extracts the corresponding point from each detected building image. The compound eye photographing device described in the item. A shooting step of shooting a subject by a plurality of imaging units integrated into the camera body;
A corresponding point extracting step for extracting corresponding points indicating common characteristic points existing in the respective captured images from the captured images obtained in the capturing step;
A posture detection step of detecting the posture of the camera body at the time of shooting each of the captured images by comparing the positions of the corresponding points corresponding to each other in each of the captured images;
A posture detection method for a compound eye photographing apparatus, comprising:

Images