JP2013098965A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus which, when a photographer takes an image of a subject, prevents the photographer from being invisible from a subject side due to the imaging apparatus.SOLUTION: An imaging apparatus 10 comprises: a pupil position detector 18 which detects the positions of the pupils of a first subject in three-dimensional coordinates corresponding to an actual space on the basis of an image taken by an imaging unit 17; a shape detector 20 for detecting the shape information of a second subject in the three-dimensional coordinates; and a display image generator 13 which on the basis of information on an image taken by an imaging unit 19, the positions of the pupils of the first subject, the shape information of the second subject, and the sizes of the image display surface P1 of a display 14 stored in a storage unit 22, generates an image to be displayed on the display 14 including the second subject projected on the image display surface P1 of the display 14 when the second subject is viewed from the positions of the pupils of the first subject.

Description

  The present invention relates to an imaging device including a plurality of imaging units.

  2. Description of the Related Art In recent years, an imaging apparatus such as a digital camera has been proposed that includes a plurality of imaging units and can photograph not only a subject but also a photographer himself (see, for example, Patent Document 1).

  By the way, when the photographer takes a picture with the imaging device near his / her face, the photographer's face may be hidden by the imaging device when viewed from the subject. Further, when the image display surface is relatively large and the entire imaging apparatus is large, the photographer's face may be further hidden by the imaging apparatus when viewed from the subject. For example, when a mother photographs a baby, if the baby cannot see the mother's face, the baby's sense of relief may fade and the baby's facial expression may be stiff, making it difficult to photograph the baby's smile.

JP 2004-147046 A

  The present invention prevents the photographer from obscuring the photographer from the subject side when the photographer photographs the subject.

  The imaging apparatus of the present invention is configured to determine a pupil position of the first subject in three-dimensional coordinates corresponding to a real space based on a first imaging unit that images the first subject and an image captured by the first imaging unit. First pupil position detecting means for detecting, and second imaging for imaging a second subject located in the back direction when the imaging direction of the first imaging means is the front direction and the direction opposite to the front direction is the back direction. Means, first shape detecting means for detecting shape information of the second subject in the three-dimensional coordinates, first display means arranged so that a display direction for displaying an image is the front direction, First storage means for storing the size of the image display surface of the first display means, information on an image captured by the second imaging means, and the first subject detected by the first pupil position detection means The pupil position of the From the pupil position of the first subject based on the shape information of the second subject detected by the one shape detection means and the size of the image display surface of the first display means stored in the first storage means. First display image generation means for generating an image to be displayed on the first display means including the second subject projected on the image display surface of the first display means when the second subject is viewed; Prepare.

  According to the present invention, when shooting the first subject, the second subject being the photographer, the projected image of the second subject is displayed on the first display means, so that the second subject from the first subject side is displayed. This can be prevented from being invisible by the imaging apparatus.

It is a figure which shows the imaging device of 1st Embodiment of this invention. It is a figure which shows the usage example of the imaging device of 1st Embodiment. It is a figure which shows an example of a projection point. It is a flowchart which shows an example of operation | movement of a display image generation part. It is a figure which shows the modification (the 1) of the imaging device of 1st Embodiment. It is a figure which shows the relationship between point Pb1, Pb2, and distance L1. It is a figure which shows the modification (the 2) of the imaging device of 1st Embodiment. It is a figure which shows an example of one camera and the other camera of an imaging part. It is a figure which shows an example of the image imaged with one camera. It is a figure which shows an example of the image imaged with the other camera. It is a figure which shows an example of the image after correction | amendment. It is a figure which shows the modification (the 3) of the imaging device of 1st Embodiment. It is a figure which shows the imaging device of 2nd Embodiment of this invention. It is a figure which shows the usage example of the imaging device of 2nd Embodiment. It is a figure which shows an example of a projection point. It is a figure which shows the modification (the 1) of the imaging device of 2nd Embodiment. It is a figure which shows the modification (the 2) of the imaging device of 2nd Embodiment. It is a figure which shows the modification (the 3) of the imaging device of 2nd Embodiment.

<First Embodiment>

  FIG. 1 is a diagram illustrating an imaging apparatus according to a first embodiment of the present invention.

  An imaging apparatus 10 illustrated in FIG. 1 is, for example, a smartphone or a tablet PC, and includes a first subject information detection unit 11, a second subject information detection unit 12, a display image generation unit 13, a display unit 14, and the like. A recording determination unit 15 and a recording unit 16 are provided.

  The first subject information detection unit 11 includes an imaging unit 17 and a pupil position detection unit 18.

  The second subject information detection unit 12 includes an imaging unit 19 and a shape detection unit 20.

  The display image generation unit 13 includes a projection image generation unit 21 and a storage unit 22.

  In addition, as shown in FIG. 2, let the imaging direction of the imaging part 17 be a front direction. Moreover, let the direction opposite to the front direction be a back direction.

  The recording unit 16 is configured by, for example, a hard disk, a CD (Compact Disc), a DVD (Digital Versatile Disc), a memory card, or a USB memory.

  The storage unit 22 includes, for example, a ROM (Read Only Memory), a RAM (Random Access Memory), a nonvolatile memory, or a flash memory. The storage unit 22 may be provided outside the display image generation unit 13.

  Further, the first subject information detection unit 11, the second subject information detection unit 12, the display image generation unit 13, and the like execute a program stored in a memory (not shown) by a DSP (Digital Signal Processor), for example. Alternatively, it may be realized using an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or the like.

  The imaging unit 17 images the first subject located in the front direction, and the imaging unit 19 images the second subject located in the back direction.

  Each of the imaging units 17 and 19 converts, for example, a subject image formed on an imaging element such as a CCD (Charge Coupled Device) by a lens into an electrical signal in the imaging element, and sets the image data. In contrast, various image processes are performed using a sharpness correction circuit, a contrast correction circuit, a luminance / color difference signal generation circuit, a white balance correction circuit, and the like.

  The pupil position detector 18 detects the pupil position of the first subject in the three-dimensional coordinates corresponding to the real space. For example, when the imaging unit 17 is configured with two cameras and these cameras are arranged at different positions in the imaging device 10 on the front direction side, the pupil position detection unit 18 captures images with the two cameras of the imaging unit 17. Using each image, the pupil position of the first subject in the three-dimensional coordinates corresponding to the real space is obtained by the principle of triangulation or the stereo matching method. Note that the imaging unit 17 may be composed of three or more cameras in order to increase the detection accuracy of the pupil position of the first subject in the three-dimensional coordinates corresponding to the real space.

  The shape detection unit 20 detects shape information of the second subject in the three-dimensional coordinates corresponding to the real space (all points constituting the second subject in the three-dimensional coordinates corresponding to the real space). For example, when the imaging unit 19 is configured by two cameras and these cameras are arranged at different positions in the imaging device 10 on the back side, the shape detection unit 20 is captured by the two cameras of the imaging unit 19. Using the respective images, all points constituting the second subject in the three-dimensional coordinates corresponding to the real space are obtained by the principle of triangulation or the stereo matching method. Note that the imaging unit 19 may be configured with three or more cameras in order to increase the detection accuracy of the shape information of the second subject in the three-dimensional coordinates corresponding to the real space. Further, the method of detecting the shape information of the second subject in the three-dimensional coordinates corresponding to the real space may employ an infrared pattern projection method or the like in addition to the method described above, and is not particularly limited.

  The projection image generation unit 21 includes information on an image captured by the imaging unit 19 (for example, luminance and color of each pixel), a pupil position of the first subject detected by the pupil position detection unit 18, and a shape detection unit. 20 based on the shape information of the second subject detected at 20 and the size of the image display surface P1 of the display unit 14 in the three-dimensional coordinates corresponding to the real space stored in the storage unit 22. A projected image displayed on the display unit 14 is generated including the second subject projected on the image display surface P1 of the display unit 14 when the second subject is viewed from the position. For example, as shown in FIG. 3, the pupil position PA (for example, the first subject) in three-dimensional coordinates (X, Y, Z) corresponding to the real space and having the origin at the center point of the image display surface P1 of the display unit 14 The intermediate point of the left and right pupils is (Xpa, Ypa, Zpa), and any point among all the points constituting the second subject in the three-dimensional coordinates (X, Y, Z) is (Xb1, Yb1, Zb1). ) And a projection point on the image display surface P1 of the display unit 14 at the three-dimensional coordinates (X, Y, Z) (intersection of the straight line l1 connecting the pupil position PA to an arbitrary point and the image display surface P1) ( Xb2, Yb2, 0). In this case, the projection image generation unit 21 sets projection points respectively corresponding to all points constituting the second subject to Xb2 = (− Zb1 × (Xpa−Xb1) / Zpa−Zb1) + Xb1 and Yb2 = ( It is obtained by calculating −Zb1 × (Ypa−Yb1) / Zpa−Zb1) + Yb1. Then, the projection image generation unit 21 displays the second subject when viewing the second subject from the pupil position of the first subject based on the obtained projection points and information on the image captured by the imaging unit 19. A projection image including the second subject projected on the image display surface P1 is generated. Then, the display unit 14 displays the projection image generated by the projection image generation unit 21.

  When an image recording instruction is input by operating the first subject or the like, the recording determination unit 15 records an image captured by the imaging unit 17 at that time in the recording unit 16.

  The recording determination unit 15 is configured to record an image captured by the imaging unit 17 at that time in the recording unit 16 when the display unit 14 configured by a touch panel display is touched by the first subject or the like. May be.

  FIG. 4 is a flowchart illustrating an example of the operation of the display image generation unit 13.

  First, when the display image generation unit 13 acquires information on the image captured by the imaging unit 19, the pupil position of the first subject, and the shape information of the second subject (S1 is Yes), all of the constituents of the second subject are obtained. Projection points respectively corresponding to these points are calculated (S2, S3).

  Then, the display image generation unit 13 generates a projection image based on the information of the image captured by the imaging unit 19 and the projection points respectively corresponding to all the points constituting the second subject (S4), After the generated projection image is sent to the display unit 14 (S5), the process returns to S1 again.

As described above, the imaging apparatus 10 according to the first embodiment illustrated in FIG. 1 includes the second subject projected onto the image display surface P1 of the display unit 14 when the second subject is viewed from the pupil position of the first subject. Therefore, even if the imaging device 10 and the second subject overlap each other when viewed from the first subject side, the second subject is seen through from the first subject side. Can be displayed on the image display surface P <b> 1 of the display unit 14. For this reason, when the mother photographs the baby, an image can be displayed on the display unit 14 as if the mother is seeing through the baby from the baby side. This makes it easier to capture the baby's smile.
<Modification of First Embodiment (Part 1)>

  FIG. 5 is a diagram illustrating a modification (No. 1) of the imaging device 10 according to the first embodiment. In addition, the same code | symbol is attached | subjected to the same structure as the structure shown in FIG.

  The imaging apparatus 10 shown in FIG. 5 is different from the imaging apparatus 10 shown in FIG. 1 in that a line-of-sight direction detection unit 23 is further provided. Note that the line-of-sight direction detection unit 23 may be realized, for example, by a DSP executing a program stored in a memory (not shown), or may be realized using an FPGA, an ASIC, or the like.

  The gaze direction detection unit 23 detects the gaze direction of the first subject. For example, the gaze direction detection unit 23 detects the gaze direction of the first subject based on the position of the black eye portion with respect to the white eye portion of the first subject in the image captured by the imaging unit 17.

  The recording determination unit 15 includes the pupil position of the second subject in the projection image displayed on the image display surface P1 of the display unit 14, the gaze direction of the first subject detected by the gaze direction detection unit 23, and the display unit 14. When the distance from the intersection with the image display surface P1 is equal to or smaller than the threshold, the image captured by the imaging unit 17 is recorded in the recording unit 16. For example, as shown in FIG. 6, the recording determination unit 15 includes the pupil position (midpoint between the left and right pupils) Pb1 of the second subject in the projection image displayed on the image display surface P1 of the display unit 14, and the line-of-sight direction. When the distance L between the line-of-sight direction of the first subject detected by the detection unit 23 and the intersection Pb2 between the image display surface P1 of the display unit 14 is equal to or less than the threshold value Lth, the image captured by the imaging unit 17 is stored in the recording unit 16. Let me record.

According to the imaging device 10 illustrated in FIG. 5, even when the face of the first subject is hidden behind the imaging device 10 when viewed from the second subject side, the face of the first subject looking at the camera can be recorded. Therefore, when the mother photographs the baby, the baby's face as viewed from the camera can be recorded even when the baby's face is hidden by the imaging device 10 when viewed from the mother's side.
In addition, according to the imaging device 10 illustrated in FIG. 5, it is possible to perform imaging when the first subject and the second subject are in line of sight.
<Modification of First Embodiment (Part 2)>

  FIG. 7 is a diagram illustrating a second modification of the imaging device 10 according to the first embodiment. In addition, the same code | symbol is attached | subjected to the same structure as the structure shown in FIG.

  The imaging apparatus 10 shown in FIG. 7 is different from the imaging apparatus 10 shown in FIG. 5 in that a correction determination unit 24 and a correction unit 25 are further provided.

  The correction determination unit 24 determines whether it is necessary to correct the image captured by the imaging unit 17.

  When the correction determination unit 24 determines that the image captured by the imaging unit 17 needs to be corrected, the correction unit 25 corrects the image.

  For example, as illustrated in FIG. 8A, one of the two cameras 17-1 and 17-2 constituting the imaging unit 17 is disposed at the upper end of the imaging device 10 on the front direction side. It is assumed that the other camera 17-2 is disposed at the lower end of the imaging device 10 on the front direction side. In this case, first, the correction determination unit 24 obtains the distance d1 between the pupil position PA of the first subject and the camera 17-1 based on the image captured by the camera 17-1, and is captured by the camera 17-2. The distance d2 between the pupil position PA of the second subject and the camera 17-2 is obtained based on the image obtained. Then, when the difference (absolute value) between the distances d1 and d2 is greater than or equal to the threshold value dth1, the correction determination unit 24 distorts the captured image of the image capturing unit 17 that is generated when the image capturing apparatus 10 is tilted with respect to the first subject. It is determined that correction is necessary. For example, when the distance d1 is longer than the distance d2 and the difference between the distances d1 and d2 is greater than or equal to the threshold value dth1, the cameras 17-1 and 17-2 each capture the first subject from the lower side. Then, the black eye portion of the first subject in the image picked up at that time is closer to the lower side than the white eye portion as shown in FIG. 8B. In addition, when the distance d2 is longer than the distance d1 and the difference between the distances d1 and d2 is equal to or greater than the threshold value dth1, the cameras 17-1 and 17-2 are respectively capturing the first subject from above. Thus, the black eye portion of the first subject in the image picked up at that time is closer to the upper side than the white eye portion. In these cases, as illustrated in FIG. 8D, the correction unit 25 corrects the image so that the black eye portion of the first subject is positioned at the center with respect to the white eye portion. Thus, based on the difference between the distances d1 and d2, the vertical distortion of the image captured by the imaging unit 17 can be corrected.

  Note that one of the two cameras constituting the imaging unit 17 is arranged at the right end of the imaging device 10 on the front direction side, and the other camera is arranged on the left end of the imaging device 10 on the front direction side. May be. In this case, the lateral distortion of the image captured by the imaging unit 17 can be corrected based on the difference between the distances d1 and d2.

  Further, when the correction determination unit 24 is configured by an acceleration sensor, a gyro, or the like, and the inclination θ of the imaging device 10 with respect to the ground is inclined more than the threshold θth, it is determined that the image captured by the imaging unit 17 needs to be corrected. May be. In this case, the correction unit 25 corrects the image captured by the imaging unit 17 based on the inclination θ of the imaging device 10.

According to the imaging device 10 illustrated in FIG. 7, even when the imaging device 10 is tilted with respect to the first subject, the recording unit 16 does not distort the face of the first subject in the image captured by the imaging unit 17. Can be recorded. Therefore, when the mother photographs the baby, even if the imaging device 10 is tilted with respect to the baby, the baby's face or the like in the image captured by the imaging unit 17 is recorded in the recording unit 16 without being distorted. Can do.
<Modification of First Embodiment (Part 3)>

  FIG. 9 is a diagram illustrating a modification (No. 3) of the imaging device 10 according to the first embodiment. In addition, the same code | symbol is attached | subjected to the same structure as the structure shown in FIG.

  The imaging apparatus 10 shown in FIG. 9 is different from the imaging apparatus 10 shown in FIG. 7 in that a photometric unit 26 is further provided.

  The photometry unit 26 measures the back side.

  The projection image generation unit 21 includes information on an image captured by the imaging unit 19, a pupil position of the first subject detected by the pupil position detection unit 18, and shape information of the second subject detected by the shape detection unit 20. Based on the size of the image display surface P1 of the display unit 14 in the three-dimensional coordinates stored in the storage unit 22 and the photometric result of the photometric unit 26, a projection image is generated.

According to the image pickup apparatus 10 shown in FIG. 9, since the luminance and color of the second subject can be accurately detected by the photometry unit 26, the second subject in the projection image can be reproduced with a more natural color. . Therefore, when the mother photographs the baby, the mother's brightness and color can be detected with high accuracy, so that the mother in the projected image can be reproduced with a more natural color.
Second Embodiment

  FIG. 10 is a diagram illustrating an imaging apparatus according to the second embodiment of the present invention. In addition, the same code | symbol is attached | subjected to the same structure as the imaging device 10 of 1st Embodiment shown in FIG.

  An imaging apparatus 100 illustrated in FIG. 10 includes a first subject information detection unit 11, a second subject information detection unit 12, a display image generation unit 13, a display unit 14, a recording determination unit 15, and a recording unit 16. A first subject information detection unit 51, a second subject information detection unit 52, a display image generation unit 53, a display unit 54, a recording determination unit 55, and a recording unit 56 are provided.

  The first subject information detection unit 11 includes an imaging unit 17 and a pupil position detection unit 18. The second subject information detection unit 12 includes an imaging unit 19 and a shape detection unit 20. The display image generation unit 13 includes a projection image generation unit 21 and a storage unit 22.

  The first subject information detection unit 51 includes an imaging unit 57 and a pupil position detection unit 58. The second subject information detection unit 52 includes an imaging unit 59 and a shape detection unit 60. The display image generation unit 53 includes a projection image generation unit 61 and a storage unit 62.

  As shown in FIG. 11, the imaging direction of the imaging unit 17 is the front direction. Moreover, let the direction opposite to the front direction be a back direction. In this case, the imaging direction of each of the imaging units 19 and 57 is the back direction, and the imaging direction of the imaging unit 59 is the front direction.

  Further, in the imaging apparatus 100 of the second embodiment shown in FIG. 10, the first subject information detection unit 11, the second subject information detection unit 12, the display image generation unit 13, the display unit 14, the recording determination unit 15, and the recording unit. Each operation of 16 includes a first subject information detection unit 11, a second subject information detection unit 12, a display image generation unit 13, a display unit 14, and a recording determination unit 15 of the imaging apparatus 10 according to the first embodiment shown in FIG. , And the operations of the recording unit 16 are the same.

  The imaging units 57 and 59 convert, for example, a subject image formed on an imaging element such as a CCD by a lens into an electrical signal in the imaging element, and a sharpness correction circuit for the image data. Various image processing is performed using a contrast correction circuit, a luminance / color difference signal generation circuit, a white balance correction circuit, and the like.

  In addition, the image display surfaces P1 and P2 of the display units 14 and 54 have substantially the same size.

  The pupil position detector 58 detects the pupil position of the second subject in the three-dimensional coordinates corresponding to the real space. For example, when the imaging unit 57 is composed of two cameras and these cameras are arranged at different positions in the imaging device 100 on the back side, the pupil position detection unit 58 captures images with the two cameras of the imaging unit 57. Using each image, the pupil position of the second subject in the three-dimensional coordinates corresponding to the real space is obtained by the principle of triangulation or the stereo matching method. Note that the imaging unit 57 may be configured with three or more cameras in order to increase the detection accuracy of the pupil position of the second subject in the three-dimensional coordinates corresponding to the real space.

  The shape detection unit 60 detects shape information of the first subject in the three-dimensional coordinates corresponding to the real space (all points corresponding to the first subject in the three-dimensional coordinates corresponding to the real space). For example, when the imaging unit 59 is configured by two cameras and these cameras are arranged at different positions in the imaging device 100 on the front direction side, the shape detection unit 60 is captured by the two cameras of the imaging unit 59. Using the respective images, all points constituting the first subject in the three-dimensional coordinates corresponding to the real space are obtained by the principle of triangulation or the stereo matching method. Note that the imaging unit 59 may be configured with three or more cameras in order to increase the detection accuracy of the shape information of the first subject in the three-dimensional coordinates corresponding to the real space. Further, the method for detecting the shape information of the first subject in the three-dimensional coordinates corresponding to the real space may employ an infrared pattern projection method in addition to the above-described method, and is not particularly limited.

  The projection image generation unit 61 includes information on an image captured by the imaging unit 59 (for example, luminance and color of each pixel), a pupil position of the second subject detected by the pupil position detection unit 58, and a shape detection unit. Based on the shape information of the first subject detected at 60 and the size of the image display surface P2 of the display unit 54 in the three-dimensional coordinates corresponding to the real space stored in the storage unit 62, the pupil of the second subject A projection image that is displayed on the display unit 54 is generated including the first subject that is projected on the image display surface P2 of the display unit 54 when the first subject is viewed from the position. For example, as shown in FIG. 12, the pupil position PB of the second subject in three-dimensional coordinates (X, Y, Z) corresponding to the real space and having the origin at the center point of the image display surface P2 of the display unit 54 (for example, An intermediate point between the left and right pupils) is (Xpb, Ypb, Zpb), and any point among all the points constituting the first subject in the three-dimensional coordinates (X, Y, Z) is (Xa1, Ya1, Za1). ) And a projection point on the image display surface P2 of the display unit 54 at the three-dimensional coordinates (X, Y, Z) (intersection of the straight line l2 connecting the pupil position PB to an arbitrary point and the image display surface P2) ( Xa2, Ya2, 0). In this case, the projection image generation unit 61 sets projection points respectively corresponding to all points constituting the first subject to Xa2 = (− Za1 × (Xpb−Xa1) / Zpb−Za1) + Xa1 and Ya2 = ( It is obtained by calculating −Za1 × (Ypb−Ya1) / Zpb−Za1) + Ya1. The projection image generation unit 61 then displays the display unit 54 when viewing the first subject from the pupil position of the second subject based on the obtained projection points and information on the image captured by the imaging unit 59. A projection image including the first subject projected on the image display surface P2 is generated. The display unit 54 displays the projection image generated by the projection image generation unit 61.

  When an image recording instruction is input by operating the second subject or the like, the recording determination unit 55 records an image captured by the imaging unit 57 at that time in the recording unit 56.

  The recording determination unit 55 is configured to record an image captured by the imaging unit 57 at that time in the recording unit 56 when the display unit 54 configured by a touch panel display is touched by the second subject or the like. May be.

  In addition, the recording determination unit 15 records characters and pictures written by touching the display unit 14 configured by a touch panel display with the first subject or the second subject on an image captured by the imaging unit 17. You may comprise so that it may memorize | store in the part 16. FIG. In addition, the recording determination unit 55 records characters and pictures written by touching the display unit 54 configured by a touch panel display with the first subject or the second subject on the image captured by the imaging unit 57. You may comprise so that it may memorize | store in the part 56. FIG.

  As described above, the imaging apparatus 100 according to the second embodiment includes the second subject that is projected on the image display surface P1 of the display unit 14 when the second subject is viewed from the pupil position of the first subject. A projection image displayed on the image display surface P1 is generated and displayed including the first subject projected on the image display surface P2 of the display unit 54 when the first subject is viewed from the pupil position of the second subject. This is a configuration for generating a projection image displayed on the unit 54. Therefore, even when the imaging device 100 and the second subject overlap each other when viewed from the first subject, an image as if the second subject is seen through from the first subject side is displayed on the image display surface P1 of the display unit 14. And the display unit displays an image as if the first subject is seen through from the second subject side even if the imaging device 100 and the first subject overlap when seen from the second subject side. 54 image display surfaces P2. Thereby, when the mother photographs the baby, an image as if the mother is seen through the baby from the baby side can be displayed on the display unit 14. The baby's smile can be easily captured without hiding it. In addition, since an image as if the baby is seen through from the mother's side can be displayed on the display unit 54, it is possible to take a picture while watching the state of the baby.

In the imaging apparatus 100 of the second embodiment, the imaging unit 59 may be the same as the imaging unit 17 and the imaging unit 57 may be the same as the imaging unit 19. In this case, the pupil position detection unit 58 detects the pupil position of the second subject in the three-dimensional coordinates corresponding to the real space based on the image captured by the imaging unit 19. Further, the shape detection unit 60 detects the shape information of the first subject in the three-dimensional coordinates corresponding to the real space based on the image captured by the imaging unit 17. In addition, the projection image generation unit 61 generates a projection image using information on an image captured by the imaging unit 17. As described above, by making the imaging unit 59 the same as the imaging unit 17 and making the imaging unit 57 the same as the imaging unit 19, it is possible to suppress an increase in size and cost of the entire imaging device 100.
<Modification of Second Embodiment (Part 1)>

  FIG. 13 is a diagram illustrating an imaging apparatus according to the third embodiment of the present invention. In addition, the same code | symbol is attached | subjected to the same structure as the imaging device 100 of 2nd Embodiment shown in FIG.

  The imaging device 130 of the third embodiment shown in FIG. 13 is different from the imaging device 100 of the second embodiment shown in FIG. 10 in that the first subject information detection unit 11, the first subject information detection unit 52, and the display image generation unit. 53, the display unit 14, the recording determination unit 15, and the recording unit 16 are provided in the housing 131, and the second subject information detection unit 12, the second subject information detection unit 51, the display image generation unit 13, the display unit 54, and the recording determination The portion 55 and the recording portion 56 are provided in the housing 132, and the housing 131 and the housing 132 are connected to each other via a coupler 133. Note that the housing 131 and the housing 132 are, for example, smartphones and tablet PCs, and are attachable to and detachable from the coupler 133. Further, the data transfer between the casings 131 and 132 may be wireless communication or wired communication.

  The projection image generation unit 21 acquires the pupil position of the first subject detected by the pupil position detection unit 18 from the housing 131 via the coupler 133.

  The display unit 14 acquires the projection image generated by the projection image generation unit 21 from the housing 132 via the coupler 133.

  The projection image generation unit 61 acquires the pupil position of the second subject detected by the pupil position detection unit 58 from the housing 132 via the coupler 133.

  The display unit 54 acquires the projection image generated by the projection image generation unit 61 from the housing 131 via the coupler 133.

The imaging apparatus 130 according to the third embodiment connects the casings 131 and 132 to each other by the coupler 133 even if no imaging unit is disposed on the rear side of the casings 131 and 132. The see-through image of the first subject can be displayed on the display unit 54 while the see-through image is displayed on the display unit 14.
<Modification of Second Embodiment (Part 2)>

  FIG. 14 is a diagram illustrating an imaging apparatus according to the fourth embodiment of the present invention. In addition, the same code | symbol is attached | subjected to the same structure as the imaging device 100 of 2nd Embodiment shown in FIG.

  The imaging apparatus 140 of the fourth embodiment shown in FIG. 14 is different from the imaging apparatus 100 of the second embodiment shown in FIG. 10 in that the line-of-sight direction detection units 141 and 142, the correction determination units 143 and 144, and the correction It is a point provided with the part 145,146. The operations of the correction determination unit 143 and the correction unit 145 are the same as the operations of the correction determination unit 24 and the correction unit 25 in the modified example (No. 2) of the first embodiment shown in FIG.

  The line-of-sight direction detection unit 141 detects the line-of-sight direction of the first subject based on the image captured by the imaging unit 17. For example, the gaze direction detection unit 141 detects the gaze direction of the first subject based on the position of the black eye portion with respect to the white eye portion of the first subject in the image captured by the imaging unit 17.

  The line-of-sight direction detection unit 142 detects the line-of-sight direction of the second subject based on the image captured by the imaging unit 57. For example, the gaze direction detection unit 142 detects the gaze direction of the second subject based on the position of the black eye portion with respect to the white eye portion of the second subject in the image captured by the imaging unit 57.

  The correction determination unit 144 determines whether it is necessary to correct the image captured by the imaging unit 57.

  When it is determined that the image captured by the imaging unit 57 needs to be corrected, the correction unit 146 corrects the image.

  For example, one of the two cameras constituting the imaging unit 57 is arranged at the upper end of the imaging device 100 on the rear side, and the other camera is placed on the lower end of the imaging device 100 on the rear side. Suppose it is placed. In this case, first, the correction determination unit 144 obtains the distance d3 between the second subject and one camera based on the image captured by one camera, and the second based on the image captured by the other camera. A distance d4 between the subject and the other camera is obtained. Then, when the difference (absolute value) between the distances d3 and d4 is equal to or greater than the threshold value dth2, the correction determination unit 144 determines that the image captured by the imaging unit 57 needs to be corrected. For example, when the distance d3 is longer than the distance d4 and the difference between the distances d3 and d4 is equal to or larger than the threshold value dth2, the two cameras are respectively capturing the second subject from the lower side, and then The black eye portion of the second subject in the image picked up at the end of the image is shifted downward with respect to the white eye portion. Further, when the distance d4 is longer than the distance d3 and the difference between the distances d3 and d4 is equal to or larger than the threshold value dth2, the two cameras are capturing the second subject from the upper side, and the imaging is performed at that time. The black-eye portion of the second subject in the image to be moved is closer to the upper side than the white-eye portion. In these cases, the correction unit 146 corrects the projection image so that the black eye portion of the second subject is positioned at the center with respect to the white eye portion. As described above, based on the difference between the distances d3 and d4, it is possible to correct the distortion in the vertical direction of the image captured by the imaging unit 57.

  Note that one of the two cameras constituting the imaging unit 57 is arranged at the right end of the imaging device 100 on the rear side, and the other camera is arranged on the left side of the imaging device 100 on the rear side. May be. In this case, the lateral distortion of the image captured by the imaging unit 57 can be corrected based on the difference between the distances d3 and d4.

  Further, the correction determination unit 144 may be configured by an acceleration sensor, a gyro, or the like, and it may be determined that the projection image needs to be corrected when the inclination θ of the imaging device 100 with respect to the ground is inclined by a threshold θth or more. In this case, the correction unit 146 corrects the image captured by the imaging unit 57 based on the inclination θ of the imaging device 100.

  In addition, the recording determination unit 15 determines that when the line-of-sight direction of the first subject detected by the line-of-sight direction detection unit 141 and the line-of-sight direction of the second subject detected by the line-of-sight direction detection unit 142 match or substantially match each other, Alternatively, the image captured by the imaging unit 17 may be recorded in the recording unit 16. In addition, when the line-of-sight direction of the first subject detected by the line-of-sight direction detection unit 141 and the line-of-sight direction of the second subject detected by the line-of-sight direction detection unit 142 match or substantially match, the recording determination unit 55 Alternatively, the image captured by the imaging unit 57 may be recorded in the recording unit 56.

  When the correction determining unit 143 determines that the correction needs to be corrected, the correcting unit 145 detects the gaze direction of the first subject detected by the gaze direction detecting unit 141 and the second detected by the gaze direction detecting unit 142. The image captured by the imaging unit 17 may be corrected based on the line-of-sight direction of the first subject when the line-of-sight direction of the subject matches or substantially matches. For example, when it is determined that the correction determination unit 143 needs to correct the correction, the correction unit 145 detects the first subject's gaze direction detected by the gaze direction detection unit 141 and the second gaze direction detection unit 142 detects the second gaze direction. The intersection between the line-of-sight direction of the first subject and the image display surface P1 of the display unit 14 when the line-of-sight direction of the subject coincides with or substantially coincides with the pupil position of the first subject in the image captured by the imaging unit 17 The images picked up by the image pickup unit 17 are corrected so that and become the same position.

  When the correction determining unit 144 determines that the correction needs to be corrected, the correcting unit 146 detects the first subject's gaze direction detected by the gaze direction detecting unit 141 and the second gaze direction detecting unit 142 detects the second gaze direction. The image captured by the imaging unit 57 may be corrected based on the line-of-sight direction of the second subject when the line-of-sight direction of the subject matches or substantially matches. For example, when the correction unit 146 determines that the correction needs to be corrected by the correction determination unit 144, the gaze direction of the first subject detected by the gaze direction detection unit 141 and the second gaze direction detection unit 142 detects the second gaze direction detected by the gaze direction detection unit 142. The intersection between the line-of-sight direction of the second subject and the image display surface P2 of the display unit 54 when the line-of-sight direction of the subject coincides with or substantially coincides with the pupil position of the second subject in the image captured by the imaging unit 57 The images picked up by the image pickup unit 57 are corrected so that and become the same position.

According to the imaging device 100 illustrated in FIG. 14, even when the imaging device 100 is inclined with respect to the first subject, the recording unit 16 does not distort the face of the first subject in the image captured by the imaging unit 17. In addition, even when the imaging device 100 is inclined with respect to the second subject, the face of the second subject in the image captured by the imaging unit 57 is recorded in the recording unit 56 without being distorted. be able to.
<Modification of Second Embodiment (Part 3)>

  FIG. 15 is a diagram illustrating an imaging apparatus according to the fifth embodiment of the present invention. In addition, the same code | symbol is attached | subjected to the same structure as the imaging device 100 of 2nd Embodiment shown in FIG.

  The imaging apparatus 100 according to the fifth embodiment illustrated in FIG. 15 is different from the imaging apparatus 100 according to the second embodiment illustrated in FIG. 10 in that photometric units 151 and 152 are further provided.

  The photometry unit 151 measures the front side.

  The photometry unit 152 measures the back side.

  The projection image generation unit 21 includes information on an image captured by the imaging unit 19, a pupil position of the first subject detected by the pupil position detection unit 18, and shape information of the second subject detected by the shape detection unit 20. A projection image is generated based on the size of the image display surface P1 of the display unit 14 in the three-dimensional coordinates corresponding to the real space stored in the storage unit 22 and the photometric result of the photometric unit 151.

  The projection image generation unit 61 includes information on an image captured by the imaging unit 59, a pupil position of the second subject detected by the pupil position detection unit 58, and shape information of the first subject detected by the shape detection unit 60. A projection image is generated based on the size of the image display surface P2 of the display unit 54 in the three-dimensional coordinates corresponding to the real space stored in the storage unit 62 and the photometric result of the photometric unit 152.

  According to the imaging apparatus 100 shown in FIG. 15, since the photometry unit 151 can accurately detect the luminance and color of the first subject, the first subject in the projection image can be reproduced with a more natural color. At the same time, the luminance and color of the second subject can be accurately detected by the photometry unit 152, so that the second subject in the projection image can be reproduced with a more natural color.

DESCRIPTION OF SYMBOLS 10 Imaging device 11, 52 1st subject information detection part 12, 51 2nd subject information detection part 13 53 Display image generation part 14, 54 Display part 15, 55 Recording judgment part 16, 56 Recording part 17, 57 Imaging part 18, 58 pupil position detection unit 19, 59 imaging unit 20, 60 shape detection unit 21, 61 projection image generation unit 22, 62 storage unit 23 gaze direction detection unit 24 correction determination unit 25 correction unit 26 photometry unit

Claims (18)

First imaging means for imaging a first subject;
First pupil position detection means for detecting a pupil position of the first subject in three-dimensional coordinates corresponding to real space based on an image captured by the first imaging means;
A second image pickup means for picking up an image of a second subject located in the back direction when the image pickup direction of the first image pickup means is a front direction and a direction opposite to the front direction is a back direction;
First shape detecting means for detecting shape information of the second subject in the three-dimensional coordinates;
First display means arranged so that a display direction for displaying an image is the front direction;
First storage means for storing the size of the image display surface of the first display means;
Information on the image captured by the second imaging means, the pupil position of the first subject detected by the first pupil position detection means, and the shape of the second subject detected by the first shape detection means The first display means when the second subject is viewed from the pupil position of the first subject based on the information and the size of the image display surface of the first display means stored in the first storage means. First display image generation means for generating an image to be displayed on the first display means including the second subject projected on the image display surface of
An imaging apparatus comprising: The imaging apparatus according to claim 1,
First gaze direction detection means for detecting a gaze direction of the first subject based on an image captured by the first imaging means;
First record judging means;
First recording means;
With
The first recording determining means includes a pupil position of the second subject in an image displayed on the image display surface of the first display means, and a line of sight of the first subject detected by the first gaze direction detecting means. When the distance between the direction and the intersection of the image display surface of the first display unit is equal to or less than a threshold, the image captured by the first image capture unit is recorded in the first recording unit.
An imaging apparatus characterized by that. The imaging apparatus according to claim 1 or 2,
Correction determination means for determining whether or not it is necessary to correct an image captured by the first imaging means;
A first correction unit that corrects an image captured by the first imaging unit when the correction determination unit determines that an image captured by the first imaging unit needs to be corrected;
An imaging apparatus comprising: The imaging apparatus according to claim 3,
The correction determination unit determines whether or not it is necessary to correct an image captured by the first imaging unit based on the inclination of the imaging device.
An imaging apparatus characterized by that. The imaging apparatus according to claim 4,
The first imaging means includes first and second cameras disposed at different positions,
The correction determination means is configured to determine a first distance between the first camera and the first subject and a second distance between the second camera and the first subject based on the images captured by the first and second cameras. Determining the distance, and determining that the image captured by the first imaging means needs to be corrected when the difference between the first distance and the second distance is equal to or greater than a threshold;
An imaging apparatus characterized by that. The imaging apparatus according to any one of claims 1 to 5,
First metering means for metering the back direction side;
With
The first display image generating means further projects on the image display surface of the first display means when viewing the second subject from the pupil position of the first subject based on the photometric result of the first photometric means. Generating an image to be displayed on the first display means including the second subject to be
An imaging apparatus characterized by that. The imaging apparatus according to claim 1,
Third imaging means for imaging the second subject;
Fourth imaging means for imaging the first subject;
Second pupil position detection means for detecting a pupil position of the second subject in the three-dimensional coordinates based on an image captured by the third imaging means;
Second shape detecting means for detecting shape information of the first subject in the three-dimensional coordinates;
A second display means arranged so that a display direction for displaying an image is the back direction;
Second storage means for storing the size of the image display surface of the second display means;
Information on the image captured by the fourth imaging means, the pupil position of the second subject detected by the second pupil position detection means, and the shape of the first subject detected by the second shape detection means The second display means when the first subject is viewed from the pupil position of the second subject based on the information and the size of the image display surface of the second display means stored in the second storage means Second display image generation means for generating an image to be displayed on the second display means including the first subject projected on the image display surface of
An imaging apparatus comprising: The imaging apparatus according to claim 7,
First gaze direction detection means for detecting a gaze direction of the first subject based on an image captured by the first imaging means;
First record judging means;
First recording means;
Second gaze direction detection means for detecting a gaze direction of the second subject based on an image captured by the third imaging means;
A second record judging means;
A second recording means;
With
The first recording determination unit may be configured such that a line-of-sight direction of the first subject detected by the first line-of-sight direction detection unit and a line-of-sight direction of the second subject detected by the second line-of-sight direction detection unit coincide with each other. If substantially coincident, the image captured by the first imaging means is recorded in the first recording means,
The second recording determination unit may be configured such that a line-of-sight direction of the first subject detected by the first line-of-sight direction detection unit and a line-of-sight direction of the second subject detected by the second line-of-sight direction detection unit coincide with each other. When substantially matching, the image captured by the third imaging unit is recorded in the second recording unit.
An imaging apparatus characterized by that. The imaging apparatus according to claim 8, wherein
First correction determining means for determining that an image captured by the first imaging means needs to be corrected;
Second correction determination means for determining that an image captured by the third imaging means needs to be corrected;
A first correction unit that corrects an image captured by the first imaging unit when the first correction determination unit determines that an image captured by the first imaging unit needs to be corrected;
A second correction unit that corrects an image captured by the third imaging unit when the second correction determination unit determines that an image captured by the third imaging unit needs to be corrected;
An imaging apparatus comprising: The imaging device according to claim 9,
The first correction determination unit determines that the image captured by the first imaging unit needs to be corrected based on the inclination of the imaging device,
The second correction determination unit determines that the image captured by the third imaging unit needs to be corrected based on the inclination of the imaging device.
An imaging apparatus characterized by that. The imaging apparatus according to claim 10,
The first imaging means includes first and second cameras disposed at different positions,
The third imaging unit includes third and fourth cameras arranged at different positions,
The first correction determination unit is configured to determine a first distance between the first camera and the first subject and a distance between the second camera and the first subject based on the images captured by the first and second cameras. A second distance is obtained, and when the difference between the first distance and the second distance is greater than or equal to a first threshold, it is determined that the image captured by the first imaging unit needs to be corrected,
The second correction determining means is configured to determine a third distance between the third camera and the second subject and a distance between the fourth camera and the second subject based on each image captured by the third and fourth cameras. Determining the fourth distance, and determining that the image captured by the third imaging means needs to be corrected when the difference between the third distance and the fourth distance is equal to or greater than a second threshold;
An imaging apparatus characterized by that. The imaging device according to claim 9,
The first correcting means matches or substantially matches the line-of-sight direction of the first subject detected by the first line-of-sight direction detecting means and the line-of-sight direction of the second subject detected by the second line-of-sight direction detecting means. Correcting the image picked up by the first image pickup means based on the line-of-sight direction of the first subject when
The second correcting means matches or substantially matches the line-of-sight direction of the first subject detected by the first line-of-sight direction detecting means with the line-of-sight direction of the second subject detected by the second line-of-sight direction detecting means. Correcting the image picked up by the third image pickup means based on the line-of-sight direction of the second subject when
An imaging apparatus characterized by that. The imaging apparatus according to any one of claims 7 to 12,
The first and fourth imaging means are the same, and the second and third imaging means are the same,
An imaging apparatus characterized by that. An imaging apparatus according to any one of claims 7 to 13,
The first and second display means have substantially the same size.
An imaging apparatus characterized by that. The imaging apparatus according to any one of claims 7 to 14,
A first housing comprising the first imaging means, the fourth imaging means, the first pupil position detection means, the second shape detection means, the first display means, and the second display image generation means;
A second housing comprising the second imaging means, the third imaging means, the second pupil position detection means, the first shape detection means, the second display means, and the first display image generation means;
Connecting means for connecting the first and second housings to each other;
An imaging apparatus comprising: The imaging apparatus according to any one of claims 7 to 15,
First metering means for metering the back direction side;
A second photometric means for measuring the front direction side;
With
The first display image generating means further projects on the image display surface of the first display means when viewing the second subject from the pupil position of the first subject based on the photometric result of the first photometric means. Generating an image to be displayed on the first display means including the second subject to be
The second display image generating means further projects on the image display surface of the second display means when viewing the first subject from the pupil position of the second subject based on the photometric result of the second photometric means. Generating an image to be displayed on the second display means including the first subject to be
An imaging apparatus characterized by that. The imaging apparatus according to claim 1,
The second imaging means includes fifth and sixth cameras arranged at different positions,
The first shape detection means detects shape information of the second subject in the three-dimensional coordinates based on the images captured by the fifth and sixth cameras.
An imaging apparatus characterized by that. The imaging apparatus according to claim 7,
The second imaging means includes fifth and sixth cameras arranged at different positions,
The first shape detection means detects shape information of the second subject in the three-dimensional coordinates based on the images captured by the fifth and sixth cameras,
The fourth imaging means includes seventh and eighth cameras arranged at different positions,
The second shape detection means detects shape information of the first subject in the three-dimensional coordinates based on the images picked up by the seventh and eighth cameras.
An imaging apparatus characterized by that.

Images