JP4432462B2 - Imaging apparatus and method, imaging system - Google Patents

Description

  The present invention relates to an image pickup apparatus and method, and an image pickup system for reconstructing an image picked up for a desired photographing target so as to display the image on a stereoscopic display or the like.

  Conventionally, a stereoscopic imaging display method using a compound eye imaging device has been proposed. In this method, two imaging optical systems are arranged on the left and right sides at intervals given by the base line length, and images are taken from two viewpoints.

  The left and right eyes of humans are said to have an average distance of about 65 mm, and the baseline length of the two imaging optical systems is set to 65 mm even in stereoscopic imaging display. As described above, when a desired subject is imaged from the two left and right viewpoints, the position and the luminance level of the subject in the image captured by each imaging system are different from each other. That is, this is parallax, and the user can view a stereoscopic image by viewing the parallax in stereo.

  Incidentally, in order to stereoscopically view the images obtained from the two left and right viewpoints, the left and right images are alternately output on the display. On the user side, a stereoscopic image can be tasted by visually recognizing it with liquid crystal shutter glasses that switch the left and right shutters in synchronization with the left and right image display switching.

  Further, as another method of viewing the obtained image in stereo, two images on the left and right are alternately arranged in every other line in the horizontal direction in a region of one stereoscopic image prepared in advance. There is a method of generating an image on a stripe composed of two left and right images. In this method, similarly to the generated stereoscopic image, a polarizing plate whose polarization direction is alternately changed every other line in the horizontal direction is provided on the display, and an image in which the polarization directions are different from each other in a stripe shape is displayed. As a result, only the polarization component in one direction is transmitted and displayed on the image captured by the right imaging optical system, and only the polarization component in the other direction is transmitted and displayed on the image captured by the left imaging optical system. Will be. The user can enjoy a stereoscopic image by wearing polarized glasses having a function of transmitting only the same polarization component as that of the image displayed on the display for each of the left and right sides.

  As described above, stereoscopic imaging display uses the parallax of images captured from different viewpoints. That is, the user creates an image with a stereoscopic effect by superimposing two images having parallax in relation to a desired subject.

  By the way, in such a stereoscopic imaging display, the user is always obliged to wear spectacles, so that there is a problem that the user feels bothered. Therefore, particularly in recent years, as a stereoscopic display that does not require dedicated glasses, a display has been proposed in which optical elements are combined on a liquid crystal panel to display different left-eye parallax images and right-eye parallax images.

  In such a stereoscopic display that enables compound eye stereoscopic vision, a double image splitter method based on the parallax barrier method is adopted, and the left and right eye parts are arranged on the back side of a thin vertical striped barrier plate in which slits are continuously arranged. A stereogram is formed by alternately displaying a parallax image for the left eye and a parallax image for the right eye having a parallax in the form of vertical stripes. When the parallax image for the left eye and the parallax image for the right eye are actually displayed, the parallax, the display direction, the transparency, the luminance level (RGB), and the like in the stereoscopic view are made variable according to the shooting target.

  Note that, in such a stereoscopic display, as shown in FIG. 10A, when the same shooting target X is captured in different shooting directions by a camera with a fixed viewpoint, each of the shooting targets X is displayed. As shown in FIG. 10B, stereoscopic viewing is realized only when the relative positional relationship of the cameras matches the relative positional relationship between the viewpoint positions of the left and right eyes of the user viewing the stereoscopic display. Will be.

  However, when the viewpoint position of the user is away from the stereoscopic display as shown in FIG. 11A, or closer to the stereoscopic display as shown in FIG. As shown in c), when moving to the left and right with respect to the stereoscopic display, the relative positional relationship with the virtually displayed photographing target X ′ will be deviated, resulting in a shift in parallax. . In such a case, an unnatural image will be displayed to the user, which will give a sense of discomfort and eye strain.

  For this reason, conventionally, a method of generating a virtual image according to the relative positional relationship of the user's viewpoint with respect to the stereoscopic display and displaying the image has been proposed (for example, see Patent Document 1).

JP-A-8-116556

  However, in the disclosed technique in the above-described Patent Document 1, a left-eye parallax image and a right-eye parallax image are virtually generated after performing image matching based on straight line detection. For this reason, in order to execute the correlation between the images with high accuracy, it is necessary to arrange a plurality of cameras for imaging a desired imaging target in a straight line, and the imaging surface of each camera is the straight line. Therefore, there is a problem that the entire system becomes large and complicated.

  In addition, since it was difficult to detect the user's viewpoint position and line-of-sight direction with high accuracy, in the case of actually generating the left-eye parallax image and the right-eye parallax image, the parallax deviation or the like There was also a problem that it could not be solved and gave a sense of incongruity.

  Therefore, the present invention has been devised in view of the above-described problems, and an object of the present invention is to generate a virtual image according to the relative positional relationship of the user's viewpoint with respect to the stereoscopic display. At the same time, it is an object to provide an imaging apparatus and method, and an imaging system capable of accurately eliminating a parallax shift based on a viewpoint position specified from a three-dimensional position on a user's head.

In order to solve the above-described problems, the present invention captures images of photographing targets from different viewpoints, associates the captured images with each photographing position while associating the captured images with each other, and visually recognizes the display screen. Refers to statistical measurement data on the relative positional relationship between the human head and both eyes based on the three-dimensional position and size of the head detected from images captured by multiple cameras that capture the human head Then, the left eye position and the right eye position of the user are estimated, and relative position information in which the left eye position and the right eye position of the user viewing the estimated display screen are relatively identified in relation to the display screen is generated. Then, a left-eye parallax image and a right-eye parallax image are generated from the pixel position and the luminance component associated with each other according to the generated relative position information, and displayed on the display screen. To.

That is, an imaging apparatus to which the present invention is applied associates an imaging unit including a plurality of cameras that capture an imaging target from different viewpoints, and an image captured by each camera in the imaging unit with the imaging target. While having a matching means for associating each pixel position and a plurality of cameras for capturing the head of the user viewing the display screen of the stereoscopic display , a three-dimensional view of the head detected from the captured image of the camera A binocular position estimator that estimates a user's left eye position and right eye position with reference to statistical measurement data on the relative positional relationship between the human head and both eyes based on the position and size based on the left eye position and a right eye position of a user viewing the display screen of the stereoscopic display estimated by the binocular position estimation unit, a left-eye position and a right eye position of the user In accordance with the generated relative position information, information generating means for generating relative position information relatively identified in relation to the display screen, and pixel positions and luminance components associated with each other by the matching means. image generating means for generating respectively a parallax image and a right-eye parallax image for the left eye and the display screen of the stereoscopic display and the generated left-eye parallax image and the right-eye parallax image by the image generating means Display control means for displaying on the top.

The imaging method according to the present invention, and an imaging step you captured from different viewpoints captured subject by the imaging means including a plurality of cameras, among the images captured by the cameras in the imaging means, the photographing A matching step for associating each pixel position with each other while associating with a target, and a three-dimensional position and size in the head detected from captured images of a plurality of cameras that capture the head of the user viewing the display screen A binocular position estimation step for estimating a user's left eye position and right eye position with reference to statistical measurement data relating to the relative positional relationship between the human head and both eyes, and the binocular position estimation based on the left eye position and a right eye position of a user viewing the display screen which is estimated by the part, relative in relation to the display screen the left eye position and a right eye position of the user From the information generation step for generating the identified relative position information and the pixel positions and the luminance components associated with each other in the matching step, the left-eye parallax image and the right-eye image are generated according to the generated relative position information. An image generation step for generating a parallax image, and a display step for displaying the left-eye parallax image and the right-eye parallax image generated in the image generation step on the display screen.

In addition, an imaging system to which the present invention is applied associates an imaging unit including a plurality of cameras that capture an imaging target from different viewpoints, and each image captured by each camera in the imaging unit with the imaging target. While having an imaging device having a matching unit that associates each pixel position and a plurality of cameras that capture the head of the user viewing the display screen, three-dimensional in the head detected from the captured image of the camera Binocular position estimator that estimates the left eye position and right eye position of a user by referring to statistical measurement data related to the relative positional relationship between the human head and both eyes based on the specific position and size If, on the basis of the left eye position and a right eye position of a user viewing the display screen which is estimated by the binocular position estimation unit, Seki the left eye position and a right eye position of the user with the display screen The relative position information generated from the information generating means for generating the relative position information relatively identified in the above, the pixel positions associated with each other received via the network from the matching means in the imaging device, and the luminance component thereof And a left-eye parallax image and a right-eye parallax image generated by the image generation unit on the display screen. And an electronic device having display control means for displaying.

In addition, an imaging system to which the present invention is applied captures an image by an imaging apparatus having an imaging unit including a plurality of cameras that capture an imaging target from different viewpoints, and each camera received via a network from a matching unit in the imaging apparatus. A matching means for associating each image position with each other, and a plurality of cameras for capturing the head of the user viewing the display screen. The left eye position and the right eye position of the user with reference to statistical measurement data regarding the relative positional relationship between the human head and both eyes based on the three-dimensional position and size in the head detected from binocular position estimation unit that estimates a, based on the left eye position and a right eye position of a user viewing the display screen which is estimated by the binocular position estimation unit, the user And information generating means for generating a relatively identified relative position information left eye position and a right eye positions in relation with the display screen, the pixel position and luminance components are associated with each other by the matching unit, the generated Image generating means for generating a left-eye parallax image and a right-eye parallax image according to the relative position information, a left-eye parallax image and a right-eye parallax image generated by the image generating means, And an electronic device having display control means for displaying on the display screen.

That is, according to the present invention, a plurality of photographic subjects are imaged from different viewpoints, and each captured image is associated with each photographic subject while being associated with each photographic subject, and a plurality of images of the head of the user viewing the display screen is captured . Based on the three-dimensional position and size in the head detected from the captured image of the camera, the user's left eye position is referenced with reference to statistical measurement data regarding the relative positional relationship between the human head and both eyes. In addition, the right eye position is estimated, and the left eye position and the right eye position of the user who visually recognizes the estimated display screen are relatively identified in relation to the display screen to generate relative position information, and the pixels associated with each other From the position and its luminance component, a left-eye parallax image and a right-eye parallax image are generated according to the generated relative position information, and displayed on the display screen.

For this reason, in the present invention, the coordinates of the left eye position and the right eye position of the user are obtained while referring to statistical measurement data based on the coordinates specified for the head position and the identified size of the user head. By estimating, it is possible to relatively obtain the coordinates of the left eye position and the coordinates of the right eye position based on the center coordinates of the stereoscopic display. Therefore, according to the present invention, even when the user's viewpoint position moves away from or approaches the stereoscopic display, or even when the user moves to the left or right with respect to the stereoscopic display, the shift in parallax is highly accurate and smooth. Therefore, it is possible to always provide a natural stereoscopic image to the user.

  Hereinafter, the best mode for carrying out the present invention will be described in detail with reference to the drawings.

  The imaging system to which the present invention is applied, for example, as shown in FIG. 1, captures a desired shooting target 5 with a plurality of cameras, and performs processing to enable compound eye stereoscopic viewing on the obtained image. This is a system for displaying this on the stereoscopic display 20.

  That is, the imaging system 1 includes eight cameras 11_1 to 11_8 that capture the same subject 5 from different photographing directions, and an image correction unit that is supplied with images captured by the connected cameras 11_1 to 11_8. 12_1 to 12_8, a camera correction parameter supply unit 13 for supplying parameters necessary for image correction in each of the image correction units 12_1 to 12_8, and each image supplied from each of the connected image correction units 12_1 to 12_8. A matching unit 14 for associating; a binocular position estimating unit 15 for detecting a head position of a user viewing the stereoscopic display 20 and estimating a relative viewpoint position of the user based on the detected position; The user with respect to the stereoscopic display 20 according to the viewpoint position of the user estimated by the binocular position estimation unit 15 A parallax image for the left eye that is connected to the information generation unit 16 that generates relative position information indicating a relative positional relationship at the viewpoint position, the matching unit 14, and the information generation unit 16 and is visually recognized by the user's left eye. And a virtual viewpoint image generation unit 18 that generates a right eye parallax image that is visually recognized by the user's right eye, and a left eye parallax image and a right eye parallax image that are generated by the virtual viewpoint image generation unit 18. And a display control unit 19 for displaying on the connected stereoscopic display 20.

  The cameras 11_1 to 11_8 execute an automatic aperture control operation, an automatic focus control operation, and the like based on an operation signal supplied from a camera control unit (not shown). In addition, the cameras 11_1 to 11_8 convert the captured optical image to the CCD as it is by making it incident on the CCD as it is. The cameras 11_1 to 11_8 supply images represented by the converted electric signals to the image correction units 12_1 to 12_8, respectively. Incidentally, the cameras 11_1 to 11_8 are installed in a state where the shooting direction and the shooting angle of view are fixed according to the desired shooting target 5, but are not limited to a fixed viewpoint, for example, from the user Based on the input information, the shooting direction and the shooting angle of view may be freely changed.

  FIG. 2 shows an arrangement example of the cameras 11_1 to 11_8 that capture the imaging object 5 from different imaging directions. As shown in FIG. 2, the cameras 11_1 to 11_8 are arranged so as to draw an arc with respect to the shooting target 5 when the fixed viewpoint is used. Accordingly, even when the optical axes of the cameras 11_1 to 11_8 are aligned with the same position in the same photographing target 5, the imaging distance can be made uniform, and the uncomfortable feeling of the image generated by the image interpolation in the subsequent stage can be eliminated. Can do. Incidentally, the cameras 11_1 to 11_8 may be arranged in two or more rows as well as in the case where they are arranged in one row as shown in FIG. Further, the number of the cameras 11_1 to 11_8 is not limited to eight, and any number of cameras may be used as long as they are plural.

  The image correction units 12_1 to 12_8 correct the images respectively supplied from the cameras 11_1 to 11_8 based on the control information transmitted from the camera correction parameter supply unit 13. Each of the image correcting units 12_1 to 12_8 corrects each image supplied based on at least the geometric positional relationship in the camera cameras 11_1 to 11_8. The geometric positional relationship between the cameras 11_1 to 11_8 may be parameterized in any form in the camera correction parameter supply unit 13. At this time, the camera correction parameter supply unit 13 may similarly parameterize chromatic aberration, distortion, and optical axis shift in each lens of the cameras 11_1 to 11_8, and transmit them to the image correction units 12_1 to 12_8. .

  Incidentally, by storing these parameters in a ROM or RAM (not shown), the camera correction parameter supply unit 13 can refer to them as needed according to the situation, and can realize high-speed correction processing. . Further, the camera correction parameter supply unit 13 obtains these parameters every time images are supplied from the cameras 11_1 to 11_8, so that the image correction units 12_1 to 12_8 can realize highly accurate correction processing.

  In the imaging system 1 to which the present invention is applied, the configuration including the image correction units 12_1 to 12_8 and the camera correction parameter supply unit 13 may be omitted.

  The matching unit 14 is supplied with each image corrected by the image correction units 12_1 to 12_8. The matching unit 14 associates the supplied images with each other. In this association, when the same part constituting the photographing object 5 is extracted as a feature point, the pixel position and the luminance component at the part are matched between the images captured by the cameras adjacent to each other. To do.

  For example, as shown in FIG. 3, with respect to the pixel position P11 on the image Fr1 imaged by the camera 11_1, the pixel position P11 ′ existing at the same location on the image Fr2 imaged by the camera 11_2 is the corresponding point. As specified. Similarly, for the pixel position P12 on the image Fr1, the pixel position P12 'existing at the same location on the image Fr2 is specified as the corresponding point.

  That is, the matching unit 14 associates the images captured by the cameras adjacent to each other for each pixel position while associating the images with the subject 5. Incidentally, the matching unit 14 may be executed only for the image region in which the feature is extracted for this association, or may be executed for all the image regions.

  The binocular position estimation unit 15 detects a three-dimensional position and size in the head of the user viewing the stereoscopic display 20 relative to the stereoscopic display 20. The binocular position estimation unit 15 is applied by a distance measuring device that performs real-time position detection based on, for example, a stereo viewing method or a range finder method. The binocular position estimation unit 15 estimates the user's left eye position and right eye position under the three-dimensional spatial coordinates based on the detected three-dimensional position and size in the user's head. The estimation of the binocular position may be performed in real time with reference to statistical measurement data regarding the relative positional relationship between the human head and both eyes, for example.

  The information generation unit 16 generates relative position information obtained by parameterizing the user's left eye position and right eye position estimated by the binocular position estimation unit 15, and transmits the relative position information to the virtual viewpoint image generation unit 18.

  The virtual viewpoint image generation unit 18 receives the pixel positions and the luminance components associated with each other by the matching unit 14. Further, the virtual viewpoint image generation unit 18 newly generates the left-eye parallax image and the right to be generated according to the relative position information generated by the information generation unit 16 from the pixel position and the luminance component associated with each other. The pixel position constituting the parallax image for eye and the luminance component thereof are obtained. The virtual viewpoint image generation unit 18 transmits the left-eye parallax image and the right-eye parallax image configured by the obtained pixel position and the luminance component thereof to the display control unit 19, respectively.

  The display control unit 19 sets the luminance level (RGB) at each pixel position for the left-eye parallax image and the right-eye parallax image generated by the virtual viewpoint image generation unit 18 according to the display method on the stereoscopic display 20. assign.

  The stereoscopic display 20 is a display surface that combines optical elements on a liquid crystal panel to display different left-eye parallax images and right-eye parallax images, and does not require the user to wear dedicated glasses. Is excellent. In the stereoscopic display 20 that enables such compound-eye stereoscopic vision, for example, as shown in FIG. 4 (a), a double image splitter method based on the parallax barrier method is adopted, and a thin vertical stripe shape in which slits are continuously arranged. The left-eye parallax image and the right-eye parallax image having parallax for the left and right eyes on the back side of the barrier plate 25 are alternately displayed in a vertical stripe pattern. Thus, the user can visually recognize the parallax image for the left eye only with the left eye and can visually recognize the parallax image for the right eye only with the right eye. The context of the three-dimensional structure can be easily recognized.

  Incidentally, the stereoscopic display 20 is not limited to the case where the double image splitter method is adopted, and for example, a lenticular method as shown in FIG. 4B may be adopted. In this lenticular method, a lenticular plate 26 having sharp directivity is arranged on the surface of the stereoscopic display 20 according to the parallax for the left and right eyes. This also allows the user to visually recognize the left-eye parallax image with only the left eye and to visually recognize the right-eye parallax image with only the right eye, thereby realizing stereoscopic viewing.

  Next, a specific operation in the terminal device 2 will be described.

  The photographing object 5 is photographed from different viewpoints by the cameras 11_1 to 11_8. As a result, the images of the photographing object 5 generated by the cameras 11_1 to 11_8 are in a state in which the luminance component, the orientation of each surface, and the like are different from each other.

  The images from the cameras 11_1 to 11_8 are respectively supplied to the image correction units 12_1 to 12_8, and based on parameters such as lens distortion and image center of the cameras 11_1 to 11_8 obtained in advance by the camera correction parameters 13, Correction is performed so that distortion and the center of the image do not shift.

  For example, as illustrated in FIG. 5, when M points of the known imaging target 5 in the three-dimensional space are captured by the camera 11_1 and the camera 11_2, images parallel to the imaging surface of each camera 11 are generated. Here, an image generated by the camera 11_1 is an image I1, and an image generated by the camera 11_2 is an image I2. The image correction unit 12_1 and the image correction unit 12_2 perform geometric normalization on the generated images I1 and I2 so that the image planes are parallel to each other. This geometric normalization is realized by estimating camera internal parameters A1, A2, rotation matrices R1, R2, and transfer matrices T1, T2 using projection matrices P1, P2 obtained in advance by camera correction parameters 13. To do.

  As a result, the image I1 is converted into the image I1 ', and the image I2 is converted into the image I2'. The optical axes of the normalized image I1 'and image I2' are parallel to each other, and they are located on the same plane.

  Note that the respective images captured by the cameras 11_3 to 11_8 are also geometrically normalized in the image correction units 12_3 to 12_8. Further, the images I1 'and I2' geometrically normalized so as to be parallel to each other in the image correcting units 12_1 to 12_8 are supplied to the matching unit 14.

  In addition, regarding the camera parameter acquisition method for performing the correction process in each of the image correction units 12_3 to 12_8, for example, a camera calibration method disclosed in Japanese Patent Application Laid-Open No. 2000-350239 and Japanese Patent Application Laid-Open No. 11-53549. May be applied as is.

  When the normalized images I1 ′ and I2 ′ as described above are associated in the matching unit 14, for example, for the corresponding points of the pixel m1 ′ on the epipolar line L1 of the image I1 ′, the image I2 ′ The pixel m2 ′ as the corresponding point can be detected by searching on the straight line L2.

That is, since the image I1 ′ and the image I2 ′ are geometrically normalized in the image correction unit 12 in the previous stage, it is possible to efficiently associate each pixel. This is because each pixel on the epipolar line L1 to be scanned corresponds to each pixel on the epipolar line L2, so that the correspondence between the pixels arranged in each column can be solved as a shortest path search problem by Dynamic Programming. Because. The matching unit 14 uses the luminance difference dg (x, y) between the images and the color information difference dI (x, y) to obtain a matching cost function MC (x, y) expressed by the following equation (1). y) is determined.
MC (x, y) = (1−α) · dg (x, y) + α · dI (x, y) (1)
Here, α is a weighting coefficient and is a value of 0 ≦ α ≦ 1. The matching unit 14 detects corresponding points based on the obtained matching cost function MC (x, y). In such a case, input from each correction processing unit 12 to further improve the matching accuracy. The occlusion cost may be appropriately adjusted according to each image to be processed.

  That is, in the imaging system 1 to which the present invention is applied, since the matching processing described above is realized in the matching unit 14, it is possible to accurately and smoothly perform the association between the pixels. Incidentally, the pixel positions and the luminance components associated with each other by the matching unit 14 are output to the virtual viewpoint image generation unit 18, respectively.

The virtual viewpoint image generation unit 18 is based on the pixel positions associated with each other and the relative position information k (here, 0 <k ≦ 1) transmitted from the information generation unit 16 and based on the method described below. Thus, the pixel position and the luminance component thereof constituting the left-eye parallax image (right-eye parallax image) Iv ′ are obtained. For example, in the matching unit 14, as shown in FIG. 5, when the pixel m2 ′ existing on the image I2 ′ is specified as the corresponding point of the pixel m1 ′ on the image I1 ′, the pixel m1 ′ The coordinates are (x1, y1), and the coordinates of the pixel m2 ′ are (x2, y2). The virtual viewpoint image generation unit 18 uses the coordinates (xt, yt) of the pixel mv ′ on the left-eye parallax image (right-eye parallax image) corresponding to the pixel m1 ′ and the pixel m2 ′ as the relative position information k. Based on the following equation (2), it can be determined.
(Xt, yt) = k × (x1, y1) + (1−k) × (x2, y2) (2)
When the luminance components in the pixel m1 ′ and the pixel m2 ′ are J1 ′ and J2 ′, respectively, the luminance component Pt in the pixel mv ′ on the left-eye parallax image (right-eye parallax image) is expressed by the following formula ( It can be determined by 3).

Pt = k × J1 ′ + (1-k) × J2 ′ (3)
As described above, the virtual viewpoint image generation unit 18 can determine the coordinates of each pixel constituting the left-eye parallax image (right-eye parallax image) and the luminance component thereof according to the relative position information k.

  Incidentally, the relative position information k is generated according to the following procedure according to the viewpoint position of the user viewing the stereoscopic display 20.

  FIG. 6A shows the relative positional relationship of each camera 11 with respect to the imaging target 5 indicated by the dotted line superimposed on the positional relationship of the user's viewpoint with respect to the stereoscopic display 20 indicated by the solid line. is there. In the stereoscopic display 20, only when the relative positional relationship of the optical center of each camera 11 with respect to the photographing target 5 matches the positional relationship of the left and right eye viewpoint positions of the user viewing the stereoscopic display 20. Stereoscopic viewing will be realized.

  Here, when imaging is performed by aligning the optical axes of the cameras 11_1 to 11_8 to the M point of the imaging target 5 located on the coordinates (0, 0, 0), the center of the stereoscopic display 20 is superimposed on this. Thus, the coordinates of the center are set so as to be located at the origin (0, 0, 0). Further, the coordinates (xn, yn, zn) of the user's head position measured by the binocular position estimation unit 15 are specified based on the origin coordinates (0, 0, 0). Incidentally, although how to define the user's head position can be arbitrarily determined on the imaging system 1 side, for example, the center of the user's eyebrow may be defined as the head position.

  If the user's head is at position A, the binocular position estimation unit 15 specifies the coordinates (xa, ya, za). When the user's head is at position B, the binocular position estimation unit 15 specifies the coordinates (xb, yb, zb). When the user's head is at position C, the binocular position estimation unit 15 specifies the coordinates (xc, yc, zc). Incidentally, the binocular position estimation unit 15 identifies the size of the user's head at the same time as specifying the coordinates.

  The binocular position estimation unit 15 estimates the user's left eye position and right eye position based on the coordinates of the identified user's head position (xn, yn, zn). This estimation is based on the coordinates (xa, ya, za) specified for the head position A and the size of the identified user head, for example, as shown in FIG. Then, the coordinates (xaL, yaL, zaL) of the left eye position of the user and the coordinates (xaR, yaR, zaR) of the right eye position are estimated. Thereby, the coordinates (xaL, yaL, zaL) of the left eye position and the coordinates (xaR, yaR, zaR) of the right eye position based on the center coordinates (0, 0, 0) of the stereoscopic display 20 are relatively obtained. It becomes possible. Incidentally, the coordinates of the left eye position (xaL, yaL, zaL) and the coordinates of the right eye position (xaR, yaR, zaR) may be set to arbitrary values by the user.

  Incidentally, the estimated coordinates of the left eye position (xaL, yaL, zaL) and the coordinates of the right eye position (xaR, yaR, zaR) are the optical centers of the cameras 11_1 to 11_8 respectively indicated by dotted lines in FIG. 6 (a). If the image acquired from the camera is directly used as the left-eye parallax image and the right-eye parallax image, stereoscopic vision is realized. However, in other cases, the relative positional relationship of each camera 11 with respect to the subject 5 is deviated, making it difficult for the user to realize stereoscopic viewing.

  Here, the estimated coordinates of the left eye position (xaL, yaL, zaL) and the coordinates of the right eye position (xaR, yaR, zaR) are located outside the optical center of the cameras 11_1 to 11_8. And when it is located on the line Lc connecting the optical centers between the cameras 11 shown in FIG. 6A, such a position can be obtained only by constructing a left-eye parallax image (right-eye parallax image). Coordinate relationships.

  For example, as shown in FIG. 7 (a), when the user's head is at position D and the estimated left eye position and right eye position are at VcL1 and VcR1, respectively, a virtual camera is placed on VcL1 and VcR1. The left eye parallax image (right eye parallax image) is obtained as a result of installation and imaging of the M point of the subject 5 (corresponding to the center coordinates (0, 0, 0) of the stereoscopic display 20). Adjust as follows.

  In such a case, the information generation unit 16 acquires the left eye position VcL1 and the right eye position VcR1 estimated by the binocular position estimation unit 15 as described above. Then, the acquired left eye position VcL1 is set as a virtual position WL1 where the camera is virtually installed, and this is actually installed while maintaining a relative positional relationship with the object 5 to be captured. Apply to the positional relationship. Similarly, the acquired right eye position VcR1 is set as a virtual position WR1 where the camera is virtually installed, and the camera 11_1 to the camera 11_1 which are actually installed while maintaining a relative positional relationship with the imaging target 5 are used. It applies to the positional relationship of 11_8.

For example, as illustrated in FIG. 7A, when both the left eye position VcL1 and the right eye position VcR1 are between the cameras 11_4 and 11_5, the information generation unit 16 determines the left eye position VcL1 and the right eye for the cameras 11_4 and 11_5. The positional relationship of the eye position VcR1 is relatively identified, and k _L1 and k _R1 are respectively generated as relative position information corresponding thereto. K _L1 as the relative position information means that the virtual position WL1 is at a position of k _L1 : 1−k _L1 from the cameras 11_4 to 11_5. Similarly, k _R1 as the relative position information means that the virtual position WR1 is at a position of k _R1 : 1−k _R1 from the cameras 11_4 to 11_5. That is, as k _L1 and k _R1 as the relative position information become smaller, the virtual position WL1 and virtual position WR1 approach the camera 11_4, and as k _L1 and k _R1 as the relative position information become larger, the virtual position WL1. Therefore, the virtual position WR1 approaches the camera 11_5.

  Therefore, in the virtual viewpoint image generation unit 18, the coordinates (xt, yt) of the pixel mv ′ on the left-eye parallax image (right-eye parallax image) determined based on Expression (2) are the virtual position WL1 ( As the virtual position WR1) approaches the camera 11_4, it approaches (x1, y1), and as the virtual position WL1 (virtual position WR1) approaches the camera 11_5, it approaches (x2, y2).

  Further, the luminance component Pt at the pixel mv ′ on the left-eye parallax image (right-eye parallax image) determined based on Expression (3) is J1 as the virtual position WL1 (virtual position WR1) approaches the camera 11_4. Approaching J2 'as the virtual position WL1 (virtual position WR1) approaches the camera 11_5.

In particular, since the shooting directions of the camera 11_4 and the camera 11_5 are different from each other, the luminance component is different between the coordinates (xt, yt) and the coordinates (x2, y2). By either increasing or decreasing the luminance component Pt linearly according to k (k _L1, k _R1 ) as relative position information so that one of the different luminance components is the minimum value and the other is the maximum value. The luminance component of the pixel mv ′ on the left-eye parallax image (right-eye parallax image) can be determined.

  In other words, the virtual viewpoint image generation unit 18 responds to the estimated relative position information generated from the left eye position and the right eye position of the user from a plurality of images obtained as a result of imaging the imaging target from different viewpoints. By performing image interpolation, a new left-eye parallax image (right-eye parallax image) can be generated.

  The obtained left-eye parallax image (right-eye parallax image) corresponds to an image obtained as a result of virtually installing a camera on VcL1 (VcR1). By displaying these on the stereoscopic display 20 via the display control unit 19, an image obtained as a result of installing the cameras at the estimated left eye position and right eye positions VcL1 and VcR1 is obtained as a left eye parallax image and a right eye image. It can be output as a parallax image. For this reason, stereoscopic vision according to the estimated left eye position and right eye position can be realized. Note that the same processing is executed even when the identified user's head position is between the other cameras 11.

  FIG. 8 shows a configuration example of these left-eye parallax images (right-eye parallax images). If the subject 5 is a person's face, the camera 11_4, 11_5 captures this. In such a case, the camera 11_4 images a person as the shooting target 5 captured near the right end of the shooting range, and the camera 11_5 differs from the camera 11_4 for a person as the shooting target 5 captured near the left end of the shooting range. The image is taken from the viewpoint. As a result, the image captured by the camera 11_4 is in a state where a person facing the right side is shown near the right end as shown in FIG. 8, and the image captured by the camera 11_5 is the left side near the left end. The person who is facing is shown.

  When k as the relative position information is small (in the case of k1), the virtual position WL1a (virtual position WR1a) is closer to the camera 11_4. Further, for this virtual position WL1a, the left-eye parallax image (right-eye parallax image) obtained from the equations (2) and (3) has a content close to that of the camera 11_4.

  Further, as the relative position information gradually increases from k2 to k4, the virtual position gradually approaches the camera 11_5 from WL1b (virtual position WR1b) to WL1d (virtual position WR1d). Along with this, the image shown in the left-eye parallax image (right-eye parallax image) gradually moves from the vicinity of the right end to the vicinity of the left end, and the direction in which the person faces gradually changes from right to left. become.

  Incidentally, the virtual viewpoint image generation unit 18 is not limited to the case where image interpolation is performed. For example, as shown in FIG. 7 (a), when the user's head is at position E and the estimated left eye position and right eye position are at VcL2 and VcR2, respectively, the right-eye parallax image is out of image. You may produce | generate by inserting.

  Next, processing when the estimated left eye position coordinates (xaL, yaL, zaL) and right eye position coordinates (xaR, yaR, zaR) are not located on the line Lc will be described. In such a case, the imaging system 1 adjusts the positional relationship by enlarging or reducing the field of view of the configured left-eye parallax image (right-eye parallax image).

In the imaging system 1 to which the present invention is applied, stereoscopic vision can be realized even when the user's viewpoint position moves in the perspective direction with respect to the stereoscopic display 20. As shown in FIG. 7 (b), the left eye position VcL1 from the stereoscopic display 20, the distance up to the right eye position VcR1 and is z, the distance from the stereoscopic display 20 up to the line Lc and z ₀ When the user's head is identified at the position F such that z <z ₀ , the processing described below is executed.

  First, when the left eye position and the right eye position estimated based on the identified position F are at VcL1 and VcR1, respectively, the intersection of the extension line connecting the VcL1 and the M point and the line Lc is defined as VcL1 ′, and the VcR1 and the M point Let VcR1 ′ be the intersection of the extension line connecting the lines Lc and the line Lc. An image obtained as a result of imaging a point M (corresponding to the center coordinates (0, 0, 0) of the stereoscopic display 20) of the imaging target 5 by virtually installing a camera on the VcL1 ′ and VcR1 ′. Then, the left eye parallax image (right eye parallax image) is adjusted.

  Similarly, in this case, when both the estimated left eye position VcL1 ′ and right eye position VcR1 ′ are between the cameras 11_4 and 11_5, the information generation unit 16 determines the left eye position VcL1 ′ for the cameras 11_4 and 11_5. In addition, the positional relationship of the right eye position VcR1 ′ is relatively identified, and relative position information k corresponding to this is generated.

  Next, based on equations (2) and (3), a left-eye parallax image (right-eye parallax image) is generated according to k as relative position information. The obtained left-eye parallax image (right-eye parallax image) corresponds to an image obtained as a result of virtually installing a camera on VcL1 '(VcR1').

Next, the field of view is expanded until these left-eye parallax images (right-eye parallax images) become images obtained as a result of virtually installing cameras on VcL1 and VcR1. In such a case, the binocular position estimation unit 15 enlarges the left-eye parallax image (right-eye parallax image) according to the above-described distance ratio z / z ₀ . Incidentally, the measurement of the distance z may be performed when the head position is measured by the binocular position estimation unit 19.

  The left-eye parallax image (right-eye parallax image) enlarged per field of view corresponds to an image obtained as a result of virtually installing a camera on VcL1 (VcR1). By displaying these on the stereoscopic display 20 via the display control unit 19, an image obtained as a result of virtually installing the camera at the estimated left eye position VcL1 and right eye position VcR1 is displayed as a left-eye parallax image, It can be output as an eye parallax image. For this reason, stereoscopic vision according to the estimated left eye position and right eye position can be realized.

Here, as shown in FIG. 7C, when the user's head is identified at a position G such that z> z ₀ , processing described below is executed.

  First, when the left eye position and the right eye position estimated on the basis of the identified position F are at VcL1 and VcR1, respectively, the intersection of the line connecting VcL1 and the M point and the line Lc is defined as VcL1 ', and the VcR1 and the M point are defined as VcL1'. Let VcR1 ′ be the intersection of the connecting line and the line Lc. An image obtained as a result of imaging a point M (corresponding to the center coordinates (0, 0, 0) of the stereoscopic display 20) of the imaging target 5 by virtually installing a camera on the VcL1 ′ and VcR1 ′. It adjusts so that it may become a parallax image for left eyes (parallax image for right eyes).

  Similarly, in this case, when both the estimated left eye position VcL1 ′ and right eye position VcR1 ′ are between the cameras 11_4 and 11_5, the information generation unit 16 determines the left eye position VcL1 ′ for the cameras 11_4 and 11_5. In addition, the positional relationship of the right eye position VcR1 ′ is relatively identified, and relative position information k corresponding to this is generated.

  Next, based on equations (2) and (3), a left-eye parallax image (right-eye parallax image) is generated according to k as relative position information. The obtained left-eye parallax image (right-eye parallax image) corresponds to an image obtained as a result of virtually installing a camera on VcL1 '(VcR1').

Next, the visual field is reduced until these left-eye parallax images (right-eye parallax images) become images obtained as a result of virtually installing cameras on VcL1 and VcR1. In such a case, the binocular position estimation unit 15 reduces the left-eye parallax image (right-eye parallax image) in accordance with the distance ratio z / z ₀ described above.

  The left-eye parallax image (right-eye parallax image) reduced per field of view corresponds to an image obtained as a result of virtually installing a camera on VcL1 (VcR1). By displaying these on the stereoscopic display 20 via the display control unit 19, an image obtained as a result of virtually installing the camera at the estimated left eye position VcL1 and right eye position VcR1 is displayed as a left-eye parallax image, It can be output as an eye parallax image. For this reason, stereoscopic vision according to the estimated left eye position and right eye position can be realized.

  That is, in the imaging system 1 to which the present invention is applied, even when the user's viewpoint is not located on the line Lc, the field of view of the left-eye parallax image (right-eye parallax image) is expanded or reduced accordingly. It can be carried out. Thereby, since it is possible to eliminate the shift of parallax with high accuracy and smoothly without being controlled by the viewpoint position of the user viewing the stereoscopic display 20, a natural stereoscopic image is always provided to the user. Is possible.

  In particular, in the imaging system 1 to which the present invention is applied, image information at an infinite number of viewpoints can be theoretically obtained from the cameras 11_1 to 11_8 having a finite fixed viewpoint. Therefore, for the left eye using a relatively small number of cameras. A parallax image (right-eye parallax image) can be generated, and the entire system can be significantly reduced in size and cost.

Incidentally, when the field of view is enlarged or reduced, the configuration of the imaging system 2 shown in FIG. 9 may be adopted. In this imaging system 2, description of the same configuration as the imaging system 1 shown in FIG. 1 is omitted, but the field of view of the left-eye parallax image (right-eye parallax image) output from the virtual viewpoint image unit 18 is expanded. A magnification processing unit 21 is newly provided for performing reduction. The magnification processing unit 21 receives information on the distances z and z ₀ from the information generation unit 16 and performs the above-described enlargement or reduction of the field of view.

  Note that the imaging system 1 to which the present invention is applied is not limited to the above-described embodiment, and the optical element arrangement method is varied on the back surface of the barrier plate 25 of the stereoscopic display 20. Also good.

  The stereoscopic display 20 adopting the double image splitter method realizes stereoscopic vision by giving the display side the role of dedicated glasses that enable compound-eye stereoscopic vision, but depending on the viewpoint position of the user, In some cases, the left-eye parallax image and the right-eye parallax image displayed alternately in vertical stripes cannot be viewed only by the left eye and the right eye, respectively. In order to prevent such inconvenience, the interval and the number of optical elements to be arranged may be adjusted according to the interval of the slits.

  The imaging system 1 to which the present invention is applied receives an image capturing apparatus including at least the cameras 11_1 to 11_8 and a signal from the image capturing apparatus via, for example, a network via the Internet network, and at least the binocular position estimating unit 15, The information generation unit 16, the virtual viewpoint image generation unit 18, and the electronic device including the display control unit 19 may be included. That is, by dividing the entire imaging system 1 via a network, for example, the imaging target 5 is imaged by an imaging device provided on the broadcasting station side, and this is taken to an electronic device provided in each home via the network. Service can be provided. Incidentally, the configuration of the matching unit 14 may be provided on the imaging device side or may be provided on the electronic device side.

It is a figure shown about the structure in the imaging system 1 to which this invention is applied. It is a figure for demonstrating about the example of arrangement | positioning of the camera with respect to the imaging | photography object. It is a figure for demonstrating about the matching in a matching part. It is a figure for demonstrating per structure of a stereoscopic display. It is a figure for demonstrating about the example which performs geometric normalization between the imaged images. It is a figure for demonstrating about operation | movement in a binocular position estimation part. It is a figure for demonstrating about the example which installs a camera virtually according to a user's viewpoint position. It is a figure for demonstrating about the structural example of the parallax image for left eyes (parallax image for right eyes). It is a figure for demonstrating about the structure of the imaging system which expands or reduces a visual field. It is a figure for demonstrating about the example which implement | achieves a stereoscopic vision. It is a figure for demonstrating about the problem of a prior art.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Imaging system, 5 imaging | photography object, 11 camera, 12 Image correction part, 13 Camera correction parameter supply part, 14 Matching part, 15 Binocular position estimation part, 16 Information generation part, 18 Virtual viewpoint image generation part, 19 Display control part 20 Stereoscopic display

Claims (11)

An imaging means including a plurality of cameras for imaging the imaging object from different viewpoints;
Matching means for associating each image position with each other between the images captured by each camera in the imaging means,
A plurality of cameras that capture the head of a user viewing the display screen of the stereoscopic display, and based on the three-dimensional position and size of the head detected from the captured image of the camera, A binocular position estimation unit that estimates a user's left eye position and right eye position with reference to statistical measurement data regarding the relative positional relationship between both eyes,
Based on the left eye position and right eye position of the user viewing the display screen of the stereoscopic display estimated by the binocular position estimation unit , the user's left eye position and right eye position are relative to each other in relation to the display screen. Information generating means for generating the identified relative position information,
Image generation means for generating a left-eye parallax image and a right-eye parallax image according to the generated relative position information from the pixel position and the luminance component associated with each other by the matching means;
Said image generation means that an imaging device and a display control unit that the generated parallax image and the right-eye parallax image for the left eye is displayed on the display screen of the stereoscopic display by. Image size adjusting means for further enlarging or reducing the left-eye parallax image and right-eye parallax image generated by the image generating means according to the relative position information;
It said display control means, the image pickup apparatus 請 Motomeko 1, wherein the size adjusted parallax image and a right eye parallax image for the left eye by the image size adjustment unit Ru is displayed on the display screen. At least on the basis of the geometric positional relationship between the cameras in the imaging device, the imaging device further comprising Ru請 Motomeko 1 wherein the image correction means for correcting an image captured by the image pickup means. And an imaging step you captured from different viewpoints captured subject by the imaging means including a plurality of cameras,
A matching step for associating each image position with each other between each image captured by each camera in the image capturing means ,
Based on the three-dimensional position and size of the head detected from images captured by a plurality of cameras that capture the head of the user viewing the display screen of the stereoscopic display, the relative relationship between the human head and both eyes A binocular position estimating step for estimating the left eye position and the right eye position of the user with reference to statistical measurement data regarding the positional relationship;
Based on the left eye position and right eye position of the user viewing the display screen estimated by the binocular position estimation unit , the user's left eye position and right eye position are relatively identified in relation to the display screen. An information generation step for generating relative position information;
An image generation step for generating a left-eye parallax image and a right-eye parallax image according to the generated relative position information from the pixel positions and the luminance components associated with each other in the matching step;
That IMAGING METHOD having a display step for the generated parallax image and the right-eye parallax image for the left eye is displayed on the display screen of the stereoscopic display in the image generation step. An image size adjustment step of further enlarging or reducing the left-eye parallax image and the right-eye parallax image generated in the image generation step according to the relative position information;
The imaging method according to claim 4 , wherein in the display step, the parallax image for the left eye and the parallax image for the right eye that have been adjusted in the image size adjustment step are displayed on the display screen. The imaging method according to claim 4 , further comprising an image correction step of correcting an image captured by each camera in the imaging unit based on at least a geometric positional relationship between the cameras in the imaging unit . An imaging unit that includes a plurality of cameras that capture an image of a shooting target from different viewpoints, and a matching unit that associates each image captured by each camera in the imaging unit for each pixel position in association with the shooting target. An imaging device comprising:
A plurality of cameras that capture the head of a user who visually recognizes the display screen, and the relative relationship between the human head and both eyes based on the three-dimensional position and size of the head detected from the captured image of the camera A binocular position estimator that estimates a user's left eye position and right eye position with reference to statistical measurement data relating to a general positional relationship, and a user who views the display screen estimated by the binocular position estimator Information generating means for generating relative position information that relatively identifies the left eye position and right eye position of the user in relation to the display screen based on the left eye position and right eye position of the user, and matching in the imaging device The left-eye parallax image and the right-eye parallax image are respectively generated from the pixel positions and their luminance components received from the means via the network according to the generated relative position information. Image generating means for generating, an imaging system that includes a electronic device and a display control means for displaying the left-eye parallax image and the right-eye parallax image generated by said image generating means on said display screen . The electronic apparatus includes an image size adjusting unit that further enlarges or reduces the left-eye parallax image and the right-eye parallax image generated by the image generation unit according to the relative position information.
The imaging system according to claim 7 , wherein the display control unit displays the left-eye parallax image and the right-eye parallax image adjusted by the image size adjusting unit on the display screen. The imaging device, based on the geometric positional relationship between the camera in at least said imaging means, the imaging system according to claim 7, further comprising an image correction means for correcting an image captured by the image pickup means. An image pickup apparatus having an image pickup unit including a plurality of cameras for picking up a shooting target from different viewpoints;
Matching means for associating each image position with each other between the images captured by each camera received from the matching means in the imaging device via the network, and a user viewing the display screen Statistical analysis of the relative positional relationship between the human head and both eyes based on the three-dimensional position and size of the head detected from the image captured by the camera having a plurality of cameras that capture the head Binocular position estimation unit that estimates the user's left eye position and right eye position with reference to the measurement data, and the user's left eye position and right eye that visually recognizes the display screen estimated by the binocular position estimation unit based on the position, the information generating means for a left eye position and a right eye position of the user to generate a relatively identified relative position information in relation with the display screen, the matching Image generation means for generating a left-eye parallax image and a right-eye parallax image according to the generated relative position information from the pixel positions and their luminance components associated with each other by the stage, and the image generation an imaging system that includes an electronic device and generated left-eye parallax image and the right-eye parallax image and a display control means for displaying on said display screen by means. The electronic apparatus includes an image size adjusting unit that further enlarges or reduces the left-eye parallax image and the right-eye parallax image generated by the image generation unit according to the relative position information.
The imaging system according to claim 10 , wherein the display control unit displays the left-eye parallax image and the right-eye parallax image adjusted by the image size adjusting unit on the display screen.

Images