JP5478357B2 - Display device and display method - Google Patents

Description

  The present invention relates to, for example, a display device and a display method for displaying a display target in three dimensions.

For example, there is a video conference system in which an image obtained by photographing a user in a remote location with a camera is received and displayed on a display at the current location, so that a conference can be held while viewing the remote user on the display (see FIG. For example, see Document 1.)
In this system, the other party's images taken with the other party's camera can be displayed on their own display so that the user can talk while recognizing the gesture and facial expression of the other party's user. You can have a meeting in an environment close to that you are meeting.

Japanese Patent Laid-Open No. 9-107534 Japanese Patent No. 3587106

Jouppi, N .; : First Steps Towers Mutually-Immersive Mobile Telepresence. in Proc. CSCW2002, pp. 354-363.

  However, if this display is a display that displays a two-dimensional image and the other user is talking from the camera's point of view and this image is displayed on the local display, There is a problem that it seems that the other party's user shown on the display looks at all the users on the other side.

  For example, a case where a user L at a remote location side, a user M and a user N at a current location side have a conference using a video conference system will be described as an example. When the other user L is talking to the camera, the user L is looking at the camera, and the user L displayed on the current location display is looking at the user M or looking at the user N. An image that you do not know is displayed. For this reason, the users M and N who are viewing the display may misunderstand that both are viewed by the user L.

  On the other hand, when the user L on the other side is talking in a direction different from the camera, the user L is not looking at the camera, so the user L displayed on the display at the current location is the user M and the user L Both N are displayed as if they are talking in different directions. Therefore, the users M and N looking at the display feel that neither of them is seen by the user L.

  In this way, in the display that displays a general two-dimensional image, the appearance of the other user on the other side is the same, so the user on the current location side can tell who the other user is talking to. There was a problem that could not be understood correctly.

  In view of the above-described problems, an object of the present invention is to provide a display device and a display method capable of changing the appearance of a display target in accordance with the position of a user viewing the display.

In view of the above-described problems, the display device according to the present invention includes azimuth information indicating the orientation of the face of the display target with respect to the camera that captures the display target, and three-dimensional face model data obtained by the camera shooting the display target. an input unit for inputting bets, comprises a first display element and the second display device is disposed to face a display unit for stereoscopically displaying the display object, the 3-dimensional face model on the basis of the orientation information Data is converted into a three-dimensional face model of the front face when the display object is viewed from the front direction, and the first display image and the second display displayed by the first display element from the three-dimensional face model of the front face. generating a second display image element is displayed, observation person and the first display image shift amount based on the orientation information as seen from a position between the frontal faces of the person that is the display object are the relative and said second display image And a correction of shifting, and a display control unit for displaying on the first display element and the second display element.

Further, the above-described display device, the front Symbol display control unit, for the three-dimensional face model of the frontal face, by performing distribution of brightness proportional to the depth of the face of the display object where the orientation information indicates, that generates the second display image and the first display image.

Further, the display device described above is based on the camera that captures the display target and outputs image data, the distance sensor that measures the distance between the display target and the camera and outputs distance information, and the image data . further comprising a three-dimensional face model generator for generating the 3-dimensional face model data, and orientation information generation unit that generates the azimuth information on the basis of said distance information.

In view of the above-described problems, the display method according to the present invention is placed at a second point different from the first point based on information indicating a display target photographed by the camera of the first display device placed at the first point. In the display method for stereoscopically displaying the display object on the second display device, the first display device captures image data representing the face of the person who is the display object by photographing the display object with the camera. Obtaining a distance information by measuring a distance between the display object and the camera , generating three-dimensional face model data based on the image data , and obtaining the display object based on the distance information. that will be sending and generating orientation information indicating the orientation of the face of the display object relative to the camera for photographing, the three-dimensional face model data and the azimuth information to the second display device And-up, when the second display device, viewed and receiving said azimuth information and the 3-dimensional face model data, the display target the 3-dimensional face model data on the basis of the orientation information from the front A display image displayed by a display unit that includes a step of converting into a three-dimensional face model of the front face and a first display element and a second display element that are arranged to face each other and that displays the display target in a three-dimensional manner a step of generating a second display image in which the first display image and the second display device the first display element from the 3-dimensional face model of the front face is displayed is displayed, viewed from a position where there are observation's The first display element is corrected by relatively shifting the first display image and the second display image by a shift amount based on the orientation information so as to be a face in front of the person who is the display target. And the second display element And a step of displaying on.

  ADVANTAGE OF THE INVENTION According to this invention, the appearance of a display target can be changed according to the position of the user who views a display.

It is a figure which shows an example of the display system concerning embodiment of this invention. It is the schematic shown about an example of the three-dimensional display apparatus concerning embodiment of this invention. It is a figure which shows an example of the positional relationship of the three-dimensional display apparatus concerning embodiment of this invention, and a display target. It is a figure for demonstrating the display method of the three-dimensional display thing concerning embodiment of this invention. It is another figure for demonstrating the display method of the three-dimensional display thing concerning embodiment of this invention. It is a block diagram which shows an example of a structure of the display system concerning embodiment of this invention. It is a figure for demonstrating an example of the display method of the three-dimensional display apparatus concerning embodiment of this invention. It is a figure for demonstrating the relationship between the three-dimensional display apparatus concerning embodiment of this invention, a display target, and an observer. It is a flowchart for showing an example of the display method concerning embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a diagram for explaining an overview of a remote display system according to the first embodiment of the present invention.
As shown in FIG. 1, the remote display system according to the first embodiment of the present invention includes a 3D display device 100 and a 2D display device 200 connected via a network NW.
The 3D display device 100 is placed, for example, in a conference room or the like that exists at a first location, and transmits video images of the users A and B to the 2D display device 200 via the network NW.
In addition, the 2D display device 200 is placed in a conference room or the like that exists at a second point different from the first point, and transmits a video image of the user C to the 3D display device 100 via the network NW.
The three-dimensional display device 100 displays the received video in a three-dimensional manner. On the other hand, the two-dimensional display device 200 displays the received video in a planar manner.
Here, using the example in which the first point is the current location and the second point is a remote location, and the image of the user C at the second point is stereoscopically displayed on the three-dimensional display device 100 at the first point, explain.

Next, an example of the three-dimensional display device 100 will be described with reference to FIG.
As shown in FIG. 2, the three-dimensional display device 100 includes a two-layer stacked display 11, a camera 12, a microphone 13, and a speaker 14. The two-layer stacked display 11 includes a front display element 16, a rear display element 17, and a backlight 18.
The two-layer stacked display 11 is a display that displays an image three-dimensionally using, for example, a DFD (Depth-Fused 3-D) illusion. The three-dimensional display device 100 receives information indicating an image taken by a camera mounted on the two-dimensional display device 200, and adjusts the luminance ratio of the front display element 16 and the rear display element 17 based on the information. Then, a three-dimensional display object is displayed.
The display mechanism using the DFD illusion phenomenon will be described later.

Next, an example of the two-dimensional display device 200 will be described with reference to FIG.
As shown in FIG. 3A, the two-dimensional display device 200 includes a two-dimensional image display 21, a camera 22, a distance sensor 23, and a cold mirror 24.
The two-dimensional image display 21 is a device that displays a commonly used planar image, and for example, a liquid crystal display device can be used.
For example, the camera 22 is attached at a position where the face of the user who views the two-dimensional image display 21 can be photographed from the front.
The distance sensor 23 is, for example, a distance image sensor that measures a distance from a measurement target existing in the measurement area, and is an upper part of the camera 22 and is a measurement target from the display screen of the two-dimensional image display display 21. It is attached at a position where the distance to the user's face can be measured. A cold mirror 24 is attached to the front surface of the distance sensor 23 at an angle that intersects the surface direction of the display screen of the two-dimensional image display 21 at 45 degrees. This distance image sensor performs distance measurement using, for example, the TOF (Time-of-Flight) method. Specifically, using a light source such as infrared rays, the reflected light from the object based on the light from the light source is measured, and the flight time (delay time) of light and the speed of light (3 × 108 m) until reaching the sensor. / S), the distance to the measurement object is obtained, and the distance between the sensor and the measurement object in real space is acquired as an image.
This distance sensor 23 can measure the distance corresponding to each part according to the unevenness of the face of the user who is the subject of photographing. For example, the distance sensor 23 can measure the distance to the user's nose, the distance to the user's eyes, the distance to the user's mouth, and the like. In addition, according to the relative positional relationship between the camera 22 and the distance sensor 23, the distance from the object detected by the distance sensor 23 is associated in advance with an image area in the screen captured by the camera 22. Therefore, the distance information measured by the distance sensor 23 can indicate the orientation of the user's face with respect to the display screen of the two-dimensional image display display 21.

FIG. 3B shows a state in which the two-dimensional image display 21 shown in FIG. 3A is viewed from the top with respect to the orientation of the user's face.
As shown in FIG. 3B, the user is looking at the center from the front in the horizontal direction (X) on the two-dimensional image display 21. In this case, the azimuth (inclination angle) indicating the orientation of the user's face is 0 degrees. Note that the present invention is not limited to this example, and the state in which the front of the face is opposed to the surface direction of the display screen of the two-dimensional image display 21 is a state in which the face inclination angle is 0 degree. This is because the human nose is located at the center of the face and the eyes are approximately equidistant. For this reason, when the distance to both eyes is within the range where it can be determined that the distance to both eyes is equal to the position of the human nose, the face inclination angle is 0 degree.

On the other hand, as shown in FIG. 3C, the user faces his / her right direction with respect to the surface direction of the display screen of the two-dimensional image display 21. Here, since the orientation of the user's face is rotated to the right of the user by the azimuth angle α, the inclination angle of the user's face is + α. In this way, when the face is not facing the front with respect to the camera 22 and is directed to the right by the azimuth angle α, the distance between the parts of the user's face changes. That is, the distance between both eyes is not the same as the position of the user's nose, and the distance between the right eye and the nose is longer than the distance between the left eye and the nose, and is detected from the user's face that is the measurement target of the distance sensor 23. Distance becomes asymmetric about the nose. The tilt angle of the face can be calculated according to the ratio of the asymmetry.
Further, when the orientation of the user's face is rotated leftward by the azimuth angle α, the tilt angle of the user's face is −α. Also in this case, the tilt angle of the face can be calculated according to the ratio at which the distance detected from the face of the user that is the measurement target of the distance sensor 15 is asymmetric.
As described above, the process of calculating the face inclination angle based on the distance information measured by the distance sensor 23 is a process executed by the face inclination information generation unit described below.

Next, a display mechanism using the DFD illusion phenomenon will be described with reference to FIGS.
FIG. 4 is a schematic diagram illustrating a schematic configuration of an example of the two-layer stacked display display 11. As shown in FIG. 4, the front display element 16 and the rear display element 17 are, for example, transmissive liquid crystal panels, each having a plurality of pixels. The two-layer stacked display 11 is configured to transmit light from the backlight 18 to the transmissive panel in which two transmissive display elements (front display element 16 and rear display element 17) are stacked in parallel. Permeate from behind. Note that the user can view the displayed image in three dimensions by viewing the two-layer stacked display 11 from the front display element 16.
The front display element 16 and the rear display element 17 can be constituted by, for example, a transmissive liquid crystal display. If the transmissive type is used, a display element (organic EL (Electro-Luminescence), inorganic EL, liquid crystal display element) is used. As long as it is anything, it can be used.

FIG. 5 is a diagram for explaining a stereoscopic display object (three-dimensional stereoscopic image) displayed on the two-layer laminated display 11. As shown in FIG. 5, the transmissive front display surface 16 </ b> A of the front display element 16 and the transmissive rear display surface 17 </ b> A of the rear display element 17 are arranged in parallel with a surface interval L therebetween. An image displayed on the transmissive front display surface 16A of the front display element 16 is referred to as a front display image 16B, and an image displayed on the transmissive rear display surface 17A of the rear display element 17 is referred to as a rear display image 17B.
When the front display image 16B and the rear display image 17B are displayed on the front display element 16 and the rear display element 17, respectively, a three-dimensional display object (that is, perceived by an observer) between the front display image 16B and the rear display image 17B. 3D display object) O is formed. Note that the observation viewpoint (usually, the center point between the eyes of the observer) P is as illustrated.

  A two-dimensional image obtained by projecting the three-dimensional display object O from the direction of the observation viewpoint P onto the transmissive front display surface 16A and the transmissive rear display surface 17A is a pattern of the front display image 16B and a pattern of the rear display image 17B, respectively. It is said. Each two-dimensional luminance is distributed according to the display position (depth position: Z) of the stereoscopic display object O. When the front and rear display surfaces are transmissive display surfaces, the luminance distribution according to the display position (depth position: Z) of the stereoscopic display object O is the display position (depth position) of the stereoscopic display object O. : Distributes the transmittance according to Z), and means to change the luminance seen by the observer.

That is, since the depth position changes continuously by changing the ratio of the luminance of the image between the front display element 16 and the rear display element 17 according to the depth position of the stereoscopic display object O to be displayed, the viewer can The stereoscopic display object O can be perceived.
For example, when it is desired to display the stereoscopic display 19 at a position close to the front display element 16, the luminance ratio of the front display element 16 is increased and the luminance ratio of the rear display element 17 is decreased. On the other hand, when the stereoscopic display object O is to be displayed at a position close to the rear display element 17, the luminance ratio of the front display element 16 is decreased and the luminance ratio of the rear display element 17 is increased. In addition, when it is desired to display the three-dimensional display object O in the vicinity of the middle between the front display element 16 and the rear display element 17, the luminance ratio of the front display element 16 and the luminance ratio of the rear display element 17 are made equal.
Thereby, the person who saw the two-layer laminated display 11 from the front display element 16 side can recognize the depth of the three-dimensional display object O.

Note that a three-dimensional display device that displays a three-dimensional display object using the DFD illusion phenomenon is described in the following documents, and the display of the three-dimensional display object by the three-dimensional display device 100 is described in these documents. It can be realized by using existing technology.
<References>
(1) “3D display without depth fusion glasses” Hideaki Takada P1-8
(2) JP 2008-42745 A

  Next, an example of the configuration of the three-dimensional display device 100 and the two-dimensional display device 200 according to the present embodiment will be described with reference to FIG. In the present embodiment, an example will be described in which the two-dimensional display device 200 on the remote location side displays the video image of the user C at the second location on the 3D display device 100 on the current location side of the first location. To do.

  As shown in FIG. 6, in addition to the above-described configuration, the two-dimensional display device 200 on the remote site side includes a face shape generation unit 203, a texture information acquisition unit 204, a face inclination information generation unit 205, and a three-dimensional face model generation. Unit 206, communication unit 207, and microphone 208.

  For example, the camera 22 photographs the user C who is interacting with the users A and B at the first point while viewing the display screen of the two-dimensional image display 21 in front of the two-dimensional display device 200. The camera 22 outputs face image data indicating an image (image of the face of a remote participant) of the user C, who is an image capturing target, to the face shape generation unit 203 and the texture information acquisition unit 204, respectively. In this embodiment, the face image data is an image including the face of the user C. However, the present invention is not limited to this, and may be image data including at least one human face.

The distance sensor 23 is a sensor that measures position information of an imaging target, and measures, for example, the distance in the depth direction Z between the distance sensor 23 and the face of the user C. For example, as shown in FIG. 1, when the user C faces the two-dimensional display device 200, the distance in the depth direction from the vicinity of the face of the user C is indicated by the face depth distance information R1. As described above, the face depth distance R1 includes the distances to parts (nose, eyes, mouth, contour, hair, head, etc.) included in the face near the face.
Therefore, the distance sensor 23 measures the distance in the depth direction Z with respect to the user C to be imaged, measures the face depth distance information R1 as the distance in the depth direction Z from the vicinity of the face of the user C, and this face depth distance. The distance information including the information R1 is output to the face shape generation unit 203 and the texture information acquisition unit 204, respectively.

  The face shape generation unit 203 generates face shape information indicating the three-dimensional face shape of the user C based on the distance information input from the distance sensor 23. For example, based on the distance information between the closest measurement object and the distance sensor 23 among the measurement objects existing in the measurement area of the distance sensor 23, the face shape generation unit 203 performs unevenness of the measurement object. Acquire information indicating the shape. Note that the face shape generation unit 203 determines that the measurement object having the uneven shape indicating the characteristic of the human face is a human face based on the feature indicated by the uneven shape of the measurement object indicated by the acquired information. It may be. The face shape generation unit 203 generates face shape information indicating the shape of a human face based on the acquired information indicating the uneven shape of the measurement object.

The texture information acquisition unit 204 acquires texture information (information indicating skin color, eye color, hair color, and the like) about the human face image based on the face image data input from the camera 22 and obtains a three-dimensional face model. The data is output to the generation unit 206. For example, the texture information acquisition unit 204 performs face detection processing on face image data and detects a face image. The texture information acquisition unit 204 analyzes the image data corresponding to the detected face image, and acquires texture information. This texture information is information that expresses a partial change that feels the visual color and brightness uniformity of the face surface and the strength of tactile specific force. This is information indicated by texture coordinates corresponding to the vertical direction Y and the depth direction Z.
As the face detection method, for example, pattern recognition that detects an image region that approximates pattern information as compared with pattern information indicating a predetermined person's face as a face image can be used.

Based on the distance information input from the face inclination information generation unit 205 and the distance sensor 23, the azimuth angle at which the face of the person faces the display screen of the two-dimensional image display display 21 or the imaging surface of the camera 22 is determined. The face inclination information shown is calculated. As described above, the face inclination information generation unit 205 detects the azimuth indicating the orientation of the user's face according to the ratio of the distance between the parts detected from each part of the face to be measured being asymmetric. Then, the face tilt information is calculated.
Note that the face inclination information generation unit 205 may be configured to input information indicating the face of a person included in the face image data obtained by the analysis by the face forming unit 203. The face inclination information generation unit 205 estimates the position in the real space where the face of the person exists based on the information indicating the face, and distance information measured from the part corresponding to the position in the real space Based on, face inclination information indicating the azimuth angle of each person's face is calculated.

  The three-dimensional face model generation unit 206 generates three-dimensional face model data based on the texture information input from the texture information acquisition unit 204 and the face shape information input from the face shape generation unit 203. This 3D face model data pastes the texture (skin color, eyes, hair color, etc.) indicated by the texture information on the 3D face model (showing the uneven shape of the face) indicated by the face shape information. Data.

  The communication unit 207 associates the 3D face model data input from the 3D face model generation unit 206 with the face inclination information input from the face inclination information generation unit 205 and transmits the image information to the 3D display device 100 as video information. To do. The communication unit 207 inputs audio information indicating the audio acquired by the microphone 208 and transmits the audio information to the 3D display device 100 in association with the video information.

The three-dimensional display device 100 includes a communication unit 101, a display control unit 103, a panel drive unit 104, and a backlight drive unit 105 in addition to the above-described configuration.
The communication unit 101 is connected to the 2D display device 200 via the network NW and receives audio information and video information transmitted from the 2D display device 200.
The speaker 14 inputs the audio information received by the communication unit 101 and generates audio based on the audio information.

  The display control unit 103 inputs the video information received by the communication unit 101 and causes the two-layer stacked display display 11 to display a video based on the video information. The display control unit 103 is a front display element for causing the front display element 16 of the two-layer stacked display 11 to output a video based on the video information based on the luminance signal or the face tilt angle information on the video signal included in the video information. The driving signal and a rear display element driving signal for causing the rear display element 17 of the two-layer stacked display display 11 to output an image based on the video information are output to the panel driving unit 104. Further, the display control unit 103 outputs a backlight drive signal for controlling the luminance of the backlight 18 based on the luminance signal included in the video information to the backlight driving unit 105.

The panel driving unit 104 drives the front display element 16 and the rear display element 17 based on the input front display element drive signal and rear display element drive signal.
The backlight drive unit 105 drives the backlight 18 based on the input backlight drive signal.
As described above, the two-layer stacked display 11 uses the DFD illusion phenomenon, based on the fusion of the front display element 16 and the rear display element 17 in the depth direction and the continuous depth expression based on the luminance ratio. Display a 3D image.

The processing of the display control unit 103 will be described with reference to FIG.
For example, when the face of the user C faces the camera 22 of the two-dimensional display device 200 in an orientation corresponding to the azimuth angle + α, face tilt information indicating that the tilt angle of the user's face is + α is displayed on the display control unit. 103 is input.
Note that the three-dimensional face model data of the video information input to the display control unit 103 is model data facing the right side of the person so as to be photographed by the camera 22 as illustrated.
That is, the display control unit 103 inputs video information including face inclination information indicating the inclination angle + α of the user's face and three-dimensional face model data indicating the face of the user C facing the right side of the person as illustrated. To do.

  The display control unit 103 corrects the three-dimensional face model data based on the face tilt information, and generates three-dimensional model data of the front face when viewed from the front direction. For example, the display control unit 103 converts model data facing the right side of a person into model data facing the front.

  The display control unit 103 generates front image data and rear image data by distributing luminance in proportion to the depth of the model data facing the front. Specifically, the display control unit 103 determines a luminance value according to the distance detected by the distance sensor 23, and in the front image data, the luminance is increased with respect to an image portion where the detected distance is closer. Then, the luminance is lowered for the image portion where the detected distance is farther. For this reason, an image is not displayed in the farther image portion by setting the luminance value to 0. On the other hand, for the front image data, the display control unit 103, on the contrary, lowers the luminance for the image portion where the detected distance is closer, and the luminance for the image portion where the detected distance is farther. To increase.

The display control unit 103 corrects the front image data and the rear image data based on the face tilt information. The display control unit 103 performs correction to shift the overlapping angle of the front image data and the rear image data according to the azimuth angle (face inclination angle) of the face indicated by the face inclination information.
For example, the display control unit 103 determines the orientation of the user photographed by the camera 22 of the remote two-dimensional display device 200 based on the azimuth angle indicated by the face tilt information. As shown in FIG. 1, the face of the user C on the remote site side is facing the user B on the current location side in the screen displayed on the two-dimensional image display 21 of the two-dimensional display device 200. To do. In this case, the display control unit 103 corrects the front image data and the rear image data so as to be a face in front of the user C as a display target when the two-layer stacked display 11 is viewed from the position where the user B is present.

This correction will be specifically described with reference to FIG. FIG. 8 shows the relationship between the two-dimensional image display 21 of the two-dimensional display device 200 on the remote side and the user C, and the relationship between the two-layer stacked display 11 of the three-dimensional display device 100 on the current location side and the users A and B. It is a figure which shows this in the state seen from the top.
As shown in FIG. 8, the face of the user C on the remote site side is rotated to the right of the user by the azimuth angle α with respect to the two-dimensional image display display 21 and the camera 22 of the two-dimensional display device 200. Therefore, the display control unit 103 determines, based on the face tilt information, that the orientation of the user photographed by the camera 22 of the remote two-dimensional display device 200 is the azimuth angle + α.

The display control unit 103 performs correction to relatively shift the front display image and the rear display image by the azimuth angle + α indicated by the face tilt information. In other words, as shown in the figure, the display control unit 103 corrects the front display image and the rear display image so that the position where the user B is estimated is overlapped on the overlapping line so that the image contents match. In other words, in the front display image FD that is a part of the front face indicated by the three-dimensional model data of the front face and the rear display image RD that is the other part, for example, a portion corresponding to the apex of each nose is the nose portion N1. , N2. The display control unit 103 controls the front display image 103 so that the nose portion N1 of the front display image FD is positioned on a straight line connecting the nose portion N2 of the rear display image RD and the position where the user B is estimated to be present. The display positions of the display image FD and the rear display image RD are relatively corrected.
Note that the display control unit 103 can realize the correction as described above by calculating the correction amount d to be corrected according to the following equation.

d = L · tan (α) (1)

Here, d represents the amount of deviation to be corrected, and α represents the inclination angle of the user's face. L represents the distance between the front display element 16 and the rear display element 17 and is equal to the thickness of the state in which the front display element 16 and the rear display element 17 are stacked.

That is, the display control unit 103 performs correction to relatively shift the front display image FD and the rear display image LD by the amount of slip d calculated according to the azimuth angle indicated by the face tilt information.
Accordingly, when the user B looks at the two-layer stacked display display 11, it can be recognized that the line of sight is the same as the user C. When the user A looks at the two-layer stacked display display 11, Can be recognized as being inclined toward the user B.

  Next, an example of the display method according to the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart for explaining an example of the display method according to the present embodiment. Here, a display method for displaying image data captured by the two-dimensional display device 200 on the remote location side on the three-dimensional display device 100 on the current location side will be described as an example.

First, processing of the two-dimensional display device 200 on the remote site side will be described.
The camera 22 of the two-dimensional display device 200 on the remote side photographs the user C in front of the display screen of the two-dimensional image display display 21. The camera 22 acquires face image data indicating a video image of the user C, who is the imaging target, and outputs it to the face shape generation unit 203 and the texture information acquisition unit 204 (step ST1).
Next, the texture information acquisition unit 204 acquires texture information about a human face image based on the face image data input from the camera 22, and outputs the texture information to the three-dimensional face model generation unit 206 (step ST2).

  On the other hand, the distance sensor 23 measures the position information of the imaging target at the same timing when the camera 22 captures the user C (step ST3). The distance sensor 23 measures the face depth distance information R1 as the distance in the depth direction Z from the vicinity of the face of the user C, and the distance information including the face depth distance information R1 is obtained from the face shape generation unit 203 and the texture information acquisition unit. 204, respectively.

  Next, the face shape generation unit 203 generates face shape information indicating the three-dimensional face shape of the user C based on the face image data input from the camera 22 and the distance information input from the distance sensor 23. The result is output to the three-dimensional face model generation unit 206 (step ST4).

  Then, the 3D face model generation unit 206 generates 3D face model data based on the texture information input from the texture information acquisition unit 204 and the face shape information input from the face shape generation unit 203 (step ST5).

  Next, the face inclination information generation unit 205 calculates face inclination information indicating the azimuth angle of the person's face with respect to the display screen of the two-dimensional image display 21 based on the distance information input from the distance sensor 23. To do. The face inclination information generation unit 205 detects, for example, an azimuth angle + α indicating the orientation of the user's face according to a ratio at which the distance detected from each part of the face to be measured is asymmetric, Face tilt information indicating the angle + α is calculated (step ST6).

  Then, the communication unit 207 associates the 3D face model data input from the 3D face model generation unit 206 with the face tilt information input from the face tilt information generation unit 205 as video information and associates the 3D face model data with the 3D face model generation unit 206. Send to. The communication unit 207 inputs audio information indicating the audio acquired by the microphone 208 and transmits the audio information to the 3D display device 100 in association with the video information.

Next, the process proceeds to the processing of the three-dimensional display device 100.
The communication unit 101 of the 3D display device 100 receives audio information and video information transmitted from the 2D display device 200 via the network NW, and outputs them to the display control unit 103.

  The display control unit 103 corrects the 3D face model data, which is video information, based on the face tilt information, and generates 3D model data of the front face when viewed from the front direction. Then, the display control unit 103 distributes the luminance proportional to the depth indicated by the model data for the three-dimensional model data of the front face, and generates front display image data and rear display image data (step ST7).

Next, the display control unit 103 determines that the orientation of the face of the user C taken by the camera 22 of the remote two-dimensional display device 200 is the azimuth angle + α based on the face tilt information.
Specifically, the display control unit 103 determines whether or not the azimuth angle indicating the face direction is 0 degrees based on the face tilt information. If not, the display controller 103 performs the following correction. Judging. On the other hand, it is determined that the following correction is not performed in which the azimuth angle indicating the face direction is 0 degrees.

  Here, since the azimuth angle + α indicated by the face tilt information is not 0 degrees, the display control unit 103 performs correction to relatively shift the front display image and the rear display image by the azimuth angle + α, and the corrected front display image data The corrected rear display image data is output to the front display element 16 and the rear display element 17, respectively (step ST8).

  Then, the front display element 16 displays the front display image at the display position corrected based on the input corrected front display image data (step ST9). The rear display element 17 displays the rear display image at the display position corrected based on the corrected rear display image data to be input (step ST10).

As described above, the display control unit 103 performs correction to shift the overlapping angle of the front image data and the rear image data according to the angle (azimuth angle) of the face indicated by the face inclination information.
Accordingly, in the front display image FD that is a part of the front face indicated by the three-dimensional model data of the front face and the rear display image RD that is the other part, for example, the nose portions N1 and N2 corresponding to the vertices of the nose, respectively. Is displayed at a position that is connected with a position estimated to be the user B by a straight line.
Therefore, the user B located on this straight line can recognize that the user C is looking at him / her, and the user A is recognized as if the face of the user C is looking at the user B. Can do.

Further, as described above, the three-dimensional display device according to the present embodiment uses the directivity of the two-layer stacked display display 11 and the control of the direction due to the image shift between the front display element 16 and the rear display element 17. A fine direction can be expressed.
In addition, by using the three-dimensional display device according to the present embodiment in the video conference system, the user can intuitively view the direction of the face of the conference participant displayed as the video on the display. You can make people understand and understand who you are talking to correctly.

In addition, this invention is not restricted to the said structure, For example, the following structures may be sufficient.
For example, when the shift amount d between the front display screen and the rear display screen is larger than a predetermined threshold value, the unity of the front display screen and the rear display screen of the stereoscopic display object displayed in a three-dimensional manner is lost. It cannot be seen as a three-dimensional display object. Therefore, even if the threshold value is set in advance and the shift amount d calculated by the display control unit 103 is equal to or larger than the threshold value, the turntable that supports the two-layer stacked display 11 so as to rotate is operated. Good. In this way, by using the process of rotating the turntable, it is possible to turn the line of sight over a wide range, and the number of users participating in the video conference increases, and the face orientation covers a wide range. be able to.
The present invention is not limited to the above-described configuration, and the display device on the remote location side is not limited to the two-dimensional display device 200 and may be a three-dimensional display device having the same configuration as the three-dimensional display device 100.

  In the embodiment described above, a program for realizing the functions of the three-dimensional display device 100 and the three-dimensional display device 200 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is stored in a computer system. The encoding process and the decoding process may be performed by reading and executing the program. Here, the “computer system” may include an OS and hardware such as peripheral devices. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  DESCRIPTION OF SYMBOLS 100 ... Three-dimensional display apparatus of the present location side, 11 ... Two-layer laminated display, 16 ... Front display element, 17 ... Rear display element, 18 ... Backlight, 14 ... Speaker DESCRIPTION OF SYMBOLS 101 ... Communication part 103 ... Display control part 104 ... Panel drive part 105 ... Backlight drive part 200 ... Two-dimensional display apparatus of a remote place, 22 ... Camera 23 ... Distance sensor 203 ... Face shape generation unit 204 ... Texture information acquisition unit 205 ... Face tilt information generation unit 206 ... Three-dimensional face model generation unit 207 ..Communication unit, 208 ... Microphone

Claims (4)

An input unit for inputting azimuth information indicating the orientation of the face of the display target with respect to the camera that captures the display target, and three-dimensional face model data obtained by the camera capturing the display target ;
A display unit that includes a first display element and a second display element that are arranged to face each other, and that displays the display object in three dimensions;
Based on the orientation information, the 3D face model data is converted into a 3D face model of the front face when the display object is viewed from the front direction, and the first display element displays the 3D face model of the front face. second generating and displaying an image, observation have the azimuth information so that viewed from the position am with the face in front of a person which is the display target, wherein the first display image second display device displays the and relatively shifting correction and said second display image and the first display image in displacement amount based, a display control unit for displaying on the first display element and the second display element,
A display device comprising: Before Symbol display control unit,
Against 3-dimensional face model of the frontal face, by performing distribution of brightness proportional to the depth of the face of the display object where the orientation information indicates, that generates the second display image and the first display image the display device according to claim 1, wherein the this. A camera for photographing the display object and outputting image data;
A distance sensor that measures the distance between the display object and the camera and outputs distance information;
A three-dimensional face model generation unit that generates the three-dimensional face model data based on the image data ;
And azimuth information generating unit that generates the azimuth information on the basis of said distance information,
The display device according to claim 1, further comprising: The display target is three-dimensionally displayed on a second display device placed at a second point different from the first point based on information indicating the display subject photographed by the camera of the first display device placed at the first point. In the display method to display,
The first display device is
Photographing the display object with the camera to obtain image data representing the face of the person who is the display object;
Measuring the distance between the display object and the camera to obtain distance information;
Generating three-dimensional face model data based on the image data;
Generating azimuth information indicating the orientation of the face of the display target with respect to the camera that captures the display target based on the distance information;
And transmitting said azimuth information and the 3-dimensional face model data to the second display device,
The second display device
Receiving and the azimuth information and the 3-dimensional face model data,
Converting the three-dimensional face model data into a three-dimensional face model of a front face when the display object is viewed from the front direction based on the orientation information;
As a display image displayed by a display unit that includes a first display element and a second display element that are arranged to face each other and displays the display target in a three-dimensional manner, the first display from the three-dimensional face model of the front face Generating a first display image displayed by the element and a second display image displayed by the second display element ;
The observation sees from the position am with the display is the object relatively shifting correction and said second display image and the first display image in shift amount based on the orientation information so as to face the front of the person And displaying on the first display element and the second display element;
A display method comprising:

Images