JP6157077B2 - Display device with camera - Google Patents

Description

  The present invention relates to a camera-equipped display device suitably used for a videophone or a video conference.

  Matching the user who is viewing the display device with the line of sight of the other person's image displayed on the display device is an important element for realizing a natural sense of dialogue in videophones and video conferences. . Conventionally, in a videophone system or a video conference system, there has been a technique for photographing a user from the display surface direction of a display device and matching the line of sight between the users. Furthermore, as shown in Patent Document 1, in a videophone system that communicates image data between a plurality of terminals, a user A who changes the position while gazing at a display screen is imaged from the front and the imaged user is captured. There is known a technique of processing as an image to a communication partner so that the image of the image occupies almost the center of the display screen. According to this technology, the user A and the line of sight are in good agreement when viewed from the communication partner, and the conversation does not give a sense of incongruity.

JP 2009-71573 A

  In videophones and videoconferences, increasing the shooting range of the camera and increasing the degree of freedom of the face position of the user facing the display device as much as possible leads to providing good usability. That is, not only when the user is facing the display device but also when the display device is viewed from a slight angle, it is important that the camera is within the shooting range.

  On the other hand, there is a user's consciousness that it does not want to capture various objects in the background of the user, and in the vicinity of the shooting range of the wide-angle lens, for example, a lens in which distortion is completely corrected Even so, since the user is distorted due to so-called wide-angle distortion (volume distortion), there is a demand for reducing the shooting angle of view of the camera.

  Setting a sufficiently wide and practical shooting range at the expected distance from the display device to the user and narrowing the shooting angle of view is nothing more than moving the camera backward from the display surface. . If the size of the imaging surface of the camera is fixed with respect to this, a camera provided with a lens having a longer focal distance is arranged behind the display surface (in the screen direction) and spaced apart.

  FIG. 17 is an explanatory diagram regarding this arrangement. FIG. 17A is an overhead view showing the relationship between the user and the shooting range of the camera when a wide-angle camera is placed close to the display surface of the display device (display device), and FIG. It is an overhead view which shows the relationship between the user when using the camera of a narrow angle of view, and the imaging range of a camera. In both figures, the distance between the display device and the user is the same.

In FIG. 17A, 201 is an organic EL display device in which the ratio of transmitted light is increased by adopting a passive matrix system, 202 is a camera, 225 is an optical axis of the camera 202, and 203 is an imaging range of the camera 202. Reference numeral 204 denotes a user's head, 205 denotes a shooting range of the camera 202 at the position of the eyes of the user 204, and 206 denotes a background shooting range. In FIG. 17B, 211 is a camera, 226 is an optical axis of the camera 211, 212 is a shooting range of the camera 211, 213 is a shooting range of the camera 211 at the position of the user's eyes, and 214 is a background shooting range. It is. The focal length of the camera 211 is longer than the focal length of the camera 202, and the shooting angle of view θ ₂₁₁ of the camera ₂₁₁ is smaller than the shooting angle of view θ ₂₀₂ of the camera 202.

  If the shooting ranges 205 and 213 at the user's eye position are set to the same length, the camera 211 having a narrow shooting angle of view needs to be further away from the user than the camera 202 having a wide shooting angle of view. It will be separated from the display surface 201. At this time, the background photographing range 214 by the camera 211 is narrower than the background photographing range 206 by the camera 202, and the background in FIG. 17B is simplified.

Further, as the shooting angle of view becomes wider, wide-angle distortion occurs, and there is a phenomenon that the user's head is unnaturally stretched and imaged in a short portion of the screen. FIG. 18 is an explanatory diagram thereof, in FIG. 18, reference numeral 221 denotes a photographing lens of the camera, 222a to 222e correspond to the user's head, 223a to 223e correspond to the user's heads 222a to 222e, respectively. 215 is an optical image projected onto 215. Although the size of the heads 222a to 222e of the user is constant, the size of the optical images 223a to 223e is not constant, and the size of the optical image increases as the distance from the optical axis 224 increases. The image will be stretched. This phenomenon is not caused by the distortion of the taking lens 221, but the subject is deep even in a well-corrected aberration-correcting taking lens capable of forming a planar lattice as a similar lattice image. And this phenomenon will occur. The only way to suppress such wide-angle distortion is to narrow the shooting angle of view. When this is applied to the state of FIG. 17, it is more desirable to use a camera 211 having a narrow shooting angle of view θ _{211 because} a good optical image without wide-angle distortion can be obtained. .

  However, when the camera is arranged away from the display surface of the display device and the background is simplified and the wide-angle distortion is suppressed, there is a serious problem that the line-of-sight matching degree is lowered.

  FIG. 19 is an explanatory diagram thereof. 230 is an image of the head of the communication partner. As shown in FIG. 19A, the image of the communication partner is displayed at the center of the display device 201 where the camera 202 is located as disclosed in Japanese Patent Application Laid-Open No. 2009-71573, so that the line of sight 227 is obtained. When the user looks at the communication partner, the camera behind the communication partner is also seen, and an image with the user's line of sight toward the camera can be acquired from the camera 202. However, as shown in FIG. 19B, when the camera 211 is away from the display surface of the display device 201, even if the communication partner image is displayed in the center of the display device 201, the user's line of sight is displayed. It is not possible to acquire an image that is suitable for the camera. Reference numeral 228 denotes a user's line of sight, and the line of sight 228 connects the image 230 of the communication partner's head from the center position of the eyes of the user's head 204. The camera 211 is behind the image 230 of the head of the communication partner, but is not close to it and is located away. Therefore, since the camera 211 is not located on the extension line of the line of sight 228, the image of the user 204 acquired by the camera 211 is not that the user has his line of sight toward the camera 211. As a result, when such a camera-equipped display device is used in a videophone system or a video conference system, there is a problem that the degree of coincidence of the line of sight is lowered, the sense of reality is impaired, and sufficient communication cannot be achieved.

(Object of invention)
An object of the present invention is to provide a display device with a camera that can obtain a sufficient degree of coincidence of line of sight even when the camera is arranged in the screen direction of the display means.

In order to achieve the above object, a camera-equipped display device of the present invention is a camera-equipped display device used in an image communication system for communicating image data with each other,
Display means for displaying an image from a communication partner;
A camera that images the user observing the display means from the screen direction of the display means;
Display position calculation means for calculating the shift amount of the display position of the face image of the communication partner on the display surface of the display means based on the position of the face image of the user in the acquired image of the camera;
The display means displays the image by shifting the calculated shift amount so as to be positioned on a straight line connecting the camera position of the user and the center position of both eyes of the user .

  According to the present invention, it is possible to obtain a sufficient degree of line-of-sight matching even when the camera is arranged in the screen direction of the display means.

It is a block diagram which shows the hardware structural example of the display apparatus with a camera which is Example 1 of this invention. 3 is a flowchart illustrating processing in the first embodiment. 3 is a flowchart illustrating a transmission process according to the first embodiment. 3 is a flowchart illustrating a reception process in the first embodiment. It is a figure which shows the camera image before and behind the trimming of the user B. It is a figure which shows the camera image before and behind the trimming of the user A. FIG. 5 is an overhead view for explaining a method for calculating a display position of a camera image in the first embodiment. It is a figure which shows the state where the user B faced the display apparatus from the front side of the user B. It is a figure which shows the state where the user B faced the display apparatus from the back side of the user B. It is a figure which shows the state which the user B shifted | deviated to the display apparatus from the front side of the user B. It is a figure which shows the state which the user B shifted | deviated sideways with respect to the display apparatus from the back side of the user B. It is a figure which shows the state which added the user A to FIG. 9A from the front side of the user B. It is a figure which shows the state which added the user A to FIG. 9B from the back side of the user B. It is a figure which shows the state which shifted the shift amount which calculated the face image of the user B of FIG. 9A from the front side of the user B. It is a figure which shows the state which shifted the shift amount which computed the face image of the user B of FIG. 9B from the back side of the user B. It is a figure which shows the mode of the eyes | visual_axis matching by Example 1 from the user's B front side. It is a figure which shows the mode of the eyes | visual_axis matching by Example 1 from the back side of the user B. FIG. It is a figure which shows the mode of the eyes | visual_axis matching by Example 1 when the user A and B have shifted | deviated from both sides from the front side of the user B. FIG. It is a figure which shows the mode of a line-of-sight match by Example 1 when the users A and B have shifted | deviated from both sides from the back side of the user B. It is a figure which shows the installation state of the some camera in Example 2 of this invention. Similarly, it is a figure showing the installation state of a plurality of cameras in Example 2. FIG. FIG. 10 is an overhead view for explaining a method for calculating a display position of a camera image in the second embodiment. It is a figure which shows the mode of the eyes | visual_axis matching by Example 2 when the user A and B have shifted | deviated from both sides from the user B front side. It is a figure which shows the mode of the eyes | visual_axis matching by Example 2 when the user A and B have shifted | deviated from both sides from the back side of the user B. It is explanatory drawing of the state which has arrange | positioned the camera close | similar to and spaced apart from the display surface of a display apparatus. It is explanatory drawing of wide angle distortion. It is an overhead view which shows the case where a user's eyes | visual_axis looks at a camera, and the case where it remove | deviates.

  The mode for carrying out the invention is as described in Examples 1 and 2 below.

  First, a hardware configuration example of a camera-equipped display device that is Embodiment 1 of the present invention will be described. As shown in FIG. 1, a display device 101 (104) serving as a display device provided in an image communication system that communicates image data with each other includes a display 132 that is a display means, A camera 102 (105) is provided for photographing a user facing the display surface. FIG. 1 is an explanatory diagram of a specific hardware configuration example of a display apparatus having an information processing function for executing processing of an acquired image of the camera 102, display control processing for the display 132, or data transmission / reception processing. is there. The camera 102 (105) images the user observing the display 132 from the back of the display 132 in the screen direction.

  A CPU (Central Processing Unit) 156 controls execution of various application programs. For example, the processor functions as a control unit that executes user face detection processing, cut-out processing of an acquired image based on user position information, distortion correction, display control processing for the display device 132, and data transmission / reception processing control. The memory 157 is used as a ROM (Read-Only-Memory) for storing a program executed by the CPU 156 and fixed data, a storage area for a program executed in the processing of the CPU 156 and a parameter that changes as appropriate in the program processing, and a work area. RAM (Random Access Memory) or the like.

  The HDD 158 can be used as a program storage area, and is a storage unit having a hard disk that can be used as a transmission / reception image data storage area.

  The codec 151 performs encoding (encoding) processing and decoding (decoding) processing of image data transmitted / received via a network. Since the image data has a large amount of information, it is compressed and transmitted with the data amount reduced.

  The network interface 152 functions as an interface with various communication networks such as the Internet and a LAN. The input interface 153 functions as an interface with input devices such as a mouse 137 and a keyboard 136. The user executes various mode settings by inputting data from the keyboard 136, for example.

  The AV interface 154 and the display interface 155 perform data input / output from AV data input / output devices such as the cameras 102 and 105, the microphone 134, and the speaker 135. Control information and data are transferred between the components via the PCI bus 159. These data transfer control and other various program controls are executed by the CPU 156.

  Next, the operation of the display device in a video phone or video conference will be described. In particular, this includes the operation when the display surface of the display device and the camera are not close to each other. 2 to 4 are flowcharts showing processing by the CPU 156.

  As shown in FIG. 2 as an overall flow, in this videophone system, first, in step S101, it is checked whether or not the videophone is activated by a user's operation. If activated, the process proceeds to step S102 to perform transmission processing, and after performing reception processing in step S103, the process returns to step S101 again to check whether the videophone is activated. Here, if it is activated, the transmission process and the reception process are repeated. If not activated, step S101 is repeated to wait for activation by the user's operation. In the following description, the flow of processing will be described focusing on image data, but it goes without saying that audio data as well as image data is transmitted and received by a known method to communicate with images and sounds.

  FIG. 3 is a flowchart showing details of the transmission process. As will be described later with reference to FIGS. 13A and 13B, the description will be made assuming that the user B faces the display apparatus 101. In the first step S201, an image is acquired from the camera 102 installed behind the display. In this camera image, user B is shown as shown in FIG. In FIG. 5, 160 is a camera image, and 161 is an image of the user B.

  In step S202, the data of the camera image 160 is analyzed to detect a person's face shown in the image.

  In step S203, it is determined whether or not a face has been detected. If a face has been detected, the process proceeds to the next step S204. If not, the process proceeds to step S205 while retaining the center position coordinates of the previous eyes. .

  In step S204, the center position coordinates of both eyes are newly calculated using the detected face feature information.

  In step S205, a part of the camera image 160 is trimmed based on the coordinates of the center position of both eyes calculated in the previous step S204. The trimming range 162 shown in FIG. 5A is set so that the center position of both eyes of the user B is slightly above the center of the trimming range 162. This position is determined so as to be the same as the positional relationship between the display device and the camera in the display device 104 on the other side. FIG. 5B shows a state of an image after trimming, where 163 is a camera image and 164 is an image of the user B.

  Information on the positional relationship between the display and the camera is communicated with each other when the videophone system is activated.

  In the subsequent step S206, distortion correction is performed on the trimming range 162 in order to remove distortion and wide-angle distortion of the optical system.

  In step S207, encoding is performed to reduce the amount of data when transmitting image data via the network.

  In step S208, the image data (163) having undergone trimming and distortion correction is transmitted to the other party of the videophone, and the transmission subroutine is terminated.

  FIG. 4 is a flowchart showing details of the reception process. As will be described later with reference to FIGS. 13A and 13B, the description will be made assuming that the user B faces the first display device 101 and further the user A faces the second display device 104. Proceed.

  An image of the user A is acquired by the camera 105 also on the second display device 104 side. FIG. 6A shows an image by the camera 105, where 165 is a camera image, 166 is an image of the user A, and 167 is a trimming range. As shown in FIG. 6B, the trimmed image has the user image 169 positioned at the center of the camera image 168 and the user A's eyes positioned slightly above the center.

  In step S301, this image data (168) is received as an image on the communication partner side.

  In step S302, a decoding process is performed on the image data received in step S301.

  In step S303, the display position of the received camera image 163 is calculated based on the face position detected in step S102 of FIG. 2 and steps S201 and S202 of FIG. The CPU 156 that executes step S303 constitutes a display position calculation unit.

  FIG. 7 is a diagram for explaining a method of calculating the display position of the camera image 168, and is an overhead view of the user B, the display device, and the camera. The display position of the camera image 168 needs to be adjusted in both the vertical direction and the horizontal direction. However, since the calculation method is the same in any direction, the description here will be made taking the horizontal direction of the display as an example. Proceed.

  In FIG. 7, 301 is a display device of the display apparatus 101, 311 is a camera installed behind the display device 301, 326 is an optical axis of a photographing lens provided in the camera 311, 304 is a head of the user B, 328 Is the line of sight of user B. The camera 311 images the range of the angle of view θ indicated by the region 312 around the optical axis 326. If the user's B line of sight 328 faces the camera 311 as shown in FIG. 7, the image of the user B captured by the camera 311 points the line of sight 328 toward the camera 311.

  In this way, in order for the user B's line of sight 328 to face the camera 311, the face image of the other party, that is, the face image of the user A is displayed at the position indicated by 330 on the display 301. It ’s fine. The user B naturally matches the line of sight 328 with the image of the user A, so that the line of sight 328 of the user B is directed to the camera 311. Since the user B does not necessarily face the camera 311, the optimum position of the user A's face image 330 is determined by the position of the head 304 of the user B.

The position of the face image 330 of the user A can be obtained as follows. If the shooting screen size on the assumed face distance is Wk, the screen size on the camera image is Wg (FIG. 5), and the face shift amount on the camera image is Sg (FIG. 5), the face shift amount Sk is
Sk = Wk × Sg / Wg (1)
It becomes. Further, assuming that the assumed face distance is Dk and the distance between the display surface and the camera is Dc, the shift amount Sh of the display image is
Sh = Sk × Dc / (Dk + Dc) = (Wk × Sg / Wg) × Dc / (Dk + Dc) (2)
It can be expressed.

When Wk / (Dk + Dc) is represented by the angle of view θ of the camera,
Wk / (Dk + Dc) = 2 × tan (θ / 2) (3)
It is.

Therefore, the shift amount Sh of the display image is
Sh = 2 × (Sg / Wg) × Dc × tan (θ / 2) (4)
It can be expressed as.

  As can be seen from the equation (4), the angle of view θ of the camera set as a design value of the display device, the distance Dc between the display surface and the camera, and the center position of both eyes calculated in step S204 in FIG. The optimal position of the face image 330 of the user A in the horizontal direction is determined by the screen size Wg on the camera image and the face shift amount Sg on the camera image as the information of the above. If the vertical direction is calculated by the same method and the face image 330 is displayed at a position where they are combined, the user B's line of sight is directed toward the camera 311 in both the vertical and horizontal directions.

  In order to avoid frequent movement of the position of the face image 330 of the user A, temporal hysteresis is added to the display position information to be used or moving average processing is added in step S304 shown in FIG. It is effective.

  Next, the state of line-of-sight matching in a videophone or videoconference will be described in detail. 8A to 13B are explanatory diagrams for this purpose. 8A to 13B, A is the first display device viewed from the back and the second display device is viewed from the front, and B is the first display. The state which looked at the apparatus from the front and the 2nd display apparatus from the back is represented.

  8A to 13B, 101 is a first display device, 102 is located adjacent to and behind the display surface of the first display device 101, and performs imaging based on light transmitted through the display surface. A first camera 103 and 103 ′ which is a part of one display device 101 is a user B. Reference numeral 104 denotes a second display device. Reference numeral 105 denotes a second display device 104 which is located adjacent to and behind the display surface of the second display device 104 and performs imaging based on light transmitted through the display surface. The second camera 106, 106 ′, 106 ″ is the image of the user B displayed on the second display device 104, and 107, 107 ′ is the user A, 108, 108 ′, Reference numerals 108 ″ and 108 ′ ″ denote images of the user A displayed on the first display device 101.

  The first camera 102 and the second camera 105 are disposed slightly above the center of the display surface of the display devices 101 and 104, respectively. This is so that the camera is positioned behind the eyes when an image of the face of the communication partner is displayed on the display surface.

  The first display device 101 and the second display device 104 are connected to each other via a network, and an image acquired by the first camera 102 is displayed on the second display device 104 and acquired by the second camera 105. The obtained image is displayed on the first display device 101.

  First, the case where one user is directly facing the display device will be described. 8A and 8B show a state in which the user B103 alone is viewing the display surface of the first display device 101 from the front. Since the user B 103 faces the first camera 102, the user B 103 points the line of sight 110 toward the first camera 102 on the image acquired by the first camera 102. As a result, the image 106 of the user B displayed on the second display device 104 has the line of sight 110 facing the front as shown in the figure.

  Next, consider a case where a single user deviates from the front of the display device and looks at the center of the display surface of the display device. FIG. 9A and FIG. 9B show a state where the user B 103 ′ alone is viewing the display surface of the first display device 101 obliquely from the front. In the figure, 103 ′ is an image of the user B and 106 ′ is an image of the user B displayed on the second display device 104. Since the user B 103 ′ is viewing the first camera 102 at an angle, the user B 103 ′ points the line of sight 111 toward the first camera 102 on the image acquired by the first camera 102. As a result, the image 106 ′ of the user B displayed on the second display device 104 is in the periphery of the display surface and has a line of sight on the front as shown in the figure.

  Here, a user A who is a communication partner of the user B is added and communication by videophone is considered. FIG. 10A and FIG. 10B show a state where the user B deviates from the front of the first display device 101 and looks at the center of the display surface of the first display device 101, and the user A who is a communication partner A state is shown in which the center of the display surface of the second display device 104 is viewed facing the second display device 104. In the figure, 107 is an image of the user A 107 and 108 is an image of the user A 107 displayed on the first display device 101.

  In order to communicate, the user A 107 looks at the image 106 ′ of the user B 103 ′ displayed on the second display device 104, so that the line of sight 112 becomes an arrow. As a result, on the image of the user A 107 captured by the second camera 105, the user A 107 does not point the line of sight 112 toward the second camera 105. That is, the image 108 of the user A displayed on the first display device 101 is deflected sideways without directing the line of sight to the front. Even if the user B looks at the image 108 of the user A, the user B is not looking at him and cannot get a sense of reality. On the other hand, when the user A sees the image 106 ′ of the user B, since this is an image looking at the front, the user can feel that his / her line of sight is facing and obtain a sufficient presence. I can do it.

  Such a situation where the lines of sight between the users do not match each other can be solved by shifting the position of the image and displaying it. 11A and 11B are explanatory diagrams showing a state in which the display position of the image is changed from the state shown in FIGS. 9A and 9B. In the figure, an image 106 ′ indicated by a broken line is an image of the user B before changing the display position, and 106 ″ indicates the display position in consideration of the distance between the camera and the display as described with reference to FIG. It is the image of the user B who changed. The image 106 ″ is displayed at a position where the eye of the user B 103 ′ approximately overlaps the second camera 105. The shooting angle of view of the first camera 102 is set to be wider than the display area in advance, and an area centered on the user B is trimmed from the image, whereby the eye position of the user B 103 ′ is set to the second position. It can be displayed at a position approximately overlapping the camera 105.

  Thus, if the position of the eyes of the user B103 ′ is displayed at a position that substantially overlaps the second camera 105, as shown in FIGS. 12A and 12B, the user A107 can communicate with the second camera 107 to communicate with the second camera 105. When viewing the image 106 ″ of the user B 103 ′ displayed on the display device 104, the line of sight 113 is directed to the second camera 105 as indicated by an arrow. As described above with reference to FIGS. 11A and 11B, the image 106 ″ of the user B 103 ′ is an image looking at the front, so that the user A 107 feels that his / her line of sight is facing himself / herself. Can get a sense of realism.

  On the image in which the second camera 105 captures the user A 107, the user A 107 directs the line of sight 113 toward the second camera 105, and thus the user A 107 displayed on the first display device 101. The image 108 ′ has the line of sight 113 facing the front. As a result, when the user B looks at the image 108 ′ of the user A, the user B feels that he / she is looking at himself / herself, and the user B can also obtain a sufficient presence.

  As described above, it has been described using FIG. 11A and FIG. 11B and FIGS. 12A and 12B that shifting the position of the image displayed on the second display device 104 increases the degree of line-of-sight matching. Similarly, by shifting the display position of the image displayed on the first display device 101, it becomes possible to always maintain the line-of-sight match even if the user on either side deviates from the front of the display device. . FIG. 13A and FIG. 13B are explanatory diagrams of this state, and 107 ′ is a user A who is deviating from the front of the second display device 104 and looking at the display surface of the second display device 104. 108 ″ shown is an image of the user A before changing the display position, and 108 ′ ″ is the image after changing the display position in consideration of the distance between the camera and the display as described with reference to FIG. The image of the user A, 114 is the line of sight of the user B, and 115 is the line of sight of the user A.

  First, the description starts from displaying the position of the eye of the image 106 ″ of the user B at a position approximately overlapping the second camera 105. When the user A views the image 106 ″ of the user B displayed on the second display device 104 for communication, the line of sight 115 faces the second camera 105 as indicated by an arrow. Therefore, the image 108 ″ of the user A is an image looking at the front.

  The shooting angle of view of the second camera 105 is set wider than the display area in advance, and an area centered on the user A 107 ′ is trimmed from the image, whereby the position of the eye of the user A is set to the first position. It is possible to display at a position approximately overlapping the camera 102. Using this processing, the image 108 ″ ″ is displayed at a position where the eye position of the user A substantially overlaps the second camera 102.

  Further, when the user B views the image 108 ″ ″ of the user A displayed on the first display device 101 for communication, the line of sight 114 becomes an arrow. At this time, since the image 108 ″ ″ of the user A is an image looking at the front, the user B can feel that his / her line of sight is facing himself and can obtain a sufficient sense of reality.

  On the image of the user B captured by the first camera 102, the user B is pointing the line of sight 114 toward the first camera 102, so that the image of the user B displayed on the second display device 104 is displayed. 106 ″ is also looking forward. As a result, when the user A sees the image 106 ″ of the user B, he / she feels that he / she is looking at himself / herself and deviates from the front of the second display device 104 to display the second display device 104. The user A who is looking at the surface can also obtain a sufficient presence. Therefore, both parties having a videophone conversation can get a sense of realism by matching their line of sight.

  The second embodiment of the present invention is a display device provided with a plurality of cameras so that the line of sight can be matched even when the other party's image is displayed in a region other than the central portion of the display.

  FIG. 14A and FIG. 14B are explanatory views showing an installation state of a plurality of cameras. In the figure, 301 is a first display device, 302 is a first camera group installed on the first display device 301, 304 is a second display device, and 305 is a second display device 304 installed on the second display device 304. It is the second camera group. Each of the first camera group 302 and the second camera group 305 includes 10 cameras.

  The camera closest to the position where the information display window for displaying the image of the other party of the videophone call is selected, and the image is transmitted to the other party.

  FIG. 15 is a diagram for explaining a method for calculating the display position of the camera image, and is an explanatory diagram of the user B, the display device, and the camera as viewed from above. The display position of the camera image needs to be adjusted in both the vertical direction and the horizontal direction. However, since the calculation method is the same in any direction, the description will be given here taking the horizontal direction of the display as an example. .

  In FIG. 15, 401 is a display unit of the display device 301, 411 is a camera installed behind the display unit 301, 426 is an optical axis of a photographing lens provided in the camera 411, 404 is a head of the user B, 428 Is the line of sight of user B. The camera 411 images the range of the angle of view θ indicated by a region 412 with the optical axis 426 as the center. Further, when the user B's line of sight 428 faces the camera 411 as illustrated in FIG. 15, the image of the user B captured by the camera 411 has the line of sight toward the camera 411.

  In this way, in order for the user B's line of sight 428 to face the camera 411, it is only necessary that the other party's face, that is, the face of the user A is displayed on the display 401 at the position indicated by 430. .

  By setting the shooting angle of view of the camera 305a (FIG. 16B) wider than the display area in advance and trimming the area centered on the user A307 ′ from the image, the eye position of the user A307 ′ can be changed to the camera. It can be displayed at a position approximately overlapping with 302a.

  Since the positional relationship between the user B's head and the camera is the same as that in the case where the camera shown in FIG. 7 is in the center, the shift amount Sh of the display image can be expressed by Expression (4). .

  In addition, for the image displayed on the other display device, the camera's shooting angle of view is set wider than the display area in advance, and the area centered on the user is trimmed from that image. Even if a person deviates from the front of the display device and looks at the display surface of the display device, the line-of-sight state can be maintained.

  16A and 16B show a state where the user B 303 ′ is looking at the periphery of the display surface of the first display device 301, and the user A 307 ′ is looking at the periphery of the display surface of the second display device 304. It shows the state. Information display windows 351, 352, and 353 are displayed on the first display device 301, and information display windows 354 and 355 are displayed on the second display device 305. Among these, the information display windows 351 and 354 are displayed by reducing the image of the other party of the videophone.

  Since the information display window 351 is displayed at a position overlapping one camera 302 a in the first camera group 302, the user B 303 ′ is the user A of the user A in the information display window 351 as indicated by the line of sight 314. At the same time as viewing the image 308 ′ ″, the user is looking in the direction of the camera 302a. On the image acquired by the camera 302a, the user B303 ′ points the line of sight 314 toward the camera 302a. As a result, the image 306 ″ of the user B displayed on the second display device 304 has the line of sight 314 facing the front as shown in FIG. 16A.

  Similarly, since the information display window 354 is displayed at a position overlapping with one camera 305 a in the second camera group 305, the user A 307 ′ uses the information display window 354 in the information display window 354 as indicated by the line of sight 315. At the same time as viewing the image 306 ″ of the person B, the direction of the camera 305a is viewed. On the image acquired by the camera 305a, the user A307 ′ points the line of sight 315 toward the camera. As a result, the image 308 ″ ″ of the user A displayed on the first display device 301 has the line of sight 315 facing the front as shown in FIG. 16B.

  In the first and second embodiments, the camera is arranged behind the display. However, the camera is arranged around the display, and a viewpoint without an actual camera is obtained from images from a plurality of cameras using view interpolation technology. You may generate a video that can be seen from. In this case, the screen direction of the display device is not limited to an arrangement in which the camera is behind the display surface as viewed from the user, but an arrangement in which the camera protrudes to the user side is also possible.

101 First display device 102 First camera 103, 103 ′ User B
104 Second display device 105 Second camera 107, 107 ′ User A
156 CPU
Sg User face image shift amount Sh Display position shift amount

Claims (5)

A display device with a camera used in an image communication system for communicating image data with each other,
Display means for displaying an image from a communication partner;
A camera that images the user observing the display means from the screen direction of the display means;
Display position calculation means for calculating the shift amount of the display position of the face image of the communication partner on the display surface of the display means based on the position of the face image of the user in the acquired image of the camera;
The display device with a camera , wherein the display means displays the image by shifting the calculated shift amount so as to be positioned on a straight line connecting the camera position of the user and the center position of both eyes of the user . The shift amount Sh of the display position of the communication partner of the face image is, the direction of which is perpendicular to the optical axis of the user of the camera from the optical axis of the user of the camera to the center position of the eyes of the face image of the user distance Sk, said from the face of the user of the user of the camera to the display surface of the display means of the user the direction of the optical axis of the distance Dk, from the user of the camera to the display surface of the display means of the user The display device with a camera according to claim 1, wherein the display device is calculated based on a distance Dc in an optical axis direction of the user 's camera.   The display device with a camera according to claim 1, wherein the position of the user's face image is represented by a shift amount when the face image is shifted to the center of the acquired image of the camera.   4. The display device with a camera according to claim 1, further comprising: a transmission unit configured to transmit a camera image in a state where the face image of the user is shifted to the center of the image by trimming. 5. A plurality of the cameras are provided,
The display position calculating means shifts the shift amount of the display position of the face image of the communication partner on the display surface of the display means to the position of the face image of the user in the acquired image of the selected camera among the cameras. The camera-equipped display device according to claim 1, wherein the display device is calculated based on the calculation result.

Images