JP2014233045A - Video display system and video display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain smooth communication by processing a display image on the basis of the positional relationship among a plurality of users and the direction of faces, at the execution of dialogs among a multiplicity of persons using a video communication system.SOLUTION: A main point device 100 and a subordinate point device 200 connected through a network calculate the position and the direction of faces in imaged images, and deform the images on the basis of the calculation result, to transmit to an opposite device. The device, on receiving the image, synthesizes and displays the image and a background image on the basis of the positional relationship and the direction of the faces of the users.

Description

  The present invention relates to video communication technology.

  Conventionally, a video conference system has been used as a system in which a plurality of users participate in discussions and conversations. In the video conference system, discussions and conversations are performed between a plurality of users by transmitting a user's video imaged by a single fixed camera to the other party. However, in such a method, there is a problem that it is difficult for the user's line-of-sight direction to be transmitted to the conversation partner. In order to solve such a problem, a technique has been proposed in which a user is photographed with a plurality of cameras from multiple directions so that the direction of the user's line of sight can be transmitted to a conversation partner and communication can be naturally performed (eg Non-Patent Document 1). In addition, a user having a window frame used as an interface between the user side and the conversation partner is photographed using a single camera, and the user's space is displayed in the window frame on the screen viewed by the conversation partner. A video communication technique for displaying has also been proposed (see, for example, Non-Patent Document 2).

Vertegaal, R., et al., “GAZE-2: conveying eye contact in group video conferencing using eye-controlled camera direction”, CHI ‘03, pp. 521-528. 2002. Shingo Fujita, Osamu Yoshino, “Three-dimensional movement of frame and effect of existence in superimposed video chat”, Information Processing Society of Japan

  In the conversation, it is important to know the direction of the other party's line of sight and the direction of interest, since it affects the ease of natural conversation and the smooth conversation by being able to grasp the interest of the other party. In a conventional video communication system with a large number of people, conversations with a partner who has recognized in advance or a video communication system has been used, and a one-on-one conversation in a sufficiently mastered state is easy. However, it is difficult to recognize a partner positioned other than the front position of the screen, the positional relationship between a plurality of users, and the line-of-sight direction.

  In the technique of Non-Patent Document 1, the user's line-of-sight direction is obtained by photographing the user with cameras installed in a plurality of directions and changing the direction of the window frame for displaying the user's video according to each user's line-of-sight direction. Is communicated to the other party. For this reason, the position of the camera to be photographed is fixed, and the communication environment can be realized only in a state where the positional relationship between users is known. Furthermore, in an environment using a small device such as a terminal device (for example, a smartphone or a tablet terminal), there is a problem that it is difficult to prepare a plurality of cameras. Further, in the technique of Non-Patent Document 2, by displaying the window frame on the screen, it is recognized that the user and the partner of the conversation are clearly present in different spaces. Furthermore, it has been difficult to convey clear instructions and line of sight due to the difference between the work area in the real space and the observation position on the screen. Therefore, there was a problem that smooth communication could not be achieved.

  In view of the above circumstances, an object of the present invention is to provide a technique that enables smooth communication.

  One aspect of the present invention is an acquisition unit that acquires the orientation of a user who is a conversation partner, a processing unit that processes the video of the user based on the acquired orientation, and a display that displays the processed video A video display system.

  One aspect of the present invention is the video display system described above, wherein the processing unit changes the transparency of an area of the video that does not include the user when the orientation is in a screen direction.

  One aspect of the present invention is the video display system described above, wherein the processing unit changes a color of a frame of the video when the orientation is in a screen direction.

  One aspect of the present invention is the video display system described above, in which the processing unit includes an area that does not include the user of the video and a frame of the area when the orientation is in a screen direction. The blur process is performed on the image.

  One aspect of the present invention is an acquisition step of acquiring a direction of a user who is a partner of a conversation, a processing step of processing the user's video based on the acquired direction, and a display for displaying the processed video And a video display method.

  According to the present invention, smooth communication can be achieved.

It is a block diagram which shows the system configuration | structure of the video display system in this embodiment. It is a figure which shows the specific method which deform | transforms an image based on the coordinate of a person's face position. It is a figure which shows the specific example of a face position deformation | transformation image. It is a figure which shows the specific example at the time of changing the transparency of a background area | region. It is a figure which shows the example which has arrange | positioned the image of the other party of dialogue in virtual space. It is a figure which shows the specific example which displays the frame which added the color around the image. It is a figure which shows the specific example of the image which blurred the background area | region and the area | region of the edge. It is a figure which shows a mode that the user in each base observes the object in a main point, and interacts using the video communication system by this embodiment. It is a sequence diagram which shows the flow of operation | movement of the video display system in this embodiment. It is a sequence diagram which shows the flow of operation | movement of the video display system in this embodiment.

  In the video display system of the present embodiment, the video of the other party changes in accordance with the direction in which the other party is paying attention. Therefore, it is possible to know the direction that the other party is paying attention to and to facilitate smooth communication. Hereinafter, details of one embodiment of the present invention will be described.

  FIG. 1 is a block diagram showing a system configuration of a video display system in the present embodiment. The video display system of the present invention is configured by connecting a master location device 100 installed at a master location and a slave location device 200 installed at a slave location via a network. The main point is a point having the image generation unit 106. The image generation unit 106 generates a background image shared between users. In each device (the main point device 100 and the slave point device 200), the background image generated by the image generation unit 106 and the video of each user are displayed. There are one or more slave points, and the image generator 106 is not included. In the following description, a user at a main point is described as user A, and a user at a slave point is described as user B.

  A functional configuration of the main point device 100 will be described. The main point device 100 includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, and the like connected by a bus, and executes the main point program. By executing the main point program, the main point device 100 includes an imaging unit 101, a face position acquisition unit 102, an image deformation unit 103, a communication unit 104, a region division unit 105, an image generation unit 106, a position / direction calculation unit 107, and a determination unit. 108, a processing unit 109, a video space generation unit 110, and a display unit 111. Note that all or part of each function of the main point device 100 may be realized by using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field programmable gate array (FPGA). . The main point program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. The main point program may be transmitted / received via a telecommunication line.

The imaging unit 101 captures an image including the user A at the main location and the background. The imaging unit 101 outputs the captured image to the face position acquisition unit 102.
The face position acquisition unit 102 acquires the coordinate value of the face position of the person (user A) from the image captured by the imaging unit 101. As a method for acquiring the position of the face, the technique described in Reference Document 1 below may be used. Reference 1: Paul Viola and Michael J. Jones. Rapid Object Detection using a Boosted Cascade of Simple Features. IEEE CVPR, 2001.

  The image deformation unit 103 generates a face position deformed image based on the acquired coordinate value of the face position of the person. Specifically, the image deformation unit 103 generates a face position deformed image by deforming the acquired face position coordinates so as to be emphasized. As a method for deforming the image, a deformation method such as a Bezier curve using the left end and the right end of the image as reference points and the coordinates of the face position as control points may be used.

Hereinafter, the processing of the image transformation unit 103 will be specifically described with reference to FIG. FIG. 2 is a diagram illustrating a specific method of deforming an image based on the coordinates of the person's face position.
FIG. 2A is a diagram illustrating an image in which a user is captured and a coordinate system. As shown in FIG. 2A, the left end of the image is A and the right end is B. A is the coordinate of the left end of the image, and is represented by A (x_A, y_A). The notation x_A indicates that A is a subscript of x. The notation y_A indicates that A is a subscript of y. B is the coordinate of the right end of the image and is represented by B (x_B, y_B). The notation x_B indicates that B is a subscript of x. The notation y_B indicates that B is a subscript of y. Note that x_A <x_B and y_A <y_B. In the coordinate system shown in FIG. 2A, the horizontal direction is represented as the x axis, the vertical direction is represented as the z axis, and the depth direction is represented as the y axis.

  The image transformation unit 103 uses the left end A and the right end B of the image as reference points, and uses one point in the image, for example, the center position of the face of a person in the image, or the midpoint of the left eye and right eye as control points 2 Deform the image along the next Bezier curve. The center coordinate of the human face determined from the image is represented by C1 (x_C1, z_C1). The notation x_C1 indicates that C1 is a subscript of x. The notation z_C1 indicates that C1 is a subscript of z.

  As shown in FIG. 2B, C is a point on a straight line that intersects the midpoint of the line segment AB at 45 degrees and 315 degrees on the xy plane. This is expressed as the x coordinate of the position of the person's face in the image. The image transformation unit 103 calculates y_C by substituting x_C1 for x_C in Expression (1). Note that the notation x_C indicates that there is C as a subscript of x. The notation y_C indicates that there is C as a subscript of y. Thereafter, the image deforming unit 103 calculates x_C by substituting the calculated y_C into Expression (1). The image transformation unit 103 calculates C (x_C, y_C) by performing such processing.

After calculating the x-coordinate and y-coordinate of the three points A, B, and C, the image transformation unit 103 performs a point P (x_P, y_P) on the secondary Bezier curve with A as the start point, B as the end point, and C as the control point. ) Is calculated based on Equation (2) and Equation (3). Note that the notation x_P indicates that P is a subscript of x. The notation y_P indicates that P is a subscript of y.

Note that t in the equations (2) and (3) is a parameter that increases from 0 to 1, and is represented by the equation (4). Further, X in Expression (4) corresponds to the position X of the image sequence in the captured image. The image deformation unit 103 deforms and arranges an image along a curve obtained by obtaining P corresponding to the position X of the image sequence.

FIG. 3 shows a specific example of the face position deformed image. As shown in FIG. 3, it can be seen that the image is deformed so that the face position of the person is emphasized.
After generating the face position deformed image, the image deforming unit 103 rotates the face position deformed image in accordance with the line of sight of the user A. Note that the method for deforming an image need not be limited to the above-described method. For example, by changing the image using the viewpoint position, eyeball position, head position, etc. of the person in the image, it is possible to reinforce the transmission of the person's position and the person's gaze direction.

The communication unit 104 transmits and receives data to and from the follower device 200. The communication unit 104 transmits, for example, the face position deformed image data generated by the image deforming unit 103 to the follower device 200. Further, for example, the communication unit 104 receives the face position deformed image data from the follower device 200.
The region dividing unit 105 performs region division on the face position deformed image generated by the image deforming unit 203 at another point (for example, the follower device 200). Specifically, the area dividing unit 105 divides the person area and the background area by taking the difference between the background area and the person area information acquired in advance from the received face position deformation image. To do. The person area is an area of a person captured in front of the image. The background area is an area other than the person area. A specific example of the background area is a wall. In addition, the region dividing unit 105 may determine a person region and a background region by a method of determining a person region such as the above-described Reference Document 1 and perform region division. The area dividing unit 105 outputs the person area image and the background area image to the processing unit 109.

  The image generation unit 106 captures a space as a main point with a camera in advance, and a background image (hereinafter referred to as “virtual space environment”) pasted on a hemispherical surface or polyhedron prepared so as to surround a person in the virtual space. Is generated. For the generation of the virtual space environment, it is possible to use a still camera image prepared in advance or an image taken in advance. Furthermore, real-time video can be used by installing a plurality of cameras in an environment shared by many users.

  The position / direction calculation unit 107 acquires the position and line-of-sight direction of the user's viewpoint with respect to the environment (background image) at each point. In addition, the position / direction calculation unit 107 acquires the position and direction of the display. As a method of acquiring a three-dimensional position, for example, there are a method of using a device that measures a three-dimensional position with a plurality of cameras, a device that measures a three-dimensional position using magnetism, and the like. Specifically, the position / direction calculation unit 107 calculates the position and line-of-sight direction of the user A's viewpoint with respect to the environment from the head position of the person in the image and the positions of both eyes of the person.

  For the origin (reference point) serving as a reference for the three-dimensional position, a point set by the user A at the start of the system may be used, or a preset value may be used as the reference point. The reference point can be reset even while the system is operating. The position of the viewpoint represents the distance between the reference point and the user and the direction of the user with respect to the environment. In addition, the position of the user's viewpoint and the line-of-sight direction with respect to the reference point may be acquired using the user or a three-dimensional position sensor attached around the user.

In addition, the position / direction calculation unit 107 acquires the display position of the main point (user A) with respect to the environment. The position of the display indicates the distance between the upper end, the lower end, the left end, and the right end of the display with respect to the reference point, and the direction of the display with respect to the reference point. For example, image processing, a position sensor, an ultrasonic wave, an infrared light sensor, or the like can be used for acquiring the position.
The position / direction calculation unit 107 detects which part of the communication environment the user (user A) is observing in accordance with the position of the display with respect to the position of the person, and the detected result is the video space generation unit 110 and the processing unit 109. And output to the follower device 200.

The determination unit 108 determines whether or not an observation target such as a person or an object exists with respect to the user B's line-of-sight direction calculated by the position / direction calculation unit 206 of another point (for example, a slave point). The observation target is a person participating in the conversation or an object listed on the conversation agenda.
The processing unit 109 processes the image captured by the user B based on the determination result of the determination unit 108. Specifically, when an observation target exists with respect to the user B's line-of-sight direction, the processing unit 109 processes the background region of the image in which the user B is captured. There are three processing methods as specific examples of the processing method. For example, there are a process for changing the transparency of the background area, a process for displaying a colored frame around the image, a process for blurring the frame around the background area and the image, and the like. On the other hand, when there is no observation target with respect to the user B's line-of-sight direction, the processing unit 109 does not perform processing.

  The user's interest and interest can be determined by the presence of an observation target in the user's line-of-sight direction. In addition, the degree of interest can be voluntarily manipulated by changing the degree of interest and the degree of interest by operating buttons, slide bars, etc. from the user side participating from a remote location.

Hereinafter, three processes will be described as specific examples of the process of the processing unit 109.
(1. Processing to change the transparency of the background area)
When there is an observation target such as a person participating in the conversation or an object listed on the conversation agenda, the processing unit 109 changes the transparency of the background area of the image of the conversation partner. Let Specifically, the processing unit 109 increases the transparency of the background area. By performing this process, it is possible to make the user feel as if the video of only the other party of the conversation is displayed in the virtual space. Thereby, the presence of the other party in the virtual space can be enhanced. This makes it easier to communicate. On the other hand, when there is no observation target for the line-of-sight direction of the conversation partner, the processing unit 109 does not change the transparency of the background area of the image of the conversation partner. Therefore, the video and background area of the conversation partner are displayed in the virtual space. FIG. 4 shows a specific example when the transparency of the background region is changed.

FIG. 4 is a diagram illustrating a specific example when the transparency of the background region is changed.
As shown in FIG. 4, the transparency increases in the order of FIG. 4 (A), FIG. 4 (B), and FIG. 4 (C). FIG. 4A is a diagram illustrating an image in which the transparency is not changed. In FIG. 4A, since the transparency is not changed, the conversation partner area and the background area are clearly displayed. FIG. 4B is a diagram illustrating an image in which the transparency of the background region is higher than that in FIG. Since FIG. 4B has higher transparency than FIG. 4A, the color of the background region is lighter than that of FIG. FIG. 4C is a diagram illustrating an image in which the transparency of the background region is higher than that in FIG. Since FIG. 4C has higher transparency than FIG. 4B, the color of the background region is lighter than that of FIG. 4B. As shown in FIG. 4, when the transparency is increased, the color of the background area is lowered, and it is possible to make the user feel as if the video of only the other party of the conversation is displayed in the virtual space.

FIG. 5 is a diagram illustrating an example in which images of a conversation partner are arranged in a virtual space.
As shown in FIG. 5, two images 10 and 20 are arranged in the virtual space. In the image 10, the conversation partner area and the background area are clearly displayed. Therefore, it feels as if another space exists in the virtual space. On the other hand, since the image 20 has a lighter background area than the image 10, it feels as if the video of only the other party of the conversation is displayed in the virtual space.

(2. Processing to display a colored frame around the image)
When there is an observation target with respect to the line-of-sight direction of the conversation partner, the processing unit 109 changes the color of the frame of the conversation partner image. On the other hand, when there is no observation target for the line-of-sight direction of the conversation partner, the processing unit 109 does not change the color of the frame of the image of the conversation partner. FIG. 6 shows a specific example in the case of changing the colored frame around the image.

FIG. 6 is a diagram illustrating a specific example of displaying a frame with a color around the image.
In FIG. 6, two images are arranged. The processing unit 109 changes the color of the frame of an image (for example, the image 30) arranged in the virtual space when there is an observation target with respect to the line-of-sight direction of the conversation partner. For example, the processing unit 109 changes the color of the frame of the image to green. For example, the processing unit 109 changes the color of the frame of the image to red. By performing such processing, the user can understand the state of the other party at a glance.

(3. Processing to blur the background area and the edge area)
When there is an observation target for the line-of-sight direction of the conversation partner, the processing unit 109 performs a blurring process on the background area of the conversation partner image and the edge of the background area. On the other hand, when there is no observation target for the line-of-sight direction of the conversation partner, the processing unit 109 does not perform the blurring process. FIG. 7 shows a specific example when the background area and the edge area are blurred.

FIG. 7 is a diagram illustrating a specific example of an image in which the background region and the edge region are blurred.
As shown in FIG. 7A to FIG. 7C, the blurring process is performed on the background area of the conversation partner image arranged in the virtual space and the edge of the background area. . By performing such processing, it is possible to increase the degree of joining between the conversation partner area and the virtual space area, and to make it feel as if they exist in the same space. In addition, this process may be executed in combination with the above-described process of changing the transparency and the process of displaying the frame of the background area with a color.

Above, the description about the specific example of the process of the process part 109 is complete | finished.
The three processes described above can be arbitrarily changed according to user settings.

  The video space generation unit 110 includes an image that is an output result of the processing unit 109, an image of another point (for example, a slave point) that is an output result of the position / direction calculation unit 107, and a user B located at another point. By using the viewpoint position and viewpoint direction (three-dimensional coordinate position), a virtual space environment including a person's video is generated by arranging images of users at other points in the virtual space environment.

  The display unit 111 is an image display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro Luminescence) display. The display unit 111 displays the generated video for the user A at the main point on the display. Specifically, the display unit 111 displays the three-dimensional virtual space environment generated by the video space generation unit 110 on a two-dimensional screen based on the position of the line of sight and the line-of-sight direction of the user A who is the local point. Perspective projection conversion is performed and the drawing result is displayed on the display.

  Next, the functional configuration of the follower device 200 will be described. The follower device 200 includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and executes a follower program. By executing the follow-up program, the follow-up device 200 includes an imaging unit 201, a face position acquisition unit 202, an image transformation unit 203, a communication unit 204, a region division unit 205, a position / direction calculation unit 206, a determination unit 207, and a processing unit 208. , Function as an apparatus including a video space generation unit 209 and a display unit 210. Note that all or part of the functions of the follower device 200 may be realized using hardware such as an ASIC, a PLD, or an FPGA. The follower program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. The main point program may be transmitted / received via a telecommunication line.

The imaging unit 201 captures an image including the user B and the background at the slave point. The imaging unit 201 outputs the captured image to the face position acquisition unit 202.
The face position acquisition unit 202 acquires the coordinate value of the face position of the person (user B) from the image captured by the imaging unit 201 by the same processing as the face position acquisition unit 102.

The image deforming unit 203 generates a face position deformed image based on the acquired coordinate value of the face position of the user B by the same processing as the image deforming unit 103.
The communication unit 204 transmits / receives data to / from the main point device 100. In addition, the communication unit 204 transmits and receives data to and from another slave device 200.

The region dividing unit 205 performs region division on the face position deformed image generated by the image deforming unit 103 at another point (for example, the main point device 100) by the same processing as the region dividing unit 105.
The position / direction calculation unit 206 acquires the position and line-of-sight direction of the user B's viewpoint at the slave point relative to the reference point in the slave point determined in advance. In addition, the position / direction calculation unit 206 acquires the display position of the slave point (user B) with respect to the reference point. The display is installed in an arbitrary line-of-sight direction such as an object or user A and is movable.

The determination unit 207 determines whether or not there is an observation target such as a person or an object with respect to the line-of-sight direction of the user A calculated by the position / direction calculation unit 107 of another point (for example, the main point).
The processing unit 208 processes the image captured by the user A based on the determination result of the determination unit 207.

  The video space generation unit 209 includes an image that is an output result of the processing unit 109, an image of another point (for example, a main point) that is an output result of the position / direction calculation unit 107, and an observation position of the user A located at the other point. And a direction (three-dimensional coordinate position) are used to generate a virtual space environment including a person's video by arranging images of users at other points in the virtual space environment.

  The display unit 210 is an image display device such as a CRT display, a liquid crystal display, or an organic EL display. The display unit 210 displays the video for the user B at the slave point generated by the video space generation unit 209 on the display. Specifically, the display unit 210 displays the three-dimensional virtual space environment created by the video space generation unit 209 on a two-dimensional screen based on the position of the line of sight and the line of sight direction of the user B who is the local point. Perspective projection conversion is performed and the drawing result is displayed on the display.

FIG. 8 is a diagram showing a situation in which three users at three locations, the master location, the follower location 1 and the follower location 2, observe and interact with an object at the master location using the video display system of the present invention. is there.
Reference numeral 300 indicates a positional relationship between the object in the virtual space and the user at the master point, the user at the slave point 1, and the user at the slave point 2. Reference numeral 301 denotes the state of the main point, and reference numeral 302 denotes the state of the slave point 1. At each of the main point and the slave point 1, the user observes the state of the object and the partner of the conversation using a large display. Reference numeral 303 denotes the state of the slave point 2, and the user observes the state of the object and the other party using a tablet-type terminal. The user at each base glances at the gaze direction represented by the display position of the other party in the virtual space centered on the object, the facial expression, the gaze, and the body direction from the video displayed on the display. Can be grasped.

  In this way, the video display system of the present invention displays the video of the other party of conversation in an expression according to the size of the display used by the user, the distance and angle with the user, and further displays the person, object, and space. By expressing continuously, communication can be activated.

9 and 10 are sequence diagrams showing the flow of operations of the video display system in the present embodiment. 9 and 10, for the sake of simplicity of explanation, the case of two points where the main point device 100 and one slave point device 200 exist will be described. First, a specific example in which an image is displayed on the main point device 100 will be described with reference to FIG.
First, the imaging unit 201 of the follower device 200 takes an image including the follower user B and the background (step S101). The face position acquisition unit 202 acquires the coordinate value of the person's face position from the captured image (step S102). The image deforming unit 203 generates a face position deformed image by deforming the captured image based on the acquired coordinates of the face position of the person (step S103).

  The position direction calculation unit 206 acquires the position of the viewpoint of the user B and the line-of-sight direction with respect to the environment. Further, the position / direction calculation unit 206 acquires the position of the slave point display with respect to the environment (step S104). The position / direction calculation unit 206 outputs the acquired viewpoint position, line-of-sight direction, and display position of the user B to the display unit 210. In addition, the position / direction calculation unit 206 outputs the acquired viewpoint position and line-of-sight direction of the user B and the display position to the communication unit 204. The communication unit 204 transmits the generated face position deformed image data, the viewpoint position and line-of-sight direction of the user B, and the display position to the main point device 100 (step S105).

  The communication unit 104 of the master location device 100 receives the information transmitted from the slave location device 200 (step S106). The communication unit 104 obtains user B's face position deformed image data, user B's viewpoint position and line-of-sight direction, and display position from the received information (step S107). The processing unit 109 processes the face position deformed image captured by the user B based on the determination result of the determination unit 108 (step S108). The video space generation unit 110 includes an image that is an output result of the processing unit 109, an image of another point (for example, a slave point) that is an output result of the position / direction calculation unit 206, and a user B located at another point. By using the position of the viewpoint and the line-of-sight direction (three-dimensional coordinate position), a virtual space environment including a person's video is generated by arranging an image of the user B at another point in the virtual space environment (step S109). . The position / direction calculation unit 107 acquires the position of the viewpoint of the user A with respect to the environment and the line-of-sight direction. The display unit 111 is a virtual space environment generated by the video space generation unit 110 by performing perspective projection conversion on a two-dimensional screen based on the acquired position of the viewpoint of the user A at the main point and the line-of-sight direction. Is displayed on the display (step S110).

Next, a specific example in which an image is displayed on the follower device 200 will be described with reference to FIG.
First, the imaging unit 101 of the main point device 100 captures an image including the user A and the background at the main point (step S201). The face position acquisition unit 102 acquires the coordinate value of the person's face position from the captured image (step S202). The image deforming unit 103 generates a face position deformed image by deforming the captured image based on the acquired face position coordinates of the person (step S203).

  The position / direction calculation unit 107 acquires the position of the viewpoint of the user A and the line-of-sight direction with respect to the environment. Further, the position / direction calculation unit 107 acquires the position of the display of the main point with respect to the environment (step S204). The position / direction calculation unit 107 outputs the acquired viewpoint position, line-of-sight direction, and display position of the user A to the display unit 111. Further, the position / direction calculation unit 107 outputs the acquired position of the viewpoint of the user A, the line-of-sight direction, and the display position to the communication unit 104. The communication unit 104 transmits the generated face position deformation image data, the position and line-of-sight direction of the user A's viewpoint, and the display position to the follower device 200 (step S205).

  The communication unit 204 of the slave point device 200 receives the information transmitted from the master point device 100 (step S206). The communication unit 204 acquires the face position deformed image data of the user A, the position and line-of-sight direction of the user A, and the display position from the received information (step S207). The processing unit 208 processes the face position deformed image captured by the user A based on the determination result of the determination unit 207 (step S208). The video space generation unit 209 includes an image that is an output result of the processing unit 208, an image of another point (for example, a main point) that is an output result of the position / direction calculation unit 107, and a user A located at another point. Using the viewpoint position and the line-of-sight direction (three-dimensional coordinate position), a virtual space environment including a person's video is generated by arranging images of user A at other points in the virtual space environment (step S209). . The position / direction calculation unit 206 acquires the position of the viewpoint of the user B with respect to the environment and the line-of-sight direction. The display unit 210 converts the virtual space environment generated by the video space generation unit 209 by performing perspective projection conversion on the two-dimensional screen based on the acquired position of the viewpoint of the user B as the slave point and the line-of-sight direction. It is displayed on the display (step S210).

  According to the video display system configured as described above, a plurality of users can share the same space and communicate while taking a free positional relationship. Specifically, the displayed background area of the user is processed according to whether there are other users or objects with respect to the user's line-of-sight direction. Therefore, the user can easily grasp the state of the user just by looking at the image of the other user displayed on the screen. Therefore, smooth communication can be achieved.

Further, the image deforming units 103 and 203 deform the two-dimensional image based on the user's face position in the acquired image and generate a face position deformed image. Thereafter, the generated image is displayed in the virtual space, so that it becomes possible to easily transmit the line-of-sight direction of each user.
Moreover, one space can be shared by arranging a person in the virtual space according to the user's observation position and line-of-sight direction. In addition, when there is a person or object in the space that the user wants to observe, the user can view a desired video just by moving or changing the orientation of the display.

<Modification>
The main point device 100 may be configured to include an image acquisition unit instead of the imaging unit 101. The image acquisition unit acquires an image including the user A and the background from one fixed camera installed around the main point device 100. The follower device 200 may be configured to include an image acquisition unit instead of the imaging unit 201. The image acquisition unit acquires an image including the user B and the background from one fixed camera installed around the slave point device 200.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 100 ... Master point apparatus, 200 ... Slave point apparatus, 101 ... Imaging part, 102 ... Face position acquisition part, 103 ... Image deformation part, 104 ... Communication part, 105 ... Area division part, 106 ... Image generation part, 107 ... Position Direction calculation unit 108 ... Judgment unit 109 ... Processing unit 110 ... Video space generation unit 111 ... Display unit 201 ... Imaging unit 202 ... Face position acquisition unit 203 ... Image deformation unit 204 ... Communication unit 205 ... area division unit, 206 ... position / direction calculation unit, 207 ... determination unit, 208 ... processing unit, 209 ... video space generation unit, 210 ... display unit

Claims (5)

An acquisition unit for acquiring the orientation of the user who is the partner of the dialogue;
A processing unit that processes the video of the user based on the acquired orientation;
A display unit for displaying the processed image;
A video display system comprising:   2. The video display system according to claim 1, wherein the processing unit changes the transparency of an area of the video that does not include the user when the orientation is in a screen direction.   The video display system according to claim 1, wherein the processing unit changes a color of a frame of the video when the orientation is directed to a screen direction.   The video display system according to claim 1, wherein the processing unit performs a blurring process on a region that does not include the user of the video and a frame of the region when the orientation is in a screen direction. An acquisition step of acquiring the orientation of the user who is the partner of the dialogue;
A processing step of processing the video of the user based on the acquired orientation;
A display step for displaying the processed video;
A video display method comprising: