JP7306765B2 - Communication device, communication program and storage medium - Google Patents

Description

The present invention relates to a communication device or the like for displaying participants at other locations on a screen for video conferences or conversations.

As a communication device of this type, there is a video conference device in which a plurality of terminal devices are connected via a network, and video conferences are held by displaying images of participants from imaging devices installed in the terminal devices on the display screens of each terminal device. . As shown in FIG. 1 of Japanese Unexamined Patent Application Publication No. 2002-100000, for example, a conventional video conference apparatus displays the front faces of a plurality of participants in a grid pattern on the display screen of the terminal device.

Japanese Patent Application Laid-Open No. 2017-005616 Japanese Patent Application Laid-Open No. 2001-136501

However, with a device that simply enumerates and displays front faces like in Patent Document 1, it is not possible to move the face and see only the part one wants to see, as in an actual conference room or venue. It's hard to get a sense of realism like that. In addition, as disclosed in Patent Document 2, for example, by using a plurality of imaging devices, a side face can be captured and displayed. However, the techniques disclosed in Patent Document 1 and Patent Document 2 are intended solely for estimating a conversation partner and matching their line of sight. Movements that do not necessarily require eye contact, such as talking to people or looking over other participants, are not assumed. Therefore, it is difficult to convey the presence of being in an actual conference room or venue.

In consideration of such circumstances, the present invention provides a communication device or the like that can give a sense of realism as if one is actually there by changing the appearance of one's own face and the appearance of the surroundings according to the movement of one's face. intended to provide

In order to solve the above problems, the device of the present invention is a communication device for displaying images of a plurality of participants on a terminal device, wherein the plurality of images including images of faces facing different directions for each of the plurality of participants. a motion detection unit that detects the facial movement of at least one of the participants from the facial video acquired by the acquisition unit; An image selection unit that selects an image of another participant's face orientation, and an image generation that generates an image for displaying at least an image of the other participant's face on a terminal device from the image selected by the image selection unit. and According to the communication device of this aspect, it is possible to change the appearance of other participants in accordance with the movement of one's own face, thereby providing a sense of realism as if one were there.

In a preferred aspect of the present invention, a virtual conference room for arranging a plurality of participants and a virtual conference room storage unit for storing the positions of the plurality of participants in the virtual conference room are provided. A position-specific image is generated to display the participant according to the position of the participant in the room. According to this aspect, since the position where the participant is displayed is specified according to the position of the participant in the virtual conference room, the position of the other participants looks as if they were in the actual conference room. You can have video conferences and conversations with

In a preferred aspect of the present invention, the image selection unit selects images of face orientations according to the positions of the participants in the virtual conference room. According to this aspect, for example, a participant next to him/herself can display a side view image, so that the appearance of other participants can be seen as if they were in an actual conference room. It can be performed.

In a preferred aspect of the present invention, the video selection unit selects the display range of the virtual conference room according to the facial movements of the participants detected by the motion detection unit, and the video generation unit selects the A video is generated for displaying the participants included in the display range on the terminal device. According to this aspect, by moving one's face, it is possible to display a desired range of the virtual conference room. For example, if you want to see only the participants on the right side of the table, you can move your face to the right so that only the participants on your right can be seen. It is also possible to display only the participant by moving the face toward the participant to have a conversation, just like going near the participant and having a conversation in an actual conference room. This gives a sense of realism as if you were having a conversation in an actual conference room.

In a preferred aspect of the present invention, the motion detection unit detects whether the face orientation of at least one of the participants is front, right, or left from the face image acquired by the acquisition unit, and selects the image. When the face direction of a participant detected by the motion detection unit is facing the front, the display area of the virtual conference room should not include the images of other participants located to the right and left of that participant. , if a participant's face is facing right as detected by the motion detection unit, the display area of the virtual conference room will include the video of the other participant located to the right of the participant, and the motion detection unit will If the face of the detected participant faces left, the image of the other participant positioned to the left of the participant is included. According to this aspect, when conversing with the participant on the right side of oneself, it is necessary to turn to the right, and when conversing with the participant on the left side of oneself, it is sufficient to turn to the left, so it is as if talking with the next participant in the conference room. You can get a sense of reality like a case. Also, if there are multiple participants on your right side, you can turn to the right when you want to see them, and if you want to see multiple participants on your left side, you can turn to the left when you want to see them. You can get a sense of realism as if you changed the direction of your face in a room.

In a preferred aspect of the present invention, when the position of the participant in the virtual conference room changes, the image selection unit selects an image of the face direction corresponding to the changed position, and the image generation unit A position-specified video is generated that displays the participant according to the position. According to this aspect, by moving the seats in the virtual conference room, it is possible to achieve the same appearance as in the actual conference room in the video conference or conversation, thereby enhancing the sense of realism.

In a preferred embodiment of the present invention, a voice selection unit that selects voices of other participants according to facial movements of participants detected by the motion detection unit; and a voice generator for generating voices for outputting the voices of the participants from the terminal device. According to this aspect, it is possible to select a sound according to the movement of the face of the participant in accordance with the video. For example, it is possible to select and output only the voice of the participant whose video is displayed according to the movement of the face.

In a preferred aspect of the present invention, the acquisition unit acquires images of faces facing different directions and surrounding images facing different directions for each of a plurality of participants, and the image selection unit obtains images of the participants detected by the motion detection unit. The image generation unit selects at least the image of the other participant's face and the image of the surroundings from the images selected by the image selection unit. A video for displaying the video on the terminal device is generated. According to this aspect, the image of the other participant's face direction and the surrounding image are selected according to the movement of the face of the participant from the images of the face of the different direction and the surrounding images of the different direction. If you move the , the surrounding image will move and be displayed in conjunction with the movement of the face. This makes it possible to move the surrounding image so that invisible parts can be seen just by moving the face, so you can experience a realistic feeling as if you were there.

In a preferred embodiment of the present invention, the motion detection unit detects movement and direction of the face as the movement of the face of the participant, and the image selection unit selects surrounding images according to the movement of the face, and detects the movement of the face. The display range of the surrounding image is selected according to the orientation of the image, and the image generation unit generates an image for displaying the surrounding image on the terminal device in the display range selected by the image selection unit. According to this aspect, the surrounding image is selected according to the movement of the face of the participant, and the display range of the surrounding image is selected according to the orientation of the face. You can also change the display range of the surrounding image by pressing With this, you can change the display range of the surrounding image so that you can see the invisible part just by moving your face, so you can experience the realism as if you were there.

In a preferred aspect of the present invention, a virtual conference room for arranging a plurality of participants and a virtual conference room storage unit for storing the positions of the plurality of participants in the virtual conference room and the positions of surrounding images, The selection unit changes the display position of the participant according to the direction of the surrounding image selected according to the movement of the participant's face. According to this aspect, by changing the display position of the participant in accordance with the orientation of the surrounding image selected according to the movement of the participant's face, it is possible to display the surrounding image as if it were being viewed from that display position. Therefore, it is possible to have a realistic experience as if you were there.

In a preferred aspect of the present invention, the storage medium of the present invention is a computer-readable storage medium storing a communication program that causes a computer to execute video processing performed by a communication device, the video processing being performed by a plurality of participants. obtaining a plurality of videos, each including videos of faces in different orientations; detecting facial movements of at least one of the participants from the obtained facial videos; and detecting facial movements of the participants. and generating an image for displaying at least the image of the other participant's face from the selected image on the terminal device. . Video processing can be executed by reading and executing the program stored in the storage medium of this embodiment, and the computer can function as a communication device.

In order to solve the above problems, a program of the present invention is a communication program that causes a computer to execute video processing performed by a communication device, wherein the video processing includes images of faces facing different directions for each of a plurality of participants. acquiring a plurality of images; detecting facial movements of at least one of the participants from the acquired facial images; It includes the steps of selecting an orientation image, and generating from the selected image an image for displaying at least an image of the other participant's face on the terminal device. By executing the program of this aspect, video processing can be executed, and the computer can function as a communication device.

According to the present invention, by changing the appearance of other participants and the appearance of the surroundings according to the movement of one's face, it is possible to have a video conference or conversation with a sense of realism as if one were there. can.

1 is a diagram showing the configuration of a video conference system according to a first embodiment of the present invention; FIG. 1 is a block diagram of a video conference device (communication device) and a terminal device; FIG. 3 is a diagram showing a specific example of virtual meeting room configuration information in FIG. 2; FIG. 3 is a diagram showing a specific example of participant information in FIG. 2; FIG. 3 is a diagram showing a specific example of virtual conference room display information in FIG. 2; FIG. 1 is a diagram showing a schematic configuration of a video conference system (two participants); FIG. FIG. 7 is a schematic diagram of the terminal device of FIG. 6 viewed from the front; FIG. 7 is a top view of the terminal device in FIG. 6 and shows an example of an image from an imaging device; FIG. 7 is a view of the terminal device in FIG. 6 viewed from the left, and is a view showing an example of an image from an imaging device; 4 is a flowchart showing a specific example of video conference processing; FIG. 11 is a flow chart showing a specific example of the video processing shown in FIG. 10; FIG. FIG. 10 is an explanatory view of the effect when detecting no movement of the face; FIG. 10 is an explanatory view of action when detecting presence of movement of the face; FIG. 12 is a diagram showing a schematic configuration of a video conference system (four participants) according to the second embodiment; It is a figure which shows the specific example of the virtual meeting room display information of 2nd Embodiment. FIG. 10 is a diagram showing the relationship between the view from participant A and the imaging devices of other participants; FIG. 10 is a diagram showing the relationship between the view from participant B and the imaging devices of other participants; 9 is a flowchart showing a specific example of video processing according to the second embodiment; FIG. 11 is an explanatory view of the effect when detecting that there is movement of the face in the second embodiment; FIG. 20 is a diagram showing a specific example of virtual conference room display information in FIG. 19; FIG. 11 is a flowchart showing a specific example of video processing according to the third embodiment; FIG. It is a figure which shows the specific example of the virtual meeting room display information in 3rd Embodiment. FIG. 23 is a diagram showing how the virtual meeting room display information of FIG. 22 is viewed from participant A; FIG. 10 is a diagram showing virtual conference room display information when the face direction moves to the left; FIG. 25 is a diagram showing how the virtual meeting room display information of FIG. 24 is viewed from the participant A. FIG. FIG. 10 is a diagram showing virtual conference room display information when the face direction moves to the right; FIG. 27 is a diagram showing how the virtual meeting room display information of FIG. 26 is viewed from participant A; FIG. 10 is a diagram showing a specific example of virtual conference room display information according to the first modified example; FIG. 10 is a diagram showing how the participant A sees after the participant D moves his/her seat. FIG. 10 is a diagram showing how the participant D sees the image before the participant D moves his/her seat. FIG. 10 is a diagram showing how the participant D sees after the participant D moves his/her seat. FIG. 11 is a diagram showing a specific example of virtual conference room display information according to the second modification; FIG. 33 is a diagram showing how the virtual meeting room display information in FIG. 32 is viewed from participant A1. It is a figure which shows schematic structure of the communication system of 4th Embodiment. 35 is a schematic diagram of the terminal device of FIG. 34 viewed from the front; FIG. FIG. 35 is a top view of the terminal device of FIG. 34 and shows a specific example of surrounding images from each imaging device; It is a block diagram of a communication device and a terminal device of a fourth embodiment. FIG. 37 is a diagram showing a specific example of the virtual meeting room configuration information of FIG. 36; FIG. 37 is a diagram showing a specific example of participant information in FIG. 36; FIG. 37 is a diagram showing a specific example of virtual conference room display information in FIG. 36; FIG. 10A is a diagram showing an example of an image from an imaging device 24g of a participant B, (a) is an omnidirectional image, and (b) is an output image generated from the omnidirectional image. FIG. 10 is a diagram showing a display image that participant A can see from the front without moving his/her face. FIG. 10A is a diagram showing an example of an image from an imaging device 24f of a participant B, (a) is an omnidirectional image, and (b) is an output image generated from the omnidirectional image. FIG. 10 is a diagram showing a display image seen by Participant A as he moves his face to the left; FIG. 10A is a diagram showing an example of an image from an imaging device 24e of a participant B, (a) is an omnidirectional image, and (b) is an output image generated from the omnidirectional image. FIG. 10 is a diagram showing a display image seen when participant A moves his face to the right; FIG. 10 is a diagram showing changes in the display range when participant A turns his face to the left; FIG. 10 is a diagram showing a display image in which Participant A can be seen with his face turned to the left; FIG. 10 is a diagram showing changes in the display range when participant A turns his face to the right; FIG. 11 shows a display image in which Participant A looks to the right;

<First embodiment>
A first embodiment of the present invention will be described below with reference to the drawings. In the first embodiment, a video conference system 100 (communication system) including a video conference device 10 as an example of the communication device of the present invention is taken as an example. FIG. 1 is a diagram showing the configuration of a video conference system 100 according to the first embodiment. A video conference system 100 in FIG. 1 includes a video conference device 10 and terminal devices 20 .

The video conference system 100 displays a plurality of participants on the screen of the terminal device 20 to hold a video conference (web conference, video conference). The video conference device 10 of this embodiment changes the appearance of other participants according to the movement of their own face (in conjunction with the movement of their own face) and displays them on the screen. Since the appearance of other participants changes in conjunction with the movement of your face (face movement, face direction, etc.), you can have a realistic meeting as if you were there.

The video conference device 10 of the first embodiment is configured by a server computer having the terminal device 20 as a client. The video conference device 10 may be configured to perform distributed processing by a plurality of devices, or may be configured by a plurality of virtual machines provided in one server device. Also, the video conference device 10 may be configured by a personal computer or may be configured by a cloud server. The video conference device 10 and the terminal device 20 are configured to communicate with each other via a network N such as the Internet.

The terminal device 20 is an information processing device used by a user. The terminal device 20 is, for example, a mobile terminal such as a smart phone, a tablet, or a PDA (Personal Digital Assistant), a desktop personal computer, a notebook personal computer, or the like. Although two terminal devices 20 are connected to the network N, three or more terminal devices 20 may be connected. Each terminal device 20 is used by participants at separate points, but may include terminals used by other participants at the same point.

FIG. 2 is a block diagram showing a specific configuration example of the video conference device 10 (communication device) and the terminal device 20 in FIG. As shown in FIG. 2, the video conference device 10 includes a communication section 11, a control section 12, and a storage section . The communication unit 11, the control unit 12, and the storage unit 14 are each connected to the bus line 10L, and can mutually exchange information (data).

The communication unit 11 is connected to the network N by wire or wirelessly, and transmits and receives information (data) to and from the terminal device 20 . The communication unit 11 functions as a communication interface for the Internet or an intranet, and is capable of communication using TCP/IP, for example.

The control unit 12 centrally controls the entire video conference device 10 . The control unit 12 is configured by an integrated circuit such as an MPU (Micro Processing Unit). The control unit 12 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The control unit 12 loads a necessary program into the ROM and executes the program using the RAM as a work area to perform various processes (such as video conference processing).

The storage unit 14 is a computer-readable storage medium that stores various programs executed by the control unit 12 and data used by these programs. The storage unit 14 is configured by a storage device such as a hard disk, an optical disk, or a magnetic disk. The configuration of the storage unit 14 is not limited to these, and the storage unit 14 may be configured by a semiconductor memory such as RAM or flash memory. For example, the storage unit 14 can be configured with an SSD (Solid State Drive).

The storage unit 14 includes a program storage unit 15, a data storage unit 16, a virtual conference room storage unit 18, and the like. The program storage unit 15 stores various programs executed by the control unit 12 . The control unit 12 reads necessary programs from the program storage unit 15 and executes various processes.

The data storage unit 16 stores, for example, user information 161 and the like. The user information 161 includes pre-registered user IDs, company names, names, and the like. Note that the user information 161 is not limited to this.

The virtual conference room storage unit 18 stores the preset configuration of the virtual conference room, the position information of the participants in the virtual conference room, and the like. Specifically, the virtual conference room storage unit 18 stores virtual conference room configuration information 181, participant information 182, virtual conference room display information 183, and the like. In addition, image data such as a table 183c and a whiteboard 183d are stored in the virtual conference room storage unit 18 as conference room components to be displayed in the virtual conference room. Images and moving images for presentation can also be displayed on this whiteboard 183d. In this specification, the virtual conference room may be a virtual conversation room.

The virtual conference room configuration information 181 is configuration information of the virtual conference room for arranging the images of the participants to be displayed on each terminal device 20, and includes, for example, a data table as shown in FIG. Specifically, they include the type of virtual conference room, the number of people accommodated, the presence or absence of a table 183c, the presence or absence of a whiteboard 183d, the number of columns and rows in which participants can be displayed, and the like. In this embodiment, the configuration information of a plurality of virtual conference rooms with different number of participants, the number of columns in which participants are arranged, and whether or not to display the table 183c is stored.

FIG. 3 illustrates a small conference room R1, a medium conference room R2, a large conference room R3, a face-to-face conference room R4, and the like. The capacity of the small conference room R1, the medium conference room R2, and the large conference room R3 are different, and the appearance of the participants differs between the face-to-face conference room R4 and the other conference rooms. For example, in the small conference room R1, the middle conference room R2, and the large conference room R3, the images of the participant and the other participants are displayed in a frame around the table 183b as shown in FIG. 6, which will be described later. It is also possible not to display the table 183b. In the two-person meeting room R4, neither the table 183b nor the user himself is displayed, and only the image of the other party is displayed. Note that the virtual conference room is not limited to the illustrated one. In addition, the user can change not only the preset virtual conference room, but also the type of virtual conference room, the number of people that can be accommodated, the presence or absence of a table 183c, the presence or absence of a whiteboard 183d, and the number of columns and rows in which participants can be displayed. or set.

The participant information 182 stores information about participants who participate in the conference, and is composed of a data table as shown in FIG. 4, for example. Specifically, the participant information 182 includes the ID of the conference room in which the participant participates, the user ID, the location, the presence or absence of entry into the room, the presence or absence of exit from the room, and the position of the participant on the virtual conference room (row y01, column t01, etc.). , facial movements of the participants, the number of video data, the type of audio data, and the like. Video data and audio data are data received from the terminal device 20, and the number of video data is four for the front, right, left, and bottom sides, for example. The type of audio data received from the terminal device 20 is stereo audio. The participant information 182 is not limited to those described above, and may store, for example, IP addresses, screenshot images, and the like.

The virtual conference room display information 183 is display information of the virtual conference room that each participant enters. For example, as shown in FIG. 5, the virtual conference room display information 183 includes the columns in which each participant is arranged, the position of each participant (the position of the frame in which the video of the participant is fitted), and the conference information such as a table 183c and a whiteboard 183d. The position of the element, the display extent 183b of the virtual conference room, etc. are included. The virtual conference room display information 183 is provided for each type of virtual conference room in the virtual conference room configuration information 181 in FIG. The host participant can set in advance which virtual conference room to use.

FIG. 5 illustrates a case where two participants A and B use the virtual conference room of the small conference room R1 in FIG. As shown in FIG. 5, an outer frame 183a indicates the entire virtual conference room. The thick frame inside it indicates the display range 183b. The small conference room R1 in FIG. 3 has a capacity of two people, a table, two rows and two columns. In FIG. 5, a table 183c is placed in the center. The lower row of the table 183c corresponds to the horizontal row y01, and the upper row of the table 183c corresponds to the horizontal row y02. The left column of the table 183c corresponds to the column t01, and the right column of the table 183c corresponds to the column t02. Note that the positions of the table 183c, the whiteboard 183d, columns, rows, etc. shown in FIG. .

A video of each participant can be inserted into the A frame arranged in the horizontal row y01 and the B frame arranged in the horizontal row y02 in FIG. For example, FIG. 6, which will be described later, exemplifies a case where the image of participant A is fitted in the A frame and the image of participant B is fitted in the B frame using the small conference room R1. It should be noted that it is possible to select in advance whether the virtual conference room should be displayed two-dimensionally or three-dimensionally for display on the terminal device 20 . When the two-dimensional display is selected, the display range 183b of FIG. 5 is displayed as it is on the display screen 252, and when the three-dimensional display is selected, the table 183c and the like are stereoscopically displayed as shown in FIG. In FIG. 5, the display range 183b is the range including the columns in which the participants are arranged and the table 183c. Since the whiteboard 183d is not included in the display range 183b in FIG. 5, it is not displayed. If displayed, the display range 183b can be adjusted to include the whiteboard 183d from the beginning or during the meeting.

The control unit 12 in FIG. 2 includes an acquisition unit 121 , a motion detection unit 122 , an image selection unit 123 , an image generation unit 124 , an audio selection unit 125 , an audio generation unit 126 and an output unit 127 . Each component of the control unit 12 may be configured by a physical circuit, or may be configured by a program executable by the CPU. The configuration of the control unit 12 is not limited to the configuration shown in FIG.

The acquisition unit 121 receives, via the communication unit 11 , a plurality of video data and audio data of the participants captured by the imaging device 24 for each terminal device 20 . Specifically, the acquisition unit 121 receives video data including facial images of different orientations of each participant (for example, front, right, left, upper, lower, etc.) and stereo audio data (left audio, right audio). do. The acquisition unit 121 does not matter which imaging device 24 the image is captured by, but in addition to the image of the front face, if there is a person who is looking at the person there, the image of the person who should be seen when the person moves his or her face. A face image (either right, left, top, bottom, etc.) is also input. For example, when moving the face left and right to change the image of another participant, the image from the right side of the face (right face image) and the image from the left side (left face image) are input.

The movement detection unit 122 detects movement of the participant's face from the video data acquired by the acquisition unit 121 . The motion detection unit 122 also detects whether or not the face of the participant moves. The "movement of the face" here includes movement of the face (change in position of the face) and rotation of the face (change in orientation of the face). The movement detection unit 122 can detect movement of the face (change in position of the face) in, for example, the horizontal direction (horizontal direction), the vertical direction (vertical direction), and the front-rear direction. In addition, the motion detection unit 122 may be configured to detect rotation of the face such as rightward, leftward, upward, downward, etc. (change in face direction). When the movement detection unit 122 of the first embodiment detects that there is a movement of the face, it also detects in which direction the face has moved: the horizontal direction (right direction, left direction) or the vertical direction (upward direction, downward direction). do.

The motion detection unit 122 may detect the movement of the face from the image of the front of the face, or may detect the movement of the face from the image of another direction (image of the face facing right or left). . For example, by recognizing the face part from the image of the front of the face, vectorizing the direction of the face (or the direction of the nose), detecting the movement of the face (change in the position of the face) from the change in the position of the vector, The rotation of the face (change in face orientation) may be detected from the change in angle of the vector. In this case, for example, the movement of the face from the image of the imaging device 24 is detected from the image of the imaging device 24 using a trained model obtained by machine learning with AI (artificial intelligence) or an existing trained model. You may make it The detection of the movement of the face is not limited to the above method, and any method may be used as long as it can be detected from the image of the face.

The image selection unit 123 selects images of other participants to be displayed on the terminal device 20 according to the facial movements detected by the movement detection unit 122 from images of faces in different directions acquired by the acquisition unit 121 for each participant. Select a video. For example, in an actual meeting, when there is a person facing each other, if you move your face to the left, you should be able to see the right side of the person's face. Therefore, for example, when the movement detection unit 122 detects a movement of the participant's face in the left direction (left movement), the image selection unit 123 acquires the image of the other participant to be displayed at the position facing the participant. An image of the right side of the face is selected from the images of the face in different orientations (front, right, left, upper, lower, etc.) acquired by the unit 121 . The image generation unit 124 inserts the image of each participant's face selected by the image selection unit 123 into a predetermined position (the position of the face image frame) in the virtual conference room, and generates an output image synthesized with an image such as a table. do. This allows you to see the images of other participants in the same way as if you moved your face in an actual meeting.

The voice selection unit 125 selects voices of other participants according to the facial movements of the participants detected by the motion detection unit 122 . For example, in an actual conference, when speaking to a participant on the left side by moving one's head, the left side should sound stronger. Therefore, for example, when the movement detection unit 122 detects the movement of the participant's face to the left, the stereo sound acquired by the acquisition unit 121 is used as the voice of the other participant displayed on the left side of the participant. The left voice is selected from the data. The audio generation unit 126 generates output audio in which the left and right audio and the overall volume are adjusted. When the movement detection unit 122 detects the movement of the participant's face to the left as described above, the volume of the left audio is set higher than that of the right audio for the audio of the other participants displayed to the left of the participant. Generates louder output audio.

The output unit 127 outputs the output image generated by the image generation unit 124 and the output sound generated by the sound generation unit 126 to the communication unit 11 as moving image data (video data). Note that the control unit 12 stores and updates the participant information 182 shown in FIG. 4 with the movement of the face detected by the movement detection unit 122 . The video selection unit 123 and the audio selection unit 125 may select video and audio based on facial movements in the participant information 182 .

The output video and output audio from the output unit 127 are transmitted to the terminal device 20 via the communication unit 11 as moving image data. The terminal device 20 displays the received output video on the display screen 252 of the display device 25 and outputs the output audio from the speaker 27 . Note that the output video and output audio may be web video data. In this case, the video conference device 10 displays the output video and output audio as moving image data on the web, and the terminal device 20 receives the moving image data from the web and displays it on the browser.

Next, a configuration example of the terminal device 20 will be described with reference to FIG. A terminal device 20 shown in FIG. These are connected to the bus line 20L, respectively, and can exchange information (data) with each other.

The communication unit 21 is connected to the network N by wire or wirelessly, and transmits and receives information (data) to and from the video conference device 10 . The communication unit 21 functions as a communication interface for the Internet or an intranet, and is capable of communication using TCP/IP, for example.

The control unit 22 controls the terminal device 20 as a whole. The control unit 22 is configured by an integrated circuit such as an MPU (Micro Processing Unit). The control unit 22 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory). The control unit 22 loads necessary programs into the ROM and executes the programs using the RAM as a work area, thereby performing various processes.

The storage unit 23 is an example of a storage medium that stores various programs executed by the control unit 22 and data used by these programs. The storage unit 23 is configured by a storage device such as a hard disk, an optical disk, or a magnetic disk. The configuration of the storage unit 23 is not limited to these, and the storage unit 23 may be configured by a semiconductor memory such as RAM or flash memory. For example, the storage unit 23 can be configured with an SSD (Solid State Drive).

The image pickup device 24 is a plurality of cameras that photograph the faces of the participants from different directions. The imaging devices 24 may be two, three, or four or more. The imaging device 24 of the present embodiment is configured by two in the left and right sides of the terminal device 20 and two in the top and bottom of the terminal device 20, for a total of four. Details will be described later. The imaging device 24 may be a camera built into the terminal device 20 or an external camera. Also, any camera such as a CCD (Charge Coupled Device) camera, a web camera, an IoT camera, etc., that can output an image may be used.

The display device 25 is a liquid crystal display, an organic EL display, or the like, and displays various information according to instructions from the control section 22 . The control unit 22 causes the display screen 252 of the display device 25 to display the moving image data received from the video conference device 10 via the communication unit 21 .

A microphone 26 is a stereo microphone that captures the voices of the participants. The microphone 26 may be a microphone built into the terminal device 20 or an external microphone. The speaker 27 is a stereo speaker and outputs various sounds and music according to instructions from the control unit 22 . The speaker 27 may be a speaker built into the terminal device 20 or an external speaker.

The input device 28 is a keyboard, a mouse, or the like, and receives an operation input from the user and transmits a control signal corresponding to the operation content to the control unit 22 . The input device 28 may be a touch panel provided on the display device 25 . In the present invention, the appearance of other participants on the display screen 252 can be changed by facial movements without using a mouse or keyboard.

FIG. 6 is a diagram showing a schematic configuration of the video conference system 100 when there are two participants. A video conference system 100 of FIG. 6 is configured by connecting a terminal device 20A of a participant A and a terminal device 20B of a participant B to a server constituting a video conference device 10 via a network N. FIG. Since the terminal devices 20A and 20B have the same configuration, the elements having the same functions are denoted by the same reference numerals.

Terminal devices 20A and 20B in FIG. 6 are personal computers in which a main body 20a and a display device 25 are integrated. A microphone 26 is provided in the center and speakers 27 are provided on the left and right sides of the front surface of the display device 25 . A main body 20 a is provided on the back of the display device 25 . The body 20a incorporates a communication section 21, a control section 22, a storage section 23, and the like. Four imaging devices 24 a, 24 b, 24 c, and 24 d are provided on the periphery of the display device 25 . The four imaging devices 24a, 24b, 24c, and 24d are for acquiring images including images of faces facing in different directions, respectively, and are connected to the main body 20a so that images can be transmitted from the communication unit 21. . Note that since the orientations of the four imaging devices 24a, 24b, 24c, and 24d are different, not only the face but also the background look different. Therefore, the background changes depending on which image is selected and displayed, so that the sense of presence can be enhanced.

Note that the imaging devices 24 (24a, 24b, 24c, and 24d) do not necessarily have to be installed in the display device 25, and can be placed on any place such as a wall or a desk as long as video data including videos of faces in different directions can be acquired. may be installed. Also, the imaging device 24 does not necessarily have to be electrically connected to the terminal devices 20A and 20B. The image capturing device 24 may be any image capturing device 24 such as a CCD camera, a web camera, an IoT camera, etc., as long as the image data including images of faces facing different directions is transmitted to the video conference device 10 via the network N. may be used. Also, the number of imaging devices 24 is not limited to four.

Here, installation positions of the imaging devices 24a, 24b, 24c, and 24d and image examples will be described with reference to FIGS. 7 to 9. FIG. FIG. 7 is a front view of the terminal device 20B in FIG. 6 and shows the position of the imaging device. FIG. 8 is a top view of the terminal device 20B in FIG. 7 and shows an example of an image from the imaging device 24. In FIG. FIG. 9 is a view of the terminal device 20B in FIG. 7 viewed from the left, showing an example of an image from the imaging device 24. As shown in FIG. Since the terminal device 20A has the same configuration, the terminal device 20B will be described here as a representative.

As shown in FIG. 7, the imaging device 24a is a right imaging device (first imaging device) arranged on the right side of the display device 25, and images the right side of the participant A's face as shown in FIG. The imaging device 24a is supported by a supporting portion 242a so as to be freely rotatable in the left and right direction around a rotating shaft 244a along the vertical direction of the display device 25. As shown in FIG. The imaging angle of the right side of the face can be adjusted by turning the imaging device 24a left and right. As shown in FIG. 7, the imaging device 24b is a left imaging device (second imaging device) arranged on the left side of the display device 25, and images the left side of the participant A's face as shown in FIG. The imaging device 24b is supported by a supporting portion 242b so as to be freely rotatable in the left and right direction around a rotation shaft 244b along the vertical direction of the display device 25. As shown in FIG. By turning the imaging device 24b left and right, the imaging angle of the left side of the face can be adjusted.

As shown in FIG. 7, the imaging device 24c is a central imaging device (third imaging device) arranged on the upper side of the display device 25, and as shown in FIGS. do. The imaging device 24c is supported by a supporting portion 242c so as to be vertically rotatable about a rotating shaft 244c along the horizontal direction of the display device 25. As shown in FIG. The imaging angle of the front of the face can be adjusted by rotating the imaging device 24c up and down. Here, a case where the image capturing device 24c captures an image of the front of the face is exemplified, but the image capturing angle of the image capturing device 24c may be changed to image the upper side of the face. As shown in FIG. 7, the imaging device 24d is a lower imaging device (fourth imaging device) arranged on the lower side of the display device 25, and images the lower side of the face of the participant A as shown in FIG. . The imaging device 24d is supported by a supporting portion 242d so as to be vertically rotatable about a rotating shaft 244d along the horizontal direction of the display device 25. As shown in FIG. The imaging angle of the lower side of the face can be adjusted by rotating the imaging device 24d up and down.

By the way, a communication device such as the video conference device 10 displays images of participants from the imaging device 24 installed in the terminal device 20 on the display screen 252 of the display device 25 of each terminal device 20 to hold a meeting or have a conversation. is now possible. In this case, for example, the front faces of a plurality of participants can be displayed in a grid pattern on the display screen 252 of the display device 25 of the terminal device 20 .

However, by simply listing and displaying the front face, it is not possible to move the face and see only the part you want to see as if you were in the actual meeting room or venue, so it is difficult to create a realistic feeling as if you were there. I can't get it. Also, by using a plurality of imaging devices, it is possible to image and display a side face. However, simply estimating the conversation partner and matching their gaze does not support movements such as moving the face to look into the other person's profile, talking to the person next to them while watching a presentation, or looking over other participants. Can not. Therefore, it is difficult to convey the presence of being in an actual conference room or venue.

In addition, VR (Virtual Reality) conferences allow the viewer to see what they want to see, so it is possible to obtain a sense of realism. However, a VR conference requires a head-mounted display (such as VR goggles), and it is troublesome to hold a conference while wearing this. In addition, since a large amount of communication data is required, if all participants do not have a high-spec terminal device 20 or a communication environment, there is a problem that the voice is interrupted or the screen freezes, and smooth meetings and conversations cannot be conducted. There is also

According to the video conference device 10 of the present embodiment, by making it possible to change the appearance of other participants according to the movement of their own faces, participants can feel as if they are there without using VR. You can get a sense of realism.

Video conference processing performed by the video conference device 10 will be described below with reference to the drawings. FIG. 10 is a flow chart showing a specific example of video conference processing. 10 is performed by the control unit 12 (acquisition unit 121, motion detection unit 122, video selection unit 123, video generation unit 124, audio selection unit 125, audio generation unit 126, output unit 127, etc.). A necessary program is read from 15 and executed.

A case will be exemplified where a Web conference is held by the terminal devices 20 of a plurality of participants connected to the video conference device 10 via a network by this video conference processing. The video conference device 10 can hold a web conference by receiving a request to enter a preset virtual conference room from the terminal device 20 of each participant.

In the following, the video conference processing will be described by exemplifying the case where two participants hold a video conference or have a conversation as shown in FIG. 12A and 12B are diagrams for explaining the action when no face movement is detected, and FIGS. 13A and 13B are diagrams for explaining the action when the presence of face movement is detected. FIG. 6 illustrates the case of a small conference room R1 (see FIG. 3) that displays the table 183c and himself as a virtual conference room, but in FIGS. The case of the meeting room R4 (see FIG. 3) is exemplified. Only the participant B is displayed on the display screen 252 of the participant A in FIGS. 12 and 13, and only the participant A is displayed on the display screen 252 of the participant B. FIG. The display of the table 183c, the whiteboard 183d, etc. can be switched during the conference.

First, the control unit 12 determines whether or not an entry request has been received from the terminal device 20 in step S110 of FIG. Specifically, it accepts a login to the video conference system 100 from the user and a request to enter the virtual conference room. By entering the virtual conference room provided on the server of the video conference device 10, a two-way web conference can be held with another terminal device 20 already in the room. The virtual conference room here is a conference room selected in advance from FIG. 3 by the participant who will be the host.

Upon accepting entry into the virtual conference room, the control unit 12 registers the user as a participant, stores the participant information shown in FIG. Execute the process. In video processing, movement of a participant's face is detected, and moving image data is generated by selecting video and audio from imaging devices of other participants according to the movement of the face. Details of the video processing will be described later.

Subsequently, in step S<b>150 , the control unit 12 determines whether or not a request to leave the room has been received from the terminal device 20 . The processes of steps S130 and SS140 are repeated until the exit request is received. When the controller 12 determines that it has received a request to leave the room from the terminal device 20, in step S160 it ends the acquisition of the video and audio from the terminal device 20 that requested to leave the room, and logs out the participant. For other terminal devices 20, the processing of steps S130 and S140 is continued. When all the terminal devices 20 request to leave the room and the acquisition of the video and audio ends, the series of videoconference processing ends.

Next, the video processing of step S140 in FIG. 10 will be described in detail with reference to FIG. FIG. 11 is a flow chart showing a specific example of the video processing shown in FIG. This video processing is exemplary of the communication program of the present invention. In the video processing of the present embodiment, first, in step S142 shown in FIG. 11, the control unit 12 detects the movement of the participant's face from at least one image among the plurality of images acquired by the acquisition unit 121. FIG.

Specifically, the acquiring unit 121 acquires four images (images of faces in different directions) from all the imaging devices 24a, 24b, 24c, and 24d from the terminal devices 20A and 20B of the participants A and B, respectively, by the imaging device 24. are acquired for each of participants A and B. For each of the participants A and B, the movement detection unit 122 detects the movement of the faces of the participants A and B from at least one of the four images acquired. For example, the movement of the face is detected from the image of the front face captured by the imaging device 24c.

The movement of the face detected by the movement detection unit 122 of the present embodiment is not the movement of the line of sight or the direction of the face, but the movement of the face in the horizontal direction (rightward or leftward) or vertical direction (upward or downward). is. For example, FIG. 13 shows a case where the motion detection unit 122 detects that the face of participant A is moving (moving the face to the left), and that the face of participant B is not moving.

When the movement of the face is detected in step S142, the control unit 12 selects one video and audio from four videos of the other participants in step S144 according to the movement of the face, and in step S146. An output video and an output audio are generated from the selected video and audio, and the output video and the output audio are output as moving image data in step S148.

Note that the motion detection unit 122 detects that the face has moved when the distance over which the face has moved is greater than or equal to a predetermined distance. It is also possible to detect that there is no movement of the According to this, it is possible to prevent the image from changing even if the face moves slightly. By adjusting the predetermined distance for detecting facial movements, it is possible to prevent the image from changing too frequently due to subtle movements of the face, or to change the image only when the face is intentionally moved. can be adjusted to

Specifically, as shown in FIG. 12, while the movement of the face is not detected, in step S144, both the participants A and B receive an image of the front of the face from the imaging device 24c (image of the imaging device 24c). Selection is made by the video selection unit 123 , and left and right sounds from the microphones 26 of both the participants A and B are selected by the sound selection unit 125 . In step S146, the video generation unit 124 generates output video from the selected video, and the audio generation unit 126 generates output audio from the selected audio. In step S148, the output unit 127 outputs the generated output video and output audio as image data.

In this way, on the display screen 252 of the participant A, the image of the front of the face of the participant B (image of the imaging device 24c) is displayed, and the left and right sounds are output as they are. On the display screen 252 of the participant B, the front image of the face of the participant A (image of the imaging device 24c) is displayed, and the left and right sounds are output as they are.

12 during the conference, the movement of the face of participant A to the left is detected in step S144, and the movement of the face of participant A to the left is detected in step S146. For the person B, the image on the right side of the face (image of the imaging device 24 a ) is selected by the image selection unit 123 , and the sound on the right from the microphone 26 is selected by the sound selection unit 125 . In step S146, the image generator 124 generates an output image of the right side of the face from the selected image. The sound generator 126 generates output sound from the selected sound in which the sound from the participant B is output to the terminal device 20A of the participant A such that the sound on the right side is stronger than the sound on the left side. At this time, as shown in FIG. 13, it appears to participant B that participant A has moved to the right. Produces an output sound that is stronger. In step S148, the output unit 127 outputs the generated output video and output audio as image data.

In this way, on the display screen 252 of the participant A, the image of the right side of the face of the participant B (the image of the imaging device 24a) is displayed, and the sound on the right side is stronger than that on the left side. On the display screen 252 of the terminal device 20B of the participant B, the front image of the face of the participant A (image of the imaging device 24c) is displayed, and the left and right sounds are output as they are. Then, when the movement of participant A's face toward the right direction is detected, on the display screen 252 of participant A, the image of the left side of participant B's face (image of the imaging device 24b) is displayed, and the right side is displayed. A stronger sound is output than the left. On the other hand, on the display screen 252 of the participant B, the image of the front of the face of the participant A (the image of the imaging device 24c) is shifted to the right, and the sound on the right side is stronger than the sound on the left side. be done.

According to this embodiment, for example, the image of the other participant B is switched to the image of the image pickup device in which the participant A moves according to the movement of the face of the participant A, thereby realizing the actual conference. Participant B's profile can be seen in the room as if it were moving in that direction. In this way, it is possible to change the appearance of other participants in accordance with the movement of one's own face, so that one can feel as if one is actually there. In addition, depending on the movement of the face and the angle of the imaging device 24, it is also possible to match the line of sight, making it easier to have a conversation.

<Second embodiment>
A second embodiment of the present invention will be described. Constituent elements having substantially the same functional configuration in each form illustrated below are denoted by the same reference numerals, thereby omitting redundant description. In the first embodiment, the case of switching the video of the other participant according to the "movement of the face" on the display screen 252 was exemplified. A case where the video of another participant is switched by pressing a button or a case where the display range 183b of the virtual conference room is changed according to the "movement of the face" will be exemplified. The configuration of the video conference device 10 of the second embodiment is the same as that of the first embodiment, so detailed description thereof will be omitted.

FIG. 14 is a diagram showing a schematic configuration of the video conference system 100 of the second embodiment, in the case of four participants. In the video conference system 100 of FIG. 14, the server constituting the video conference device 10 includes a terminal device 20A of participant A, a terminal device 20B of participant B, a terminal device 20C of participant C, and a terminal device 20D of participant D. are connected by a network N. Since the terminal devices 20A, 20B, 20C, and 20D all have the same configuration as in FIG. 6, detailed description thereof will be omitted.

FIG. 15 is a diagram showing a specific example of the virtual conference room display information 183 of the second embodiment, and is an example of using the virtual conference room R2 in the middle of FIG. Each participant's video can be inserted into A-frame to D-frame in FIG. In the configuration of FIG. 14, the image of participant A is fitted in the A frame arranged in row y01, the image of participant B is fitted in the B frame arranged in row 02, and the C frame arranged in column t01. , and the video of participant D is inserted in the D frame arranged in column t02. In FIG. 14, the display range 183b in FIG. 15 is stereoscopically displayed. In FIG. 5, the display range 183b is the range including the columns in which the participants A to D are arranged and the table 183c. In FIG. 15, as in FIG. 5, the whiteboard 183d is not included in the display range 183b, so it is not displayed.

Depending on the positions of the participants A to D in the virtual conference room of FIG. 15, the positions and face orientations displayed on the display screen 252 of the participants A to D differ. Specifically, on the display screen 252 of each participant, the face position and face orientation are displayed in substantially the same manner as in the actual conference room. FIG. 16 is a diagram showing the relationship between how participant A appears on the display screen 252 and the imaging devices of other participants. FIG. 17 is a diagram showing the relationship between how participant B appears on the display screen 252 and the imaging devices of other participants.

Regarding the positions of the participants A to D, the other participants are arranged in the same positions as those seen on the back side, right side, and left side when the participant is placed in front of the table 183c in the virtual conference room of FIG. . For example, as shown in FIG. 16, on the display screen 252 of participant A, he (participant A) is on the front side of the table 183c, participant B is on the back side, participant C is on the left side, and participant D is on the left side. displayed on the right side. On the other hand, as shown in FIG. 17, on the display screen 252 of the participant B, the participant B (himself) is on the front side of the table 183c, the participant A is on the back side, the participant D is on the left side, and the participant C is on the left side. are displayed on the right side respectively.

The face directions of the participants A to D were set to be the same as the face directions seen from the back, right, and left sides of the virtual conference room shown in FIG. Display the image of the face orientation of the participant. For example, as shown in FIG. 16, on the display screen 252 of participant A, for participant A (himself) and participant B, images of the front of the face from the imaging device 24c are displayed, and for participant C, the imaging device 24a is displayed. The image of the right side of the face of the participant D is displayed, and the image of the left side of the face of the participant D from the imaging device 24b is displayed. On the other hand, on the display screen 252 of the participant B as shown in FIG. For C, an image of the left side of the face from the imaging device 24b is displayed, and for participant D, an image of the right side of the face is displayed from the imaging device 24a.

16 and 17, it can be seen that the positions and face directions of participants C and D are left-to-right reversed. As described above, according to the second embodiment, the video of the imaging device is switched so that the positions and face directions of the other participants A to D when it is assumed that the participant is in front in the virtual conference room are switched. It is displayed on the display screens 252 of the participants AD. As a result, the display screens 252 of the participants A to D can be displayed in the same position and orientation as in the actual conference room. Therefore, it is possible to hold a Web conference as if you were in the conference room.

FIG. 18 is a flowchart showing a specific example of video processing according to the second embodiment, which is a modification of the video processing shown in FIG. In the video processing of FIG. 18, the display range 183b can be changed according to the movement of the participant's face (movement of the face). 19A and 19B are diagrams for explaining the action when the movement of the face is detected in the second embodiment, and FIG. 20 is a diagram showing a specific example of the virtual conference room display information 183 at that time.

In the video processing of the second embodiment, first, at step S242 shown in FIG. Specifically, the movement detection unit 122 detects the movement of the participant's face from at least one of the four images acquired for each participant by the acquisition unit 121 (for example, the front face image). Since step S242 is the same as step S142 in FIG. 11, detailed description thereof will be omitted. When the movement of the face is detected in step S242, the control unit 12 selects video and audio of other participants according to the movement of the face in step S244.

In step S244 of the second embodiment, the image selection unit 123 selects the display range 183b according to the movement of the face. For example, when the participant A moves his/her face to the left as shown in FIG. 19, the image selection unit 123 selects a display range 183b' shifted in the same direction (to the left) as the movement of the face as shown in FIG. . If participant A moves his face to the right, a right-shifted display range (not shown) is selected. When the participant A moves his/her face upward, an upward display range (not shown) is selected. selected. As for the voices, the voices of participants B and C included in the display range 183b' are selected by the voice selection unit 125, and the voices of the participant D not included in the display range 183b' are not selected by the voice selection unit 125.

In step S246, the control unit 12 generates output video and output audio from the video and audio in the selected display range 183b, and outputs the output video and output audio as moving image data in step S248. Thus, as shown in FIG. 19, the display range 183b' of FIG. 20 is reflected and displayed on the display screen 252 according to the facial movement of the participant A.

In FIG. 20, the display range 183b before detecting the movement of the face is indicated by a dotted line. The display range 183b' includes the participants A to C closer to the right than the display range 183b. Moreover, the display range 183b includes the participant D, whereas the display range 183b' does not include the participant D. Thereby, the participant A can make the specific participant D invisible on the display screen 252 by moving his/her face. Moreover, since the voice of participant D that is not included in the display range 183b' is not selected by the voice selection unit 125, it cannot be heard by the terminal device 20 of participant A. FIG.

Thus, in the second embodiment, the display range 183b can be changed according to the movement of the face with respect to the display screen 252. FIG. As a result, it is possible to have a conversation with just the participant you want to talk to by moving your face. 19 and 20 illustrate the case where the display range 183b is shifted to the left by moving the face to the left with respect to the display screen 252, but the present invention is not limited to this. ) or up and down (vertical direction), the display range 183b can also be shifted left and right or up and down. Specifically, the display range 183b can be shifted to the right by moving the face to the right. By moving the face further upward, the display range 183b can be shifted upward. The display range 183b can also be shifted downward by moving the face downward.

Also, the shift amount of the display range 183b can be changed according to the distance the face moves. For example, the more the face is moved, the greater the shift amount of the display range 183b. As a result, if the display range 183b is regarded as the field of view of the participant, the participant can see according to the movement of the face, so that it is possible to obtain a sense of realism as if being in an actual conference room.

According to the second embodiment as described above, since a video is generated that identifies the position where the participant is displayed according to the position of the participant in the virtual conference room, the positions of the other participants can be displayed as if they were in the actual conference. You can conduct video conferences and conversations as if you were in the room. In addition, since the video of the participant is selected according to the position of the participant in the virtual conference room, for example, the video of the side face of the participant next to the participant can be displayed. This makes it possible to conduct video conferences and conversations as if other participants were in the actual conference room.

In addition, in the second embodiment, the display range 183b of the virtual conference room is selected according to the facial movements of the participants, and the participants included in the display range 183b among the plurality of participants are displayed on the terminal device 20. image can be generated. According to this, by moving one's face, it is possible to display a desired range of the virtual conference room. For example, if it is desired to see only the participants on the right side of the table 183c, only the participants on the right side can be seen by moving the face to the right. It is also possible to display only the participant by moving the face toward the participant to have a conversation, just like going near the participant and having a conversation in an actual conference room. In this way, it is possible to obtain a sense of realism as if you were having a conversation in an actual conference room.

<Third Embodiment>
A third embodiment of the present invention will be described. The motion detection unit 122 of the first and second embodiments has exemplified the case of detecting "face movement" (horizontal and vertical movement with respect to the display screen 252) as "face movement". Not limited. The movement detection unit 122 of the third embodiment detects "face orientation" as "face movement", and exemplifies a case where the appearance of the participants in the virtual conference room is changed according to the face orientation. The configuration of the video conference device 10 of the third embodiment is the same as that of the first embodiment, so detailed description thereof will be omitted. A video conference system 100 according to the third embodiment will be described by exemplifying the case of four participants (FIG. 14) as in the second embodiment.

FIG. 21 is a flowchart showing a specific example of video processing according to the third embodiment, which is another modified example of the video processing shown in FIG. In the video processing of the third embodiment, first, in step S342 shown in FIG. 21, the control unit 12 detects the orientation of the face of the participant from the acquired video. When the orientation of the face is detected in step S342, the control unit 12 selects video and audio of other participants according to the orientation of the face in step S344. The control unit 12 generates output video and output audio from the video and audio in the display range 183b selected in step S346, and outputs the output video and output audio as moving image data in step S348.

In such a video processing of FIG. 21, by changing the display range 183b according to the movement of the face of the participant (orientation of the face), the appearance of the participants in the "same line" in the virtual conference room display information 183 is changed. be able to. A specific description will be given below with reference to FIGS. 22 to 27. FIG. FIG. 22 is a diagram showing a specific example of the virtual conference room display information 183 in the third embodiment. FIG. 23 is a diagram showing how the virtual meeting room display information 183 of FIG. 22 is viewed from the participant A. FIG. FIG. 24 is a diagram showing the virtual conference room display information 183 when the direction of the face moves from the front to the left, and FIG. 25 is a diagram showing how participant A sees the virtual conference room display information 183 in FIG. is. FIG. 26 is a diagram showing virtual conference room display information 183 when the direction of the face moves from the front to the right, and FIG. 27 is a diagram showing how participant A sees the virtual conference room display information 183 in FIG. is.

According to FIG. 22, participants C and D are in the same row as participant A (row y01), participant C is arranged to the left of participant A, and participant D is arranged to the right of participant A. It is In the third embodiment, participants in the same row can be displayed only when the direction of the face is changed. That is, when participant A faces forward as shown in FIG. 22, the display range 183b in FIG. As shown in FIG. 25, when the motion detection unit 122 detects that the face of the participant A faces left, only the left participant C in the same row is displayed according to the virtual conference room display information 183 in FIG. is output. On the other hand, as shown in FIG. 27, when the motion detection unit 122 detects that the face of the participant A faces rightward, only the right participant D in the same row is displayed in the display range 183b of FIG. Sound is output.

As described above, in the third embodiment, when participant A is facing the front and no facial movement is detected, participants C and D are not displayed and only participant B can have a conversation. . On the other hand, when the participant A turns left, he/she can talk with the participant C, and when the participant A turns right, he or she can talk with the participant D. This makes it possible to talk to participants in the same row only when necessary in the actual conference room on the Web conference.

For example, in an actual conference room, if participants from your company and participants from other companies hold a negotiation meeting, the participants from your company line up in the front row across the table, and the participants from other companies line up in the back row of the table. may line up. In this case, there is no need to talk to the participants of the company while negotiating with the participants of other companies. Your company's participants may want to speak only when necessary. Such a situation can also be realized in the third embodiment. That is, by changing the orientation of the face, the participants of the company can be displayed and the conversation can be conducted. In this way, even in the third embodiment, it is possible to experience a sense of realism as if one were in an actual conference room.

As described above, according to the third embodiment, the motion detection unit 122 determines whether the face direction of the participant is front, right, or left from at least one image among the four images acquired by the acquisition unit 121. It is possible to detect whether or not When the detected face orientation is the front, the image selection unit 123 selects the display range 183b of the virtual conference room that does not include the images of the participants positioned on the right and left sides, and selects the detected face orientation to the right. In the case of , select the display range 183b of the virtual conference room that includes the video of the participant located on the right side, and if the detected face direction is left, select the virtual conference room that includes the video of the participant located on the left side. display range 183b can be selected.

This allows you to turn to the right when talking to the participant on the right, and turn to the left when talking to the participant on the left, so you can feel as if you were talking to the participants next to you in a conference room. Obtainable. Also, if there are multiple participants on your right side, you can turn to the right when you want to see them, and if you want to see multiple participants on your left side, you can turn to the left when you want to see them. You can get a sense of realism as if you changed the direction of your face in a room. In addition, in the third embodiment, the case where there are one participant on the left side and one on the right side of the participant A (himself) has been exemplified, but the present invention is not limited to this. For example, it can also be applied when there are multiple participants on the left and right sides of oneself. In this case, looking to the right will allow you to see all participants to your right, and looking to your left will allow you to see all participants to your left.

<Modified Examples of First to Third Embodiments>
(1) In the first to third embodiments, the positions (seats) of the participants are set in advance in the virtual conference room display information 183, but the present invention is not limited to this. The positions (seats) of the participants in the virtual conference room display information 183 may be changed during the conference. For example, FIG. 28 is a diagram showing a specific example of the virtual conference room display information 183 according to the first modification, in which the position (seat) of participant D in column t02 in FIG. 15 is moved to row y02. FIG. 29 is a diagram showing how participant A sees after participant D moves his/her seat. FIG. 30 is a diagram showing how the participant D looks before the participant D moves, and FIG. 31 shows how the participant D looks after the participant D moves.

As shown in FIG. 29, when the seat of participant D is moved, the seat of participant A appears to move from the right side of the table 183c to the side of participant C on the back side, so that the movement can be understood at a glance. From the perspective of participant D, before the seat change, participant C could be seen on the far side of table 183c as shown in FIG. becomes visible to participant A. According to such a first modification, by moving the seats in the virtual conference room display information 183, it is possible to achieve the same appearance as in the actual conference room even in the video conference, so it is possible to enhance the sense of presence. .

(2) In the first embodiment, the case of two participants, and in the second and third embodiments, the case of four participants is illustrated, but the present invention is not limited to this. It may be a case of holding a conference by displaying a large number of participants. For example, FIG. 32 is a diagram showing a specific example of the virtual conference room display information 183 according to the second modification, exemplifying the display range of the large conference room R3 in FIG. 3 (in the case of 16 participants). FIG. 33 is a diagram showing how the virtual conference room display information 183 of FIG. 32 is viewed from the participant A1. In FIG. 32, there are three participants (A2, A1, A3) in row y01, four participants (B1, B2, B3, B4) in row y02, and four participants (C1, C2, C3, C4), four participants (D1, D2, D3, D4) and one participant (A4) as a presenter are arranged in column t02. As shown in FIG. 33, on the display screen 252 of participant A1, three participants (A2, A1, A3) are displayed side by side, and four participants (B1, B2, B3, B4) are viewed side by side on the back side of the table 183c. Images of the right faces of the four participants (C1, C2, C3, C4) are arranged on the left side of the table 183c, and images of the left faces of the four participants (D1, D2, D3, D4) are arranged on the right side of the table 183c. are displayed side by side. Furthermore, since the image of participant A4 as a presenter can be seen on the upper right of display screen 252, it can be seen at a glance that participant A4 is giving a presentation. Note that the whiteboard 183d may be displayed on the table 183c as shown in FIGS. In this case, the orientation of the whiteboard 183d can be adjusted so that all participants can see it in the same way.

In FIG. 32, a column p01 of presenters (an example of a column of speech areas) is provided separately from other rows y01 and y02 and columns t01 and t02. By providing a column p01 for displaying the participants in such a comment area and having the participants move to this column p01 to speak, it becomes easy to understand who is speaking. FIG. 32 illustrates a case where participant A4 is the presenter, and participant A4 has moved from the horizontal row y01 to the presenter row p01. In FIG. 32, the column p01 of presenters (speech area column) is for one person, but it may be for two or more presenters. This is because there are cases where two or more people make a presentation. In this way, even if there are a large number of participants, by moving to the presenter column p01 (speech area column) and giving a presentation or making a statement, it is possible to see who is giving the presentation, who is speaking, etc. can be seen at a glance, and the appearance can be similar to that of presenting or speaking in front of participants in an actual conference room, so the presence can be enhanced.

<Fourth Embodiment>
A fourth embodiment of the present invention will be described. In the first to third embodiments and their modifications, images of the faces of the participants facing different directions are acquired from the imaging device 24, so that the "faces" of other participants are captured in accordance with the movements of the faces of the participants. A case in which the "image of the In the fourth embodiment, an omnidirectional (360-degree) camera is used as the image pickup device 24 to acquire omnidirectional images in different directions, so that other participants'"surroundingimages" are captured according to the movement of the participant's face. ” is also changed. The "surrounding image" here includes not only the background image of the participant but also images such as the front image, the right image, and the left image. In the fourth embodiment, the appearance of such surrounding images can be changed according to the movement of the participant's face, thereby enabling a more realistic experience.

In the past, for example, when a product development meeting was held by video conference, when the video of the prototype placed on the table was to be viewed from a different direction, the participants moved the prototype so that it could be seen by the camera. I couldn't see it unless I had someone hold me or move the camera. However, it is surprisingly difficult to have the prototype pointed in the direction you want to see, or to have the camera move in the direction you want to see. In conventional video conferencing, even if the participant who is watching the video tells the other participants to move to the right or left instead of there, it is difficult to see the video they want to see. There was frustration.

In this respect, according to the fourth embodiment, even if the participants on the spot do not move the prototype or the camera, the participant who is watching the video simply moves his/her face in the desired viewing direction. You can switch to the direction you want to see the surrounding images including the image of , and you can also move the surrounding images. According to this, it can be used not only in product development meetings, but also in various situations such as when you want to talk remotely with a site supervisor while watching a video of a building at a construction site, or when you want to talk to a clerk while looking at a product in a showroom. . What's more, by simply moving the face of the participant watching the video in the direction they want to see, the appearance of the surrounding video, including the video of the building and the video of the product, can be changed, giving a sense of realism as if they were there. communication can be carried out smoothly.

The communication system 100 of such a fourth embodiment will be described in detail with reference to FIGS. 34 and 35. FIG. FIG. 34 is a diagram showing a schematic configuration of the communication system 101 of the fourth embodiment. FIG. 35 is a schematic diagram of the terminal device 20B of FIG. 34 as viewed from the front, showing a specific example of the installation position of the imaging device. A communication system 101 of FIG. 34 is configured by connecting a terminal device 20A of a participant A and a terminal device 20B of a participant B to a server constituting a communication device 11 via a network N. FIG. Since the terminal devices 20A and 20B have substantially the same configuration as in FIG. 6, elements having similar functions are denoted by similar reference numerals.

FIG. 34 is an example in which not only images of the faces of participants A and B in front of the imaging device 24 but also images of the structure 30 behind the imaging device 24 are displayed on the terminal devices 20A and 20B. Since the image pickup device 24 of the first to third embodiments is an inner camera facing toward the participant who is looking at the terminal device 20, it is not possible to photograph an object behind the image pickup device 24. FIG. Therefore, in FIG. 34, by applying an omnidirectional camera to the imaging device 24, not only the image of the participant's face but also the surrounding image including the image of the structure T can be viewed on the terminal devices 20A and 20B. It is what I did. Hereinafter, such an imaging device 24 will be specifically described.

The terminal devices 20A and 20B in FIG. 34 are respectively provided with imaging devices 24 (imaging devices 24e, 24f, and 24g) configured by omnidirectional cameras. The imaging devices 24e, 24f, and 24g capture omnidirectional images including images of the faces of the participants in different orientations and images of the surroundings in different orientations, as shown in FIG. 41, which will be described later. The "surrounding image" here is a horizontal 360-degree omnidirectional image including the image of the participant's face. For example, the participant's background image, front image, right image, left image, etc. are included. .

The imaging devices 24 (24e, 24f, and 24g) do not necessarily have to be installed in the display device 25. If it is possible to acquire video data including images of faces in different directions and images of the surroundings in different directions, they can be used on walls, desks, and so on. may be installed in any location. Also, the imaging device 24 does not necessarily have to be electrically connected to the terminal devices 20A and 20B. The imaging device 24 may have any configuration as long as image data including a face image in a different direction and a surrounding image in a different direction are transmitted to the communication device 11 via the network N.

It should be noted that the imaging device 24 may not necessarily be a camera capable of capturing images in all directions (360 degrees). For example, a wide-angle camera capable of capturing an image of a predetermined angle or more may be used. The imaging device 24 may have a configuration in which two wide-angle cameras are combined with an inner camera and an outer camera to capture an omnidirectional video, or an inner camera and an outer camera such as a smartphone may be used as the imaging device 24. In the fourth embodiment, for example, as shown in FIG. 36, an imaging device 24 capable of capturing an omnidirectional video with an outer camera 246 and an inner camera 247 is exemplified. According to this, an image of the face of the participant and an image of the background can be obtained from the inner camera 247 , and an image in front of the participant can be obtained from the outer camera 246 . Also, the imaging device 24 may be a camera with a single fisheye lens or a convex mirror. The orientation of the imaging device 24 is also not limited to that illustrated. As long as the image of the participant's face and the image of the surroundings can be captured, the lens may be set upward or downward. For example, if an imaging device mounts an omnidirectional lens, the lens may be arranged upward. Also, the number of imaging devices 24 in FIG. 34 is not limited to three.

Here, installation positions of the imaging devices 24e, 24f, and 24g and image examples will be described with reference to FIGS. 35 and 36. FIG. FIG. 35 is a schematic diagram of the terminal device of FIG. 34 viewed from the front, showing a specific example of the installation position of the imaging device. FIG. 36 is a top view of the terminal device 20B in FIG. 34, showing a specific example of surrounding images from the imaging devices 24e, 24f, and 24g. Since the terminal device 20A has the same configuration, the terminal device 20B will be described here as a representative.

As shown in FIG. 35, the imaging devices 24e, 24f, and 24g are arranged side by side above the display device 25. As shown in FIG. As shown in FIG. 36, the imaging device 24g is a central imaging device (third imaging device) arranged in the upper center of the display device 25. As shown in FIG. The imaging device 24g can image a horizontal 360-degree omnidirectional video 41g as shown in FIG. 41(a), which will be described later. The omnidirectional image 41g includes a front image of the face of the participant B as shown in FIG. 8 and a front image 40g of the structure 30 as shown in FIG.

As shown in FIG. 36, the imaging device 24e is a right imaging device (first imaging device) that is arranged above the display device 25 and spaced to the right of the imaging device 24g. The imaging device 24e can capture a horizontal 360-degree omnidirectional image 41e as shown in FIG. 45(a), which will be described later. The omnidirectional image 41e includes an image of the right side of the face of the participant B as shown in FIG. 8 and an image 40e from the front right side of the structure 30 as shown in FIG.

As shown in FIG. 36, the imaging device 24f is a left imaging device (second imaging device) arranged above the display device 25 and spaced leftward from the imaging device 24g. The imaging device 24f can capture a horizontal 360-degree omnidirectional video 41f as shown in FIG. 43(a), which will be described later. The omnidirectional image 41f includes an image of the left side of the face of the participant B as shown in FIG. 8 and an image 40f of the front left side of the structure 30 as shown in FIG. In this manner, the imaging devices 24e, 24f, and 24g can capture surrounding images in different directions (front, rightward, leftward, etc.).

As shown in FIG. 36, the structure 30 comprises, for example, a substrate 31 and a plurality of objects 32, 34, 36, 38 arranged thereon. Object 32 is a rectangular parallelepiped, object 34 is a square pyramid, object 36 is a hexagonal prism, and object 38 is a square prism. Objects 32 and 34 are arranged substantially in the front center of substrate 31 , and object 34 is arranged behind object 32 . When the object 32 is viewed from the front, an object 36 is arranged on the right side of the front (right side when facing), and an object 38 is arranged on the left side of the front (left side when facing).

By the way, according to the front image 40g of FIG. 36, it can be seen that the object 34 is hidden by the object 32 and only the triangular portion at the tip can be seen. With this, the shape of the object 34 is also unknown. If only the front image 40g is seen, the object 34 looks like a triangular plate placed on top of the object 32 .

On the other hand, the right side of the object 34 can be seen in the image 40e shifted rightward from the front, and the left side of the object 34 can be seen in the image 40f shifted leftward in the front. According to this, it can also be seen that the object 34 is arranged behind the object 32 separately from the object 32, and that the object 34 is a pyramid, not a triangular plate. As described above, according to the present embodiment, it is possible to obtain an image to the right of the front and an image to the left of the front from an omnidirectional image including an image oriented in a direction different from the front. come to understand.

Images 40e, 40f, and 40g in FIG. 34 are unfolded images obtained by extracting images of display ranges including the structure 30 from the omnidirectional images of the imaging devices 24e, 24f, and 24g, respectively. A major feature of the fourth embodiment is that the display range cut out from the omnidirectional video can be changed according to the "face movement" of the participant.

The configuration of the communication system 101 according to the fourth embodiment will be described below with reference to FIGS. 37 to 40. FIG. FIG. 37 is a block diagram of the communication system 101 of the fourth embodiment, showing a specific configuration example of the communication device 11 and the terminal device 20. As shown in FIG. Since the configuration of the terminal device 20 in FIG. 37 is substantially the same as in FIG. 2, detailed description thereof will be omitted. The configuration of the communication device 11 in FIG. 37 is the same as that of the video conference device 10 in FIG.

The communication device 11 of FIG. 37 includes a video selection unit 123A, a video generation unit 124A, a sound selection unit 125A, and a sound generation unit 126A. The image selection unit 123A includes a face image selection unit 123a and a surrounding image selection unit 123b. The face image selection unit 123 a selects “face images” of other participants according to the “face movements” of the participants detected by the motion detection unit 122 . The surrounding image selection unit 123 b selects a “surrounding image” according to the “face movement” of the participant detected by the motion detection unit 122 .

The image generation unit 124A includes a face image generation unit 124a and a surrounding image generation unit 124b. The face image generation unit 124a generates a participant image to be displayed on the terminal device 20 from the face image selected by the face image selection unit 123a. The surrounding image generation unit 124b generates a surrounding image to be displayed on the terminal device 20 from the surrounding image selected by the surrounding image selection unit 123b. Thus, the image selection unit 123A and the image generation unit 124A can select and generate not only the "face image" but also the "surrounding image".

The sound selection section 125A includes a sound selection section 125a and an ambient sound selection section 125b. The voice selection unit 125a selects voices of other participants according to the "face movements" of the participants detected by the motion detection unit 122. FIG. The ambient sound generator 126 b selects ambient sounds according to the “face movements” of the participants detected by the motion detector 122 .

The sound generation section 126A includes a sound generation section 126a and an ambient sound generation section 126b. The sound generator 126a generates a sound to be output by the terminal device 20 from the sound selected by the sound selector 125a. The ambient sound generator 126b generates ambient sounds to be output by the terminal device 20 from the ambient sounds selected by the ambient sound selector 125b. In this way, the sound selection unit 125A and the sound generation unit 126A can select and generate not only "voice" but also "surrounding sound". In this case, the acquisition unit 121 may acquire the voice and the ambient sound from separate microphones 26, or may acquire the voice and the ambient sound separately from the input sound of one microphone 26. good. Also, the ambient sound selection unit 125b and the ambient sound generation unit 126b may not necessarily be provided. In that case, only the sound may be output from the terminal device 20, or the sound with the surrounding sounds may be output from the terminal device 20. FIG.

The data table in FIG. 38 is virtual conference room configuration information 181 for displaying images of participants and surrounding images on the terminal device 20 using the virtual conference room, and corresponds to FIG. The virtual conference room configuration information 181 in FIG. 38 differs from that in FIG. 3 in that face images and surrounding images are included instead of tables and boards. The items in FIG. 38 are not limited to those shown in the figure, and items such as tables and boards may be included. According to FIG. 38, for example, in the small conference room R1, there is a participant's face image and a surrounding image, so the display range including the face image and the surrounding image is set. The data table in FIG. 39 is the participant information 182 and corresponds to FIG. The image data received in this embodiment are three omnidirectional images from the imaging devices 24e, 24f, and 24g.

The virtual conference room display information 183 of FIG. 40 is display information of the virtual conference room into which each participant enters. For example, in the virtual conference room display information 183 in FIG. 40, in addition to the row in which each participant is arranged and the position of each participant (the position of the face image frame in which the image of the participant is inserted), the surrounding image frame in which the surrounding image is inserted The position of r01, the display range 183b of the virtual conference room, and the like are included. The virtual conference room display information 183 is provided for each type of virtual conference room in the virtual conference room configuration information 181 of FIG. The host participant can set in advance which virtual conference room to use.

FIG. 40 illustrates a case where two participants A and B use the virtual conference room of the small conference room R1 in FIG. As shown in FIG. 40, an outer frame 183a indicates the entire virtual conference room. The thick frame inside it indicates the display range 183b. The small conference room R1 in FIG. 38 has a capacity of two people, a face image, a surrounding image, and one row. In FIG. 40, a surrounding image frame r01 is arranged on almost the entire surface, and a row y01 including the face image frame yA of the participant A and the face image frame yB of the participant B is arranged below it. Note that the configuration of the virtual conference room display information 183 is not limited to that illustrated.

The communication device 11 of the fourth embodiment also performs videoconference processing similar to that of FIG. The video processing of step S140 in FIG. 10 will be described with reference to FIG. FIG. 11 is a flow chart showing a specific example of the video processing shown in FIG. This video processing is exemplary of the communication program of the present invention. Also in the video processing of the fourth embodiment, similar processing is performed from step S142 to step S148. The video processing of the fourth embodiment differs from that of the first embodiment only by selecting video and audio of other participants' faces according to the movement of the faces of the participants detected in steps S144 and S146. In addition, the surrounding image and surrounding sound are also selected to generate the output image and the output sound.

The video processing of FIG. 11 of the fourth embodiment will be described in detail below with specific examples of FIGS. 41 to 46. FIG. Here, a case is exemplified in which the images of the faces of participants A and B and the image of the structure 30 in front of participant B are shared and displayed by the terminal devices 20A and 20B. 41 to 46 are diagrams for explaining how the image of participant B and the image of the structure 30 change according to the movement of participant A's face. FIGS. 41 and 42 show the cases where no movement of the face of the participant A is detected, FIGS. 43 and 44 show the cases where the movement of the face of the participant A to the left is detected, and FIGS. is a case where participant A detects that the face moves to the right. 41(a), 43(a), and 45(a) are omnidirectional images from the terminal device 20B of the participant B. FIG. 41(b), 43(b), and 45(b) are output images generated from the omnidirectional images. FIG. 42 is a diagram showing a display image viewed from the front by participant A without moving his/her face. FIG. 44 is a diagram showing a display image seen when participant A moves his face to the left. FIG. 46 is a diagram showing a display image seen when participant A moves his face to the right.

In the video processing of the present embodiment, first, in step S142 shown in FIG. 11, the control unit 12 detects the movement of the participant's face from at least one image among the plurality of images acquired by the acquisition unit 121. FIG. Specifically, the acquisition unit 121 acquires three omnidirectional images (surrounding images in different directions) from the terminal devices 20A and 20B of the participants A and B, and all the imaging devices 24e, 24f, and 24g. are acquired for each of participants A and B.

For example, from the imaging device 24g of the participant B, an omnidirectional image including the front image of the structure 30 and the front face image of the participant B as shown in FIG. From the imaging device 24f of the participant B, an omnidirectional image including the front left image of the structure 30 and the left face image of the participant B as shown in FIG. 43(a) is obtained. From the imaging device 24e of the participant B, an omnidirectional image including the front right image of the structure 30 and the right face image of the participant B as shown in FIG. 45(a) is acquired. An omnidirectional image including a face image and surrounding images is also acquired from the imaging devices 24e, 24f, and 24g of the participant A.

For each of the participants A and B, the facial image of the participant is recognized from at least one of the three omnidirectional images acquired, and the motion detection unit 122 detects the facial movements of the participants A and B from the facial image. detect. For example, the movement of the face of participant B included in the omnidirectional video shown in FIG. 41(a) is detected. In this case, it is not always necessary to use an omnidirectional image to detect the movement of the face, and an image obtained by developing the face image of the participant may be used.

The movement of the face detected by the movement detection unit 122 of the present embodiment is a movement of the face to the right or left and a change in orientation such as left or right. When the movement of the face is detected in step S142, the control unit 12 selects one video and audio from the three omnidirectional videos of the other participants in step S144 according to the movement of the face, and selects one video and audio in step S146. The output video and output audio are generated from the omnidirectional video and audio selected in step S148, and the output video and output audio are output as moving image data.

Specifically, when the motion detection unit 122 detects that the participant A's face does not move as shown in FIG. The azimuth image 41g is selected. Specifically, for the face image, the face image selection unit 123a selects the display range 42g of the face image of the participant B from the omnidirectional image 41g shown in FIG. 42g is cut out and the facial image of participant B is developed as shown in FIG. 41(b). As for the surrounding image, the surrounding image selection unit 123b selects a display range 43g of the surrounding image including the image of the structure 30 from the omnidirectional image 41g shown in FIG. 43g is cut out, and the surrounding image including the image of the structure 30 is developed into a rectangle as shown in FIG. 41(b). The face image of the participant A is similarly cut out from the omnidirectional image 41g from the imaging device 24g of the participant A and developed into a rectangle.

Based on the virtual conference room display information 183 of FIG. 40, the video generation unit 124A generates the peripheral video including the face video of the participant A, the face video of the participant B, and the video of the structure 30 developed as described above. to generate the output video from. Specifically, the image generation unit 124A inserts the surrounding image including the image of the structure 30 into the surrounding image frame r01 of FIG. The face image of the participant B is fitted in the face image frame of the participant B in the row y01. In this way, the image generation unit 124A generates the output image of FIG. 41(b) from the omnidirectional image 41g of FIG. 41(a). For both the voices of the participants A and B and the ambient sounds, the left and right voices and the ambient sounds from the microphones 26 are selected by the voice selector 125a and the ambient sound selector 125b, and the voice generator 126a and the ambient sound generator 126b are selected. Generate output audio from speech and ambient sound. The output unit 127 outputs the output video and output audio shown in FIG. 41(b) as image data.

Then, as shown in FIG. 42, the front image of the participant B's face and the front image of the structure 30 are displayed on the display screen 252 of the participant A, and the left and right sounds are output as they are. A similar image is displayed on the display screen 252 of the participant B as well. As described above, in the fourth embodiment, not only the image of the participant's face but also the surrounding image can be displayed. This allows the participants to have a conversation while watching the surrounding images.

However, in the image of the front of the structure 30 in FIG. 42, the object 34 is hidden behind the object 32 and cannot be seen. Therefore, when the participant A moves his face to the left as shown in FIG. 44, the movement of the face is detected by the movement detection unit 122, and the participant A participates from the omnidirectional video 41f of FIG. A display range 43f of the face image and the surrounding image of the person B is selected and developed into a rectangle, and the output image and the output sound shown in FIG. 43(b) are generated and output as image data. Then, as shown in FIG. 44, on the display screen 252 of the participant A, an image of the front left side of the structure 30 is displayed. As a result, the object 34, which was hidden behind the object 32 in the front image, becomes visible. That is, if the user moves his/her face to the left, the image of the structure 30 can be seen from the left side of the front, so that it is possible to have a realistic experience as if the player were there.

In this case, the display position of participant A may be moved leftward as shown in FIG. As a result, for example, as shown in FIG. 44, it is possible to create an image in which the participant A looks at the structure 30 from the left side of the front. Also, as shown in FIG. 43(b), the image of the participant B is on the left side of the face, so it looks as if the participant B is facing the participant A while conversing. By changing the display position of the participant in accordance with the orientation of the surrounding image selected in accordance with the movement of the participant's face in this manner, it is possible to display the surrounding image as if it were being viewed from that display position. As a result, it is possible to have a realistic experience as if you were there.

On the other hand, when the participant A moves his face to the right as shown in FIG. 46, the movement of the face is detected by the movement detection unit 122, and the omnidirectional image shown in FIG. A display range 43e of the face image and surrounding image of participant B is selected from 41e and developed into a rectangle, and the output image and output sound shown in FIG. 45(b) are generated and output as image data. Then, as shown in FIG. 46, on the display screen 252 of the participant A, an image of the front right of the structure 30 is displayed. This also makes it possible to see the object 34 that was hidden behind the object 32 in the front image. That is, if the user moves his/her face to the right, the image of the structure 30 can be seen from the right side of the front, so that it is possible to have a realistic experience as if the player were there.

In this case, the display position of participant A may be arranged to the right as shown in FIG. As a result, for example, as shown in FIG. 46, it is possible to create an image in which the participant A looks at the structure 30 from the right side of the front. Also, as shown in FIG. 45(b), since the image of participant B is on the right side of the face, it seems as if the participant B is facing the participant A and conversing with him/her. By changing the display position of the participant in accordance with the orientation of the surrounding image selected in accordance with the movement of the participant's face in this manner, it is possible to display the surrounding image as if it were being viewed from that display position. As a result, it is possible to have a realistic experience as if you were there.

By the way, in the image of the front of the structure 30 shown in FIG. 42, only a part of the object 38 can be seen, so the shape of the object 38 cannot be clearly understood. Therefore, since the object 38 is on the right side of the front, as shown in FIG. 45, the participant A turns his face to the left without moving his face. 47(b) is generated from the omnidirectional video 41g of FIG. output. At this time, the display range 43g in FIG. Then, as shown in FIG. 48, on the display screen 252 of the participant A, an image in which the entire structure 30 is moved to the left is displayed. As a result, the entire object 38, which was only partially visible in the front image, can be seen. That is, when the user turns his/her face to the left, the image on the left side of the structure 30 can be seen.

Also, in the image of the front of the structure 30 shown in FIG. 42, only a part of the object 36 is visible, so it is difficult to know what shape it is. Therefore, since the object 36 is on the right side of the front, as shown in FIG. 50, the participant A turns his face to the right without moving his face. 49(b) is generated from the omnidirectional video 41g of FIG. output. At this time, the display range 43g in FIG. Then, as shown in FIG. 50, on the display screen 252 of the participant A, an image in which the entire structure 30 is moved to the right is displayed. As a result, the entire object 36, which was only partially visible in the front image, can be seen. That is, when the user turns his/her face to the right, the image on the right side of the structure 30 can be seen, so that it is possible to have a realistic experience as if the user were actually there.

<Other Modifications>
The present invention is not limited to the above-described embodiments, and various applications and modifications described below are possible. Moreover, it is also possible to appropriately combine one or a plurality of arbitrarily selected aspects of these modifications, each of the above-described embodiments, and modifications thereof. In addition, it is obvious that a person skilled in the art can conceive various other modifications or modifications within the scope described in the claims, and these naturally belong to the technical scope of the present invention. understood as a thing.

(1) In the first to fourth embodiments and their modifications, images (including facial images and surrounding images) from the imaging device 24 acquired by the acquisition unit 121 and sounds (surrounding sounds are stored in the storage unit 14. The image from the imaging device 24 acquired from the acquisition unit 121 is not limited to acquisition in real time, and the image stored in the storage unit 14 is acquired. According to this, for example, a participant who cannot attend the video conference on the day can use the video and audio stored in the storage unit 14 to hold the video conference with the same face image and surrounding image as on the day. In this case, as in the case of real-time video, it is possible to change the image of the participant's face and the surrounding image according to the movement of the participant's face, so it is possible to experience as if you were there at the time you want to see it. It becomes possible.

(2) In the first to fourth embodiments and their modifications, the motion detection unit 122 detects the movement of the face of the participant in the left-right direction and the up-down direction. It may also be possible to detect movement of the face in the front-rear direction. For example, the movement of the face in the front-rear direction may be detected based on the change in the distance between the position of the face vector and the imaging device 24 . According to this, for example, when the participant approaches the display screen 25 of the terminal device 20 and the movement detection unit 122 detects the movement of the face in the forward direction. It will also be possible to enlarge the image of the face and the image of the surroundings. For example, when an object in the surrounding image has small characters that cannot be read, if the face is brought close to the display screen 25, the image is enlarged and the characters can be seen and read large. Conversely, if the face is moved away from the display screen 25, the image on the display screen 25 may be reduced. For example, when the image is large and only part of the object can be seen, moving the face away from the display screen 25 reduces the image so that the entire object can be seen. As a result, you can experience a sense of realism as if you were looking at the real thing.

(3) In the above-described first to third embodiments and their modified examples, drawings showing only face images are used for the purpose of making the contents easier to understand, but the present invention is not limited to this. For example, the image may include not only the face but also the upper half of the body as in the fourth embodiment. When the video conference device 10 receives a video including the upper half of the body and the face, the facial movement may be detected by recognizing the facial portion from the video. For example, by using a trained model machine-learned by AI (artificial intelligence) or an existing trained model, facial recognition and facial movement are detected from an image.

(4) In the above embodiment and modification, the case where the communication device of the present invention is applied to the video conference device 10 was exemplified, but the present invention is not limited to this. For example, it can be applied to various communication devices such as a web conference device, a video conference device, a video phone device, an online conference device, a video communication device, and a video communication device. In addition, the use of the present invention is not limited to conferences, and can be used in various situations such as conversations and dialogues between exhibition venues, nursing care facilities, hospital facilities, parents' homes, etc., between nursing facilities, between hospitals, etc. available for communication.

DESCRIPTION OF SYMBOLS 100... Video conference system (communication system), 10... Video conference apparatus (communication apparatus), 11... Communication part, 12... Control part, 121... Acquisition part, 122... Detection part, 123 (123A)... Video selection part, 123a Face image selection unit 123b Surrounding image selection unit 124 (124A) Image generation unit 124a Face image generation determination unit 124b Surrounding image generation unit 125A Sound selection unit 125 (125a) Sound selection Part 125b... Ambient sound selection part 126A... Sound generation part 126 (126a)... Sound generation part 126b... Ambient sound generation part 127... Output part 14... Storage part 15... Program storage part 16... Data Storage unit 161 User information 18 Virtual conference room storage unit 181 Virtual conference room configuration information 182 Participant information 183 Virtual conference room display information 183a Outer frame 183b Display range 183c Table 183d Whiteboard 20 (20A, 20B, 20C, 20D) Terminal device 20a Main body 20L Bus line 21 Communication unit 22 Control unit 23 Storage unit 24 (24a, 24b) , 24c, 24d)... Imaging devices 24a, 24e... First imaging device (right imaging device) 24b, 24f... Second imaging device (left imaging device) 24c, 24g... Third imaging device (central imaging device) , 24d... fourth imaging device (lower side imaging device), 242a, 242b, 242c, 242d... support portion, 244a, 244b, 244c, 244d... rotating shaft, 246... outer camera, 247... inner camera, 25... display device , 252... display screen, 26... microphone, 27... speaker, 28... input device, 40e, 40f, 40g... video, 41e, 41f, 41g... omnidirectional video, 42e, 42f, 42g... display range of face video, 43e , 43f, 43g... display range of surrounding video, t01...column, t02...column, y01...horizontal line, y02...horizontal line, A to D...participant, N...network, R1...small conference room, R2...medium conference room, R3: large conference room, R4: face-to-face conference room.

Claims (10)

A communication device for displaying images of a plurality of participants on a terminal device,
an acquisition unit that acquires a plurality of images including images of faces facing different directions for each of the plurality of participants;
a motion detection unit that detects movement of the face of at least one of the participants from the face image acquired by the acquisition unit;
an image selection unit that selects an image of the face direction of another participant according to the movement of the participant's face detected by the movement detection unit;
an image generation unit configured to generate an image for displaying at least an image of the other participant's face on the terminal device from the images selected by the image selection unit;
A virtual conference room for arranging the plurality of participants and a virtual conference room storage unit for storing the positions of the plurality of participants in the virtual conference room,
The image selection unit selects the display range of the virtual conference room according to the movement of the participant's face detected by the movement detection unit,
The image generation unit generates an image specifying a position to display the participant according to the position of the participant in the virtual conference room, and the participant included in the display range among the plurality of participants. generate images to be displayed on the terminal device, and
communication device. 2. The communication device according to claim 1 , wherein the image selection unit selects an image of a face orientation according to the position of the participant in the virtual conference room. The motion detection unit detects whether the face orientation of at least one of the participants is front, right, or left from the face image acquired by the acquisition unit,
When the face of the participant detected by the motion detection unit faces the front, the video selection unit selects the other participants located on the right and left sides of the participant in the display range of the virtual conference room. , and if the face direction of the participant detected by the motion detection unit is facing right, the other participant located on the right side of the participant in the display range of the virtual conference room and when the participant's face direction detected by the motion detection unit is facing left, the video of the other participant positioned to the left of the participant is included. 3. The communication device according to 2 . When the participant's position in the virtual meeting room changes,
The image selection unit selects an image of the face orientation corresponding to the changed position,
4. The communication device according to any one of claims 1 to 3 , wherein the image generating unit generates an image specifying a position where the participant is to be displayed according to the changed position. a voice selection unit that selects the voice of the other participant according to the face movement of the participant detected by the motion detection unit;
5. Any one of claims 1 to 4 , further comprising a voice generation unit that generates a voice for outputting the voice of the other participant from the terminal device based on the voice selected by the voice selection unit. Communication device as described. The acquisition unit acquires a face image in a different orientation and a peripheral image in a different orientation for each of the plurality of participants,
The image selection unit selects the image of the face direction of the other participant and the surrounding image according to the movement of the face of the participant detected by the movement detection unit,
The image generation unit generates an image for displaying at least the image of the face of the other participant and the image of the surroundings on the terminal device from the image selected by the image selection unit. Item 6. The communication device according to any one of items 5 . The movement detection unit detects movement and orientation of the face as the movement of the face of the participant,
The image selection unit selects the surrounding image according to the movement of the face, selects a display range of the surrounding image according to the orientation of the face,
7. The communication device according to claim 6 , wherein the image generating section generates an image for displaying the surrounding image on the terminal device in the display range selected by the image selecting section. a virtual conference room for arranging the plurality of participants;
the positions of the plurality of participants and the positions of the surrounding images in the virtual conference room;
Equipped with a virtual conference room storage unit that stores
8. The communication device according to claim 7 , wherein the image selection unit changes the display position of the participant according to the orientation of the surrounding image selected according to the movement of the face of the participant. A computer-readable storage medium storing a communication program that causes a computer to execute video processing performed by a communication device,
The communication device is
A virtual conference room for arranging a plurality of participants and a virtual conference room storage unit for storing the positions of the plurality of participants in the virtual conference room,
The video processing includes:
obtaining a plurality of videos, including videos of different face orientations for each of a plurality of participants;
detecting movement of the face of at least one of the participants from the captured video of the face;
selecting an image of the facial direction of another participant and a display range of the virtual conference room according to the detected facial movement of the participant;
generating an image for displaying at least the image of the other participant's face on the terminal device from the selected image;
The image to be displayed on the terminal device includes the image specifying the position of the participant to be displayed according to the position of the participant in the virtual conference room, and the display range of the plurality of participants. and a video for displaying the participant on the terminal device.
A storage medium containing A communication program that causes a computer to execute video processing performed by a communication device,
The communication device is
A virtual conference room storage unit that stores a virtual conference room in which a plurality of participants are arranged and the positions of the plurality of participants in the virtual conference room,
The video processing includes:
obtaining a plurality of videos, including videos of different face orientations for each of a plurality of participants;
detecting movement of the face of at least one of the participants from the captured video of the face;
selecting an image of the facial direction of another participant and a display range of the virtual conference room according to the detected facial movement of the participant;
generating an image for displaying at least the image of the other participant's face on the terminal device from the selected image;
The image to be displayed on the terminal device includes the image specifying the position of the participant to be displayed according to the position of the participant in the virtual conference room, and the display range of the plurality of participants. and a video for displaying the participant on the terminal device.
communication program.

Images