JP2024043175A - communication device - Google Patents

Abstract

[Problem] To provide a communication device that allows multiple people to communicate two-way through vision, hearing and touch with minimal discomfort. [Solution] Each Sync Sofa 1A, 1B is equipped with a camera, microphone, acceleration sensor, speaker, vibration actuator and large display 10, and users of each Sync Sofa 1A, 1B can communicate two-way through a control system using vision, hearing and touch. Each Sync Sofa 1A, 1B has a sofa shape with a seat and backrest, and the large display 10 is positioned on at least one of the left and right sides of the seat so as to face the seat side. [Selected Figure] Figure 1

Description

The present invention relates to a communication device that allows a plurality of people to communicate interactively through visual, auditory, and tactile senses, and particularly relates to a communication device suitable for a sofa-type two-way communication system.

Recently, remote communication devices and software that realize interpersonal communication online between remote locations have been widely used. Also, progress is being made in the use of physical communication devices and software that involve presentation of the five senses.

Non-Patent Document 1 discloses a sofa-type physical communication device that utilizes haptic technology in addition to audiovisual technology to realize physical communication involving the whole body, such as sitting next to someone or patting their back, online between remote locations.

"Developing ``Sync Sofa,'' a sofa-type communication system that connects you even when you are far away using tactile technology," KDDI Research Institute Press Release 2021112501

The physical communication devices and acoustic signal adjustment programs using audio-tactile technology disclosed in the prior art are intended for unidirectional presentation of the five senses, and cannot desirably perform bidirectional presentation of the five senses.

Specifically, in bidirectional five sense presentation where the acquisition and presentation of five sense information including visual information, auditory information, and tactile information are performed simultaneously, even if unidirectional five sense presentation is performed mutually, there is no difference in acquisition and presentation. There is a problem in that howling occurs due to physical interference between the devices and their respective information, making it impossible to obtain desired results.

SUMMARY OF THE INVENTION An object of the present invention is to provide a physical communication device capable of solving the above technical problems and realizing two-way remote communication through visual, auditory, and tactile senses using adjacent communication modes.

In order to achieve the above object, the present invention is characterized in that a communication device provided for remote communication includes the following configuration.

(1) It has a chair shape and is equipped with at least visual information acquisition means and presentation means among tactile information acquisition means and presentation means, auditory information acquisition means and presentation means, and visual information acquisition means and presentation means, The visual information presenting means is provided on at least one of one end side and the other end side of the chair shape.

(2) The information presented by each acquisition means is transmitted to other communication devices via the network, and each presentation means receives and presents the information acquired by each acquisition means of the other communication devices via the network. did.

(3) A first loop suppression means is provided that switches the microphone used to obtain the auditory information to be transmitted based on the distance between the speaker that presents the received auditory information and each microphone.

(4) Equipped with a first loop prevention means that switches the microphone used to acquire the auditory information to be transmitted based on the distance between the speaker presenting the received auditory information and each microphone and the frequency characteristics of the received auditory information. did.

(5) The first loop suppression means further includes means for suppressing presentation of auditory information.

(6) A second loop inhibiting means is provided for switching the acceleration sensor used to obtain the transmitted tactile information based on the distance between the vibration actuator presenting the received tactile information and each acceleration sensor.

(7) A second loop suppression means is provided that switches the acceleration sensor used to obtain the tactile information to be transmitted based on the distance between the vibration actuator that presents the received tactile information and each acceleration sensor and the frequency characteristics of the received tactile information.

(8) The second loop suppression means further includes means for suppressing presentation of tactile information.

According to the present invention, it is possible to realize remote communication accompanied by two-way presentation of the five senses using adjacent communication modes.

Furthermore, according to the present invention, it is possible to prevent howling that may occur due to looping of auditory information and tactile information between communication devices, so it is possible to realize high-quality two-way communication with less discomfort.

1 is a diagram schematically showing a sofa-type communication system according to an embodiment of the present invention. FIG. 1 shows the configuration of the main parts of Sync Sofa. FIG. 7 is a diagram showing an example in which users of the other Sync Sofa are displayed on the large display of one Sync Sofa. FIG. 2 is a functional block diagram showing the configuration of a control system that realizes two-way communication between Sync Sofas. FIG. 2 is a diagram schematically showing an example of two-way communication using three or more Sync Sofas.

Embodiments of the present invention will be described in detail below with reference to the drawings. FIG. 1 is a diagram schematically showing a sofa-type communication system 1 according to an embodiment of the present invention, which includes a pair of remotely located two-way communication sofas (hereinafter referred to as Sync Sofa) 1A and 1B. It is configured by connecting with a network.

Each Sync Sofa 1A, 1B is equipped with a camera as a visual information acquisition means, a microphone as an auditory information acquisition means, an acceleration sensor as a tactile information acquisition means, a speaker as an auditory information presentation means, and a vibration actuator as a tactile information presentation means (Fig. 1, both are omitted) and a large display 10 as a visual information presentation means, and users of each Sync Sofa 1A and 1B can communicate visually, audibly, and tactilely via the control system. Can be done.

Each of the Sync Sofas 1A and 1B has a sofa shape with a seat and a backrest, and the large display 10 is arranged on at least one of the left and right sides of the seat so as to face the seat. The arrangement position of the large display 10 in one Sync Sofa is notified in advance to the other Sync Sofa.

Figure 2 shows the main components of each Sync Sofa 1A, 1B, with the various cushioning materials and surface covering fabrics removed. The Sync Sofa is equipped with at least one large display 10, multiple cameras 11, multiple microphones 12, multiple acceleration sensors 13, multiple speakers 14, and multiple vibration actuators 15, as well as a control system that controls each component to realize two-way communication.

The camera 11 mainly photographs the user seated on the seat of each Sync Sofa. It is desirable that the camera 11 includes a depth sensor, but it is not essential. The microphone 12 mainly detects the voice of the user sitting on the seat of each Sync Sofa and the sound of movements such as rustling clothes. In this embodiment, an audio microphone that mainly detects audio and a gesture sound microphone that mainly detects gesture sounds are provided separately.

The acceleration sensor 13 mainly detects the behavior of the user seated on the seat of each Sync Sofa. As the acceleration sensor 13, various sensors capable of detecting the user's behavior, such as a vibration sensor, can be used. In the description of this embodiment, various sensors will be collectively referred to as acceleration sensor 13.

The speaker 14 reproduces the voice and movement sounds detected by the microphone 12 in the communication partner's Sync Sofa. The vibration actuator 15 presents the vibration detected by the acceleration sensor 13 in the communication partner's Sync Sofa. The image taken by the camera 11 on one Sync Sofa is displayed on the large display 10 on the other Sync Sofa.

FIG. 3 shows an example in which users of Sync Sofa 1A are displayed on the large display 10 of Sync Sofa 1B. The users of Sync Sofa 1B are also displayed on the large display 10 of Sync Sofa 1A, realizing two-way communication regarding visual information.

In this embodiment, vibration actuators 15 are provided in a total of ten locations: four on the seat surface 8 and six on the backrest 9. Each vibration actuator 15 is positioned so as to come into contact with the left, right and center of the back, and the left and right buttocks, respectively, based on the person's shoulder width and buttock width. This is to stimulate the tactile perception of each side of the person's back and buttocks.

The vibration actuator 15 does not come into direct contact with the human body, and is covered with a cushioning material approximately 2 cm thick and a sturdy cloth approximately 1 mm thick, and the vibrations presented are transmitted through the cushioning material and the cloth. Ru. The vibration presented to one Sync Sofa is a signal that is a composite of the vibration detected by the acceleration sensor 13 placed on the seat 8 and backrest 9 of the other Sync Sofa and the movement sound detected by the microphone 12. It is.

FIG. 4 is a functional block diagram showing the configuration of the control system 20 that realizes bidirectional communication between the Sync Sofas 1A and 1B.

In addition to the well-known control functions of controlling the large display 10, camera 11, microphone 12, acceleration sensor 13, speaker 14, vibration actuator 15, and communication control unit 16, the control system 20 includes a motion parallax image generation unit 201, a display The Sync Sofas 1A and 1B are equipped with a position control section 202 and a howling suppression section 203, and are connected to each other via a communication control section 16 and a network. The control system 20 may be provided for each Sync Sofa, or may be provided on the cloud so that each Sync Sofa can share it.

The motion parallax image generation unit 201 generates a motion parallax image based on a plurality of camera images from different viewpoints and the estimated position of the subject person. The presentation position control unit 202 localizes the auditory information and tactile information at a predetermined position based on the estimated position of the subject person. The howling suppression unit 203 includes a first loop suppression unit 2031 and a second loop suppression unit 2032, and suppresses howling of auditory information and tactile information.

Such a control system 20 can be configured by installing an application (program) that realizes each function described in detail below on a general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated machine or single-function machine in which part of the application is converted into hardware or software.

In this embodiment, two-way communication is realized by Sync Sofa 1A and Sync Sofa 1B, which have substantially the same configuration, working together via the control system 20, so here we mainly focus on the operation on the Sync Sofa 1A side. The operation of this embodiment will be explained. Furthermore, in order to make the explanation easier to understand, each configuration on the Sync Sofa 1A side is given a suffix A, and each configuration on the Sync Sofa 1B side is given a suffix B to distinguish between the two.

The motion parallax video generation unit 201A of the Sync Sofa 1A edits the visual information received from the Sync Sofa 1B into a video that reflects the motion parallax of the user UA of the Sync Sofa 1A. In this embodiment, a dynamic parallax image is generated by rendering polygon models of foreground and background placed in a 3D space using a virtual camera.

The foreground is a person object of the user UB, which is a person area extracted from the visual information sent by Sync Sofa1B and mapped onto one rectangular polygon. The background is preferably an object that imitates a sofa, wall, floor, etc. from the viewpoint of providing a sense of adjacency and presence. The human object is arranged in the 3D space so that it is displayed life-sized on the large display 10A.

The position of the virtual camera can be obtained as a result of tracking the position information of the head of user UA obtained from camera 11A with depth sensor. This allows user UA on Sync Sofa 1A to realize eye contact communication with user UB on Sync Sofa 1B while reflecting his/her own motion parallax. Note that motion parallax images can also be generated by switching foreground images captured simultaneously using multiple cameras according to the viewpoint position, or by synthesizing intermediate images.

The presentation position control unit 202A localizes each auditory information of the voice and movement sound received from the Sync Sofa 1B at a predetermined position. The presentation position control unit 202A can also localize each tactile information of vibration and movement sound received from the Sync Sofa 1B at a predetermined position.

Regarding auditory information, in this embodiment, the Sync Sofa 1A is equipped with full-range speakers 14 at six locations, four on the left and right of the seat 8 and two on the left and right of the backrest 9. Each speaker 14 is placed at a distance sufficient to localize the sound image.

In this embodiment, the movement sounds include the sound of the user's clothing rubbing against the surface fabric of Sync Sofa 1 when changing posture, the friction sound emitted when the user rearranges or fidgets with his/her legs, and the sound of the user sitting on Sync Sofa 1. This assumes the sound of a collision, etc. The sound of each user's actions is localized to the sitting position of the user, and the sound is localized to the mouth of each user displayed on the large display 10.

Although tactile information has a lower frequency than auditory information, the presentation position of tactile information may be varied depending on the position of each user, as in the case of auditory information.

For example, when a motion of the user's hand, back, buttocks, etc. is detected from the received visual information, the vibration presentation position may be changed, or the presentation position may be continuously changed in accordance with the motion. Therefore, if a motion such as patting the back is detected from visual information, the vibration actuator 15 corresponding to that position is selectively driven.

Howling suppression unit 203A includes a first loop suppression unit 2031A that suppresses a loop of auditory information and a second loop suppression unit 2032A that suppresses a loop of tactile information, in order to prevent howling during two-way communication.

Howling of auditory information in Sync Sofa 1A is unnecessary in the auditory information transmission path because the movement sound and voice received from Sync Sofa 1B are output from speaker 14A, acquired by microphone 12A, and further transmitted to Sync Sofa 1B and presented. This occurs because a loop occurs and this loop oscillates.

Here, it is known that the frequency of the auditory information loop depends on the distance between the speaker 14A and the microphone 12A and the impulse response. Therefore, by increasing the distance between the presentation position of the auditory information controlled by the presentation position control unit 202A and the position of the microphone 12A, or by switching the microphone 12A according to the frequency characteristics of the auditory information, the loop can be reduced. The occurrence can be suppressed.

In this embodiment, the first loop suppression unit 2031A monitors the position where the movement sound and voice received from the Sync Sofa 1B are localized (auditory information presentation position), and based on the distance between the localization position and each microphone 12A. It is possible to switch between the microphone 12A that captures movement sounds and the microphone 12A that captures voices.

Alternatively, based on the frequency characteristics (for example, high range, middle range, low range) within a short frame of auditory information and the distance between the position where the sound and voice are localized and each microphone 12A, for example, the high range is 12A_1, microphone 12A_2 for the middle range, microphone 12A_3 for the low range, and so on, the microphone 12A may be selected at a position where loops are less likely to occur for each frequency band of auditory information.

Alternatively, if the power in a short frame of auditory information exceeds a preset threshold, more specifically, the ratio of the average power of all frequencies in a short frame to the power of a certain frequency exceeds a preset threshold. If the threshold is exceeded, it may be determined that a sound has occurred, and the output of the speaker 14A may be suppressed.

Howling of tactile information in Sync Sofa 1A is unnecessary in the tactile information transmission path because the tactile information received from Sync Sofa 1B is output from vibration actuator 15A, acquired by acceleration sensor 13A, and further transmitted to Sync Sofa 1B and presented. This occurs because a loop occurs and this loop oscillates.

Here, it is known that the frequency of the tactile information loop depends on the distance between the vibration actuator 15A and the acceleration sensor 13A and the impulse response. Therefore, by increasing the distance between the presentation position of the tactile information controlled by the presentation position control unit 202A and the position of the acceleration sensor 13A, or by switching the acceleration sensor 13A according to the frequency characteristics of the tactile information. The occurrence of loops can be suppressed.

In this embodiment, the second loop suppression unit 2032A monitors the position where the vibrations and movement sounds received from the Sync Sofa 1B are localized (tactile information presentation position), and the distance between the localization position and each acceleration sensor 13A is determined. Based on this, the acceleration sensor 13A that acquires vibrations can be switched.

Alternatively, based on the frequency characteristics (for example, high range, middle range, low range) within a short time frame of tactile information and the distance between the position where the vibration and movement sound is localized and each acceleration sensor 13A, for example, the high range is The acceleration sensor 13A at a position where loops are unlikely to occur may be selected for each frequency band of tactile information, such as the acceleration sensor 13A_1, the acceleration sensor 13A_2 for the middle range, and the acceleration sensor 13A_3 for the low range.

Alternatively, if the power of the tactile information within a short-time frame exceeds a set threshold, more specifically, if the ratio of the average power of all frequencies within the short-time frame to the power of a certain frequency exceeds a pre-set threshold, it may be determined that a sound has been generated and the output of vibration actuator 15A may be suppressed.

Note that although the above embodiment has been described as one that aims for two-way communication through visual information, auditory information, and tactile information, the present invention is not limited to this only, and at least two-way communication is achieved through visual information. The same can be applied to any communication system that aims to communicate towards the target.

Further, in the above embodiment, the present invention has been explained by taking as an example the application to a sofa-shaped Sync Sofa, but the present invention is not limited to this, and the present invention is not limited to this, but has at least a seat surface like a bench or a chair. The same applies to structures. Further, although it is desirable to have a backrest, it is not an essential configuration for the present invention.

Further, in the above embodiment, the large display 10 is provided on at least one of the left and right sides of the seat, but the present invention is not limited to this, and can be applied to a chair with a circular seat or a balance ball. If the chair has no concept of left and right, such as the one shown in FIG.

Furthermore, in the above embodiment, the present invention was explained using two-way communication using two Sync Sofas as an example, but the present invention is not limited to this, and as shown in FIG. The same can be applied to the two-way communication using the Sync Sofa described above.

According to the above embodiment, it is possible to achieve two-way remote communication through visual, auditory, and tactile senses using adjacent communication modes, so that goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations is achieved. It will be possible to contribute to ``Developing sustainable infrastructure and promoting inclusive and sustainable industrialization'' and Goal 11, ``Making cities inclusive, safe, resilient and sustainable''.

1...Sofa type communication system, 1A, 1B, 1C...Sync Sofa, 8...Seat, 9...Backrest, 10...Large display, 11...Camera, 12...Microphone, 13...Acceleration sensor, 14...Speaker, 15...Vibration Actuator, 16...Communication control unit, 20...Control system, 201...Motion parallax image generation unit, 202...Presentation position control unit, 203...Howling suppression unit

Claims (13)

In a communication device used for remote communication,
It has a chair shape and is equipped with at least visual information acquisition means and presentation means among tactile information acquisition means and presentation means, auditory information acquisition means and presentation means, and visual information acquisition means and presentation means,
A communication device characterized in that the visual information presenting means is provided on at least one of one end side and the other end side of the chair shape. The information presented by each of the acquisition means is transmitted to another communication device via a network, and each of the presentation means receives and presents the information acquired by each acquisition means of the other communication device via the network. The communication device according to claim 1. Claim characterized by comprising a presentation position control means for controlling an image displayed on the display based on a relationship between visual information received from the other communication device and visual information transmitted to the other communication device. 2. The communication device according to 2. Communication according to claim 3, wherein the presentation position control means controls the auditory information to be presented based on position information of a person area estimated based on visual information received from the other communication device. Device. 5. The presentation position control means controls the presented tactile information based on position information of a person area estimated based on visual information received from the other communication device. communication device. 3. The apparatus according to claim 1, further comprising a first loop suppression means for switching the microphone used for acquiring the auditory information to be transmitted based on the distance between the speaker presenting the received auditory information and each microphone. Communication device as described. The present invention includes a first loop suppression means for switching the microphone used for acquiring the auditory information to be transmitted based on the distance between the speaker presenting the received auditory information and each microphone and the frequency characteristics of the received auditory information. The communication device according to claim 1 or 2. 7. The communication device according to claim 6, wherein the first loop suppression means further comprises means for suppressing presentation of the auditory information. 8. The communication device according to claim 7, wherein the first loop suppression means further comprises means for suppressing presentation of the auditory information. A claim characterized by comprising second loop suppression means for switching the acceleration sensor used to obtain the transmitted tactile information based on the distance between the vibration actuator presenting the received tactile information and each acceleration sensor. The communication device according to item 1 or 2. a second loop suppression means for switching the acceleration sensor used to obtain the tactile information to be transmitted based on the distance between the vibration actuator presenting the received tactile information and each acceleration sensor and the frequency characteristics of the received tactile information; 3. The communication device according to claim 1, further comprising a communication device. The communication device according to claim 10, characterized in that the second loop suppression means further comprises means for suppressing the presentation of the tactile information. 12. The communication device according to claim 11, wherein the second loop suppression means further comprises means for suppressing presentation of the tactile information.