JP7286303B2 - Conference support system and conference robot - Google Patents

Description

The present invention relates to technology for supporting conference communication.

Patent Literature 1 discloses a guidance robot control system that determines whether or not to provide guidance based on the user's viewing direction and the user's moving direction.

JP 2017-159410 A

The technique disclosed in Patent Literature 1 determines whether or not to provide guidance by using the behavior that the user performs when he or she desires guidance.

However, such behavior cannot be expected in a conference in which multiple people participate. In a conference, it is necessary to determine the appropriateness of interrupting communication while the conference is in progress, and to determine the timing of speaking.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to find a suitable timing for speaking in a meeting.

In order to solve the above problems, a conference support system according to a first aspect is a conference support system used in a conference by a plurality of participants, and includes a participation state acquisition unit that acquires the conference participation states of the plurality of participants. and, based on the conference participation states of the plurality of participants obtained by the participation state acquisition unit, the likelihood of interrupting communication by the plurality of participants is obtained, and based on the obtained likelihood of interruption, at the timing of speaking and a control unit that determines whether or not there is.

A second aspect is the conference support system according to the first aspect, wherein the control unit includes a speech schedule recording unit that records speech schedule content, and when it is determined that it is time to speak, the speech The contents of the speech schedule recorded in the schedule recording unit are spoken through the speaker.

A third aspect is the conference support system according to the second aspect, wherein the control unit records the speech schedule content of at least one of the plurality of participants in the speech schedule recording unit. .

A fourth aspect is the conference support system according to the third aspect, wherein the plurality of participants hold a conference in a real space with a plurality of first participants who can observe each other's conference participation states. , and a second participant who participates in the conference via a communication line, and the control unit records the speech schedule content of the second participant in the speech schedule recording unit.

A fifth aspect is the conference support system according to any one of the first to fourth aspects, wherein the participation state acquisition unit includes a plurality of individual a participation state acquisition unit, wherein the control unit obtains the ease of interruption of each of the plurality of participants based on the conference participation states of the plurality of participants obtained by the plurality of individual participation state acquisition units; Whether or not it is time to speak is determined based on the obtained degree of obstruction.

A sixth aspect is the conference support system according to any one of the first to fifth aspects, wherein a speech command reception for acquiring a speech intention command by at least one of the plurality of participants and a pseudo-conference participation device capable of participating in the conference in the real space where the conference is held, wherein the control unit determines that it is not time to speak after obtaining the intention command to speak. Then, the pseudo-conference participation device is caused to perform a first operation for increasing the likelihood of interruption.

A seventh aspect is the conference support system according to the sixth aspect, wherein, after the execution of the first operation, the control unit instructs the pseudo-conference participation device to interrupt when it is determined that it is not time to speak. This is to execute a second operation that increases ease.

An eighth aspect is the conference support system according to the seventh aspect, wherein the second action is more effective than the first action in increasing the likelihood of interruption.

In order to solve the above-described problems, a conference robot according to a ninth aspect is a conference robot that participates in a conference of a plurality of participants, and is a conference robot for inputting a conference participation state of the plurality of participants. and based on the conference participation states of the plurality of participants input from the participation state input unit, the likelihood of interrupting communication by the plurality of participants is obtained, and the timing of utterances is determined based on the obtained likelihood of interruption. and a control unit that determines whether or not it is time to speak, and executes processing for speaking through the speaker when it is determined that it is time to speak.

A tenth aspect is the conference robot according to the ninth aspect, wherein the conference participation state of each of the plurality of participants is input to the participation state input unit, and the control unit controls the Based on the state of participation in the conference of each participant, the likelihood of interruption for each of the plurality of participants is obtained, and whether or not it is time to speak is determined based on the obtained likelihood of interruption.

An eleventh aspect is the conference robot according to the ninth or tenth aspect, further comprising a robot action unit that performs an action as a robot, wherein the control unit controls the content of a scheduled speech to be generated. Later, when it is determined that it is not time to speak, the robot action section is caused to perform a first action to increase the likelihood of interruption.

A twelfth aspect is the conference robot according to the eleventh aspect, wherein the control unit instructs the robot operation unit to interrupt when it is determined that it is not time to speak after executing the first operation. This is to execute a second operation that increases ease.

A thirteenth aspect is the conference robot according to the twelfth aspect, wherein the second motion is more effective than the first motion in increasing the likelihood of interruption.

According to the first aspect, based on the conference participation states of the plurality of participants obtained by the participation state acquisition unit, the likelihood of interrupting communication by the plurality of participants is obtained, and a statement is made based on the obtained likelihood of interruption. In order to determine whether or not it is time, it is possible to find the appropriate timing for speaking in a meeting.

According to the second aspect, if the content of scheduled speech is recorded in advance, it is possible to speak at an appropriate timing.

According to the third aspect, the participants of the conference can speak at appropriate timing by recording the content of scheduled speeches in advance.

According to the fourth aspect, it is difficult for the second participant who participates in the conference via the communication line to observe the conference participation states of the plurality of first participants. Therefore, by recording the planned utterance contents of the second participant in the utterance schedule recording section, the second participant can speak at an appropriate timing if the planned utterance contents are recorded in advance.

According to the fifth aspect, it is possible to more appropriately determine whether or not it is time to speak based on the conference participation status of each of the plurality of participants.

According to the sixth aspect, when it is determined that it is not time to speak, the pseudo-conference participation device is caused to perform the first operation to increase the likelihood of interruption, so that it is easier to speak in the conference.

According to the seventh aspect, furthermore, when it is determined that it is not time to speak, the pseudo-conference participation device is caused to perform the second operation to increase the likelihood of interruption, thereby making it easier to speak in the conference.

According to the eighth aspect, furthermore, after the execution of the first action, if it is determined that it is not time to speak, the second action that is more effective in increasing the likelihood of being interrupted is executed in the conference. becomes easier.

According to the ninth aspect, based on the conference participation states of the plurality of participants input from the participation state input unit, the ease of interrupting communication by the plurality of participants is obtained, and based on the obtained ease of interruption In order to determine whether or not it is time to speak, the robot can speak at a suitable timing in the conference.

According to the tenth aspect, it is possible to more appropriately determine whether or not it is time to speak based on the conference participation status of each of the plurality of participants.

According to the eleventh aspect, when it is determined that it is not time to speak, the robot action unit is caused to perform the first action that increases the likelihood of interruption, so that it becomes easier to speak in the conference.

According to the twelfth aspect, after the execution of the first action, if it is further determined that it is not time to speak, the robot action unit is caused to perform the second action that increases the likelihood of interruption, so that the user can speak more easily in the conference. Become.

According to the thirteenth aspect, after the execution of the first action, if it is further determined that it is not time to speak, the second action that is more effective in increasing the likelihood of interruption is executed. Become.

FIG. 2 is an explanatory diagram showing a state of a conference to which the conference support system according to the first embodiment is applied; FIG. 11 is an explanatory diagram showing an example of the operation of a substitute participation robot; FIG. 4 is an explanatory diagram showing an example of imaging by a camera; 1 is a block diagram showing an electrical configuration of a conference support system; FIG. It is a figure which shows an example of a learned model. It is a figure which shows an example of a model during learning. 4 is a flowchart showing an operation example of a control unit; FIG. 10 is an explanatory diagram showing a state of a conference to which the conference robot according to the second embodiment is applied; Fig. 3 is a block diagram showing an electrical configuration of the conference robot;

{First Embodiment}
A conference support system according to the first embodiment will be described below. FIG. 1 is an explanatory diagram showing a state of a conference to which the conference support system 30 is applied.

A conference support system 30 is used for a conference by a plurality of participants 10A and 10B. A plurality of participants 10A and 10B are subjects who participate in the conference, make some remarks, and listen to the remarks of others. The participants 10A and 10B may exist at the same conference place, or may exist dispersedly in a plurality of places.

FIG. 1 shows an example in which a plurality of (five) first participants 10A are present in the conference room 20, and a second participant 10B is present at a location away from the conference room 20. . That is, a plurality of first participants 10A hold a conference in a state where they can observe each other's conference participation states in the physical space. In the conference room 20, a proxy participation robot 60 exists as a pseudo start participation device. The second participant 10B can participate in the conference of the first participants 10A in the conference room 20 through the communication line 70 and the proxy participation robot 60. FIG.

A conference desk 22 is placed in the conference room 20, and a plurality of chairs 24 are provided around it. A plurality of first participants 10A and proxy participation robots 60 are seated in respective chairs 24. As shown in FIG.

The proxy participation robot 60 is a robot capable of participating in the conference as a proxy for the second participant 10B. For example, as shown in FIG. 2, the proxy participation robot 60 can execute an act of speaking in the conference room 20 (in FIG. 2, saying "Excuse me.") through a built-in speaker or the like. In addition, the proxy participation robot 60 can perform actions other than speaking, which may be performed during a meeting (the action of raising a hand is exemplified in FIG. 2). Therefore, the second participant 10B can smoothly participate in the conference through the proxy participation robot 60. FIG. The proxy participation robot 60 incorporates a control unit 40, which will be described later.

Returning to FIG. 1, a camera 58 and a microphone 59 are provided corresponding to each first participant 10A. The camera 58 and the microphone 59 are used to acquire the meeting participation state of the first participant 10A as video data. For example, the microphone 59 is used to acquire speech data of the first participant 10A. A camera 58 is used to capture the motion of the first participant 10A. For example, with the camera 58, as shown in FIG. ) etc. can be imaged. The camera 58 and microphone 59 are connected to the controller 40 .

The second participant 10B is seated on a chair 24B provided in front of the desk 22B. A microphone 86, a speaker 87 and a display device 88 are provided in front of the second participant 10B. The voice of the first participant 10A is input to the microphone 86 . The state of the conference in the conference room 20 is displayed on the display device 88 , and the speech in the conference is reproduced through the speaker 87 . These microphone 86 , speaker 87 and display device 88 are connected to the terminal device 80 .

The proxy participation robot 60 and the terminal device 80 are connected via a communication line 70 so as to be mutually communicable. The communication line 70 may be wired or wireless. Also, the communication line 70 may be a public communication network or a dedicated line.

The proxy participation robot 60 determines communication (conference) by the plurality of participants 10A based on the conference participation states of the plurality of first participants 10A obtained by each camera 58 and each microphone 59 as a participation state acquisition unit. A control unit 40 (to be described later) is provided which determines whether or not it is time to speak based on the obtained likelihood of interruption. Then, when it is determined that it is time to speak, the second participant 10B can participate in the conference in the conference room 20 via the proxy participation robot 60 .

FIG. 4 is a block diagram showing the electrical configuration of the conference support system 30. As shown in FIG.

The conference support system 30 includes a control unit 40, a proxy participation robot 60, a plurality of sets of cameras 58 and microphones 59 provided in the conference room 20, a terminal device 80 provided at a position away from the conference room 20, a microphone 86, A speaker 87 and a display device 88 are provided. The control unit 40 and the terminal device 80 are connected through a communication line 70 so as to be able to communicate with each other.

A plurality of sets of cameras 58 and microphones 59 are an example of a participation state acquiring unit that acquires conference participation states of a plurality of participants (here, the first participant 10A). Here, a set of camera 58 and microphone 59 is provided corresponding to each first participant 10A. Therefore, each set of camera 58 and microphone 59 is an example of an individual participation state acquisition unit that acquires the conference participation state of each of the plurality of first participants 10A. The camera 58 images the part including the face of the corresponding first participant 10A, here the upper half of the body. Thereby, the camera 58 can image the movement of the line of sight and the inclination of the face of the first participant 10A. The microphone 59 converts the sound mainly made by the corresponding first participant 10A into an electric signal. The microphone 59 may convert sounds of surrounding participants and the like into electrical signals. However, with the microphone 59, the sounds of the surrounding participants are reduced, and the influence thereof is small.

It is not essential that the participation state acquisition unit is provided for each first participant. One participation state acquisition unit may be provided for all of the plurality of participants. In this case, the camera may be provided so as to be able to capture an image of all of the participants. Also, it is preferable that the microphones be provided so as to be able to convert the sounds of conversations of all the participants into electrical signals. Also, a plurality of participants may be divided into a plurality of groups, and one participation state acquisition unit may be provided for each group. A participation state acquisition unit may be provided for the second participant as well.

As described above, the proxy participation robot 60 is a robot capable of participating in the conference as a proxy for the second participant 10B. Here, the proxy participation robot 60 includes a robot action unit 67 that operates as a robot. Here, the robot action section 67 includes a speaker 68 and an arm drive section 69 . Under the control of the control unit 40 , the substitute participation robot 60 can emit sound through the speaker 68 . The arm driving section 69 also includes a motor or the like for driving at least one arm up and down. Under the control of the control unit 40 , the arm drive unit 69 can raise (raise) or lower the arm of the proxy participation robot 60 . The substitute participation robot 60 may be a humanoid robot, or may be another articulated robot or the like. If the proxy participation robot 60 is a humanoid robot, the act of the second participant 10B participating in the conference through the proxy participation robot 60 is less likely to cause discomfort to the first participant 10A. Therefore, smooth progress of the conference can be expected.

The proxy participation robot 60 is an example of a pseudo-conference participation device capable of participating in a conference in the real space where the conference is held. As the operation to participate in the conference, an operation indicating that there is a scheduled speech content in the conference, an operation to speak, an operation to indicate approval or disapproval, and the like are assumed. These actions can be realized by speech through a speaker, action by a robot, and display action by a display device. For this reason, a pseudo-conference participation device capable of participating in a conference includes a robot capable of some action, a speaker that emits voice, a display device that can display the presence or absence of planned speech content, speech content, approval or disapproval, etc. may be used.

Based on the conference participation states of the plurality of first participants 10A obtained by the plurality of sets of cameras 58 and microphones 59, the control unit 40 obtains the ease of interrupting communication by the plurality of first participants 10A, and Whether or not it is time to speak is determined based on the likelihood of interruption. Here, the control unit 40 includes a CPU (Central Processing Unit) 41 as at least one processor, a RAM (Random Access Memory) 42, a storage unit 43, an input/output unit 44, a communication circuit 45, a robot control input/output unit 46, and the like. are composed of computers interconnected via a bus line 48 .

The storage unit 43 is composed of a nonvolatile storage device such as a flash memory or a hard disk device. In the storage unit 43, based on the conference participation states of the plurality of first participants 10A obtained by the plurality of sets of cameras 58 and microphones 59, the ease of interrupting communication by the plurality of first participants 10A is obtained, and the obtained A program 43a is stored for determining whether or not it is time to speak based on the likelihood of interruption. Here, the program 43a includes a learned model 49, which will be described later. based on, a processing program for determining how easily communication is interrupted by a plurality of first participants 10A. Further, the program 43a executes a process of storing the utterance of the second participant 10B as the voice data 43c in the storage unit 43, and a process of driving and controlling the robot operation unit 67 when it is determined that it is time to utter. Also includes a program for That is, the control unit 40 includes the storage unit 43 as a speech schedule recording unit that records the speech schedule contents of the second participant 10B, which is at least one of the participants 10A and 10B. An example of processing by the program 43a will be described later.

The RAM 42 is used as a working area when the CPU 41 performs predetermined processing. The input/output unit 44 is connected to a plurality of sets of cameras 58 and microphones 59 , and signals from the cameras 58 and microphones 59 are input to the main control unit 40 via the input/output unit 44 . The control unit 40 and the plurality of sets of cameras 58 and microphones 59 may be connected for wired communication, or may be connected for wireless communication. The communication circuit 45 is connected to the terminal device 80 through the communication line 70 so as to be mutually communicable. The robot control input/output unit 46 is connected to the robot operation unit 67 so as to control the operation of the robot operation unit 67 . Therefore, the controller 40 can control the operation of the robot operation unit 67 .

In this control unit 40, the CPU 41 performs arithmetic processing according to the procedure described in the program 43a, the operation related to the speech of the second participant 10B input through the terminal device 80, the input from the multiple sets of cameras 58 and the microphone 59, etc. , the function of controlling the robot operation unit 67 is executed.

In this example, the control unit 40 is incorporated in the proxy participating robot 60 . The control unit 40 may be provided outside the proxy participation robot 60 .

The program 43a is stored in advance in the storage unit 43, but may be recorded in a recording medium such as a CD-ROM, a DVD-ROM, or an external flash memory, or transmitted via a network. It may be provided to an existing computer by downloading from an external server or the like.

Some or all of the functions realized by the control unit 40 may be processed by one processor, or may be processed by a plurality of processors in a distributed manner. Some or all of the functions realized by the control unit 40 may be realized by a computer outside the conference room, such as a terminal device on the side of the second participant 10B.

As with the control unit 40, the terminal device 80 is configured by a computer including a CPU as at least one processor, a RAM 42, a storage unit, an input/output unit, a communication circuit, and the like. The terminal device 80 is provided on the side of the second participant 10B. The terminal device 80 is connected to the control unit 40 through the communication line 70 so as to be able to communicate with each other.

The microphone 86 is provided at a position capable of converting the sound produced by the second participant 10B into an electrical signal. The speaker 87 is provided at a position capable of providing sound to the second participant 10B, and the display device 88 is provided at a position capable of providing display content to the second participant 10B. A microphone 86 , a speaker 87 and a display device 88 are connected to the terminal device 80 .

Then, the speech by the second participant 10B is acquired through the microphone 86, and voice data based on this speech is transmitted from the terminal device 80 to the control section 40 through the communication line 70. FIG. Further, for example, the utterances and video data of each first participant 10A obtained through the plurality of sets of cameras 58 and microphones 59 are transmitted from the control unit 40 to the terminal device 80 via the communication line 70, and each utterance is It is reproduced through the speaker 87, and the video data of each first participant 10A is displayed on the display device 88. FIG. When the second participant 10B wants to make a predetermined statement, when he or she makes a statement, the content of the planned statement is acquired by the microphone 86 . The contents of the speech schedule are not reproduced as they are in the conference room, but are obtained as an intention command of speech by the second participant 10B, and are spoken through the proxy participation robot 60 at the speech timing described later. In this regard, the microphone 86 functions as a speech command reception unit that acquires a speech intention command from the second participant 10B. The speech command receiving unit may be another switch such as a keyboard, or may be a switch dedicated to speech.

FIG. 5 is a diagram showing an example of the trained model 49. As shown in FIG. The learned model 49 is a model generated by machine learning, and is used in the control unit 40, which is a kind of computer. The trained model 49 includes a feature amount extraction unit 49a and a discrimination layer 49b.

The feature amount extraction unit 49a extracts feature amounts from audio data and video data that are input data. For example, audio data is acquired through the microphone 59 and video data is acquired through the camera 58 . For example, volume is extracted as a feature amount from the audio data 43c. For example, motion of the line of sight (eyes) and inclination of the face are extracted as feature amounts from the moving image data. The discrimination layer 49b is composed of a well-known multilayer neural network in which multiple layers of perceptrons are combined.

As shown in FIG. 6, the trained model 49, in a simulated or actually held conference, for input data (speech data, video data at a predetermined time) corresponding to one of the first participants 10A, This is a trained model learned based on a plurality of learning data that distinguishes whether or not it is appropriate as a timing at which communication is likely to be interrupted. Such a learning model, for example, determines whether the designer of this model actually sees and hears the input data of the conference or the conference and determines whether it is the timing when communication is likely to be interrupted, or the first participant 10A actually It can be generated by judging whether or not a statement has been made.

Then, as shown in FIG. 5, the CPU 41 of the control unit 40 performs learning in the identification layer 49b for input data (audio data, moving image data) corresponding to one of the first participants 10A, according to instructions of the program 43a. By performing calculations based on the weighting coefficients and the response function, etc., which have already been used, the identification layer 49b can output a result indicating how easily communication is interrupted. Output 1 may be output when communication is likely to be interrupted, and output 2 may be output when communication is difficult to be interrupted. Alternatively, an index value (suitability) that quantifies the likelihood of interrupting communication may be output as output 1, and an index value (suitability) that quantifies the difficulty of interrupting communication may be output as output 2. Here, the former case will be explained.

When applying machine learning to determine the ease of interrupting communication, the technique is not limited to neural networks, and other techniques such as support vector machines may be applied. It is not essential to apply machine learning to find the ease of blocking communication. A predetermined conditional expression may be set in advance, and the degree of interruption of communication may be obtained based on the conditional expression. In addition, it is possible to determine whether or not the participant has looked at the smartphone or the document based on eye movements, facial movements, etc., and based on the determination results, determine how easily communication is interrupted. . For example, when looking at a smartphone, when looking at a document, it can be said that there is a high possibility of interrupting communication in a meeting.

Based on the output of the learned model 49, the control unit 40 determines whether or not it is time to speak. When output 1 and output 2 are selectively output from the trained model 49, the output may be followed. When an index value quantified in multiple stages is output from output 1 or output 2, the index value is compared with a preset threshold to determine whether or not it is time to speak. good too. For example, when an index value that quantifies the likelihood of interrupting communication is output from the output 1, it may be determined that it is time to speak when the index value is equal to or greater than a threshold value or exceeds the threshold value.

FIG. 7 is a flow chart showing an operation example of the control unit 40. As shown in FIG.

Processing begins with the start of the conference. In step S1, it is determined whether or not there is an intention to speak. Here, when the second participant 10B makes a speech for the conference through the microphone 86, the speech is acquired through the microphone 86. FIG. Voice data corresponding to the utterance is sent to the control section 40 through the terminal device 80 and the communication line 70 . When voice data of the second participant 10B is transmitted from the terminal device 80 side, the control unit 40 stores the voice data (contents of scheduled speech of the second participant 10B) in the storage unit 43 as voice data 43c. Also, the reception of such voice data is input as a speech intention command of the second participant 10B. Therefore, the control unit 40 determines that there is an intention to speak when voice data is received from the second participant 10B, and determines that there is no intention to speak when no voice data is received. The command of the intention to speak may be input by operating a switch on the side of the second participant 10B, not based on the voice data. The scheduled speech content may be stored as character data input from a keyboard or the like, in addition to being stored as voice data. If it is determined in step S1 that there is no intention to speak, in step S2, after waiting for a predetermined waiting time, the process returns to the start and the processes after step S1 are repeated. If it is determined that there is an intention to speak in step S1, the process proceeds to step S3.

In this example, an example will be described in which the storage unit 43 serves as a statement schedule recording unit and records the statement schedule contents of at least one of the participants 10 . However, the speech schedule recording section may record the speech schedule contents by an artificial computer system instead of the speech schedule contents of the participants. For example, when an artificial meeting support system using a computer, etc., has content such as "Please finish the meeting in 5 minutes" as information (scheduled speech content) that you want to interrupt the meeting and convey to the participants. , the contents of the speech schedule may be recorded in the speech schedule recording section.

In step S3, the control unit 40 inputs the output from each set of the camera 58 and the microphone 59 to the learned model 49 and obtains the degree of obstruction. Here, the control unit 40 obtains the easiness of obstruction for each of the plurality of first participants 10A.

In the next step S4, it is determined whether or not it is time to speak based on the likelihood of interruption obtained in step S3. As described above, when the learned model 49 outputs two outputs indicating the likelihood of interruption, whether or not it is time to speak may be determined according to the outputs. When the learned model 49 outputs an index value that quantifies the likelihood of interruption in multiple stages, it is sufficient to determine whether or not it is time to speak based on the index value and a predetermined threshold value. In this case, the threshold may be changed for each first participant 10A. Also, the threshold may be changed according to the atmosphere of the conference. For example, when the number of statements is small, the threshold value may be made small so that the second participant 10B can easily make statements. Also, the threshold may be changed manually by the conference chair or moderator, or may be changed by the computer in consideration of the amount of speech. Further, here, the control unit 40 determines whether or not it is time to speak for each of the first participants 10A based on the obtained degree of obstruction. Then, when it is determined that it is time to speak for all the first participants 10A, it is determined that it is time to speak in the conference. That is, when it is determined that the timing is not suitable for speaking for at least one of the first participants 10A, it is determined that it is not the timing to speak.

Here, the likelihood of interruption and whether or not it is time to speak is determined for each first participant 10A. It may be determined whether or not

If it is determined in step S4 that it is time to speak, the process proceeds to step S5. In step S<b>5 , the control unit 40 speaks through the speaker 68 the planned speech content (here, the voice data 43 c ) stored in the storage unit 43 . When the contents of the scheduled speech are stored as character data, synthesized speech or the like may be spoken through the speaker 68 . Since the speaker 68 is built into the proxy participation robot 60, the surrounding first participant 10A can accept the utterance of the second participant 10B participating through the proxy participation robot 60 with little sense of incongruity. After finishing the speech, it returns to the start and repeats the processing after step S1.

If it is determined in step S4 that it is not time to speak, the process proceeds to step S6. In step S6, the control unit 40 causes the robot motion unit 67 to perform the first motion. The first operation is an operation for increasing the ease of blocking. The first action is a movement indicating that there is a statement (for example, raising a hand), a short statement indicating that there is a statement (for example, “Excuse me”, “Excuse me”, “I would like to say something”, “Speak”). There is something I want to do,” etc.) is assumed. When the pseudo-conference participation device is a humanoid robot or when there is no such device, the similar short speech is reproduced through the speaker, and the fact that there is a speech is displayed on the display device (display of the short speech, light-emitting display indicating the speech) etc.), generating a beep sound or the like, generating vibration, and the like.

In the next step S7, similarly to step S3, the likelihood of obstruction is determined.

In the next step S8, similarly to step S4, it is determined whether or not it is time to speak based on the likelihood of interruption obtained in step S7. If it is determined that it is time to speak, the process proceeds to step S5, and speech is made. If it is determined that it is not time to speak, the process proceeds to step S9.

In step S9, it is determined whether or not a predetermined time has passed after the first action. If it is determined that the predetermined time has not passed yet, the process proceeds to step S10, waits for a predetermined time, returns to step S17, and repeats the processes after step S7. If it is determined that the predetermined time has elapsed, the process proceeds to step S11. If the elapsed time after the first action is exactly the predetermined time, the process may proceed to either step S10 or S11.

In step S11, the control unit 40 causes the robot motion unit 67 to perform the second motion. The second action is, like the first action, an action that increases the likelihood of blocking. The second action may be the same as or different from the first action. The second action may be an action that is more effective than the first action in increasing the likelihood of obstruction. For example, it is assumed that the first action is a non-speech action and the second action is a short speech. More specifically, the first action is raising the hand, displaying on the display device, or emitting light from the light-emitting display device, and the second action is saying "Excuse me", "Excuse me", "Uh~", etc. is assumed. Also, it is assumed that the first action is a meaningless utterance and the second action is a request utterance requesting one's own utterance. For example, the first action is "Excuse me", "Excuse me", "Ano~", etc., and the second action is "I would like to say something", "I have something to say", etc. is assumed. Also, it is assumed that the first action is a polite utterance and the second action is an unpolite utterance. Also, it is assumed that the first action is a short utterance and the second action is a long utterance. Furthermore, it is also assumed that the volume of the utterance of the second action is louder than that of the utterance of the first action.

In the next step S12, similarly to step S3, the likelihood of obstruction is obtained.

In the next step S13, similarly to step S4, it is determined whether or not it is time to speak based on the likelihood of interruption obtained in step S12. If it is determined that it is time to speak, the process proceeds to step S5, and speech is made. If it is determined that it is not time to speak, the process proceeds to step S14.

In step S14, it is determined whether or not a predetermined time has passed after the second action. If it is determined that the predetermined time has not passed yet, the process proceeds to step S15, waits for a predetermined time, returns to step S12, and repeats the processes after step S12. If it is determined that the predetermined time has elapsed, the process proceeds to step S16. Note that if the elapsed time after the second action is exactly the predetermined time, the process may proceed to either step S15 or S16.

In step S16, the control section 40 causes the robot motion section 67 to perform the third motion. The third action is an action that increases the likelihood of obstruction, similar to the first action. The third action may be the same as or different from the first and second actions. The third action may be an action that is more effective than the second action in increasing the likelihood of obstruction. In other words, the later the action is, the more effective the action may be in increasing the likelihood of blocking. For example, it is assumed that the first action is a non-utterance action, the second action is a short utterance, and the second action is a request utterance requesting one's own utterance. For example, the first action is to raise your hand, the second action is to say "Excuse me", "I'm sorry", "Uh~", etc., and the third action is "I would like to say something", "I want to say something". It is assumed that it is a statement such as "There is In addition, the same idea as explained in step S11 can be applied to the contents of the operation that are effective in increasing the susceptibility to obstruction.

In the next step S17, similar to step S3, the likelihood of obstruction is obtained.

In the next step S18, similarly to step S4, it is determined whether or not it is time to speak based on the likelihood of interruption obtained in step S17. If it is determined that it is time to speak, the process proceeds to step S5, and speech is made. If it is determined that it is not time to speak, the process proceeds to step S19.

In step S19, it is determined whether or not a predetermined time has passed after the third action. If it is determined that the predetermined time has not elapsed, the process proceeds to step S20, waits for a predetermined time period, returns to step S17, and repeats the processes after step S17. If it is determined that the predetermined time has elapsed, the process proceeds to step S5. If the elapsed time after the third action is exactly the predetermined time, the process may proceed to either step S5 or S20. That is, when a predetermined time has passed after the third action, the speaker is forced to speak regardless of the degree of interruption and regardless of whether or not it is time to speak.

In the above process, if a predetermined time has passed after the third action, the player may return to the start without speaking. Further, the process for performing the third action and the processes for performing the second and third actions may be omitted.

According to the conference support system 30 configured as described above, based on the participation states of the first participants 10A obtained through the camera 58 and the microphone 59, which are the participation state acquisition unit, the plurality of first participants 10A Since it is determined whether or not it is time to speak based on the obtained ease of interruption, it is possible to find a suitable timing for speaking in a conference.

Further, if the second participant 10B stores the scheduled speech contents in the storage unit 43 in advance, the control unit 40 speaks the planned speech contents through the speaker 68 when it is determined that it is time to speak. . Therefore, the second participant 10B can speak at an appropriate timing by recording the content of the speech to be made in advance.

In particular, since the second participant 10B participates in the conference through the communication line 70, it is difficult to observe the conference participation states of the first participants 10A. Even in such a situation, the second participant 10B is likely to speak at an appropriate timing.

In addition, the control unit 40 grasps the conference participation status of each of the plurality of first participants 10A based on outputs from the cameras 58 and microphones provided for each of the plurality of first participants 10A, and determines each participation status. Then, it is determined whether or not it is time to speak. Therefore, it is possible to more appropriately determine whether or not it is time to speak based on the conference participation status of each of the plurality of first participants 10A.

Also, when it is determined that it is not time to speak, the proxy participation robot 60, which is a pseudo-conference participation device, is made to perform the first action of increasing the ease of interruption. Therefore, it becomes easier for the second participant 10B to speak in the conference.

After the first action, if it is determined that it is not time to speak, the proxy participation robot 60, which is a pseudo-conference participation device, is made to perform the second action of increasing the ease of interruption. Therefore, it becomes easier for the second participant 10B to speak in the conference.

Further, after the execution of the first action, if it is determined that it is not time to speak, the second participant 10B is allowed to speak more in the conference in order to execute the second action more effective in increasing the likelihood of interruption. becomes easier.

Various modifications will be described on the premise of the first embodiment.

In the first embodiment, instead of or in addition to the function of storing the scheduled speech content of the second participant 10B in the storage unit 43, the control unit 40 determines that it is time to speak. A message to that effect may be displayed on the display device 88 . In this case, the second participant 10B may speak into the microphone 86 while viewing the display. It is preferable that the speech be made from the proxy participation robot 60 via the terminal device 80 and the communication line 70 . Since the second participant 10B can speak after seeing the indication that it is time to speak, the second participant 10B can speak at an appropriate timing. In particular, the first participant 10A is holding a conference in a state where they can observe each other's participation in the conference in the real space, while the second participant 10B is participating in the conference through the communication line 70. , it is difficult to observe the conference participation state of the first participant 10A. Even in this case, there is an advantage that the second participant 10B can speak at an appropriate timing.

In the above example, an example in which there is one second participant 10B has been described, but there may be a plurality of second participants 10B.

In the above example, the first participant 10A is provided with the camera 58 and the microphone 59, which are the participation state acquisition unit, but the second participant 10B is provided with the participation state acquisition unit, Based on the conference participation state of the second participant 10B, it may be determined whether or not it is an appropriate speech timing by obtaining the degree of interruption.

Further, in the above example, the second participant 10B participates in the conference through the communication line 70, but the second participant 10B can participate in the conference in the real space together with the first participant 10A. It can be a participant.

Further, in the above example, when the second participant 10B enters the scheduled speech content, it is determined that there is an intention to speak. In this case, it may be determined that there is an intention to speak. For example, when the second participant 10B inputs the planned speech content as character information, the planned speech content is displayed on the terminal device at hand of the other participant, and when the other participant agrees with the planned speech content, enter that you agree. Then, when the number of approvals exceeds a predetermined number, it may be determined that there is an intention to speak, and the processing from step S3 onwards may be executed.

{Second embodiment}
A conference robot according to the second embodiment will be described. In the description of the present embodiment, the same reference numerals are given to the same components as those described in the first embodiment, and the description thereof will be omitted. FIG. 8 is an explanatory diagram showing a conference to which the conference robot 130 is applied, and FIG. 9 is a block diagram showing the electrical configuration of the conference robot.

This conference robot 130 differs from the conference support system 30 in that the second participant 10B does not exist as a premise of the conference, and instead the conference robot 130 autonomously makes remarks for the conference. .

That is, the conference robot 130 participates in the conference instead of the proxy participation robot 60 .

The conference robot 130 includes a control unit 140 , a robot operating unit 167 and multiple sets of cameras 58 and microphones 59 .

A plurality of sets of cameras 58 and microphones 59 is an example of a participation state acquisition unit that acquires conference participation states of a plurality of participants (here, the first participant 10A), as described above.

The robot operation unit 167 performs the operation as a robot in the same manner as the robot operation unit 67 described above. Here, the robot action section 167 includes a speaker 168 and an arm drive section 169 . Under the control of the control unit 140 , the robot operation unit 167 can emit sound through the speaker 168 . The arm driving section 169 also includes a motor or the like for driving at least one arm up and down. Under the control of the control unit 140 , the arm driving unit 169 can raise (raise) or lower the arm of the conference robot 130 . The conference robot 130 may be a humanoid robot, or may be another articulated robot or the like.

The control unit 140 is connected to the multiple sets of cameras 58 and microphones 59 and the robot operation unit 167 , and controls the operation of the robot operation unit 167 based on inputs from the multiple sets of cameras 58 and microphones 59 . In particular, the control unit 140 obtains the ease of interrupting communication by the plurality of first participants 10A based on the conference participation states of the plurality of first participants 10A obtained by the plurality of sets of cameras 58 and microphones 59, Whether or not it is time to speak is determined based on the obtained degree of obstruction. Then, when it is determined that it is time to speak, speech processing through the speaker 68 is executed.

The control unit 140 differs from the control unit 40 in that the main control unit 140 executes the conversation program 140b. The conversation program 140b is a program stored in the storage unit 43, and the CPU 41 executes conversation processing based on the procedures described in the conversation program 140b and the inputs from the plurality of microphones 59. do. For the conversation program 140b itself, a program including a well-known program that performs a process of issuing a statement related to the statement content based on the statement content of the plurality of first participants 10A can be applied. The content of the statement issued by the conversation program 140b may be created before it is determined that it is time to speak, and may be recorded in the statement schedule recording unit as in the first embodiment, or it may be determined that it is time to speak. may be generated when

According to this conference robot 130, the conference robot 130, which autonomously makes a statement for a conference, can make a statement at an appropriate timing. In addition, except for the effect of the second participant 10B making a statement, it is possible to obtain the same effects as those of the first embodiment. In addition, the conference robot 130 can speak for the conference, making the conference lively and supporting the conference.

{Modification}
In addition, each configuration described in each of the above-described embodiments and modifications can be appropriately combined as long as they do not contradict each other. For example, the first embodiment and the second embodiment may be combined so that both the second participant 10B and the conference robot 130 participate in the conference in addition to the plurality of first participants.

Although the present invention has been described in detail as above, the above description is illustrative in all aspects, and the present invention is not limited thereto. It is understood that numerous variations not illustrated can be envisioned without departing from the scope of the invention.

10A first participant 10B second participant 30 conference support system 40, 140 control unit 43a program 43c voice data 45 communication circuit 46 robot control input/output unit 58 camera 59 microphone 60 proxy participation robot 67, 167 robot operation unit 68, 168 Speakers 69, 169 Arm drive unit 70 Communication line 80 Terminal device 86 Microphone 130 Conference robot 140b Conversation program

Claims (11)

A conference support system used in a conference by a plurality of participants,
a participation state acquisition unit that acquires the conference participation states of the plurality of participants;
Based on the conference participation states of the plurality of participants obtained by the participation state acquisition unit, the likelihood of interrupting communication by the plurality of participants is obtained, and whether it is time to speak based on the obtained likelihood of interruption. A control unit that determines whether
with
The control unit includes an utterance schedule recording unit that records utterance schedule contents, and when it is determined that it is time to utter, the utterance schedule contents recorded in the utterance schedule recording unit are said through a speaker. system. The meeting support system according to claim 1 ,
The conference support system, wherein the control section records, in the speech schedule recording section, speech schedule contents of at least one participant among the plurality of participants. The conference support system according to claim 2 ,
The plurality of participants includes a plurality of first participants who hold a conference in a state in which each other's conference participation status can be observed in real space, and a second participant who participates in the conference via a communication line,
The conference support system, wherein the control section records the speech schedule content of the second participant in the speech schedule recording section. The conference support system according to any one of claims 1 to 3 ,
The participation state acquisition unit includes a plurality of individual participation state acquisition units that acquire the conference participation state of each of the plurality of participants,
The control unit obtains the ease of interruption for each of the plurality of participants based on the conference participation states of the plurality of participants obtained by the plurality of individual participation state acquisition units. A conference support system that determines whether or not it is time to speak based on the A conference support system used in a conference by a plurality of participants,
a participation state acquisition unit that acquires the conference participation states of the plurality of participants;
Based on the conference participation states of the plurality of participants obtained by the participation state acquisition unit, the likelihood of interrupting communication by the plurality of participants is obtained, and whether it is time to speak based on the obtained likelihood of interruption. A control unit that determines whether
a speech command reception unit that acquires a speech command by at least one of the plurality of participants;
a virtual conference participation device capable of participating in a conference in the real space where the conference is held;
with
The conference support system, wherein the control unit causes the pseudo-conference participation device to perform a first operation to increase the likelihood of interruption when it is determined that it is not the time to speak after obtaining an intention command to speak. The meeting support system according to claim 5 ,
The conference support system, wherein, after executing the first operation, the control unit causes the pseudo-conference participation device to execute a second operation to increase the likelihood of interruption when it is determined that it is not time to speak. The meeting support system according to claim 6 ,
The conference support system, wherein the second motion is a motion more effective than the first motion in increasing the likelihood of interruption. A conference robot that participates in a conference with a plurality of participants,
a participation state input unit for inputting conference participation states of the plurality of participants;
Based on the conference participation states of the plurality of participants input from the participation state input unit, the likelihood of interrupting communication by the plurality of participants is obtained, and whether it is time to speak based on the obtained likelihood of interruption. a control unit that determines whether or not, and executes processing for speaking through a speaker when it is determined that it is time to speak;
a robot action unit that performs actions as a robot;
with
The robot for conference, wherein the control unit causes the robot action unit to perform a first action to increase the likelihood of interruption when it is determined that it is not the timing to speak after the expected speech content to be generated occurs. A conference robot according to claim 8 ,
the conference participation state of each of the plurality of participants is input to the participation state input unit;
The control unit obtains the likelihood of interruption for each of the plurality of participants based on the conference participation status of each of the plurality of participants, and determines whether or not it is time to speak based on the obtained likelihood of interruption. A conference robot. The conference robot according to claim 8 or 9 ,
The conference robot, wherein, after the execution of the first action, the control section causes the robot action section to perform a second action that increases the likelihood of interruption when it is determined that it is not time to speak. A conference robot according to claim 10 ,
The conference robot, wherein the second action is more effective than the first action in increasing the likelihood of interruption.

Images