JP4845183B2 - Remote dialogue method and apparatus - Google Patents

Description

  The present invention relates to a method and apparatus for performing remote dialogue by transmitting and receiving audio and video between terminals connected via a network.

Systems such as VC (video chat) and videophone are widely used for remote dialogue by transmitting and receiving video together with sound.
According to the current general VC equipment set, it is possible to recognize the position of the face of the subject and track whether the eyes are open or closed. However, in terms of the number of pixels, the accuracy, the computational power of the PC, etc., the level that can track the direction of the line of sight of the subject has not actually been reached.

In the VC, the bust-up image is basically shared by both parties during the dialogue. Therefore, it is not known how the other party's hand operates the keyboard or mouse during the conversation. In addition, it is not uncommon for people to face down when operating these equipment, which can lead to a stagnation of dialogue.
If information about calling a PC function and what is being called can be shared by both parties, it is useful for a smooth and effective dialogue. However, there is no prior art that makes this possible.

Related arts include the following documents.
Patent Document 1 relates to a video conference system, and is characterized by including a display panel that displays received data in RGB and a circuit that displays received video information on the display panel.
Japanese Patent Application Laid-Open No. 2002-271863 “Video Conference System”

This is used in a device used for presentations, etc., and is a content that always recognizes the presenter's voice, accepts a specific keyword as a command, and executes a specific operation in accordance with the command.
For example, the presenter utters “Now” in a scene to be changed, and performs a specific gesture toward the camera in a predetermined timing range. As a result, the image recognition means is activated with the voice “Now” as a trigger, and a gesture command for superimposing instructions is recognized from the received image within a predetermined timing range.

According to this method, since it is necessary to always perform speech recognition, an apparatus for analyzing speech inevitably analyzes all utterances of the presenter. If the device is installed at a position away from the presenter, the size of the device, power consumption, heat and noise generated from the device will not be questioned so much, but generally it is installed near the presenter. However, it would be better if it was small, power-saving, low heat generation and no noise. For that purpose, it is necessary to devise such as reduction and simplification of the total calculation amount of analysis performed by the apparatus.
In speech recognition, it is necessary to analyze what vowels and consonants are uttered. If the speech recognition algorithm is simplified and the burden is reduced, there is a high possibility that an important command will be mistaken.
Also, depending on speech recognition, words spoken appropriately without being conscious may be misunderstood as commands, so the user must always try to pronounce correctly. Therefore, it is suitable for formal remarks such as presentations, but is not suitable for informal chats.
From the above, it is difficult to say that performing voice recognition constantly or using voice as a trigger is suitable for VC.

If constant computation is required, it can be executed with a very light load depending on the algorithm. There are several methods, for example, a method of reducing the amount of information processed per unit time, and a method of configuring the main calculation itself by simple addition and subtraction.
In the case of speech recognition, there is a method of reducing the sampling frequency and its bit amount in order to reduce the amount of information processed per unit time. In this method, the content of the utterance itself becomes slightly unclear and we humans will make many mistakes. It is still better to focus only on “change in voice volume” and “change in voice pitch (frequency)”, but it becomes difficult to identify and extract utterance content (words, etc.). It will be difficult to recognize words by simple addition and subtraction.
In the case of motion recognition, in order to reduce the amount of information processed per unit time, the number of frames per second and the size of the frame are reduced, the color information is discarded, only the luminance information, and the bit amount of the luminance information is increased. There are ways to reduce it. In this method, fine operations are discarded, but if the operation is large or easy to predict, it can be reduced to a considerable amount. In general, motion recognition is generally based on simple addition and subtraction between preceding and following frames.
Therefore, motion recognition can be executed with a very light load depending on the algorithm.
If a special action that is unlikely to occur unless intended by the person is used as a trigger, there is an advantage that misidentification can be suppressed.

Japanese Patent Application Laid-Open No. 2004-133867 discloses a conventional technique that uses an operation as a trigger.
Patent 3160108 "Driving support system"

This relates to a system that supports driving of an automobile or the like, and recognizes a driver's gaze from various out-of-vehicle information and the driver's line of sight detected by a line-of-sight detection circuit, and outputs desired out-of-vehicle information to a CRT. It is characterized by being output to.
When presenting the situation outside the vehicle to the driver and contributing to safe driving, a specific action of the driver is used as a trigger for information presentation to the driver.
The driving of a car is under some restraint conditions and is suitable for continuing to acquire line-of-sight information, but it is difficult to apply this directly to VC. The line-of-sight extraction, which is an existing technique, is possible in principle, but is still expensive, and the apparatus generally invades the user as the line of sight is strictly followed. It is difficult to think that this is easily accepted by general VC users, and further ingenuity is required.

Non-Patent Document 1 is a prior art that uses a voice of a different mode from VC as a trigger in VC.
Proceedings of the European Conference on Speech Communication and Technology (Eurospeech 2003), 1201-1204, Sep. 2003

This is characterized by using the pitch of the voice as a trigger, and uttering a normal utterance as it is, leaving the utterance uttered intentionally as a command.
The speech recognition apparatus is operated on the PC, and the content of speech in normal speech is recognized and converted into text. Then, if there is an intentional utterance such as “save” in a high voice, the “save” is recognized as a command, and the command is transferred to appropriate software. In this case, the text data until “save” is spoken is “saved” on the PC.

However, in the case of a dialogue between two or more people such as VC, it is difficult to intentionally control sound quality such as voice pitch. For example, in a dialogue between two parties, there is a possibility that the voice is turned upside down and emotions are expressed such as laughter, which may be mistaken as a command.
Therefore, it is effective when you are monologized in a quiet room alone, but is not suitable for dialogue.
Moreover, since voice recognition is always performed and voice is used as a trigger, it is like a voice constraint that avoids a trigger word, so it is difficult to say that it is suitable for a VC mainly using free dialogue. .

  Therefore, the present invention provides a remote interaction method that can recognize a command accurately while sharing a light load on the apparatus, and can also share a command with a conversation partner in order to be suitable for a VC, and can perform a natural conversation. It is an object to provide an apparatus to be implemented.

  In order to solve the above problems, a remote interaction method of the present invention includes at least a voice / image input / output unit and a recognition unit, a communication unit, a terminal having a computer, and a network that exchanges data among the plurality of terminals. In a system for performing remote dialogue by transmitting and receiving audio and video between each terminal, when a predetermined subject specific operation is recognized by the image recognition unit with respect to the subject imaged by the image input unit, The command voice recognition means in the voice recognition means is activated with specific action recognition as a trigger, and the command voice recognition means recognizes the command from the voice collected by the voice input means.

  Here, the voice recognized as the command by the command voice recognition means may be a predetermined keyword.

  The specific operation may be an operation that continues for a period of time, and a voice that is collected from the start to the end may be regarded as a command.

  The command may be output to software provided in the terminal, and the command content may be executed by the software.

  The command may be transmitted to the terminal of the conversation partner, and the command content may be executed by software provided in the partner terminal to contribute to information sharing.

  The specific action may be made to contribute to suppression of misperception as a special mode action that hardly occurs unless the subject himself / herself intends.

  The command content for the same voice command may be changed according to the difference in the specific operation.

  A wink is useful as the specific operation.

  The right wink may be a specific operation that specifies execution of a subsequent command at its own terminal, and the left wink may be a specific operation that specifies execution of a subsequent command at the terminal of the conversation partner. .

  As the specific operation, an operation of rotating the head to a range where one eye is not imaged by the image input means is also useful.

  In addition, a specific operation of placing a finger on a predetermined part of the head is also useful.

Further, it is also useful to substitute a motion of covering a specific eye with a palm instead of a wink. In addition, the action of raising a specific shoulder, showing a specific nail of the hand, and covering a specific nostril is also useful. In order to increase the precision, headsets that use retroreflective members (microphones and speakers worn on the head), earphones, glasses, rings, bracelets, necklaces that are easy to identify optically It is also useful to set a specific operation such as an operation of showing, hiding, or moving a mounted product such as earrings, earrings such as ears, a processed nail or a part of the mounted product.
As described above, as the specific action, an operation of showing a predetermined part of the body or an attached product that did not appear on the counterpart terminal, or hiding the predetermined part of the body or the attached article that appeared on the counterpart terminal can be used.

  The remote interaction apparatus of the present invention comprises at least terminals having voice and image input / output means and recognition means, communication means, and a computer, and a network through which data is exchanged among the plurality of terminals. In addition, in a system for performing remote dialogue by transmitting and receiving video, when a specific action of a predetermined subject is recognized by the image recognition means with respect to the subject imaged by the image input means, the specific action recognition is activated as a trigger. And command voice recognition means for recognizing a command from the voice collected by the voice input means.

  As the voice input / output means, a microphone and a speaker that are used by being worn on the head may be used, and a light emitting member may be provided at a position near the ear at the time of wearing.

  As the voice input / output means, a microphone and a speaker which are used by being mounted on the shoulder portion, and those provided with light emitting members at both shoulder positions at the time of the mounting are also useful.

  According to the present invention, since a specific operation is used as a trigger for inputting a command, the apparatus is lightly loaded and command misunderstanding can be suppressed. In addition, since the command can be shared with the conversation partner in a natural form, it contributes to the smooth progress of the conversation.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is an explanatory diagram showing an overview of a VC system.
The PC itself is equipped with various applications such as voice recognition, image recognition, and communication. Input devices such as a mouse and keyboard and output devices such as a monitor are connected, and VC equipment such as a microphone, speaker, and video camera is connected. Is provided. Each of such terminals is connected to other terminals via a communication line such as the Internet, and enables real-time VC.

The present invention is based on the fact that an operation in a special state that is not likely to occur unless intended normally is used as a command input trigger. If this is compared with the said nonpatent literature 1, it respond | corresponds to shifting to command input mode by specific operation instead of the change of a voice pitch.
FIG. 2 is an explanatory diagram showing a main part of a conventional VC system according to Non-Patent Document 1. As shown in FIG.
There is a similar device on the other side of the communication line, and VC is executed between the two parties. The communication means includes a network card, network software, and the like. The VC software exchanges the video and audio of the other party and oneself with the communication means.
Note that the other software in the figure specifically indicates that a browser is searching through the net.
On the other hand, FIG. 3 is an explanatory view showing the main part of the VC system of the present invention.
In the configuration of the present invention, the moving image audio used in the VC software is utilized. It also directly communicates with communication means and other software.
In this way, Non-Patent Document 1 describes a high voice command input mode, a command of “save”, a high voice, for example, an utterance in which the voice level is changed so that “save” is spoken with a high voice. The command input mode is terminated at the same time. On the other hand, the present invention is different in that the operation (maintaining) represents on / off of the command mode state, and the utterance is a command.

FIG. 4 is a flowchart showing an outline of a system for recognizing command voice in the present invention.
In a state where an image input device such as a camera and image recognition software are activated, a subject that is a conversation person facing the camera is imaged. When a predetermined specific action of the subject is recognized by the image recognition software, a voice input device such as a microphone and the voice recognition software are activated using the specific action recognition as a trigger. The command voice is recognized by the command voice recognition software from the voice collected by the voice input device. Then, a command is generated from the command voice and output.

Similarly, FIG. 5 is a flowchart showing a system outline of another embodiment.
In the example in FIG. 4, the specific action recognition is used only as a trigger, but in this example, the specific action recognition is also reflected in command generation.
That is, for example, a specific operation content difference, such as a difference between a right eye wink and a left eye wink, and a specific sound are combined, and commands having different contents are generated according to each.

FIG. 6 is also a flowchart showing a system outline of another embodiment.
In this example, both the other party and himself / herself are recognized. The other party's recognition interrupt process and one's own recognition interrupt process are performed by a sequential interrupt type program, and the other party's and one's own actions are processed in parallel.

FIG. 7 is a flowchart showing a system overview of the prior art.
The voice input device is activated from the beginning, which is different from the present invention.
While many of the prior arts are clear voice waiting loops, the present invention is a rough operation waiting loop that can reduce arithmetic processing.

Similarly, FIG. 8 is a flowchart showing an outline of the system of the prior art.
In the example in FIG. 7, specific voice recognition is used as a trigger, but in this example, specific change recognition of voice is used as a trigger.

In any case, in the present invention, the process proceeds to a step in which a command is output by a combination of a specific action that is unlikely to occur unless it is normally intended and a command by a specific voice such as a clear keyword defined in advance. Therefore, it is possible to relay commands accurately with low load.
The output command is input to software provided in the terminal to which the command voice is input or the terminal of the conversation partner via the communication line, and the command content is executed by the software.

An example of the specific operation is wink.
Wink is an operation that is difficult to express in a normal dialogue unless intended by the person. Furthermore, the operation of maintaining the wink state for several seconds is very rare.
When an operation that continues for a time is selected as a specific operation in this way, it is possible to contribute to the recognition of the command voice, for example, the voice collected from the start to the end of the action is regarded as a command.

9 to 11 are explanatory diagrams showing the mode of command transfer according to the state of the wink. FIG. 9 outputs a command to the terminal by the right eye wink, and FIG. 10 shows the other party by the left eye wink. The command is output to the terminal, and FIG. 11 shows that the command is output to both the terminal and the other terminal by winking both eyes.
The wink as the specific operation is preferably an operation of closing one eye.
Then, it becomes possible to change the command content for the same voice command according to the difference in specific operation, and it is easy to give diversity to the command.
For example, if it is the right wink, it means a specific operation that specifies that the subsequent command is executed on its own terminal, and if it is the left wink, it specifies that the subsequent command is executed on the terminal of the conversation partner. It is possible to give different meanings to the winks of the left and right eyes, for example, by meaning a specific action.
In this case, it is only necessary to set the apparatus so that the left wink of the other party is ignored, the left wink of the other party is valid, and the command at that time is the other party's voice.
On the other hand, if it is desired to reduce misrecognition, a setting may be made such that only commands that are interpreted in the same way by both the own device and the other device are recognized as authentic commands. As described above, when recognition processing is multiplexed in parallel, recognition accuracy can be improved.

Even when a VC is not actually used, an existing VC device can be used alone. For example, the meaning of each wink can be changed as appropriate, for example, the right wink means “desktop ~” and the left wink means “Internet ~”.
The wink is an operation of closing one eye for a time t or more. t is appropriately set, for example, 0.5 seconds, and operations shorter than this are ignored. Blinks are repeated for a short time at the same time for both eyes, so there is no risk of misunderstanding.

When wink is used as a specific operation, it can be used as follows, for example.
Speech during normal operation is ignored by the speech recognition device. If only one eye is kept closed, the speech recognition device is activated, and the speech that is made until the next time the eyes are opened is recognized. Then, the voice-recognized command is transmitted to predetermined software according to the recognized content.
For example, in the middle of a normal dialogue with both eyes open, keep one eye closed and say “Search Fukuzawa Yukichi”, then open your eyes. In this case, “Fukuzawa Yukichi” is searched by the search engine, and the search result is displayed on the monitor.

According to this system, the conversation partner can recognize what he / she is doing, so there is an advantage that the situation can be shared. In the traditional case of pointing down and typing “Fukuzawa Yukichi” on the keyboard, you don't know what the conversation partner is doing. Then, it becomes tedious for a few seconds and bored. Also, it is usually redundant, such as "wait a little ... (the sound of hitting the keyboard and the keyboard)", "what did you do now", "searched Yukichi Fukuzawa. The first one ..." Dialogue is required.
The wink-like operation has the advantage of not obstructing command input even if the hand is buried with a mouse or keyboard.
In addition, when the conversation partner closes his eyes, he can understand that he has entered the command input mode, so that he can easily understand the content of the command that is subsequently spoken and can maintain a sense of reality.
And if the conversation partner is also equipped with the same application, the conversation partner can also search for “Fukuzawa Yukichi” and see the search results. . This makes it possible to explain the functions of the PC from a remote location to an elderly person who is not familiar with the operation even though the PC can be turned on, and can be applied to correspondence education combined with application software.

It is also effective to set the specific action as an action of rotating the head until one eye is not captured by the camera.
FIGS. 12 to 14 are explanatory views showing specific operations of another embodiment. FIG. 12 shows a command output to its own terminal by rotating the head to the right, and FIG. FIG. 14 shows that a command is output to both the own terminal and the partner terminal by rotating upward.
Such a rough movement can be sufficiently recognized by an inexpensive device at present. In addition, since the monitor is generally almost flat, it is possible to recognize a picture and characters projected on the monitor with one eye without load for a short time. Therefore, it is possible to know in real time whether or not a command by utterance is correctly recognized.

It is also effective to set the specific action as an action of applying a specific finger to a specific part of the head.
FIGS. 15 to 17 are explanatory views showing a specific operation of another embodiment. FIG. 15 outputs a command to its own terminal by an operation of placing a finger on the right temple, and FIG. 16 applies a finger to the left temple. The command is output to the partner terminal by the operation, and FIG. 17 shows that the command is output to both the terminal and the partner terminal by the operation of placing the finger on the forehead.

18 to 20 are explanatory views showing specific operations of another embodiment, in which FIG. 18 outputs a command to his / her terminal by the operation of placing the finger of the right hand on the nose, and FIG. 19 shows the finger of the left hand on the nose. A command is output to the partner's terminal by the operation of hitting the terminal, and FIG. 20 shows that the command is output to both the terminal of the partner and the partner by the action of hitting a plurality of fingers to the nose.
As described above, since the part to be touched by the finger or the like and the part to be touched can be set in various ways, the change in the command content due to the difference in the specific operation can be changed in various ways easily.

A member that emits light may be attached to the head or the like to contribute to recognition of a specific action.
FIG. 21 is an explanatory view showing a state in which the light emitting member is mounted in the vicinity of the ear, and FIG. 22 is an explanatory view showing a state in which the head is rotated in that state.
In the example shown in the figure, a speaker connected with LEDs is attached to the left ear.
According to this, the operation can be recognized by the light of the LED even when the face is not illuminated. When the head is rotated counterclockwise, the LED light does not reach the camera with his nose or right cheek. Therefore, if no LED light is input for a certain period of time, the command input mode can be set to start.

FIG. 23 is an explanatory view showing a state where the light emitting member is attached to both shoulder positions, and FIG. 24 is an explanatory view showing a state where the right shoulder is raised in that state.
In the example shown in the figure, a set of microphones and speakers provided with LEDs is mounted on both shoulder positions.
Usually, it is assumed that two LED lights are reflected on the camera almost horizontally. If the right shoulder is raised or the upper body is tilted to the left as shown, the LED light on the right shoulder rises. As described above, the specific operation can be recognized based on the positional relationship between the plurality of LED lights.
In addition, when this type of voice input / output device to be worn on the shoulder is used, there is no pain in the ear as in conventional headphones even when used for a long time. If the voice is sent and received wirelessly as a battery built-in type, you can leave the seat as it is, and it is convenient.

If a plurality of LED emission wavelengths are prepared and used as IDs, even if there are a plurality of people in front of the camera, they can be identified without recognizing each person's face.
If function assignment information dedicated to each person is stored in the server, the same function assignment can be used by any PC in a predetermined network.

According to the present invention, a command can be relayed accurately and can be shared with a conversation partner in a natural form while being able to be configured with a light load by combining existing devices.
Since the interface can be used naturally, even those who are not familiar with PC operations, such as elderly people, can easily learn, and it is possible to impersonate the application and penetrate it into the living space.
In addition, it can be applied to correspondence education and the like, and is widely useful in industrial applications.

Explanatory diagram showing an overview of the VC system Explanatory drawing which shows the principal part of the system of the conventional VC by a nonpatent literature 1. Explanatory drawing which shows the principal part of the system of VC of this invention Flow chart showing the system outline for recognizing command voice Same example diagram Same example diagram Flow chart showing system overview of the prior art Same example Explanatory diagram showing command transfer mode according to wink mode Same state diagram Same state diagram Explanatory drawing which shows specific operation | movement of another Example using rotation of a head. Same state diagram Same state diagram Explanatory drawing which shows specific operation | movement of another Example instruct | indicated with a finger | toe Same state diagram Same state diagram Explanatory drawing which shows the specific operation | movement of another Example which instruct | indicates a nose Same state diagram Same state diagram Explanatory drawing which shows the state which mounted | wore the light emitting member in the ear vicinity position Same state diagram Explanatory drawing which shows the state which mounted | wore the light emitting member in both shoulder positions Same state diagram

Claims (17)

At least a voice and image input / output means and recognition means, a communication means, a terminal having a computer, and a network through data exchange between the plurality of terminals,
Audio and video obtained by the respective audio input unit and the image input means at each terminal, a method of performing remote dialogue by mutually transmitting and receiving each other between the respective terminals,
Regard the subject captured by the image input means, a specific operation of the subject to a predetermined, the step of recognizing the image recognition unit,
Remote Interactive method of as a trigger that particular operation recognition, the speech is picked up by the voice input means, characterized in that it comprises a step of recognizing the command by the speech recognition means. The remote dialogue method according to claim 1, wherein the voice recognized as the command by the voice recognition means is a predetermined keyword. The remote interaction method according to claim 1 , wherein the specific operation is a continuous operation, and a command is recognized from a voice collected from the start to the end of the specific operation. The remote interaction method according to any one of claims 1 to 3, wherein a command recognized by the voice recognition means is output to software provided in the terminal, and the command content is executed by the software. The remote dialogue method according to any one of claims 1 to 4, wherein a command recognized by the voice recognition means is transmitted to a terminal of a conversation partner, and the contents of the command are executed by software provided in the partner terminal. The remote interaction method according to any one of claims 1 to 5, wherein command contents for the same voice command are changed according to a difference in specific operation. The remote interaction method according to any one of claims 1 to 6, wherein the specific operation is winking. The right wink is a specific action that specifies that subsequent commands are executed on your terminal,
The remote interaction method according to claim 6 or 7, wherein the left wink is a specific operation that specifies execution of a subsequent command at a terminal of an interaction partner. The remote interaction method according to claim 1, wherein the specific operation is an operation of rotating the head to a range where one eye is not imaged by the image input means. The remote interaction method according to claim 1, wherein the specific action is an action of placing a finger on a predetermined part of the head. The remote dialogue method according to claim 1, wherein the specific action is an action of showing a predetermined part of the body that has not appeared. The remote interaction method according to claim 1, wherein the specific action is an action of hiding a predetermined part of the body that has appeared. The remote interaction method according to any one of claims 1 to 6, wherein the specific action is an action of showing a body wearing item that has not appeared. The remote interaction method according to any one of claims 1 to 6, wherein the specific action is an action of hiding the worn body accessory. At least a voice and image input / output means and recognition means, a communication means, a terminal having a computer, and a network through data exchange between the plurality of terminals,
Audio and video obtained by the respective audio input unit and the image input means at each terminal, in the apparatus for performing remote dialogue by mutually transmitting and receiving each other between the respective terminals,
Regard the subject captured by the image input means, an image recognition means for recognizing the specific operation of the subject to a predetermined,
As a trigger that particular motion recognition remote interactive device characterized by having a voice recognition means for recognizing a command from the speech that is picked up by the voice input means. The voice input / output means is a microphone and a speaker that are used on the head,
The remote interaction device according to the near-ear position, in Claim 15 in which the light emitting member Ru is attached at the time when the subject wearing it is imaged by the image input means. The voice input / output means is a microphone and a speaker that are used by attaching to the shoulder,
The remote interaction device according to the shoulders vicinity at the time when the subject wearing it is imaged by the image input means, to claim 15 in which the light emitting member Ru is attached.

Images