JP2019220145A - Operation terminal, voice input method, and program - Google Patents

Abstract

To make an operation terminal ready to input a voice without troubling a user.SOLUTION: An operation terminal comprises: an imaging part which images a space; a person detection part which detects a user from information on the imaged space; a voice input part which inputs an utterance voice made by the user; a coordinate detection part which detects, when the person detection part detects the user, first coordinates of a predetermined first region included in the upper limb of the user and second coordinates of a predetermined second region included in the upper half of the body excluding the upper limb of the user on the basis of information obtained by predetermined means; and a condition determination part which compares position relation between the first coordinates and second coordinates, and makes the voice input part ready to input the voice when the position relation meets a predetermined condition at least once.SELECTED DRAWING: Figure 4

Description

  The present disclosure relates to an operation terminal operated by a user's uttered voice, a voice input method in the operation terminal, and a program for causing a computer to execute the voice input method.

  When a user operates a specific terminal by voice, the terminal needs to collect the voice of the user, and the method is roughly classified into two types. One is a method of judging the start of a user's voice input by a user's operation and starting sound collection. The other is a method of constantly collecting sound and extracting sound from the collected sound. In the latter method, the user always feels that voice is being collected by the terminal, so that the user may have a concern that privacy is leaked. Therefore, a method of collecting sound only when the user has indicated his / her intention to input a voice, as in the former method, is effective.

  Also, in recent years, a technique of instructing a robot by detecting a gesture of a user has been known. For example, Patent Document 1 discloses that in order to perform a pointing operation in a natural state and perform highly accurate pointing position detection, the position of a person's head and the tip of a hand are determined from images captured by a plurality of cameras. There is disclosed a pointing position detection device that detects a position and a hand direction, detects a direction pointed by a person based on the detection results, and detects a position pointed by the person from the detected direction.

  Patent Document 2 specifies a range image sensor capable of appropriately recognizing a gesture using an arm from among a plurality of range image sensors in order to appropriately recognize a gesture using an arm performed at an arbitrary position. A gesture management system that registers a gesture recognized using a specified range image sensor is disclosed.

Patent No. 4149213 Patent No. 6303918

  However, in Patent Literatures 1 and 2, a troublesome gesture such as turning a user's arm correctly in a specific direction in a space is required for the user, and further improvement is required.

  An object of the present disclosure is to provide an operation device or the like that brings an operation terminal into a state in which a voice input can be accepted without giving a user trouble.

An operation terminal according to an aspect of the present disclosure is an operation terminal operated by a user's uttered voice,
An imaging unit for imaging a space;
From the information of the imaged space, a person detection unit that detects the user,
A voice input unit that receives an input of an uttered voice by the user;
When the user is detected by the human detection unit, the first coordinate of a predetermined first portion included in the upper limb of the user based on information obtained by predetermined means and included in the upper body excluding the upper limb of the user A coordinate detection unit that detects a second coordinate of a predetermined second part to be
Comparing the positional relationship between the first coordinates and the second coordinates, and, at least once, when the positional relationship satisfies a predetermined first condition, a condition determination that sets the voice input unit to a state in which a voice input can be accepted. Unit.

  According to the present disclosure, the operation terminal can be set to a state in which voice input can be accepted without giving the user any trouble.

FIG. 3 is a diagram illustrating an example of a positional relationship between an operation terminal and a user according to Embodiment 1 of the present disclosure. It is a figure showing an example of appearance composition of an operation terminal. FIG. 4 is a diagram illustrating an example of skeleton information of a user measured by an imaging device. FIG. 2 is a block diagram illustrating an example of a configuration of an operation terminal according to Embodiment 1 of the present disclosure. 6 is a flowchart illustrating an example of a process of a start condition determination unit according to Embodiment 1 of the present disclosure. 11 is a flowchart illustrating an example of a process of a management unit according to the embodiment of the present disclosure. FIG. 5 is a diagram illustrating skeleton information of a user for explaining a start condition. It is a figure showing an example of a gesture possible range. FIG. 9 is a diagram illustrating a case where a plurality of users make a gesture on the operation terminal. It is a figure showing the first example of a state notice. It is a figure showing the 2nd example of a state notice. It is a figure showing the 3rd example of a state notice. It is a figure showing the 4th example of a state notice. It is a figure showing the 5th example of a state notice. FIG. 15 is a block diagram of the operation terminal when the display device and the playback device illustrated in FIGS. 10 to 14 are added to the block diagram of the operation terminal illustrated in FIG. 4. FIG. 9 is a block diagram illustrating an example of a configuration of an operation terminal according to Embodiment 2. 13 is a flowchart illustrating an example of a process of a termination condition determination unit according to Embodiment 2 of the present disclosure. FIG. 7 is a diagram illustrating skeleton information of a user in order to explain an end condition. 13 is a flowchart illustrating an example of a process of a timeout determination unit according to Embodiment 2 of the present disclosure. 15 is a flowchart illustrating an example of a process of a management unit according to the second embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of a configuration in a case where an imaging device, a playback device, and a display device are configured by devices different from the operation terminal 1.

(Knowledge underlying the present disclosure)
The present inventor is studying, for example, an operation terminal that leans against a wall in a house and operates various electric devices installed in the house by recognizing a voice from a user. In such an operation terminal, it is necessary to recognize that the user is performing a voice input or is attempting to perform a voice input. Many terminals that can be operated by voice always pick up voice so that a specific phrase can always be recognized, and trigger voice recognition of phrases other than the specific phrase when triggered by recognition of the specific phrase It is common to provide. However, in this configuration, since the voice is always collected, there is a possibility that the user may have a concern such as invasion of privacy. Therefore, there is a need for a mechanism for determining a user's intention to start voice input without constantly collecting voice.

  In addition, if a configuration in which the user utters a specific phrase is applied to the operation terminal as it is, the user is required to utter a specific phrase every time the home appliance is operated, and the user is facing the operation terminal. Nevertheless, since the user needs to utter a specific phrase, the user is bothered and unnatural.

  On the other hand, there is a method of operating a device such as a robot by using a gesture performed by a user as a trigger, as described in Patent Documents 1 and 2 described above.

  However, the gesture to be detected in Patent Literature 1 is a gesture for the user to make the robot pick up an object or move the robot, and is not a gesture for indicating intention to start voice recognition. Therefore, in Patent Literature 1, a gesture is required for the user to turn his arm in a specific direction in space. Therefore, when the technique of Patent Document 1 is applied to the operation terminal, the user needs to make a gesture of turning his / her arm in a specific direction in order to start voice recognition, which gives the user trouble.

  Further, Patent Literature 2 is a technology for managing a gesture performed by a user using an arm performed at an arbitrary position in a space such as a shopping mall, a museum, an exhibition hall, and the like. It's not a gesture management technology. In addition, since the gesture to be managed in Patent Literature 2 is a gesture in which the arm is pointed at an object such as an exhibit, the direction of the arm is important, and if the direction is different, the gesture is determined to be different. Therefore, when the technique of Patent Literature 2 is applied to the operation terminal as it is, the user needs to perform a gesture of turning his / her arm in the same direction as the managed gesture, which gives the user trouble. Further, in Patent Document 2, when trying to start voice recognition with a simple gesture irrespective of the direction of the arm, the user needs to perform various gestures with different arm directions that the user wants to use to start the voice recognition in advance. It is necessary to register, again giving the user trouble.

  Therefore, the present inventor has obtained the knowledge that a simple gesture such as irrespective of the exact arm direction is effective in order to start voice recognition without giving the user annoyance, and disclosed the present disclosure. I came to imagination.

An operation terminal according to an aspect of the present disclosure is an operation terminal operated by a user's uttered voice,
An imaging unit for imaging a space;
From the information of the imaged space, a person detection unit that detects the user,
A voice input unit that receives an input of an uttered voice by the user;
When the user is detected by the human detection unit, the first coordinate of a predetermined first portion included in the upper limb of the user based on information obtained by predetermined means and included in the upper body excluding the upper limb of the user A coordinate detection unit that detects a second coordinate of a predetermined second part to be
Comparing the positional relationship between the first coordinates and the second coordinates, and, at least once, when the positional relationship satisfies a predetermined first condition, a condition determination that sets the voice input unit to a state in which a voice input can be accepted. Unit.

  According to this configuration, when the positional relationship between the first coordinates of the first part included in the upper limb of the user and the second coordinates of the second part included in the upper body excluding the upper limb of the user satisfies a predetermined first condition. Then, the voice input unit is set to a state in which voice input can be accepted. Therefore, in this configuration, for example, the user can perform a simple gesture irrespective of the direction of the arm, such as raising the arm slightly above the neck, so that the voice input unit can be in a state where the voice input can be accepted. . As a result, the operation terminal can be set to a state in which voice input can be accepted without giving the user any trouble.

In the above aspect, the apparatus further includes a skeleton information extracting unit that extracts skeleton information of the user from the information of the space,
The information obtained by the predetermined means may be the skeleton information.

  According to this aspect, since the first coordinates and the second coordinates are detected based on the skeleton information of the user, the first coordinates and the second coordinates can be accurately detected.

  In the above aspect, the imaging unit may be a visible light camera, an infrared camera, a TOF sensor, an ultrasonic sensor, or a radio wave sensor.

  According to this configuration, since the imaging unit is configured by the visible light camera, the infrared camera, the TOF sensor, the ultrasonic sensor, or the radio wave sensor, the spatial information includes the distance information, and the user is in the surrounding space. A user can be detected accurately.

  In the above aspect, the positional relationship may be a positional relationship between the first coordinates and the second coordinates in a vertical direction.

  If the positional relationship between the first coordinate and the second coordinate in the vertical direction satisfies the first condition, the voice input can be accepted, so the user performs a simple gesture such as raising the upper limb in the vertical direction, for example. Just by doing so, it can be set in the acceptable state.

  In the aspect described above, the positional relationship may be a positional relationship between the first coordinates and the second coordinates in a trunk axis direction of the user.

  According to this configuration, when the positional relationship between the first coordinates and the second coordinates satisfies the first condition, the receivable state is established. Therefore, for example, the user can easily move the upper limb in the trunk axis direction. The gesture can be brought into the acceptable state only by performing a gesture. In addition, since the positional relationship is determined based on the trunk axis direction, the user raises the upper limb in the trunk axis direction without worrying about the current posture such as a lying state or a standing state, for example. Thus, the above-mentioned receivable state can be set.

In the above aspect, the coordinate detection unit further detects a third coordinate of a third part in the upper body,
The first condition is that the angle formed by the first coordinate, the second coordinate, and the third coordinate exceeds a predetermined threshold, is lower than the predetermined threshold, or falls within a predetermined range. Good.

  According to this configuration, the third coordinate of the third part in the upper body is further detected, and the angle formed by the first coordinate, the second coordinate, and the third coordinate exceeds, falls below, or falls within a predetermined range. If it is, it is determined that the positional relationship satisfies the first condition. Therefore, the user can be brought into the receivable state by, for example, a gesture in which the upper limb is at a predetermined angle with respect to the trunk axis direction.

In the above aspect, the first site includes a plurality of sites included in the upper limb,
The first coordinates may be determined based on coordinates of one or more of the plurality of parts.

  According to this configuration, since the first coordinates are determined based on the coordinates of each of the plurality of parts constituting the first part, the first coordinates can be determined flexibly.

In the above aspect, the second portion includes a plurality of portions included in the upper body excluding the upper limb,
The second coordinates may be determined based on coordinates of one or more of the plurality of parts.

  According to this configuration, since the second coordinates are determined based on the coordinates of each of the plurality of parts constituting the second part, the second coordinates can be determined flexibly.

In the above aspect, the first condition includes a plurality of second conditions,
The condition determination unit may set the receivable state when the positional relationship satisfies at least one of the plurality of second conditions or a third condition obtained by combining a part of the plurality of second conditions. .

  According to this configuration, it is possible to flexibly determine whether the positional relationship satisfies the first condition.

  In the above aspect, a display unit or a playback unit that outputs information indicating whether the voice input unit is in the receivable state may be further provided.

  According to this configuration, it is possible to visually or audibly notify the user of information indicating whether or not the voice input unit is in a receivable state.

  In the above aspect, the display unit may be a display.

  According to this configuration, it is possible to notify the user of information indicating whether the voice input unit is in the receivable state using the display.

  In the above aspect, the information indicating whether the voice input unit is in the receivable state may be a color, a text, or an icon.

  According to this configuration, it is possible to notify the user whether or not the voice input unit is in the receivable state using the color, the text, or the icon.

  In the above aspect, the display unit may be a light emitting device that emits light indicating that the voice input unit is in the receivable state.

  According to this configuration, the user can be notified of whether or not the voice input unit is in a receivable state by the light emitted from the light emitting device.

  In the above aspect, the playback unit may output a sound indicating whether the sound input unit is in the receivable state.

  According to this configuration, the user can be notified by voice whether or not the voice input unit is in a receivable state.

  In the above aspect, the reproducing unit may output a sound indicating whether or not the audio input unit is in the receivable state.

  According to this configuration, the user can be notified by sound whether or not the voice input unit is in a receivable state.

  In the above aspect, the condition determination unit may compare the positional relationship only when a distance between the operation terminal and the user satisfies a predetermined fourth condition.

  According to this configuration, only when the distance between the operation terminal and the user satisfies the predetermined fourth condition, the positional relationship between the first coordinates and the second coordinates is compared, so that the user who does not intend to operate the operation terminal can In this case, it is possible to prevent the processing for comparing the positional relationships from being executed, and to reduce the processing cost.

  In the above aspect, the condition determination unit may end the receivable state when a silent section continues for a predetermined time in the receivable state.

  According to this configuration, if the silent section continues for a certain period of time in the receivable state, the receivable state is ended, so that the receivable state is continued even though the user does not intend to operate the operation terminal. Can be prevented. As a result, the privacy of the user can be secured.

  In the above aspect, the condition determination unit may continue the receivable state as long as the positional relationship satisfies the first condition in the receivable state.

  According to this configuration, in the receivable state, as long as the positional relationship satisfies the first condition, the receivable state is continued, so that the user can continue the gesture in which the positional relationship satisfies the first condition, thereby performing an operation. It is possible to indicate the intention of the operation by voice to the terminal.

  In the above aspect, the condition determination unit may end the receivable state when the state where the positional relationship does not satisfy the first condition continues for a predetermined timeout period in the receivable state.

  According to this configuration, when the state in which the positional relationship does not satisfy the first condition in the receivable state is continued for the timeout period, the receivable state is terminated, so that the user has no intention to operate the operation terminal. Therefore, it is possible to prevent the receivable state from being continued. As a result, the privacy of the user can be secured.

  In the above aspect, the condition determination unit may extend the timeout period when determining that the positional relationship satisfies the first condition during the timeout period.

  According to the configuration, in the timeout period, the gesture in which the positional relationship satisfies the first condition is performed again, so that the state in which the voice input can be accepted can be continued.

  In the above aspect, the condition determination unit may continue the receivable state if a voice input is detected at the end of the timeout period.

  According to this aspect, even if the state in which the positional relationship does not satisfy the first condition is continued for the timeout period, if a voice input is detected at the end of the timeout period, the acceptable state is continued. It is possible to prevent the receivable state from ending even though the utterance for operation is being performed.

  In the above aspect, the condition determination unit may end the receivable state when the positional relationship satisfies a predetermined fifth condition different from the first condition.

  According to this configuration, the user can end the receivable state by performing a gesture whose positional relationship satisfies the fifth condition.

  In the above aspect, the condition determination unit may recognize a specific one as an operator of the operation terminal when the human detection unit detects a plurality of users.

  According to this configuration, when the human detection unit detects a plurality of users, a specific one is recognized as an operator of the operation terminal. Can be given the right to operate the operation terminal. As a result, the utterance related to the operation of the operator can be accurately recognized.

  In the above aspect, the operator may be a user closest to the operation terminal among the plurality of users.

  According to this configuration, when the human detection unit detects a plurality of users, the user who is closest to the operation terminal is specified as the operator. Can be identified.

  The present disclosure can also be realized as a program that causes a computer to execute each characteristic configuration included in such an operation terminal, or a voice input method that is operated by the program. Needless to say, such a program can be distributed via a non-transitory computer-readable recording medium such as a CD-ROM or a communication network such as the Internet.

  Each of the embodiments described below shows a specific example of the present disclosure. Numerical values, shapes, components, steps, order of steps, and the like shown in the following embodiments are merely examples, and do not limit the present disclosure. Further, among the components in the following embodiments, components not described in the independent claims indicating the highest concept are described as arbitrary components. Further, in all the embodiments, the respective contents can be combined.

(Embodiment 1)
FIG. 1 is a diagram illustrating an example of a positional relationship between the operation terminal 1 and the user U1 according to Embodiment 1 of the present disclosure. The operation terminal 1 is, for example, a device that is installed in a building such as a house in which the user U1 lives, collects voice uttered by the user U1, recognizes the voice, and receives an operation from the user U1. The operations accepted by the operation terminal 1 include, for example, operations on electric appliances installed in the building, operations on the operation terminal 1, and the like. The electric appliances include, for example, household electric appliances such as a washing machine, a refrigerator, a microwave oven, and an air conditioner, and AV devices such as a television, an audio device, and a recorder. When operating a certain electric appliance, the user U1 approaches the operation terminal 1 and speaks to operate the electric appliance. Then, the operation terminal 1 recognizes the uttered voice, determines the electric appliance to be operated and the operation content for the electric appliance, and transmits a control command corresponding to the operation content to the electric appliance to be operated. I do. Note that the operation terminal 1 is communicably connected to electric appliances wirelessly or by wire via a network. The network is, for example, a wireless LAN, a wired LAN, or the like. Note that the network may include the Internet.

  FIG. 2 is a diagram illustrating an example of an external configuration of the operation terminal 1. As illustrated in FIG. 2, the operation terminal 1 includes an imaging device 301 (an example of an imaging unit) and a sound collection device 307 (an example of a voice input unit). The imaging device 301 includes a person detection function for detecting that the user U1 is present around the operation terminal 1, a position detection function for detecting the position of the user U1 in the space, and a user U1 as illustrated in FIG. And a skeleton detecting function for detecting the skeleton information 201 of the above. The sound collection device 307 has a function of collecting sound uttered by the user U1 to the operation terminal 1.

  FIG. 3 is a diagram illustrating an example of the skeleton information 201 of the user U1 measured by the imaging device 301. The skeleton information 201 includes part coordinates 202 including three-dimensional coordinates of the plurality of parts of the user U1 in the respective spaces, and links 203 connecting each of the part coordinates 202 along the body of the user U1. The part coordinates include coordinates of joints such as wrists, elbows, and shoulders, and coordinates of body ends such as fingertips, toes, and heads. The part coordinates may include coordinates of a characteristic part of the body other than the center of the chest, the joint such as the navel, and the tip.

  In the example of FIG. 3, in order from the top, the tip of the head, the base of the neck and the face (neck), the base of the neck and the body (neck), the left and right elbows, the left and right wrists, the waist, the left and right knees, and the left and right ankles Are adopted as the part coordinates 202.

  The three-dimensional coordinates indicating the part coordinates 202 are, for example, a rectangular coordinate system set based on the operation terminal 1, a rectangular coordinate system set based on the user U1, a polar coordinate system set based on the operation terminal 1, or It is defined in a polar coordinate system set based on the user U1. However, this is only an example, and the coordinate system defining the three-dimensional coordinates is not limited to these coordinate systems. The link 203 is, for example, a three-dimensional vector connecting the part coordinates 202.

  The configuration of the imaging device 301 is not particularly limited as long as the imaging device 301 has a function of acquiring surrounding spatial information. For example, the imaging device 301 is configured by a sensor that measures spatial information indicating three-dimensional information of the space around the operation terminal 1, such as a visible light camera, an infrared camera, a TOF sensor, an ultrasonic sensor, and a radio wave sensor. You. Note that the imaging device 301 may realize a human detection function and a skeleton information detection function by combining any two or more of a visible light camera, an infrared camera, a TOF sensor, an ultrasonic sensor, and a radio wave sensor.

  The visible light camera is, for example, a color or monochrome camera. The infrared camera measures the reflection time of the emitted infrared light for each of a plurality of pixels. The TOF (Time of Flight) sensor measures the reflection time of the irradiated pulse light for each of a plurality of pixels. The ultrasonic sensor is, for example, an ultrasonic sensor array. The radio wave sensor is, for example, a radio wave sensor array.

  FIG. 4 is a block diagram illustrating an example of a configuration of the operation terminal 1 according to Embodiment 1 of the present disclosure.

  The operation terminal 1 includes a processor 300, an imaging device 301, a sound collection device 307, a sound collection sound recording unit 308, and a memory 309. The processor 300 is configured by an electronic circuit such as a CPU, and includes a human detection unit 302, a skeleton information extraction unit 303, a gesture extraction unit 304, a start condition determination unit 305, and a management unit 306. The memory 309 includes a collected sound recording unit 308.

  The imaging device 301 acquires spatial information at a predetermined frame rate, for example, and outputs the spatial information to the human detecting unit 302 and the skeleton information extracting unit 303. The spatial information is, for example, data in which a plurality of pixel data including RGB color components and depth components are arranged in a matrix. Note that the pixel data forming the spatial information only needs to include at least one of the depth component and the color component, and may include a luminance component instead of the color component.

  The person detection unit 302 acquires space information from the imaging device 301, detects whether or not a user is present in the space around the operation terminal 1, and outputs a detection result indicating the presence or absence of the user as a gesture extraction unit. Output to 304. Here, the human detection unit 302 may detect the user from the spatial information using various human detection techniques. For example, the person detection unit 302 may extract one or more objects from the spatial information, and determine that a user is present if any one of the one or more objects is an object indicating a person.

  The skeleton information extracting unit 303 acquires spatial information from the imaging device 301, extracts the user's skeleton information 201 from the acquired spatial information, and outputs the extracted skeleton information 201 to the gesture extracting unit 304. The skeleton information extracting unit 303 may extract the skeleton information every time the spatial information is obtained, or may use the trigger of the skeleton information extraction request from the gesture extracting unit 304 as described below to trigger the skeleton information of the user. May be extracted. In this case, the skeleton information extraction unit 303 acquires, for example, from the gesture extraction unit 304, an extraction request including the number of users in the spatial information and the area in the spatial information where the user is located. Thereby, the skeleton information extracting unit 303 can extract the skeleton information 201 from the spatial information in the area where the user is located, and can reduce the processing load as compared with the case where the skeleton information 201 is extracted from the entire spatial information. . In addition, when the human detection unit 302 detects a plurality of users, the skeleton information extraction unit 303 may acquire an area where each user is located from the gesture extraction unit 304.

  The skeleton information extracting unit 303 extracts skeleton information in real time using a technique such as skeleton tracking and motion capturing. When a plurality of users exist in the space, the skeleton information extracting unit 303 may extract the skeleton information 201 of each user in real time.

  The gesture extracting unit 304 (an example of a coordinate detecting unit) extracts first coordinates and second coordinates based on the detection result obtained from the human detecting unit 302 and the skeleton information 201 obtained from the skeleton information extracting unit 303. The gesture extraction information including the one coordinate and the second coordinate is output to the start condition determination unit 305.

  For example, when the gesture extraction unit 304 acquires a detection result indicating that a user has been detected from the human detection unit 302, the gesture extraction unit 304 acquires skeleton information from the skeleton information extraction unit 303. Alternatively, when the gesture extraction unit 304 obtains a detection result indicating that a person has been detected from the human detection unit 302, the gesture extraction unit 304 outputs a skeleton information extraction request to the skeleton information extraction unit 303, and thereby the skeleton information extraction unit 303 The skeletal information may be obtained from. In this case, the gesture extracting unit 304 may include the number of users and the area where the user is in the spatial information indicated by the detection result of the human detecting unit 302 in the extraction request and output the request to the skeleton information extracting unit 303.

  The first coordinates are coordinates of a first part constituting the upper limb. The second coordinates are coordinates of a second part constituting the upper body excluding the upper limbs. The upper limb points from the shoulder joint to the fingertip. The lower limb points from the waist to the toes. The upper body points from the waist to the tip of the head. Therefore, the second part is a specific part of the upper body excluding the upper limbs, that is, the torso, the neck, and the face. For example, the second part is a neck, a neck, a head, and the like. The first part is, for example, a wrist, an elbow, and a shoulder.

  In the present embodiment, the gesture extracting unit 304 adopts, for example, any one of the wrist, the elbow, and the shoulder (for example, the wrist) as the first part. In addition, the gesture extracting unit 304 adopts, for example, any one of the neck, the neck, and the head (for example, the neck) as the second part.

  However, this is merely an example, and the gesture extracting unit 304 may employ, for example, two or more of the wrist, elbow, and shoulder as the first part, or, for example, the neck, neck, and head. Any two or more may be adopted as the second part. In this case, the gesture extracting unit 304 may calculate, as the first coordinates, an average value or an added value of all or some of the two or more first parts. Further, in this case, the gesture extracting unit 304 may calculate the average value or the added value of all or some of the two or more second parts as the second coordinates.

  Further, the gesture extracting unit 304 may extract a third part other than the first part and the second part in the upper body. The third part is, for example, a waist, a navel, a chest, or the like. In this case, the gesture extracting unit 304 may include the third coordinates of the third part in the gesture extraction information in addition to the first coordinates and the second coordinates.

  The start condition determining unit 305 compares the positional relationship between the first coordinate and the second coordinate included in the gesture extraction information acquired from the gesture extracting unit 304, and determines that the positional relationship is a start condition (an example of the first condition) of voice input. The result of the determination as to whether or not the condition is satisfied is output to the management unit 306. The start condition is, for example, a condition indicating that the user makes a predetermined gesture indicating an intention to start voice input, such as raising the upper limb. Specifically, assuming that the first coordinate is the wrist and the second coordinate is the neck, a condition that the first coordinate is located above the second coordinate in the vertical direction is an example of the start condition.

  When acquiring the determination result indicating that the start condition is satisfied from the start condition determination unit 305, the management unit 306 sets the state flag stored in the memory 309 and outputs a start instruction to the sound collection device 307. Then, the sound collection device 307 is set to a state in which a voice input can be accepted. On the other hand, when the management unit 306 acquires a determination result indicating that the start condition is not satisfied from the start condition determination unit 305, the management unit 306 outputs an end instruction to the sound collection device 307, and ends the acceptable state. At this time, the management unit 306 manages the end of the receivable state by lowering the state flag stored in the memory 309. As a result, as long as the user makes a gesture that satisfies the start condition, the acceptable state can be continued. It should be noted that raising the status flag refers to setting the status of the status flag to an acceptable state, and lowering the status flag refers to setting the status of the status flag to a non-acceptable state (standby state). .

  The sound collection device 307 includes a microphone that acquires surrounding sounds, and an electric circuit that controls the microphone. When acquiring the start instruction from the management unit 306, the sound collection device 307 causes the microphone to collect surrounding sounds, and records a sound signal indicating the collected sounds in the sound collection sound recording unit 308. As a result, the sound collection device 307 enters a receivable state. On the other hand, when the sound collection device 307 receives an end instruction from the management unit 306, the sound collection device 307 ends sound collection. As a result, the sound collection device 307 enters a standby state.

  The sound collection sound recording unit 308 realizes a sound recording function of the operation terminal 1 by recording a sound signal acquired from the sound collection device 307. The sound collection voice recording unit 308 may be configured with, for example, a nonvolatile memory or a volatile memory.

  FIG. 5 is a flowchart illustrating an example of a process of the start condition determining unit 305 according to Embodiment 1 of the present disclosure.

  In step S401, the start condition determination unit 305 acquires gesture extraction information from the gesture extraction unit 304.

  In step S402, the start condition determination unit 305 determines whether or not a user is within the gesture possible range around the operation terminal 1 based on the gesture extraction information acquired in step S401. If it is determined that there is no user within the gesture possible range (NO in step S402), the start condition determining unit 305 returns the process to step S401, and acquires gesture extraction information. On the other hand, if it is determined that the user is within the gesture possible range (YES in step S402), step S403 is executed. The details of the gesture possible range will be described later with reference to FIG. Here, the start condition determination unit 305 determines that if the coordinates indicating the user's position, such as the first coordinates, the second coordinates, and the third coordinates, included in the gesture extraction information are within the gesture possible range, the user can perform the gesture possible range. , And if the coordinates are not within the gesture possible range, the user may determine that the user is not within the gesture possible range.

  In step S403, the start condition determination unit 305 determines whether the positional relationship between the first coordinate, the second coordinate, and the third coordinate included in the gesture extraction information satisfies the start condition (step S403). If it is determined that the start condition is satisfied (YES in step S403), start condition determination unit 305 advances the process to step S404 because the user has performed a gesture indicating intention to start voice input. On the other hand, when determining that the start condition is not satisfied (NO in step S403), the start condition determining unit 305 returns the process to step S401, and acquires the gesture extraction information.

  In step S404, the start condition determination unit 305 outputs a determination result indicating that the start condition is satisfied to the management unit 306. When step S404 ends, the start condition determining unit 305 returns the process to step S401, and acquires gesture extraction information.

  FIG. 6 is a flowchart illustrating an example of a process of the management unit 306 according to the embodiment of the present disclosure. In step S601, the management unit 306 determines whether the sound collection device 307 is in a state in which a voice input can be accepted. In this case, if the state flag stored in memory 309 is set, management unit 306 determines that the state is acceptable, and if the state flag stored in memory 309 is not set, it is not in the acceptable state. Should be determined.

  If it is determined that it is in the acceptable state (YES in step S601), management unit 306 returns the process to step S601. On the other hand, when determining that the state is not the acceptable state (NO in step S601), management unit 306 advances the process to step S602.

  In step S602, if the management unit 306 acquires a determination result indicating that the start condition is satisfied from the start condition determination unit 305 (YES in step S602), the process proceeds to step S603, and the determination result indicates that the start condition is satisfied. Is not acquired (NO in step S602), the management unit 306 returns the process to step S601.

  In step S603, the management unit 306 outputs a start instruction to the sound collection device 307, thereby setting the sound collection device 307 to an acceptable state and setting a state flag stored in the memory 309. When the processing in step S603 ends, the management unit 306 returns the processing to step S601.

  Next, the start condition will be described. FIG. 7 is a diagram exemplifying the skeleton information 201 of the user U1 for explaining the start condition. FIG. 7 illustrates the first coordinate, the second coordinate, and the third coordinate to be compared with the start condition in step S403. In the example of FIG. 7, the wrist coordinate H is adopted as the first coordinate, the neck coordinate N is adopted as the second coordinate, and the waist coordinate W is adopted as the third coordinate.

  A first example of the start condition is a condition that the wrist coordinates H are larger (higher) than the neck base coordinates N by a first threshold (for example, 20 cm) or more in the vertical direction. The vertical direction is a direction orthogonal to the ground. In this case, the user U1 performs the gesture of raising the upper limb so that the wrist coordinate H is higher than the neck coordinate N in the vertical direction by the first threshold or more, so that the sound collection device 307 can be in the acceptable state. Can be. The upper limb may be the right arm or the left arm.

  A second example of the start condition is a condition that the wrist coordinates H and the neck base coordinates N fall within a predetermined range in the vertical direction. For example, a condition that the wrist coordinate H is located within a predetermined range (for example, about ± 10 cm) in the vertical direction about the neck coordinate N can be adopted as a second example of the start condition. In this case, the user U1 performs a gesture of bending the elbow to raise the wrist coordinate H to near the chest, or a gesture of turning the entire upper limb to the outside of the torso without bending the elbow and raising the wrist coordinate H to near the chest. It can be set in the acceptable state.

  A third example of the start condition is a condition that the wrist coordinate H is larger than the neck coordinate N by a first threshold (for example, 10 cm) or more in the trunk axis direction connecting the neck coordinate N and the waist coordinate W. In this case, the user U1 can perform the gesture of raising the upper limb so that the wrist coordinate H is higher than the neck base coordinate N by the first threshold or more, so that the user U1 can enter the receivable state. In this case, regardless of the current posture in which the user U1 is lying down or standing up, the user U1 can raise the upper limb along the trunk axis direction without being conscious of the vertical direction, so that the reception state can be set. .

  A fourth example of the start condition is a condition that the wrist coordinates H and the neck base coordinates N fall within a predetermined range in the trunk axis direction. For example, a condition that the wrist coordinate H is located within a predetermined range (for example, about plus or minus 10 cm) around the neck coordinate N in the trunk axis direction can be adopted as a fourth example of the start condition. In this case, for example, the user U1 raises the wrist coordinate H to the vicinity of the chest by bending the elbow while lying down, or turns the entire upper limb to the outside of the torso without bending the elbow and raises the wrist coordinate H to the vicinity of the chest. By performing a gesture, the state can be set to be acceptable.

  A fifth example of the start condition is a line segment indicating the upper limb direction connecting the wrist coordinates H and the neck coordinates N and a line segment indicating the trunk axis direction connecting the waist coordinates W and the neck coordinates N. The condition is that the angle formed is equal to or larger than a predetermined second threshold (for example, 100 degrees, 80 degrees, etc.). In this case, regardless of the current posture, such as a standing state or a lying state, the user U1 makes a gesture of raising the hand in the trunk axis direction without being conscious of the vertical direction, thereby setting the state to the receivable state. be able to.

  The sixth example of the start condition is a line segment indicating the upper limb direction connecting the wrist coordinates H and the neck coordinates N and a line segment indicating the trunk axis direction connecting the waist coordinates W and the neck coordinates N. This is a condition that the angle formed falls within a predetermined angle range. The predetermined angle range is, for example, plus or minus 10 degrees or 20 degrees around 100 degrees. In this case, regardless of the current posture such as the standing state and the lying state, the user U1 does not care about the vertical direction so that the angle between the upper limb direction and the trunk axis direction is within a predetermined angle range. By performing a gesture of raising the upper limb, the reception state can be set.

  The start condition may be a condition in which any two or more of the first to sixth examples are combined. For example, as the start condition, a condition that any two or more of the first to sixth examples are satisfied (an example of a third condition) can be adopted. Alternatively, the start condition may be a condition that one or at least two of the first to sixth examples is satisfied (an example of a second condition). Here, the gesture of raising the upper limb is assumed in each of the first to sixth examples, but this is an example. For example, various gestures such as a gesture of lowering the upper limb and a gesture of expanding the left and right upper limb can be adopted as the start condition, and the gesture to be detected is not particularly limited. The gestures for extending the left and right upper limbs include, for example, a gesture for raising the left and right upper limbs, a gesture for lowering the left and right upper limbs, and a gesture for raising one upper limb and lowering the other upper limb.

  Next, an example of the process of step S402 will be described. FIG. 8 is a diagram illustrating an example of the gesture possible range 901. As illustrated in FIG. 8, the gesture possible range 901 is sandwiched between a gesture impossible range 902 and a gesture impossible range 903.

  The start condition determination unit 305 compares the position of the user U1 with the gesture possible range 901 and the gesture impossible range 902, 903, and determines the range in which the gesture of the user U1 is detected based on the distance between the user U1 and the operation terminal 1. Restrict.

  The non-gesture range 902 is a circular or fan-shaped area around the operation terminal 1 with the lower limit value D1 of the gesture possible range 901 as the radius. The gesture possible range 901 is a donut-shaped area obtained by removing the non-gesture-possible range 902 from a circular or fan-shaped area whose radius is the upper limit value D2. The gesture impossible range 903 is an area that is farther from the operation terminal 1 than the upper limit value D2.

  Therefore, the start condition determining unit 305 determines that the position of the user U1 is located in the range from the lower limit D1 to the upper limit D2 with respect to the operation terminal 1, that is, the user U1 is located in the gesture possible range 901 (an example of the fourth condition ), The gesture of the user U1 is detected. On the other hand, the start condition determining unit 305 determines whether the position of the user U1 is less than or equal to the lower limit D1 with respect to the operation terminal 1, or if the position of the user U1 is within a range equal to or more than the upper limit D2 with respect to the operation terminal 1. In this case, the gesture of the user U1 is not detected.

  When the position of the user U1 is too close to the operation terminal 1, there is a possibility that the gesture of the user U1 may not be detected well, and in addition, the operation terminal may be such that the user U1 happens to be performing some work near the operation terminal 1. There is a possibility that there is no intention to operate 1. If the position of the user U1 is too far from the operation terminal 1, there is a high possibility that the user U1 does not intend to operate the operation terminal 1. Therefore, in the present embodiment, the start condition determining unit 305 performs a process of detecting a gesture of the user U1 only when the user U1 is in the gesture possible range 901, that is, a process of determining whether the start condition is satisfied. I decided to do it. Accordingly, it is possible to prevent a decrease in the accuracy of gesture detection, and to prevent a process of detecting a gesture from being activated when the user U1 does not intend to operate the operation terminal 1, thereby reducing a processing load on the operation terminal 1. .

  In the above description, the gesture extraction unit 304 has been described as including one first coordinate and one second coordinate in the gesture extraction information and outputting the information to the start condition determination unit 305, but the present disclosure is not limited to this. The gesture extraction unit 304 may include one or a plurality of first coordinates and one or a plurality of second coordinates in the gesture extraction information and output the information to the start condition determination unit 305.

  For example, when the gesture extraction information includes a plurality of first coordinates and one second coordinate, the start condition determining unit 305 determines the plurality of first coordinates (for example, wrist coordinates H, elbow coordinates, and shoulder coordinates). If at least one of the first coordinates is greater than or equal to one second coordinate (for example, neck coordinate N) in the vertical direction or the trunk axis direction by a first threshold or more, it may be determined that the start condition is satisfied. When the gesture extraction information includes one first coordinate and a plurality of second coordinates, the start condition determining unit 305 determines that one first coordinate (for example, wrist coordinate H) is a plurality of second coordinates (for example, wrist coordinate H). , Torso coordinates, neck coordinates N, coordinates at the tip of the head) in the vertical direction or trunk axis direction with respect to at least one second coordinate of a first threshold or more, if it is determined that the start condition is satisfied. Good.

  FIG. 9 is a diagram illustrating a case where a plurality of users make a gesture on the operation terminal 1. When a plurality of users make gestures on the operation terminal 1 as exemplified by the user U1 and the user U2 in FIG. 9, the gesture extraction unit 304 specifies one operator and performs a gesture for the specified operator. The extraction information may be output to the start condition determination unit 305. In this case, the gesture extraction unit 304 may specify a user located closest to the operation terminal 1 among the plurality of users as the operator.

  In addition, the gesture extraction unit 304 may continue to specify the first detected user as the operator until the first detected user out of the plurality of users exits the gesture possible range 901. For example, if the user U1 enters the gesture possible range 901 first and then the user U2 enters the gesture possible range 901, the gesture extracting unit 304 determines that the user U1 is the operator as long as the user U1 is in the gesture possible range 901. To be specified. Then, when the user U1 exits the gesture possible range 901, the gesture extracting unit 304 specifies the user U2 as the operator if the user U2 exists in the gesture possible range 901. At this time, if there is a user U3 in addition to the user U2 in the gesture possible range 901, the gesture extracting unit 304 specifies a user whose distance to the operation terminal 1 is shorter among the users U2 and U3 as an operator. Is also good.

  However, these are merely examples, and the method of specifying one operator from a plurality of users is not limited to the above-described method.

  Next, a modified example of the first embodiment will be described. A modification of the first embodiment is to output a state notification indicating whether or not the apparatus is in a receivable state.

  FIG. 10 is a diagram illustrating a first example of the status notification. In the first example, the operation terminal 1 includes a display 501 on the front surface, and is attached to, for example, a wall in a room. The imaging device 301 is provided, for example, above the outer frame of the display 501. Two sound collection devices 307 are provided on both sides of the imaging device 301 above the outer frame of the display 501. These are the same in FIGS. 11 and 12.

  In the first example of the status notification, the display 501 displays the status notification by the text 502. In this example, since the sound collection device 307 is in a state where voice input can be accepted, “accepting voice input” is employed as the text 502. Thereby, the user who has performed the gesture can recognize that the operation terminal 1 is in a state in which voice input can be accepted. When the receivable state ends, the display 501 may hide the text 502 or display the text 502 indicating that the operation terminal 1 is not in the receivable state, such as “waiting for voice input”. May be. Note that the text 502 illustrated in FIG. 10 is an example, and other messages may be employed as long as the user can recognize that the operation terminal 1 is in a receivable state. The locations and numbers of the imaging devices 301 and the sound pickup devices 307 shown in FIG. 10 are only examples. This is the same in FIGS. 11 and 12.

  FIG. 11 is a diagram illustrating a second example of the status notification. In the second example of the status notification, the display 501 displays the status notification by the icon 503. In this example, since the sound collection device 307 is in a state in which voice input can be received, an icon simulating a microphone is used as the icon 503. Thereby, the user who has performed the gesture can recognize that the operation terminal 1 is in a state in which voice input can be accepted. When the receivable state has ended, the display 501 may hide the icon 503 or display an icon indicating that the voice input is in a standby state. Alternatively, the display 501 displays the icon 503 in a predetermined first color when in the acceptable state, and displays the icon 503 in a predetermined second color different from the first color when the voice input is in a standby state. It may be displayed. Note that the icon 503 illustrated in FIG. 10 is an example, and other icons may be employed as long as the user can recognize that the user is in the receivable state.

  FIG. 12 is a diagram illustrating a third example of the status notification. In the third example of the status notification, the display 501 displays the status notification using the color 504 of the entire display area. The color 504 is a color of a background displayed on the entire display area. In this example, since the sound collection device 307 is in a state in which a voice input can be accepted, a first color (for example, red, blue, yellow, or the like) indicating that the sound collection device 307 is in the acceptable state is adopted as the color 504. Thereby, the user who has performed the gesture can recognize that the operation terminal 1 is in a state in which voice input can be accepted. When the receivable state ends, the display 501 may display a second color different from the first color indicating the standby state. As the second color, for example, a default background color displayed on the display 501 can be adopted, for example, white, black, or the like. Note that the color 504 illustrated in FIG. 12 is an example, and any color may be adopted as long as the user can recognize the receivable state.

  FIG. 13 is a diagram illustrating a fourth example of the status notification. In the fourth example, the operation terminal 1 has, for example, two sound collection devices 307 and, for example, one imaging device 301 on the front surface, and has, for example, four light emitting devices 505 on the upper surface. In the fourth example, the operation terminal 1 is constituted by a stationary device placed on a desk or a floor such as a smart speaker, for example. The light emitting device 505 is, for example, a light emitting diode.

  In the fourth example, a state notification is displayed by the light emitting device 505. For example, when the operation terminal 1 is in a receivable state, the light emitting device 505 emits light. On the other hand, when the operation terminal 1 is in the standby state, the light emitting device 505 is turned off. Thereby, the user who has performed the gesture can recognize that it is in the acceptable state. However, this is merely an example, and any display mode of the light emitting device 505 may be adopted as long as the user can recognize the receivable state. For example, examples of the display mode of the light emitting device 505 in the receivable state include a mode in which the light emitting device 505 is constantly turned on, a mode in which the light emitting device 505 blinks, and a mode in which the color of emitted light is changed over time. In addition, as a display mode of the light emitting device 505, for example, a mode in which the light is constantly lit, and a mode in which the light is flashed when a receivable state is set, or a reverse mode may be adopted. Alternatively, as a display mode of the light-emitting device 505, for example, a mode in which the type of a light-emitting color is changed between when in a receivable state and when in a standby state may be adopted.

  In FIG. 13, the number of the light emitting devices 505 is four, but this is an example, and three or less and five or more may be used. Further, the light emitting device 505 is disposed on the upper surface, but this is also merely an example, and may be disposed on the front surface, the side surface, the rear surface, or the like. Furthermore, the number and arrangement of the imaging devices 301 and the sound collection devices 307 are not particularly limited.

  FIG. 14 is a diagram illustrating a fifth example of the status notification. The operation terminal 1 of the fifth example is further provided with a speaker 506 on the front side of the operation terminal 1 of the fourth example. In the fifth example, a state notification is output by sound output from the speaker 506. In FIG. 13, the number and arrangement of the speakers 506 are not particularly limited as long as the user can recognize that the speaker is in the acceptable state. In the fifth example, when the speaker 506 is in the receivable state, the speaker 506 may output a voice message indicating that the state is receivable, such as “Please perform voice input”. Alternatively, when the speaker 506 is in the receivable state, the speaker 506 may output a sound effect or may output a beep sound. As described above, the output pattern of the sound from the speaker 506 is not limited to a specific pattern. Note that the speaker 506 may stop outputting sound when in the standby state.

  The configuration of the operation terminal 1 for notifying the user whether or not it is in the receivable state illustrated in FIGS. 10 to 14, that is, a display device such as the display 501 and the light emitting device 505 and a reproducing device such as the speaker 506 And may be arbitrarily combined. For example, the operation terminal 1 may be configured by arbitrarily combining one or more types of display devices and one or more types of playback devices.

  FIG. 15 is a block diagram of the operation terminal 1 in a case where the display device 602 and the reproduction device 603 illustrated in FIGS. 10 to 14 are added to the block diagram of the operation terminal 1 illustrated in FIG.

  The operation terminal 1 illustrated in FIG. 15 further includes a playback device 603 and a display device 602 as compared to FIG. In FIG. 15, the operation terminal 1 may include at least one of the playback device 603 and the display device 602.

  In FIG. 15, the same components as those in FIG. 4 are denoted by the same reference numerals, and description thereof will be omitted. However, in FIG. 15, a reference numeral 601 is assigned to the management unit instead of the reference numeral 302.

  The start condition determination unit 305 determines whether the positional relationship among the first coordinate, the second coordinate, and the third coordinate included in the gesture extraction information acquired from the gesture extraction unit 304 satisfies the start condition, as in FIG. Then, the determination result is output to the management unit 601. The details of this processing are the same as the flow illustrated in FIG. However, in step S404, the determination result is output to the management unit 601 instead of the management unit 306.

  The management unit 601 has the following functions in addition to the functions of the management unit 306. That is, when the management unit 601 obtains the determination result indicating that the start condition is satisfied from the start condition determination unit 305, the management unit 601 outputs the output command of the status notification illustrated in FIGS. 10 to 14 to the playback device 603 and the display device 602. .

  When acquiring the start instruction from the management unit 601, the sound collection device 307 causes the microphone to collect surrounding sounds, and records a sound signal indicating the collected sound in the sound collection sound recording unit 308.

  The playback device 603 includes the speaker 506 illustrated in FIG. 14, a playback circuit that plays back a playback sound, and the like. When an output command for status notification is acquired from the management unit 306, the playback device 603 reads a predetermined playback sound from the memory 309 and plays it. . Here, the reproduction sound reproduced from the speaker 506 is a sound effect, a beep sound, a voice message, or the like illustrated in FIG. As a result, the user is notified of the status notification through hearing.

  The display device 602 is configured by at least one of the display 501 illustrated in FIGS. 10 to 14 and the light emitting device 505 illustrated in FIG. 13, and when a status notification output command is acquired from the management unit 601, the display device 602 illustrated in FIGS. The status notification illustrated in the example is output. As a result, the user is notified visually of the status notification by a message, a color, an icon, or the like.

  As described above, according to the present embodiment, a state in which a simple gesture such as spreading both hands with the hand raised to the operation terminal 1 is not a troublesome gesture such as turning the upper limb to a specific position in the space. It becomes possible to.

(Embodiment 2)
In the first embodiment, the mode in which the user starts the receivable state by gesture is mainly exemplified. Embodiment 2 shows the details of the mode of Embodiment 1 in which the sound collection device 307 ends the receivable state.

  FIG. 16 is a block diagram illustrating an example of a configuration of the operation terminal 1 according to Embodiment 2. In the present embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted.

  The operation terminal 1 of FIG. 16 further includes a timeout determination unit 702, an end condition determination unit 703, and a silent section detection unit 705 in addition to the configuration of FIG. Also, the gesture extraction unit, the start condition determination unit, and the management unit are provided with reference numerals 700, 701, and 704 because the functions are added to the first embodiment. In FIG. 16, the operation terminal 1 does not need to have the playback device 603 and the display device 602.

  The gesture extracting unit 700 extracts the first coordinate, the second coordinate, and the third coordinate based on the detection result obtained from the human detecting unit 302 and the skeleton information obtained from the skeleton information extracting unit 303. The gesture extraction information including the two coordinates and the third coordinates is output to the start condition determination unit 701 and further to the end condition determination unit 703. The details of the processing of the gesture extracting unit 700 are the same as those of the first embodiment.

  The start condition determining unit 701 determines whether the positional relationship between the first coordinate, the second coordinate, and the third coordinate included in the gesture extraction information acquired from the gesture extracting unit 700 satisfies the start condition, and determines that the condition is satisfied. In this case, a determination result indicating that the start condition is satisfied is output to the management unit 704 and the timeout determination unit 702. In the second embodiment, the details of the process of determining whether the positional relationship satisfies the start condition are the same as those in the flow of FIG.

  The timeout determination unit 702 starts counting down for a predetermined timeout period (for example, 10 seconds) when acquiring the determination result indicating that the start condition is satisfied from the start condition determination unit 701, and when the countdown is completed, A determination result indicating that the period has elapsed is output to the management unit 704. However, when the timeout determination unit 702 obtains a determination result indicating that the start condition is satisfied from the start condition determination unit 701 during the countdown, that is, within the timeout period, the timeout period is initialized, and the timeout period starts from the beginning. Perform a countdown. Thereby, even if the user fades out the gesture possible range 901 without performing the gesture indicating the intention to end the voice input in the acceptable state, it is possible to prevent the acceptable state from continuing. Therefore, if the user forgets to perform a gesture indicating the intention to end the voice input, it is possible to prevent the receivable state from continuing. As a result, the privacy of the user can be secured.

  The termination condition determination unit 703 determines whether the positional relationship between the first coordinate, the second coordinate, and the third coordinate included in the gesture extraction information acquired from the gesture extraction unit 700 satisfies the termination condition (an example of a fifth condition). Is determined, the determination result indicating that the termination condition is satisfied is output to the management unit 704. Details of the processing of the end condition determination unit 703 will be described later using the flow of FIG.

  When the management unit 704 obtains a determination result indicating that the start condition is satisfied from the start condition determination unit 701 in the standby state, the management unit 704 sets the state flag stored in the memory 309 and sets the sound collection device 307. By outputting a start instruction to the sound collecting device 307, the sound collecting device 307 is set in a receivable state.

  When the management unit 704 obtains a determination result indicating that the timeout period has elapsed from the timeout determination unit 702 in the acceptable state, it lowers the status flag stored in the memory 309 and collects the termination instruction. By outputting to the sound device 307, the receivable state is ended and the apparatus enters the standby state.

  When the management unit 704 obtains a determination result indicating that the termination condition has been satisfied from the termination condition determination unit 703 in the acceptable state, the management unit 704 lowers the status flag stored in the memory 309 and terminates the process. By outputting the instruction to the sound collection device 307, the receivable state is ended and the standby state is set. Thus, the acceptable state can be ended by the user making a gesture indicating the intention to end the voice input. In the first embodiment, when the management unit 306 obtains a determination result indicating that the start condition is not satisfied from the start condition determination unit 305, the management unit 306 ends the receivable state. However, in the second embodiment, Basically, when the management unit 704 obtains a determination result indicating that the termination condition has been satisfied from the termination condition determination unit 703, the management unit 704 ends the receivable state.

  When the management unit 704 obtains a determination result indicating that the timeout period has elapsed from the timeout determination unit 702 in the acceptable state, the silent section detection unit 705 further detects a sound section. If so, the receivable state may be continued. Thus, it is possible to avoid a situation in which the receivable state automatically ends on condition that the timeout period has elapsed, even though the user is making an utterance for operating the operation terminal 1.

  On the other hand, when the management unit 704 obtains a determination result indicating that the timeout period has elapsed from the timeout determination unit 702 in the acceptable state, the silent unit detection unit 705 may further detect a silent period. If so, the acceptable state ends.

  The silent section detecting section 705 detects whether or not the latest audio signal recorded in the collected voice recording section 308 includes a silent section. Here, the silence section detection unit 705 may determine that there is a silence section in the audio signal when the time during which the input level is equal to or less than the predetermined threshold continues for a predetermined time (for example, 300 milliseconds). The section detection method is not limited to a specific method. When detecting a silent section, the silent section detecting section 705 sets the current sound collection state to silent and outputs the state to the managing section 704. On the other hand, when the silent section detecting section 705 detects a sound section, the silent section detecting section 705 sets the current sound collecting state to sound and outputs the state to the managing section 704.

  In Embodiment 2, the gesture extraction unit 700, the start condition determination unit 701, the timeout determination unit 702, the end condition determination unit 703, and the management unit 704 correspond to an example of a condition determination unit.

  In FIG. 16, the operation terminal 1 does not need to include all of the timeout determination unit 702, the end condition determination unit 703, and the silent section detection unit 705, and may include at least one.

  FIG. 17 is a flowchart illustrating an example of a process of the termination condition determination unit 703 according to Embodiment 2 of the present disclosure.

  In step S801, the termination condition determination unit 703 acquires gesture extraction information from the gesture extraction unit 700.

  In step S802, the termination condition determination unit 703 determines whether or not the user is within the gesture possible range 901 around the operation terminal 1, based on the gesture extraction information acquired in step S801. If it is determined that there is no user within the gesture possible range 901 (NO in step S802), the process returns to step S801, and gesture extraction information is obtained. On the other hand, when the end condition determination unit 703 determines that the user is within the gesture possible range (YES in step S802), the end condition determination unit 703 executes step S803. Here, if the coordinates indicating the user's position, such as the first coordinates and the second coordinates, included in the gesture extraction information are within the gesture possible range 901, the end condition determination unit 703 determines that the user is within the gesture possible range 901. If the coordinates are not within the gesture possible range 901, the user may determine that the user is not within the gesture possible range 901.

  In step S803, the termination condition determination unit 703 determines whether the positional relationship among the first coordinate, the second coordinate, and the third coordinate included in the gesture extraction information satisfies a predetermined termination condition (an example of a fifth condition). judge. Then, when it is determined that the positional relationship satisfies the termination condition (YES in step S803), the termination condition determination unit 703 moves the process to step S804 because the user has performed a gesture indicating intention to terminate the voice input. Proceed. On the other hand, if it is determined that the positional relationship does not satisfy the termination condition (NO in step S803), the termination condition determination unit 703 returns the process to step S801, and acquires gesture extraction information.

  In step S804, the termination condition determination unit 703 outputs a determination result indicating that the termination condition is satisfied to the management unit 704. When step S804 ends, the termination condition determination unit 703 returns the process to step S801, and acquires gesture extraction information.

  Next, the termination condition will be described. FIG. 18 is a diagram exemplifying the skeleton information 201 of the user U1 for explaining the termination condition. FIG. 18 illustrates the first coordinate, the second coordinate, and the third coordinate to be compared with the start condition in step S803. In the example of FIG. 18, the wrist coordinate H is used as the first coordinate, the neck coordinate N is used as the second coordinate, and the waist coordinate W is used as the third coordinate.

  The first example of the end condition corresponds to the first example of the start condition, and the wrist coordinate H is smaller (lower) than the neck coordinate N by a first threshold (for example, 20 cm) or more in the vertical direction. Condition. In this case, the user U1 can end the receivable state by performing a gesture of lowering the upper limb so that the wrist coordinates H are lower than the neck base coordinates N by a first threshold or more in the vertical direction. The upper limb may be the right arm or the left arm.

  The second example of the end condition corresponds to the second example of the start condition, and is a condition that the wrist coordinates H and the neck coordinates N do not fall within a predetermined range in the vertical direction. In this case, the user U1 can end the acceptable state by performing a gesture of lowering or raising the wrist coordinate H raised near the chest to a position outside the threshold range in the vertical direction.

  The third example of the end condition corresponds to the third example of the start condition. In the trunk axis direction connecting the neck coordinate N and the waist coordinate W, the wrist coordinate H is more than the neck coordinate N. The condition is that the value is smaller than one threshold value. In this case, the user U1 can end the receivable state by performing a gesture of lowering the upper limb so that the wrist coordinate H is lower than the neck coordinate N in the trunk axis direction by the first threshold or more.

  The fourth example of the end condition corresponds to the fourth example of the start condition, and is a condition that the wrist coordinates H and the neck base coordinates N do not fall within a predetermined range in the trunk axis direction. In this case, the user U1 can end the receivable state by performing a gesture of raising or lowering the wrist coordinate H raised near the chest to a position outside the threshold range in the trunk axis direction.

  The fifth example of the end condition corresponds to the fifth example of the start condition, and connects a line indicating the upper limb direction connecting the wrist coordinates H and the neck coordinates N to the waist coordinate W and the neck coordinates N. The condition is that the angle formed by the line indicating the trunk axis direction is smaller than a predetermined second threshold value (100 degrees, 80 degrees, etc.). In this case, regardless of the current posture of the user U1 such as standing or lying down, the user U1 is ready to accept by performing a gesture of lowering the hand in the trunk axis direction without being aware of the vertical direction. Can be terminated.

  The sixth example of the end condition corresponds to the sixth example of the start condition, and connects the line indicating the upper limb direction connecting the wrist coordinate H and the neck coordinate N to the waist coordinate W and the neck coordinate N. This is a condition that the angle formed by the line segment indicating the trunk axis direction is within a predetermined angle range. In this case, regardless of the current posture of the user U1 such as standing or lying down, the user U1 is ready to accept by performing a gesture of lowering the hand in the trunk axis direction without being aware of the vertical direction. Can be terminated.

  The termination condition may be a condition combining any two or more of the first to sixth examples. For example, as the end condition, a condition that any two or more conditions among the first to sixth examples are satisfied can be adopted. Alternatively, the termination condition may be a condition that any one of the first to sixth examples is satisfied. Here, a gesture of lowering the upper limb is assumed in each of the first to sixth examples of the end condition, but this is an example. For example, if a gesture of lowering the upper limb or lowering both hands is adopted as the start condition, a gesture of raising the upper limb or raising both hands may be adopted as the end condition. That is, any condition may be adopted as the end condition as long as the condition that the end condition does not overlap with the start condition is satisfied.

  Next, an example of the process of step S802 will be described with reference to FIG. The end condition determination unit 703 determines that the position of the user U1 is located within the range from the lower limit value D1 to the upper limit value D2 with respect to the operation terminal 1 as in the start condition determination unit 701. If so, the gesture is detected. On the other hand, the termination condition determination unit 703 determines that the position of the user U1 is located below the lower limit value D1 with respect to the operation terminal, or that the position of the user U1 is located within the range above the upper limit value D2 with respect to the operation terminal 1. In this case, the gesture of the user U1 is not detected.

  When the position of the user U1 is too close to the operation terminal 1, there is a possibility that the gesture of the user U1 may not be detected well, and in addition, the user U1 may perform the gesture indicating that the intention to end the voice input is forgotten. The range 901 may have faded out. Further, when the position of the user U1 is too far from the operation terminal 1, the user U1 may have forgotten to perform a gesture indicating an intention to end the voice input and faded out the gesture possible range 901. Therefore, in the present embodiment, when the user U1 is in the gesture possible range 901, the end condition determination unit 703 performs a process of detecting the gesture of the user U1, that is, a process of determining whether the end condition is satisfied. I decided to do it. As a result, it is possible to prevent a decrease in gesture detection accuracy, and to prevent a state in which the accepting state is continued when the user U1 forgets to perform a gesture indicating an intention to end the voice input.

  When one or a plurality of first coordinates and one or a plurality of second coordinates are included in the gesture extraction information, the end condition determining unit 703 uses these coordinates similarly to the start condition determining unit 701. Thus, the termination condition may be determined.

  For example, when the gesture extraction information includes a plurality of first coordinates and one second coordinate, the end condition determining unit 703 determines the plurality of first coordinates (for example, wrist coordinates H, elbow coordinates, and shoulder coordinates). If at least one of the first coordinates is smaller than or equal to one second coordinate (for example, neck coordinate N) in the vertical direction or the trunk axis direction by a first threshold or more, it is determined that the end condition is satisfied. Good. When the gesture extraction information includes one first coordinate and a plurality of second coordinates, the termination condition determination unit 703 determines that one first coordinate (for example, wrist coordinate H) is a plurality of second coordinates (for example, wrist coordinate H). , Torso coordinates, neck coordinates N, coordinates of the tip of the head) in the vertical direction or the trunk axis direction with respect to at least one second coordinate, which is smaller than or equal to the first threshold value, may be determined to satisfy the termination condition. . However, the start condition and the end condition must not overlap.

  Next, details of the processing for extending the timeout period will be described. FIG. 19 is a flowchart illustrating an example of a process of the timeout determination unit 702 according to Embodiment 2 of the present disclosure.

  In step S1601, the timeout determination unit 702 determines whether the timeout period is counting down. If the timer is counting down (YES in step S1601), the timeout determination unit 702 advances the process to step S1602. If the timer is not counting down (NO in step S1601), the process returns to step S1601.

  In step S1602, the timeout determination unit 702 determines whether a determination result indicating that the start condition is satisfied is obtained from the start condition determination unit 701. If this determination result has been obtained (YES in step S1602), timeout determination unit 702 proceeds with the process to step S1603, and if this determination result has not been obtained (NO in step S1602), returns the process to step S1601.

  In step S1603, the timeout determination unit 702 extends the timeout period by returning the timeout period to the initial value and starting the countdown again. When the process in step S1603 ends, the process returns to step S1601.

  As described above, as long as the user performs the gesture indicating the intention of voice input within the gesture possible range 901, the timeout period is extended, and it is possible to avoid a situation in which the utterance for operating the operation terminal 1 is not collected.

  Next, the processing of the management unit 704 will be described. FIG. 20 is a flowchart illustrating an example of a process when the management unit 704 according to Embodiment 2 of the present disclosure terminates the receivable state. Note that the processing when the management unit 704 starts the receivable state is the same as that in FIG. 6, and thus the description is omitted here. In step S1701, the management unit 704 refers to the status flag stored in the memory 309 to determine whether the status is the acceptable status. If it is in the receivable state (YES in step S1701), the process proceeds to step S1702. If it is not in the receivable state (NO in step S1701), the process returns to step S1701.

  In step S1702, the management unit 704 determines whether the determination result indicating that the termination condition is satisfied has been obtained from the termination condition determination unit 703. If this determination result has been obtained (YES in step S1702), the process proceeds to step S1705. If this determination result has not been obtained (NO in step S1702), the process proceeds to step S1703.

  In step S1703, the management unit 704 determines whether a determination result indicating that the timeout period has elapsed has been acquired from the timeout determination unit 702. If this determination result has been obtained (YES in step S1703), the process proceeds to step S1704. If this determination result has not been obtained (NO in step S1703), the process returns to step S1701.

  In step S1704, the management unit 704 determines whether the sound collection state output from the silence section detection unit 705 indicates that there is no sound. If the sound pickup state indicates that there is no sound (YES in step S1704), the process proceeds to step S1705. If the sound pickup state indicates that there is sound (NO in step S1704), the process returns to step S1701. . Thereby, if a silent section is detected after the elapse of the timeout period, the receivable state is ended, and if a sound section is detected, the receivable state is continued.

  In step S1705, the management unit 704 ends the receivable state, and returns the process to step S1701.

  As described above, according to the second embodiment, the user causes the operation terminal 1 to start the receivable state by a simple gesture such as raising the hand, and to accept the operation terminal 1 by a simple gesture such as lowering the hand. The enabled state can be ended.

  Next, a modification of the second embodiment will be described. In the second embodiment, as in the first embodiment, when a plurality of users make gestures on the operation terminal 1 as shown in FIG. 9, the gesture extraction unit 700 specifies one operator. May be. In this case, the gesture extracting unit 700 may specify the nearest user as the operator as in the first embodiment, or may identify the user until the first detected user exits the gesture possible range 901. You may continue to specify the operator.

  In the example of FIG. 16, the operation terminal 1 includes the imaging device 301, the sound collection device 307, the playback device 603, and the display device 602, but the present disclosure is not limited to this. For example, as illustrated in FIG. 21, the imaging device 301, the playback device 603, and the display device 602 may be configured as devices different from the operation terminal 1.

  FIG. 21 is a diagram illustrating an example of a configuration in a case where the imaging device 301, the playback device 603, and the display device 602 are configured by devices different from the operation terminal 1. In FIG. 21, an imaging device 301, a playback device 603, and a display device 602 are communicably connected to each other via a network such as a LAN.

  In the example of FIG. 21, a gesture performed by the user U1 on the operation terminal 1 having the sound pickup device 307 is imaged by the imaging device 301 separate from the operation terminal 1, and the imaging device 301 is in the obtained space. A gesture indicating intention to start voice input or a gesture indicating intention to end voice input is detected from the information, and the detection result is transmitted to the operation terminal 1. The operation terminal 1 sets the sound collection device 307 to a receivable state or a standby state according to a detection result by the imaging device 301. Then, the operation terminal 1 transmits the status notification to the display device 602 and the playback device 603, and causes the display device 602 and the playback device 603 to output the status notification as illustrated in FIGS.

  In FIG. 21, the operation terminal 1, the imaging device 301, the display device 602, and the playback device 603 may be each configured by a plurality of devices, or may be integrally configured by one device. . In addition, in the example of FIG. 21, the sound collection device 307 is provided in the operation terminal 1, but may be configured as a device separate from the operation terminal 1.

  Further, in the second embodiment, when a silent section is detected in the receivable state, the management unit 704 indicates the intention to end the voice input regardless of whether the user is in the gesture possible range 901. The receivable state may be ended regardless of whether a gesture is made or whether a timeout period elapses.

  The technology implemented by the present disclosure to determine the start and end of the voice input of the operation terminal by the gesture of the user eliminates the need for the user to memorize a strict gesture, and realizes the simple start and end of the voice input by the gesture. It is useful as a technique.

1: operation terminal 300: processor 301: imaging device 302: human detection unit 303: skeleton information extraction unit 304: gesture extraction unit 305: start condition determination unit 306: management unit 307: sound collection device 308: sound collection voice recording unit 309 : Memory 501: Display 502: Text 503: Icon 504: Color 505: Light emitting device 506: Speaker 601: Management unit 602: Display device 603: Playback device 700: Gesture extraction unit 701: Start condition judgment unit 702: Timeout judgment unit 703 : End condition determination unit 704: Management unit 705: Silence section detection unit

Claims (26)

An operation terminal operated by a user's uttered voice,
An imaging unit for imaging a space;
From the information of the imaged space, a person detection unit that detects the user,
A voice input unit that receives an input of an uttered voice by the user;
When the user is detected by the human detection unit, the first coordinate of a predetermined first portion included in the upper limb of the user based on information obtained by predetermined means and included in the upper body excluding the upper limb of the user A coordinate detection unit that detects a second coordinate of a predetermined second part to be
Comparing the positional relationship between the first coordinates and the second coordinates, and, at least once, when the positional relationship satisfies a predetermined first condition, a condition determination that sets the voice input unit to a state in which a voice input can be accepted. And a part,
Operation terminal. A skeleton information extraction unit configured to extract skeleton information of the user from the information of the space;
The information obtained by the predetermined means is the skeleton information,
The operation terminal according to claim 1.   The operation terminal according to claim 1, wherein the imaging unit is a visible light camera, an infrared camera, a TOF sensor, an ultrasonic sensor, or a radio wave sensor. The positional relationship is a positional relationship between the first coordinates and the second coordinates in the vertical direction,
The operation terminal according to claim 1. The positional relationship is a positional relationship between the first coordinates and the second coordinates in the trunk axis direction of the user,
The operation terminal according to claim 1. The coordinate detection unit further detects a third coordinate of a third part in the upper body,
The first condition is that the angle formed by the first coordinate, the second coordinate, and the third coordinate exceeds a predetermined threshold, is lower than the predetermined threshold, or falls within a predetermined range.
The operation terminal according to claim 1. The first portion includes a plurality of portions included in the upper limb,
The first coordinates are determined based on coordinates of any one or more of the plurality of parts,
The operation terminal according to claim 1. The second part includes a plurality of parts included in the upper body excluding the upper limb,
The second coordinates are determined based on any one or more coordinates of the plurality of parts,
The operation terminal according to claim 1. The first condition includes a plurality of second conditions,
The condition determination unit sets the receivable state when the positional relationship satisfies at least one of the plurality of second conditions, or a third condition obtained by combining a part of the plurality of second conditions.
An operation terminal according to any one of claims 1 to 8. The audio input unit further includes a display unit or a reproduction unit that outputs information indicating whether or not in the acceptable state,
The operation terminal according to claim 1. The display unit is a display,
The operation terminal according to claim 10.   The operation terminal according to claim 11, wherein the information indicating whether the voice input unit is in the receivable state is a color, a text, or an icon. The display unit is a light emitting device that emits light indicating that the voice input unit is in the acceptable state,
The operation terminal according to claim 10. The playback unit outputs a sound indicating whether the sound input unit is in the receivable state,
The operation terminal according to claim 10. The playback unit outputs a sound indicating whether the voice input unit is in the receivable state,
The operation terminal according to claim 10. The condition determination unit compares the positional relationship only when the distance between the operation terminal and the user satisfies a predetermined fourth condition,
The operation terminal according to claim 1. The condition determination unit terminates the receivable state when a silent section continues for a predetermined time in the receivable state,
An operation terminal according to any one of claims 1 to 16. The condition determination unit continues the receivable state as long as the positional relationship satisfies the first condition in the receivable state,
An operation terminal according to any one of claims 1 to 17. The condition determination unit, when the state in which the positional relationship does not satisfy the first condition in the receivable state continues for a predetermined timeout period, ends the receivable state,
An operation terminal according to any one of claims 1 to 18. The condition determination unit, in the timeout period, if it is determined that the positional relationship satisfies the first condition, extend the timeout period,
The operation terminal according to claim 19. The condition determination unit continues the receivable state if a voice input is detected at the end of the timeout period,
The operation terminal according to claim 18. The condition determination unit ends the receivable state when the positional relationship satisfies a predetermined fifth condition different from the first condition,
The operation terminal according to claim 1. The condition determination unit, when the human detection unit detects a plurality of users, recognizes a specific one as an operator of the operation terminal,
An operation terminal according to claim 1. The operator is a user closest to the operation terminal among the plurality of users,
The operation terminal according to claim 23. A voice input method in an operation terminal operated by a user's uttered voice,
Obtain information on the space imaged by the imaging device,
Detecting the user from the information of the space,
When the user is detected, first coordinates of a predetermined first portion included in the upper limb of the user based on information obtained by predetermined means and predetermined second coordinates included in the upper body excluding the upper limb of the user Detect the second coordinate of the part and
Comparing the positional relationship between the first coordinates and the second coordinates, at least once, when the positional relationship satisfies a predetermined first condition, the audio input unit is set to a state in which a voice input can be accepted.
Voice input method.   A program for causing a computer to execute the voice input method according to claim 25.

Images