JP2018036902A - Equipment operation system, equipment operation method, and equipment operation program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an equipment operation system for preventing an erroneous input due to an input whose operation is not intended from being performed to operation target equipment to be operated on the basis of a gesture input and a voice input.SOLUTION: An equipment operation system for operating equipment on the basis of a gesture input and a voice input comprises: operation intention confirmation means for confirming the presence/absence of an operation intention without executing the operation of the equipment by the gesture input or the voice input; and operation instruction means for instructing the operation of the equipment by at least one of the gesture input and the voice input. The operation intention confirmation means is characterized to, when it is determined that a specific gesture indicating the operation intention has been detected as the gesture input, and it is detected that a specific voice keyword indicating the operation intention has been detected as the voice input, determine that the operation intention is present, and to proceed to processing by the operation instruction means.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a device operation system, a device operation method, and a device operation program.

  2. Description of the Related Art Conventionally, as a user interface for input, a gesture input device using a gesture recognition technology that acquires user gestures and imitations as an image from an imaging device such as a camera and recognizes it by image processing is known. For example, the display content on the display device provided in front of the driver's seat of the vehicle is changed by a gesture operation (Patent Document 1), or the viewpoint switching in HMD (Head Mounted Display) is performed by a gesture input (Patent Document 2). It has been known. In the HMD of Patent Document 2, the display can be switched using a plurality of input means such as button / switch input, line-of-sight input, voice input, myoelectric input, and electroencephalogram input in addition to gestures.

  The operation of a device by a plurality of input means including gesture input is also described in Patent Document 3 and Patent Document 4. The multimodal input / user interface of Patent Document 3 describes that a speech recognition result and other input results including a gesture input are integrated to output a highly likely interpretation result. Patent Literature 4 recognizes semantic information meaning an input operation for each type of input information from two or more different types of input information, and combines the two or more semantic information to perform a predetermined operation on the target device. It is described to be executed.

JP 2014-8818 A JP 2014-115457 A JP 2006-48628 A JP 2012-103840 A

  However, the conventional technique has a problem that malfunction occurs due to, for example, an unintended gesture input. When the operator performs some operation unconsciously, it is not easy for the system to determine whether the operation is an operation intended by the operator as an input operation or an operation not intended for the operation. For example, there is a possibility that the signage erroneously detects the movement of a person who simply crosses in front of a signage (signage: digital signage) as an input operation, and the signage malfunctions.

  The present invention has been made in view of the above-described problems, and it is an object of the present invention to perform an erroneous input due to an operation or speech that is not intended for an operation target device that can be operated based on gesture input and voice input. An object of the present invention is to provide a device operation system, a device operation method, and a device operation program that can prevent the occurrence of the problem.

  An invention described in an embodiment for solving the above-described problem is a device operation system that operates a device based on gesture input and voice input, and executes the operation of the device by gesture input and voice input. Without the operation intention confirmation means for confirming the presence or absence of the operation intention, and the operation instruction means for instructing the operation of the device by at least one of the gesture input and the voice input, the operation intention confirmation means When it is determined that a specific gesture indicating intention is detected as the gesture input and a specific voice keyword indicating intention to operate is detected as the voice input, it is determined that the operation intention is present, and the operation instruction It is an apparatus operation system which shifts to processing by means.

  The invention described in another embodiment is a device operation method for operating a device based on gesture input and voice input, and is a device operation method for operating a device based on gesture input and voice input. , An operation intention confirmation stage for confirming the presence or absence of an operation intention without executing the operation of the device by gesture input and voice input, and an operation for instructing operation of the device by at least one of gesture input and voice input An instruction stage, and when it is determined that the specific gesture indicating the operation intention is detected as the gesture input in the operation intention confirmation stage, and the specific voice keyword indicating the operation intention is detected as the voice input, The device operating method is characterized in that it is determined that there is an intention to operate, and the operation instruction stage is entered.

  The invention described in yet another embodiment is a program that causes a computer to execute a device operation method for operating a device based on gesture input and voice input. The device operation method is based on gesture input and voice input. An operation intention confirmation step for confirming the presence or absence of an operation intention without performing an operation of the device; and an operation instruction step for instructing an operation of the device by at least one of a gesture input and a voice input, In the confirmation stage, when it is determined that a specific gesture indicating an operation intention is detected as the gesture input, and a specific voice keyword indicating the operation intention is detected as the voice input, it is determined that the operation intention is present. Then, the apparatus operation program is shifted to the operation instruction step.

It is a block diagram which shows the structural example of the apparatus operating system as described in embodiment. It is a flowchart which shows an example of operation | movement of the apparatus operating system as described in embodiment. It is a flowchart which shows an example of a gesture input determination process. It is a flowchart which shows an example of an audio | voice input determination process. It is a flowchart which shows an example of a flag determination process. It is a figure which shows an example of the timing chart of the process of FIGS. It is a figure which shows the feedback display displayed on a display apparatus when there exists a display apparatus. It is a figure which shows the feedback display displayed on a display apparatus when there exists a display apparatus. It is a figure which shows the feedback display displayed on a display apparatus when there exists a display apparatus. The example of light emission of a light-emitting lamp is shown. It is a figure for demonstrating the setting of a priority operator. It is a figure for demonstrating switching of a priority operator. It is a figure for demonstrating the setting of multiple operators. It is explanatory drawing which shows the signage by which an apparatus operation system is mounted, and the operator who operates this. It is a figure which shows the TOP screen of signage. It is a figure which shows the relationship between an operator and a feedback display. It is a figure which shows the hierarchical structure of the menu item of signage. It is a figure which shows the structural example of the apparatus operation system in case an operation target apparatus is a household appliance. It is a figure explaining operation | movement of the apparatus operation system in case an operation target apparatus is a household appliance. It is a figure explaining operation | movement of the apparatus operation system in case an operation target apparatus is a household appliance. It is a figure explaining operation | movement of the apparatus operation system in case an operation target apparatus is a household appliance. It is a figure explaining operation | movement of the apparatus operation system in case an operation target apparatus is a household appliance. It is a figure which shows the other structural example of an apparatus operation system in case an operation target apparatus is a household appliance. It is a figure explaining operation | movement of the apparatus operation system in case an operation target apparatus is a household appliance. It is a figure explaining operation | movement of the apparatus operation system in case an operation target apparatus is a household appliance. It is a figure explaining operation | movement of the apparatus operation system in case an operation target apparatus is a household appliance. It is a figure which shows the structural example of the apparatus operation system integrated in the control system of motor vehicle equipment. It is a figure explaining the operation intention determination phase in case an operation target apparatus is a motor vehicle equipment. It is a figure explaining the operation instruction | indication phase in case operation target apparatus is a motor vehicle equipment. It is a figure explaining the operation instruction | indication phase in case operation target apparatus is a motor vehicle equipment. It is a figure explaining the operation instruction | indication phase in case operation target apparatus is a motor vehicle equipment. It is a figure explaining the operation instruction | indication phase in case operation target apparatus is a motor vehicle equipment.

  Hereinafter, embodiments of the present invention will be described in detail. The device operation system described in the embodiment is configured as a device operation system that operates various devices such as a signage, a PC, a smartphone, an HMD, a home appliance, and an automobile with gesture input and voice input.

(System configuration)
FIG. 1 is a block diagram illustrating a configuration example of a device operation system described in the embodiment. As shown in FIG. 1, the device operation system 1 includes a central processing unit (CPU) 11, a read only memory (ROM) 12, and a random access memory (RAM) 13. And an audio input unit 14, an imaging unit 15, an output unit 16, a device function processing unit 17, and connection means 10 for connecting them.

  The device function processing unit 17 is a device that performs processing for exhibiting the function of the operation target device itself, and is, for example, an illumination device for illumination. The device function processing unit 17 may be a single device, but may be integrated into the CPU 11, ROM 12, and RAM 13.

  The CPU 11 is connected to each component 11, 12, 13, 14, 15, 16, and 17 through a connection means 10 such as a bus, performs control signal and data transfer processing, and performs voice input and output in the voice input unit 14. In response to a gesture input by the imaging unit 15, various programs for executing the operation of the entire device operation system 1 are executed, arithmetic processing, and the like are performed.

  The connection means 10 is a connection means such as a bus for connecting each component of the device operation system 1. When the component of the device operation system 1 is an independent device, each component is connected by wire or wirelessly. It is means to do.

  The ROM 12 stores programs and data necessary for the operation of the entire device operating system 1. These programs are stored in a recording medium such as a DVD-ROM, HDD, or SDD, and are read into the RAM 13 to be executed by the CPU 11 to perform processing of the device control system 1 of the present embodiment. Of course, necessary programs and data can be fetched using a cloud service instead of such a ROM.

  The RAM 13 temporarily stores a program created in accordance with software for performing voice input processing and gesture input processing, which will be described later, and voice input data and gesture input data.

  The voice input unit 14 is a means for performing voice input. For example, a sound collection microphone, a directional microphone, or the like can be used.

  The imaging unit 15 is a means for performing gesture input, and can detect the movement of the operator's body, such as an RGB camera, an infrared camera, a distance image camera (Time of Flight method, etc.) that can capture the movement of the operator's body. An ultrasonic camera, a stereo camera, or the like can be used. The imaging unit 15 may determine the installation position according to what kind of movement of the operator is used for the operation determination. A gesture input is detected from the movement of the operator's body photographed by the imaging unit 15. For example, when hand movement is used as a gesture input, the hand movement is detected as a gesture input from the photographed body movement of the operator.

  The output unit 16 is for various outputs, and a display device that performs feedback by display, a speaker that performs feedback by sound, and the like can be used. When the operation target device includes a display device, a speaker, or the like, the display device, the speaker, or the like of the operation target device may also serve as the output unit 16.

(Processing flow)
FIG. 2 is a flowchart illustrating an example of the operation of the device operation system described in the embodiment. 3 is a flowchart showing an example of a gesture input determination process, FIG. 4 is a flowchart showing an example of a voice input determination process, and FIG. 5 is a flowchart showing an example of a flag determination process S1. FIG. 6 is a diagram illustrating an example of a timing chart of the processing of FIGS. The operation of the device operating system will be described with reference to FIGS. 2 to 6 with reference to FIG.

  When the device operating system 1 is turned on, the program for executing the operation shown in FIG. 2 is started, and the gesture input determination process shown in FIG. 3 and the voice input determination process shown in FIG. 4 are started. These processing programs and data necessary for the processing are stored in advance in storage means such as the ROM 12. In the device operation system 1, the CPU 11 shown in FIG. 1 cooperates with each unit of the device operation system 1 to execute the gesture input determination process and the voice input determination process based on the input from the operator, An operation intention determination phase F1 for determining an intention and an operation instruction phase F2 for actually instructing an operation are sequentially executed.

  The operation intention determination phase F1 is a phase for performing processing for determining whether or not the operator has an intention to operate when there is any input, and is a phase in which no operation processing is actually performed on the device. The next operation instruction phase F2 is a phase in which, when it is confirmed that the operator has an intention to operate, a process for actually operating the device individually and specifically according to the input of the operator.

  In the operation intention determination phase F1, the operation input using both the gesture input and voice input operation means is accepted, but the process of operating the actual device is not performed even when the input using both operation means is completed. That is, in the operation intention determination phase F1, in principle, the processing by the device function processing unit 17 is not performed. In the operation intention determination phase F1, in addition to confirming the operator's intention, it is possible to confirm that the system is in a recognition state of the operation input (gesture input and voice input) performed by the operator, or to recognize the system. It is also possible for the operator to grasp the input mode (position where a gesture should be made, the volume of the voice, etc.). Since the input in the operation intention determination phase is not regarded as an input for actually operating the device, an inadvertent operation due to an input not intended for the operation is not performed.

  In the operation intention determination phase F1, the flag determination shown in FIG. 5 is performed based on the specific gesture determination flag set by the gesture input determination process shown in FIG. 3 and the specific voice determination flag set by the voice input determination process shown in FIG. The operator's intention to operate is confirmed by executing the process S1.

  The gesture input determination process will be described. In the gesture input determination process shown in FIG. 3, it is determined that a specific gesture has been performed, and the specific gesture determination flag is changed from OFF to ON. First, as shown in FIG. 3, an image of the operator's one frame is acquired by the imaging unit 15 (S31), and gesture input detection processing is performed to detect gesture input for the acquired one-frame image (S32).

  The gesture input detection process (S32) determines the presence or absence of a gesture in the acquired image, and further determines what kind of gesture it is when it is determined that there is a gesture. `` What kind of gesture is it '' means the content of the gesture, for example, the shape of the palm such as goo, par, the palm is right, the palm is left, etc. It is the state of the arm such as bent arm, extended arm, etc., or any other state related to any body such as hand, foot, head, etc. is there.

  Next, a specific gesture determination flag setting process is performed (S33). In the specific gesture determination flag setting process, a specific gesture determination process for determining whether or not a specific gesture is performed based on the detection result of the gesture input in S32 and the history data of the detection result of the past gesture input in the current frame. The determination flag is turned ON / OFF (also referred to as valid / invalid respectively) based on the determination result.

  In the specific gesture determination process, for each gesture input of a predetermined number of gesture input detection results from the past to the current frame (only the current frame is acceptable), a match with a specific gesture stored in advance is determined. At this time, since a specific gesture is stored in advance in a database or the like provided in the ROM 12 or the like, a match (including partial match) between the gesture input and the stored specific gesture is determined. It is determined that it has been detected. In the operation intention determination phase F1, it is possible to determine a match with one or more specific gestures indicating a predetermined operation intention.

  If the detection result of a predetermined number of gesture inputs from the past to the current frame matches the specific gesture and is equal to or greater than the threshold, the determination flag for the specific gesture is changed to ON or maintained ON, and the threshold is set. If it is less than that, the determination flag of the specific gesture is changed to OFF or maintained OFF. For example, when the flag change threshold is 3, when it is determined that a specific gesture has been detected for only 3 frames among 5 consecutive frames including the current frame, the determination flag for the specific gesture is changed to ON, When the three-frame specific gesture is not detected, the determination flag of the specific gesture can be changed to OFF. If a threshold is provided and a determination is made based on a gesture over a predetermined number of frames, erroneous detection due to temporary body shake or the like can be prevented. The threshold value may be different depending on whether it is turned on or off.

  After executing the specific gesture determination flag setting process (S33), the history detection process (S34) of the gesture detection result is performed. In the history detection process of the gesture detection result, the content of the gesture detected in the gesture input detection process S32 is recorded and stored as a history. The history may be stored sequentially in a database provided in the RAM 13 or the like, for example. As another aspect, the history records the presence / absence of a specific gesture indicating an operation intention in the operation intention determination phase F1, and records the content of the gesture as the detection result in the gesture input detection process S32 in the operation instruction phase F2. It is good.

  The history detection process (S34) of the gesture detection result may be performed after the gesture input detection process S32, and may be performed before the specific gesture determination flag setting process S33. Note that, when the specific gesture determination flag setting process is performed only for the current frame regardless of the gesture detection result of a plurality of frames as described above, the history storage process S34 can be omitted.

  When the history detection process (S34) of the gesture detection result is completed, the process returns to the image acquisition process (S31) again, and the gesture input determination process for the next frame is performed.

  In the gesture input determination process, a case where one image is acquired in one frame is described as an example, but a plurality of images may be acquired in one frame. In this case, the specific gesture may be a plurality of still images or a moving image. For example, in order to detect a gesture with a time axis, such as swiping a hand to the left or right, an acquired image (a plurality of still images) in each frame within a predetermined time is stored, and a gesture is performed on a continuous image. Detection may be performed.

  In addition, the history storage may save the time stamp when the gesture input is detected together with the determination result. In the specific gesture determination flag setting process, a gesture input whose time stamp is too old may be excluded from the determination target.

  Next, the voice input determination process will be described. In the voice input determination process shown in FIG. 4, it is determined whether a specific voice input has been accepted, and the voice determination flag is changed. In the voice input determination process, first, voice is acquired by the voice input unit (S41), and voice recognition processing for performing voice recognition from the acquired voice is performed (S42). If there is any voice input from the silent state (state below a predetermined sound input level), voice acquisition in the voice acquisition process (S41) is started, and voice acquisition is continuously performed for a predetermined time. As another aspect, the voice may be continuously acquired until a silent state is reached again instead of during a predetermined time. In the voice recognition process (S42), a voice keyword is extracted from the acquired voice to convert the voice signal into a language.

  Next, a specific voice determination flag setting process is performed (S43). In the specific voice determination flag setting process, it is determined whether or not a specific voice keyword is uttered by determining whether or not a recognized voice keyword that matches the specific voice keyword is detected. Turn ON / OFF. At this time, a specific voice keyword is stored in advance in a database or the like provided in the ROM 12 or the like, and a match between the recognized voice keyword and the specific voice keyword is determined, and the recognized voice keyword is the specific voice keyword. It is determined that a specific voice keyword has been detected. In the operation intention determination phase F1, it is possible to determine a match with one or more specific voice keywords indicating a predetermined operation intention.

  Further, it is also possible to turn ON / OFF the specific voice determination flag by determining whether the history data of the detection result of the past voice keyword within a certain time including the current frame matches the specific voice keyword. If a plurality of voice keywords are not detected within a predetermined time from past history data including the current frame, the determination flag is turned on, and the determination flag is kept on for a certain time. This can prevent a normal conversation from being determined as an utterance intended by the operator. Even if there is no specific voice keyword, the flag is maintained if there is a specific voice keyword within a certain past time. When there is no need to change the determination flag, the flag state is maintained as it is.

  After the specific voice determination flag setting process (S43), a voice keyword detection result history saving process is performed (S44). The voice keyword extracted in the voice recognition process (S42) and its time stamp are recorded and stored as a history. By storing not only the voice keyword but also the time stamp, the flag can be reset when the predetermined time elapses, and the keyword within a certain past time can be easily confirmed in S43. be able to.

  The history storing process (S44) of the voice keyword detection result may be performed after S42, and may be performed before S43. Note that if the history is not used in the specific voice determination flag setting process, or if the flag is not reset when a predetermined time has elapsed, the history storage process S34 can be omitted.

  When the history storage process (S44) of the voice keyword detection result is completed, the process returns to the voice acquisition process (S41) again, and the voice input determination process is performed for the next voice.

  Returning to FIG. 2, in the flag determination process S <b> 1, the intention of operation is determined from the state of the flag indicating the results of the gesture input determination process and the voice input determination process described above according to the process shown in FIG. 5. The flag determination process S1 will be described with reference to FIG. First, the intention of operation is determined based on the determination result of gesture input and voice input (S51). In the flag determination process S1, it is possible to determine the intention to operate by looking at the ON / OFF state of the specific gesture determination flag and the specific voice determination flag. Specifically, when both the gesture determination flag and the voice determination flag are OFF, and when only one of the gesture determination flag and the voice determination flag is ON, it is determined that there is no intention to operate ( (Proceed to S53).

  However, when only one of the gesture determination flag and the voice determination flag is ON, a standby mode flag for displaying an operation input for changing the other flag can be turned ON (S54). ). When the standby mode flag is turned on, it can display on the display means or the like (corresponding to the output unit 16 in FIG. 1) that it is in the standby state of the other input.

  On the other hand, when both the gesture determination flag and the voice determination flag are ON, it is determined that there is an intention to operate (proceed to S52). If it is determined that there is an intention to operate, if there is a standby mode flag turned ON in S54, it is returned to OFF.

  Here, the relationship between the gesture input determination process of FIG. 3, the voice input determination process of FIG. 4, and the flag determination process of FIG. 5 will be described. 6A shows gesture input determination processing, FIG. 6B shows voice input determination processing, and FIG. 6C shows flag determination processing. As is clear from the fact that the number of frames is different from (a) to (c), each is an asynchronous process.

  In the gesture determination process of FIG. 6A, it is determined that “Goo” is a specific gesture indicating an operation intention, and two of the gestures detected in the past three frames including the current frame are specific gestures. In this case, the specific gesture determination flag is changed to ON, and when the determined specific gesture is 0, the specific gesture determination flag is changed to OFF. In this process, since the specific gesture is detected twice consecutively in the third frame, the specific gesture determination flag is set to ON, and the detection of “Goo” in the past three frames is 0 in the ninth frame. The flag is turned off.

  In the voice input determination process in FIG. 6B, when the “start” is a specific voice keyword indicating an intention to operate and the recognized voice keyword is a specific voice keyword, the specific voice determination flag is turned on. Is controlling. In this process, since “weather forecast” is spoken different from the specific voice keyword indicating the operation intention in the first frame, the specific voice determination flag remains OFF, but in the second frame, the specific voice keyword Since the “start” that coincides with the utterance is uttered, the specific voice determination flag is turned ON in the second frame.

  In the flag determination process of FIG. 6C, since it is confirmed that both the specific gesture determination flag and the specific voice determination flag are ON in the fifth frame, it is determined that there is an intention to operate, and the specific gesture determination flag is determined. And the specific voice determination flag are reset to OFF as necessary.

  Returning to FIG. 2 again, in the operation intention determination phase F1, when it is determined whether or not there is an operation intention by the flag determination processing S1, it is determined whether or not the operation instruction phase F2 is shifted based on the presence or absence of the operation intention. (S2). If there is an operation intention (S2: Yes), the operation instruction phase F2 is entered. If there is no operation intention (S2: No), the operation instruction phase F2 is not executed, and the process returns to the flag determination process S1 again.

  The operation instruction phase F2 is a phase in which an input by the user is interpreted as an operation instruction and processing is performed based on the interpreted operation instruction. In the operation instruction phase F2, it is not always necessary to input using both the gesture input and voice input operation means. The operation can be executed by completing at least one of the gesture input determination process of FIG. 3 and the voice input determination process of FIG.

  In the operation instruction phase F2, the specific gesture in the gesture input determination process (FIG. 3) is a gesture that means the contents of various actual operations, unlike the gesture for confirming the intention of operation. Similarly, the specific voice keyword in the voice input determination process (FIG. 4) is a voice keyword that means the contents of various actual operations, etc., unlike the voice keyword for confirming the intention of operation. When there are a plurality of operation contents, there are specific gestures and specific voice keywords according to the operation contents. Therefore, in each input determination process (FIGS. 3 and 4), the specific gesture and the specific voice keyword are included. By processing ON / OFF of each flag, it is possible to indicate which operation content has been accepted by each specific gesture determination flag and specific voice determination flag.

  In the operation instruction phase F2, it is determined whether or not there is an operation instruction based on the results of these processes (S3). The presence / absence of an operation instruction is determined based on a specific gesture determination flag or a specific voice determination flag that can be changed in each input determination process (FIGS. 3 and 4). In the operation instruction phase F2, it can be determined that there has been an operation instruction when at least one of the specific gesture input determination flag and the specific voice determination flag is ON. Of course, depending on the content of the operation, the condition for the operation instruction may be that both flags are valid.

  When it can be determined that there is an operation instruction (S3: Yes), a predetermined input pattern (operation only, sound only, operation determined as having been input in the determination processing of FIGS. 3 and 4) The operation target system is controlled by giving an operation instruction matching the operation target to the operation target (S4). Control of the operation target system (S4) is to process the function of the operation target device by the device function processing unit 17. For example, in the case of illumination, the illumination is turned on and dimmed, or in the case of TV. TV on / off, tuning, volume adjustment, etc.

  In S3, if it is determined that there is no operation instruction or if it cannot be determined (S3: No), or if the control of the operation target system (S4) ends, whether or not the operation instruction phase F2 is ended A determination is made (S5).

  Whether or not the operation instruction phase is ended is determined by whether (1) “End” is instructed as an operation instruction, (2) If there is no operation instruction within a certain time, (3) Gesture determination for a certain time When both the flag and the voice determination flag are OFF, it is determined that the operation instruction phase is finished (S5: Yes).

  When it is determined that the operation instruction phase is not finished (S5: No), the process returns to the first operation instruction determination process S3 of the operation instruction phase F2.

  When it is determined that the operation instruction phase is complete (S5: Yes), the operation intention determination phase F1 is returned again, and the flag determination process S1 is performed.

(Operation feedback)
In the device operation system of this embodiment, the operation result is fed back to the operator. The feedback includes a mode in which feedback is displayed on the display means and a mode in which feedback is output from the audio output device. The means for performing feedback display differs depending on whether or not the output unit 16 (see FIG. 1) has a display device. 7 to 9 are diagrams showing feedback displays displayed on the display device when the display device is present.

  In the operation intention determination phase F1, feedback is performed based on switching of the gesture determination flag and the voice determination flag. In the operation instruction phase, feedback is performed based on switching of at least one of the gesture determination flag and the voice determination flag. Can be performed.

  The feedback display can show the gesture input feedback display and the voice input feedback display as separate displays. Thereby, it is possible to confirm which input the feedback display relates to. As shown in FIG. 7, feedback display can be performed with a palm-shaped icon indicating feedback of gesture input and a balloon icon indicating feedback of voice input. Further, as shown in FIG. 7, the display state such as the color of the icon may be changed between the operation intention determination phase and the operation instruction phase.

  The feedback display has, for example, four types of display modes according to the statuses of a gesture (speech) input standby state, a gesture (speech) input, a gesture (speech) input completed, and a gesture (speech) input failure. Either one is displayed. The operator can confirm the status of the input by visually recognizing the display mode.

  As the feedback display, the remaining time of the operation instruction phase may be displayed. For example, when the operation instruction phase is set to automatically end when there is no input for a predetermined time, a feedback display such as “the operation instruction phase will end in 30 seconds if nothing is done” may be displayed. The remaining time may be displayed using means such as a change in color of a gesture lamp, a meter display, a separate state transition lamp (operation intention and operation instruction phase), or the like.

  In addition, for example, when a hand motion is used as a gesture, it is necessary to guide the hand to an appropriate position for hand recognition. When the operator's gesture position is not appropriate, as shown in FIGS. 8 and 9, even when the operator is too close, too far away from the camera, or out of the camera, It is possible to display on the display device indicating this and feed back to the operator.

  In the case of FIG. 8, when the position of the hand is in a middle and a spot, feedback is performed by display as being within the appropriate position range. When the hand position is shifted back and forth, a display prompting the user to move the hand to an appropriate position is displayed.

  Also, as shown in FIG. 9, even if the depth position of the hand is appropriate, if it is shifted left and right and up and down from the spot, or if an appropriate operation (gesture pose) is not performed, the hand is moved to the appropriate position, Display to prompt proper operation.

  Further, a light-emitting lamp (lamp) may be provided as the output unit 16 (see FIG. 1) instead of a display device such as a display. FIG. 10 shows a light emission example of the light-emitting lamp. These light-emitting lamps can be provided for each of the gesture and the voice, and the lighting state can be changed to indicate feedback to the operator when the determination flag for the gesture and the voice is indicated. it can. In the example of FIG. 10, a gesture (voice) input standby state in the operation intention determination phase, a gesture (voice) input standby state in the operation instruction phase, a gesture (voice) input, and a gesture (voice) input completed Each of the five display modes is displayed according to the status of the gesture (voice) input failure. Changes in the five types of display modes can be represented by display colors.

  Furthermore, when the output unit 16 includes an audio output device such as a speaker, feedback output can be performed to the operator by voice instead of feedback display. Of course, feedback display and voice feedback output may be performed together.

(Priority operator judgment)
In the device operation system of the present embodiment, the priority operator can be determined in the operation intention determination phase F1 in order to determine an operator who can operate with priority among a plurality of people. 11 is a diagram for explaining the setting of the priority operator, FIG. 12 is a diagram for explaining the switching of the priority operator, and FIG. 13 is a diagram for explaining the setting of a plurality of operators. .

  In the operation intention determination phase F1, when it is detected by image processing that only one gesture input matches the specific gesture, the person who performed the gesture input is set as an operator candidate. Further, when a voice input that matches the voice keyword is detected, it is determined which operator candidate has spoken from the direction of the sound source, the movement of the mouth in the captured image, and the like, and the operator is set as the operator candidate.

  As shown in FIG. 11, when an operator candidate surrounded by a dotted line frame performs a specific action and voice input, it is determined as a priority operator surrounded by a solid line frame, and operation authority for subsequent operations is given, The priority operator's gesture input is tracked.

  As shown in FIG. 12, the priority operator can be switched. In FIG. 12, the priority operator surrounded by the solid line frame ends the operation, and the gesture input and voice input stop. From this state, another person surrounded by a dotted line frame performs an operation input for determining the operation intention. Next, when a new operator candidate performs a specific action and voice input, it is determined as a priority operator surrounded by a solid line frame, and subsequent operation authority is given, and the priority operator's action is tracked.

  As shown in FIG. 13, there may be a plurality of priority operators surrounded by a solid line frame. Each of the plurality of persons determined as the operator in the operation intention determination is determined as a priority operator, and an operation authority is given to each.

Next, the operation in a specific operation target device will be described.
(Configuration example in signage)
First, the operation of the device operation system when the operation target device is signage will be described with reference to FIGS. FIG. 14 is an explanatory diagram showing a signage 20 on which the device operation system of FIG. 1 is mounted and an operator who operates the signage 20. Here, a case where the Map function is used in the signage 20 will be described as an example. FIG. 15 shows a signage TOP screen. FIG. 16 is a diagram showing the relationship between the operator and feedback display.

  As shown in FIG. 14, for example, the signage 20 equipped with the device operation system includes a monitor 21 that functions as an output unit 16, a camera sensor 22 that functions as an imaging unit 15, a microphone 23 that functions as an audio input unit, and an output. And a speaker 24 functioning as a unit. In the example of FIG. 14, configurations corresponding to the CPU 11, the ROM 12, and the RAM 13 of FIG. 1 are integrally incorporated in the signage 20.

  In the signage 20, as shown in FIG. 15, items “Map”, “News”, “Forecast”, and feedback display items, which are selectable on the monitor 21 as a TOP screen in the input standby state, are displayed. In this state, operation inputs (gesture input and voice input) are not accepted as operation instructions for the device. In this state, when the operator holds his hand toward the signage and speaks “OK”, which is the voice keyword for starting the operation, the gesture input is acquired from the image captured by the camera sensor 22, and from the voice acquired by the microphone 23. Get voice input. When it is determined that the acquired gesture input matches the specific gesture and the acquired voice input includes a voice keyword, both the gesture determination flag and the voice determination flag are switched to be effective, and the feedback display indicates whether the gesture input and the voice input are input. It changes to show what has been done.

  The device operation system determines that there is an intention to operate based on both the gesture determination flag and the voice determination flag, and shifts from the operation intention determination phase to the operation instruction phase. What is important here is that even if an operation input is made in the operation intention determination phase, only the operation instruction phase is entered, the item screen is not changed, and the Map function is not changed. In addition to “OK”, an arbitrary keyword such as “operation” or “start” can be used as the voice keyword for starting the operation, and the gesture indicating the intention of operation may be a gesture other than holding the hand. If the voice of the operator speaking the operation start keyword is low, the feedback display does not change to a display indicating that voice input has been made, so the operator changes the voice volume until the feedback display changes, etc. The loudness necessary for operation can be confirmed.

  Next, in the operation instruction phase, as shown in FIG. 16, when the operator speaks the keyword “Map” while holding his hand toward the signage 20, here, switching to the Map function is performed, and the content An initial screen of “Map” is displayed on the monitor 21. In this manner, in the operation instruction phase, when a gesture input or voice input by the operator is accepted, processing corresponding to the input is performed. For example, when a voice input for a keyword utterance is accepted, an icon on the screen corresponding to the keyword utterance is operated, or when a gesture input for holding the hand left or right is accepted, a cursor operation on the screen and selection of functions such as an icon are selected. Perform the operation. In addition, when a gesture input for holding a hand and stopping for a certain period of time is performed, an operation for determining a function such as an icon selected by the cursor is performed. In the example shown in FIG. 16, in the operation instruction phase, the operation of selecting the content “Map” by both the gesture input and the voice input is performed. However, the operation may be performed by either the gesture input or the voice input. Good.

  As described above, when the operation target device is signage, according to the device operation system of the present embodiment, it is possible to prevent an erroneous input due to an input that is not intended for operation.

  FIG. 17 is a diagram illustrating a hierarchical structure of signage menu items. As for the menu item of signage, “News” or “Forecast” can be selected in addition to “Map” from the TOP screen, and an item in a lower hierarchy can be further selected for each item. At this time, in addition to selecting the lower hierarchy step by step, the lower hierarchy can also be selected at once by keyword input. For example, when selecting “News-Soccer”, if the keyword “Soccer News” is spoken while the TOP screen is displayed in the operation instruction phase, “News-Soccer” is selected (jump) at once. be able to.

  Thus, the burden on the operator can be reduced by the configuration of jumping the operation hierarchy. Incidentally, if the operation menu has a hierarchical structure, a long operation time is required to obtain a desired operation result, which may increase the physical burden on the operator. In gesture input, it is common for an operator to perform an operation on a sensor by a motion indicating an operation intention. That is, not an ergonomically comfortable posture (upright immobility, etc.) but an action with some physical load is required. For example, an operation of holding a hand over the sensor is a gesture operation. Keeping your hand raised for a long time imposes a physical burden on the operator. If many instructions are to be given to the operation target, the operation time of the gesture inevitably increases. Conversely, if the number of instructions to the operation target is reduced, the number of operation items to the operation target is reduced accordingly, and convenience is impaired. Therefore, the burden on the operator can be reduced by the configuration of jumping the operation hierarchy.

(Configuration example for smartphones)
The operation of the device operation system when the operation target device is a smartphone will be described with reference to FIGS. Here, a case where the mail function is used with a smartphone will be described as an example.

  When the smartphone is in the input standby state, if you hold your hand over the smartphone and say “Operation” as a keyword, both the gesture determination flag and the voice determination flag will change effectively, and the feedback display on the smartphone screen will be the gesture input. And a display indicating that voice input has been made, and it is determined that there is an intention to operate. When it is determined that there is an intention to operate, only the operation instruction phase is entered, and the mail function is not switched.

  When a gesture input or voice input by the operator is received in the operation instruction phase, processing corresponding to the input is performed. For example, when a voice input for a keyword utterance is accepted, an icon on the screen corresponding to the keyword utterance is operated, or when a gesture input for holding the hand left or right is accepted, a cursor operation on the screen and selection of functions such as an icon are selected. Operate. In addition, when a gesture input for holding a hand for a certain period of time is performed, an operation for determining a function such as an icon selected by the cursor is performed.

  Further, in an equipment operation system configured with a smartphone as an operation target, it does not prevent the operation intention determination phase from being omitted in an emergency situation such as receiving power. In this case, when receiving a call or video call while the smartphone is in the input standby state, if the gesture input with the hand and the voice input of the keyword “Moshi” are received, the operation intention determination is omitted and the call starts. To do.

  Thus, even when the operation target device is a smartphone, according to the device operation system of the present embodiment, erroneous input due to an input that is not intended for operation is prevented.

(Configuration example in home appliances)
Here, the operation of the device operation system when the operation target device is a home appliance will be described with reference to FIGS. 1 and 2 and FIGS. 18 to 26.

[Configuration example 1]
The configuration example 1 is a case where the operation target devices are a living room television and lighting. In the configuration example 1, as illustrated in FIG. 18, the television 31 and the illumination 32 are equipped with the device function processing unit 17, and the audio input unit 14, the imaging unit 15, and the output unit 16 are You may be comprised as another apparatus. The voice input unit 14, the imaging unit 15, and the output unit 16 in the device operation system 1 of FIG. 1 are installed, for example, on a shelf or a wall in a living room so that an operator can input an operation and can be easily confirmed. be able to. The audio input unit 14, the imaging unit 15, and the output unit 16 can be configured as an apparatus integrated with the CPU 11, the ROM 12, and the RAM 13. In this case, the wired or wireless connection means 10 communicates with the television 31 and the illumination 32, and each device function processing unit 17 of the television 31 and the illumination 32 executes processing based on the operation instruction. In this example, a case will be described in which lighting is turned on from the state where both the television 31 and the illumination 32 are OFF, and the television is also turned ON.

  First, when the operator holds his hand toward the imaging unit 15 and speaks the keyword “operation”, the gesture determination flag and the voice determination flag become valid. When both the flags are thus valid, the feedback display is turned on in the output unit 16 and it is determined that there is an intention to operate, and the process proceeds to the operation instruction phase.

  Next, in the operation instruction phase, as shown in FIG. 19, when the operator holds his hand toward the imaging unit 15 and speaks the keyword “lighting”, the feedback display is turned on and the light 32 is set as the operation target. Selected. At this time, when the illumination 32 is in the off state, the illumination 32 may be in the on state when selected. Furthermore, as shown in FIG. 20, when the keyword “dimming” is spoken and the hand is held in the right direction toward the imaging unit 15, the feedback display is turned on and it is determined that an instruction to increase the intensity of illumination is given. Then, an operation for lighting the illumination 32 brighter is executed. Thereafter, when the user speaks the keyword “dimming” and holds his hand toward the imaging unit 15, the feedback display is turned on, and it is determined that an instruction to weaken the intensity of the illumination 32 is given. Lower.

  As shown in FIG. 21, when the keyword “TV” is spoken during operation of the illumination 32, the feedback display is turned on and the operation target is switched to the TV 31. When the TV 31 is originally turned off, the switch of the TV 31 may be turned on when the operation target is switched to the TV 31. Furthermore, when the keyword “volume” is spoken, the feedback display is lit and the volume can be adjusted. The volume may be adjusted by voice input or gesture input. Switching from the illumination 32 to the TV 31 may be performed by combining voice input and gesture input as shown in FIG.

  Thus, in the device operation system of the present embodiment, a plurality of home appliances can be switched and operated.

[Configuration example 2]
Configuration example 2 is a case where the operation target device is illumination. In this example, as shown in FIG. 23, unlike the configuration example 1, the device operation system 1 will be described with a configuration without a display unit. In this example, the audio output unit 16 performs feedback.

  First, as shown in FIG. 24, when the operator holds his hand toward the imaging unit 15 and speaks the keyword “lighting”, the gesture determination flag and the voice determination flag become valid. When both flags are enabled in this manner, the voice output unit 16 outputs “voice input is present” to provide feedback, determine that there is an intention to operate, and switch to the operation instruction phase.

  Next, as shown in FIG. 25, when the user speaks the keyword “light control” and holds his / her hand toward the imaging unit 15, the voice output unit 16 outputs “voice input” and “gesture input”. Thus, feedback is made, and it is determined that it is instructed to dimm the illumination, the illumination is turned on, and the dimming according to the gesture input is executed. Thereafter, when the hand is held leftward toward the image pickup unit 15, the audio output unit 16 outputs “gesture input present” to provide feedback and perform dimming according to the gesture input. Further, as shown in FIG. 26, the keyword “turn off” is spoken, it is determined that turning off the illumination is instructed, and the illumination 32 is turned off. Further, if the operator does not operate for a certain period of time after the light is turned off, the operation will return to the operation intention determination phase F1. In the case of adjustment, it starts from confirmation of the operation intention.

  Thus, even when the operation target device is a home appliance, according to the device operation system of the present embodiment, erroneous input due to an input that is not intended for operation is prevented.

(Configuration example for automobile equipment)
The equipment operation system of this embodiment can also be configured as a control system for automobile equipment. The operation of the device operation system when the operation target device is an automobile equipment will be described with reference to FIGS. FIG. 27 is a diagram illustrating a configuration example of a device operation system incorporated in a control system for automobile equipment, FIG. 28 is a diagram illustrating an operation intention determination phase, and FIGS. 29 to 32 are diagrams illustrating an operation instruction phase. FIG.

  As shown in FIG. 27, in a control system for automobile equipment, a HUD (Head Up Display) 41, a car navigation system 42, and a speaker 43 can function as the output unit 16 (see FIG. 1). The camera 44 and the microphone 45 provided on the panel at the back of the handle can function as the imaging unit 15 (see FIG. 1) and the voice input unit 14 (see FIG. 1), respectively. Moreover, you may use the instrument panel 46 as the output part 16 (refer FIG. 1).

  As car equipment, for example, a car navigation system, an audio system, a HUD, an air conditioner, a side mirror, an electronic mirror, a back monitor, a light, a power window, a trunk, a wiper, and the like can be targeted for operation. Control map operation, camera monitor switching, etc. for car navigation, control volume, channel, etc. for audio, control speed display, destination distance display, etc. for head-up display For air conditioners, air volume adjustment, wind direction, etc., for side mirrors, direction adjustment, etc., for lights, turning on and off, for power windows, It is possible to control the opening / closing adjustment, to control the opening / closing of the trunk, and to control the operation of the wiper.

  First, when the device operation system is in a standby state, as shown in FIG. 28, when a gesture is performed with both thumbs and the keyword “start” is spoken, both the gesture determination flag and the voice determination flag are effectively changed. The feedback display on the HUD 41 or the like is switched to a display indicating that a gesture input and a voice input are made, and it is determined that there is an intention to operate, and the operation instruction phase is entered.

  Next, in the operation instruction phase, when “car navigation” is spoken, the feedback display on the HUD 41 or the like is switched to a display indicating that an input of voice input has been made, and a process of selecting by turning on the car navigation 42 is executed.

  When a push operation with the thumb of the right hand is performed with the car navigation selected, the feedback display on the HUD 41 or the like is switched to a display indicating that a gesture input has been made and the destination navigation is displayed on the screen of the car navigation 42. Information is displayed.

  In addition, when a push operation with the thumbs of both hands is performed with the car navigation selected, the feedback display on the HUD 41 or the like is switched to a display indicating that a gesture input has been made and displayed on the car navigation 42. When music is played and further a push operation with the thumb of the right hand is performed, the feedback display on the HUD 41 or the like is switched to a display indicating that a gesture input has been made, and the music displayed on the car navigation system 42 is paused.

  Further, when the air volume adjustment of the air conditioner is performed from the state in which the car navigation is selected, as shown in FIG. 31, when “air conditioner” is spoken, the feedback display on the HUD 41 or the like is displayed to indicate that the voice input is input. The selection process is performed by switching the air conditioner and turning on the air conditioner. Further, when the “air volume” is spoken and a push operation with the thumb of the right hand is performed, the feedback display on the HUD 41 or the like is switched to a display indicating that the gesture input and the voice input have been made, and the air volume of the air conditioner is increased. When the push operation is stopped, the air volume adjustment of the air conditioner stops.

  When the air volume adjustment of the air conditioner is performed from the state where the car navigation system is selected, a specific voice keyword “air volume” that is specific to the air conditioner is selected with the car navigation system selected, without speaking “air conditioner”. The air volume of the air conditioner can be increased by performing a push operation with the thumb of the right hand.

  Thus, even when the operation target device is equipped with an automobile, according to the device operation system of the present embodiment, erroneous input due to an input that is not intended for operation is prevented.

1 Device operation system 11 CPU
12 ROM
13 RAM
14 Audio input unit 15 Imaging unit 16 Output unit 17 Device function processing unit 10 Connection means 20 Signage 21 Monitor 22 Camera sensor 23 Microphone 24 Speaker 31 Television 32 Illumination 41 HUD
42 Car navigation system 43 Speaker 44 Camera 45 Microphone 46 Instrument panel

Claims (7)

A device operation system for operating a device based on gesture input and voice input,
Operation intention confirming means for confirming the presence or absence of an operation intention without performing the operation of the device by gesture input and voice input;
Operation instruction means for instructing operation of the device by at least one of gesture input and voice input;
The operation intention confirmation means has the operation intention when it is determined that a specific gesture indicating the operation intention is detected as the gesture input and a specific voice keyword indicating the operation intention is detected as the voice input. The apparatus operation system is characterized in that the process is shifted to processing by the operation instruction means.   The operation intention confirmation unit determines whether or not the gesture input is a specific gesture, determines whether or not the gesture input determination unit sets a specific gesture determination flag, and determines whether or not the voice input is a specific voice keyword Voice input determination means for setting a specific voice determination flag, and flag determination means for determining whether or not there is an operation intention based on the specific gesture determination flag and the specific voice determination flag. The device operation system according to claim 1.   3. The device according to claim 2, wherein the gesture input determination unit stores a history of gesture inputs detected over a plurality of times, and sets a specific gesture determination flag based on the plurality of gesture inputs stored in the history. Operation system.   The device operation system according to claim 2 or 3, wherein the gesture input determination unit, the voice input determination unit, and the flag determination unit perform processing asynchronously with each other.   5. The device operation system according to claim 1, wherein the device to be operated is any one of a signage, a PC, a smartphone, an HMD, a home appliance, and an automobile equipment. A device operation method for operating a device based on gesture input and voice input,
An operation intention confirmation stage for confirming the presence or absence of an operation intention without performing the operation of the device by gesture input and voice input,
An operation instruction step for instructing operation of the device by at least one of gesture input and voice input,
When it is determined that the specific gesture indicating the operation intention is detected as the gesture input in the operation intention confirmation stage and the specific voice keyword indicating the operation intention is detected as the voice input, the operation intention is present. A device operation method characterized by determining and proceeding to the operation instruction step. A program for causing a computer to execute a device operation method for operating a device based on gesture input and voice input, the device operation method comprising:
An operation intention confirmation stage for confirming the presence or absence of an operation intention without performing the operation of the device by gesture input and voice input,
An operation instruction step for instructing operation of the device by at least one of gesture input and voice input,
In the operation intention confirmation step, when it is determined that a specific gesture indicating an operation intention is detected as the gesture input and a specific voice keyword indicating the operation intention is detected as the voice input, the operation intention is present. And a device operation program, wherein the operation instruction stage is determined.

Images