JP2005174356A - Direction detection method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a direction detection method for enabling a driver to perform operation safely with onboard equipment without staring at an apparatus. <P>SOLUTION: After a position of an object stands still for a specified period within a specified area or stays in a state of little movement, it moves just for a specified distance in a specified direction, and moves for a specified distance in the direction opposite to the specified direction. When the object stands still just for a specified period or stays in the state of little movement within a specified range again, a moving direction of the object is detected as a movement in the specified direction. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an operation input device and method for performing an operation input to various information devices such as in-vehicle information devices.

  In recent years, many information devices and electrical devices such as car navigation systems, audio devices, TVs, video devices, mobile phones, and air conditioners have been installed in automobiles. In addition to making calls in the car, you can read and write emails and even access the Internet. This trend will continue to progress, and automatic toll collection systems and safe driving support systems have been introduced, and the car is about to become a running computer.

  Also, it is difficult to prepare all the operation panels of these various devices, and the operation units are being shared. Currently, there are operation buttons for car audio, operation buttons for air conditioner, operation buttons for car navigation, or remote control, which are generally used for operation.

  However, in some in-vehicle devices, the status of all devices is displayed on one display, and operations can be performed using a common input device. For example, a touch panel function is mounted on the display, and an operation can be performed by touching a menu displayed on the screen. Since different screens can be displayed on the display depending on the situation, all devices can be operated. For example, there are menu icons such as “air conditioner”, “audio”, “car navigation” on the display. Touching “Audio” displays the audio operation panel, and touching the “Volume up” button in it displays the volume. growing.

  There are also many products that operate the displayed menu screen using a device such as a remote control instead of a touch panel. In that case, the remote control is typically provided with buttons for instructing four directions, and there are some which select a menu by operating these buttons.

  Thus, as the types of in-vehicle devices increase, the types of operations performed by the driver have increased dramatically. In particular, with the advent of car navigation, it can be said that the types of operations have greatly increased. For this reason, the number of cases in which the driver performs an operation while driving or in a short time such as waiting for a signal is increasing.

  Conventionally, only audio operations (volume adjustment, music selection) and air conditioner operations (temperature adjustment, air volume adjustment) were performed, but in recent years, the display screen of car navigation has been switched and facilities near the current location have been searched. You may scroll the map or use your mobile phone.

  As the number of such operations increases, attention during driving is distracted, and an accident is likely to occur. In fact, the use of mobile phones while driving increased the number of accidents, and the revision of the Road Traffic Law prohibited the use of mobile phones while driving. However, when a hands-free device that can talk without having a telephone in hand is used, use during driving is permitted.

As seen in this example, it is important for safety to be able to do other things (in this case, a telephone call) without moving the driver's line of sight.
On the other hand, regarding the operation of the in-vehicle device, since the operation is usually performed while looking at the operation panel, the driver's attention is deviated from the front. An increase in such operations indicates an increased risk during operation.

  In addition, as in the example described above, when the operation of the device is concentrated on the input device common to the display, for example, conventionally, to increase the temperature of the air conditioner, turn the temperature knob or What should have been to press the rise button, select "air conditioner" from the main menu, and after the control panel of the air conditioner is displayed, press the "temperature rise" button, increasing the driver's line of sight movement, the risk Will increase.

In order to solve these problems, it is conceivable to use hand gestures, and several devices have been proposed. Patent Literature 1 and Patent Literature 2 propose a device that performs an operation by inputting a direction by hand shaking using a hand shaking detection device using an optical sensor. However, normally, when a person inputs a movement of the hand, for example, when repeatedly waving “right” or “right”, the hand moves back to “left” during that time. According to this proposal, when such a movement is performed, “right”, “left”, and “right” are input.
Japanese Patent Laid-Open No. 11-44703 JP-A-10-148640

  The present invention is intended to solve the current situation in which it is difficult for a driver driving an automobile to stably operate an in-vehicle device while keeping a close eye on the front. It is another object of the present invention to make it possible to input shape gestures and direction gestures, which are effective at that time, more efficiently and stably.

  Further, it is assumed that not only the driver but also the passenger seat is operated, and an effective operation method when operating while looking at the screen is invented. Another object is to operate more safely and stably in a special environment of in-vehicle use. Moreover, it aims at establishment of the general purpose gesture input method which can be used with various apparatuses not only in-vehicle.

  The invention of the present application is a method for detecting the direction of the object by analyzing the locus of the position information of the object to be detected, where the position of the object is stationary within a predetermined area or has little movement After moving in a predetermined direction, moving a predetermined distance in a direction opposite to the predetermined direction, and staying still in the predetermined area for a predetermined time or in a state where there is little movement. The direction of the movement of the object is detected as the movement in the predetermined direction.

  Further, it is a method for detecting a direction of the object by analyzing a locus of position information of the object to be detected, and the position of the object is one of a plurality of predetermined directions from a predetermined region. And detecting the direction of the movement of the object a plurality of times by removing the movement returning to the predetermined area that occurs during the continuous movement in the same direction as the detected direction. It is characterized by that.

  According to the present invention, an operation is performed using two different types of gestures of gestures such as the shape and movement of a human body or a part thereof, and an operation mode is selected with the first type of gesture. An operation input device capable of changing parameters in the operation mode with the second type of gesture can be realized.

Embodiments of the present invention will be described below with reference to the drawings.
<Overview> FIG. 1 is an example of an embodiment of the present invention. The first gesture detection unit 1 and the second gesture detection unit 2 are for detecting different types of gestures. Then, the operation type selection unit 3 selects the type of operation according to the detection result of the first gesture detection unit 1. The parameter changing unit 4 changes a changeable parameter in accordance with the result of the second gesture detection unit 2 during the operation selected by the operation type selection unit 3.

  The operation input processing unit 5 determines what operation has been performed according to the results of the operation type selection unit 3 and the parameter change unit 5. The operation execution unit 6 executes the determined operation. The operation input processing unit 5 can control the first and second gesture detection units, the operation type selection unit 3, and the parameter change unit 4 so that they do not operate. Therefore, only one of the gestures can be detected depending on the situation.

  <Detailed Configuration> FIG. 2 shows FIG. 1 more specifically. Here, the first gesture detection unit is a shape gesture detection unit 7 that detects a shape gesture, and the second gesture detection unit is a direction gesture detection unit 8 that detects a direction gesture.

  Here, the “shape gesture” is a “gesture represented by a hand shape”, and the “direction gesture” is a “gesture indicating a moving direction”. Of these, in the case of shape gestures, it is not always necessary to move the hand, and in that sense the word “gesture” may be difficult to apply. Treat as.

<Details of Detection Unit> Here, the two gesture detection units will be described in detail.
As a specific example, the above-mentioned shape gesture is a shape of a hand that is presented in a barrage such as “goo” “choki” “par”. Alternatively, it refers to various shapes expressed by the hand, such as “Which finger out of five fingers” or “Make a circle with the index finger and thumb”.

  An example of a shape gesture is shown in FIG. Originally, “gesture” includes a meaning accompanied by movement, and a stationary hand shape is not necessarily called a gesture. However, when presenting these shapes, the operator must use other shapes. Since the shape of the hand is transformed into the shape of, it is treated as a gesture in a broad sense in that sense. Further, considering the detection processing method, it is relatively easy to handle these as stationary shapes, but in actuality, there may be a time-series combination of these shapes, a shape with movement and the like.

Specifically, as a means for capturing the shape of the hand, it is possible to detect the shape by imaging the hand using a small CCD camera or the like and analyzing the image.
However, especially when considering in-vehicle equipment applications, if the interior of the vehicle is bright during the day and dark at night, the fluctuation of the outside light condition is large, and the image obtained by the camera greatly fluctuates, so the detection accuracy decreases. There is. There is a risk of erroneous recognition whenever a person or object crosses the front of the camera.

  The inventors have developed an image input device that can detect an object shape at high speed by using a light emitting means and an image sensor that operates in synchronism with the light emitting means, as disclosed in JP-A-10-177449. This is an apparatus that extracts only object reflected light as an image using an image sensor that irradiates an object typically with infrared light and operates in synchronization with the infrared light. Since the intensity of reflected light is generally inversely proportional to the square of the distance to an object, a nearby object can be easily cut out from a distant background.

  When this input device is used, since the device itself has light emitting means, detection can be performed even under conditions where imaging is difficult with a normal CCD camera such as at night. Further, since the light emitting means uses near infrared light that cannot be seen by human eyes, it does not hinder the driver. In addition, since an object in the background with a distance does not reach because the reflected light does not reach, malfunction due to unrelated movement can be reduced.

  A method for recognizing the hand shape obtained from these image input devices and determining whether or not the hand shape matches a predetermined shape gesture will be described. For example, when detecting a difference in shape due to the number of fingers extended, a difference in shape can be detected by setting a region where an extended finger will cross and counting the number of fingers crossing.

  As another method, considering the contour line of the hand shape, the fingertips protrude in a mountain shape, and the gap between the fingers is dented in a valley shape. You can know the number. When the fingertip is facing the camera, the fingertip is closer to the camera than the surroundings, and the amount of reflected light is maximized. By counting the number of local maximum points, the number of fingers facing the camera can be detected.

  In addition, a technique called pattern matching can be used to recognize a free hand shape. In this method, a shape to be recognized in advance is registered as a pattern, the degree of difference from the input shape is calculated, and if there is a close shape, it is determined that it has been input. As a method of calculating the degree of difference, there are a method of simply superposing patterns, calculating the degree of overlap, mapping to a feature space, and determining by a distance there.

  <Shape Recognition Start after Standing> When recognizing a shape, the more accurate it is, the longer it takes to process. Therefore, if the recognition process is performed for each frame of the input image, the processing cost increases. When presenting the shape, it is desirable to start the recognition process when the operator presents the shape and is stationary for a certain period of time. This is because when the operator puts out the shape with the will, the operation of stopping is natural.

  In addition, the calculation process can be reduced because the recognition process only needs to be performed when a stationary state is found. In addition, a motion that is moved without the intention to present a gesture is not erroneously recognized as a gesture presentation unless it is stationary.

  On the other hand, the direction gesture can be performed by extracting a hand movement direction from an image obtained from such an image input device. For example, when the image input device disclosed in Japanese Patent Laid-Open No. 10-177449 is used, the background is not captured in the image, and only the shape of the hand is captured. Therefore, the position of the hand can be obtained by calculating the center of gravity of the image. By tracking the trajectory, it is possible to detect basic movement.

  When the center of gravity of the entire hand is calculated when moving only the fingertip and hardly moving near the wrist, the accuracy of the detected motion may be poor. In such a case, it is conceivable to perform processing in consideration of the shape of the hand.

  For example, when a finger is stretched toward the camera, the tip of the finger is closest to the camera, and therefore the reflected light is the largest, and is often the maximum point in the image. By detecting these points and calculating the closest point or the average of local maximum points, the position information is used as an evaluation value that is more sensitive to actual movement than the center of gravity of the entire hand. Can be obtained. In this case, if the correspondence between image frames of multiple local maxima is obtained, the behavior of the position becomes strange when the local maxima that existed at the edge of the image go out of the image. Can be prevented.

  In any case, it is preferable to detect a movement gesture by extracting position information from the obtained hand shape data and analyzing the locus. Even when a normal CCD camera is used, if the shape of the hand can be properly cut out by color information or the like, the position information can be extracted therefrom. A method for detecting the direction gesture will be described in detail later in the embodiment.

  <Operation State Transition> FIG. 4 shows a state transition diagram in a configured operation using these two types of gesture detection. First of all, nothing is done. Here, the first gesture detection unit detects it. Presenting any gesture will detect it and enter one of the operating modes. For example, when the gesture 1 is presented, the mode for operating the car navigation view is entered, when the gesture 2 is presented, the operation mode for the air conditioner is entered, and when the gesture 3 is presented, the mode for the audio operation is entered. Within each operation mode, several parameters can be adjusted by directional gestures in four directions.

  For example, in the audio operation mode, the volume is increased by presenting an upward gesture, and the volume is decreased by presenting a downward gesture. In addition, the next song such as a music CD can be selected by presenting the gesture in the right direction, and the previous song can be selected by presenting the gesture in the left direction. Even if only the audio is considered, there are various operation objects such as music CD, music MD, cassette tape, radio, and TV depending on circumstances. At first, the gesture operation 3 is presented to enter the audio operation mode. At that time, the operation of the currently moving object can be performed.

  For example, if a music CD is playing, the CD can be selected and the volume can be adjusted. This operation target can be switched by repeatedly presenting the gesture 3. For example, music MD is switched twice when repeated, and radio is switched when presented again. At this time, the change parameter by the direction gesture may change depending on the operation target.

  For example, in the case of CD and MD, the next song and the previous song are selected by the right and left direction gestures, but in the case of the radio, the next station and the previous station are selected in the order in which the radio stations are registered. Since the volume adjustment is frequently used for any operation target, the volume adjustment is performed in any case for the up / down direction gesture.

  If the association between the shape gesture and the operation target is remembered, the driver can select the operation mode without looking at the operation panel. In addition, the operations in the selected operation mode are very easy to remember, such as the volume of the volume, using the up / down direction gesture, so the entire operation panel can be operated while keeping the line of sight forward. Can be done without looking.

  For example, when using car navigation, a map is often displayed on the screen, but if you want to operate the audio at this time, the audio operation panel is displayed immediately by presenting the display gesture. The Normally, the main menu is displayed once, and an audio operation panel is displayed only after an audio operation item is searched from the main menu and selected.

When compared in this way, the effect of being able to quickly access frequently used functions by gesture presentation is great.
<Gesture Icons> The operations using these gestures are very simple, and frequently used shape gestures can be easily stored. In addition, movements and effects (such as raising and lowering the volume) are well associated with direction gestures, making it easy to remember. However, the user who has just started using it, or after having not used it for a long period of time, may forget what kind of gesture was performed and what could be controlled. In such a case, it would be very inconvenient if you had to take out the instruction manual and read it.

  Here, by further displaying an icon displaying a gesture on the screen at the same time, it is possible to easily know a gesture to be performed even if the user forgets the operation. This is shown in FIG. In this figure, an icon 9 indicating the shape of a gesture and an operation that can be performed when the gesture is presented are displayed at the bottom of the screen. In this way, if you can display the shape gestures you can perform and what you can do with them, if you forget, you can immediately remember by looking at this screen.

  Looking at the gesture icon on this screen for each operation is the same as looking at the operation panel, which will distract the driver's attention, but in this case it is only safe to see when you forget the operation, so Sex is preserved. Rather, it is more dangerous to forget the operation and see the instruction manual.

  <Voice Feedback> When operating without looking at the screen, sometimes it becomes difficult to know what operation mode is currently in operation and what can be operated now. This can happen by being distracted by a little bit, unlike the forgetting gestures described above. For example, when presenting gestures for audio manipulation when switching from radio to CD, suddenly hit your favorite hit song or suddenly talk in the middle of the operation. Likely to occur.

In order to deal with such a situation, the present invention adds a voice feedback function that informs the user of the current situation using voice according to the situation.
For example, in the above-described example, after presenting an audio operation gesture, when something is distracted and it is not understood, it is fed back with a voice saying “It is an audio operation”. Alternatively, the audio feedback may be changed according to the situation, such as “Radio operation” if the radio is currently active, or “CD operation” if the CD is active. Audio feedback may be performed each time an operation is performed.

  However, as you become accustomed to the operation, the audio operation gesture and the subsequent direction gesture (such as increasing the volume) are often presented in succession, which is bothersome when voice feedback occurs each time. Therefore, when there is no next operation for a while after the audio operation gesture is presented, it is desirable to determine that the user has forgotten the situation and return voice feedback. Similarly, when a certain time has passed after returning the voice feedback, what may be operated by the direction gesture may be returned by the voice feedback. For example, the voice is fed back in a manner such as “Adjust volume at the top and bottom, select the channel at the left and right”.

  FIG. 6 shows this mechanism in a block diagram. The operation input processing unit 10 starts the timer 12 when a shape gesture is input. The timer 12 notifies the operation input processing unit 10 when a predetermined time has elapsed, and accordingly, the operation input processing unit 10 instructs the audio feedback unit 13 to perform audio feedback.

  FIG. 7 is a flowchart showing this process. In the initial state, only the shape gesture is detected (S14). When the shape gesture is detected (S15), the operation mode is changed (S16), and the timer is started here (S17). Subsequently, the detection of the direction gesture (S18) is started. In this loop, it is determined whether or not a certain time has elapsed since the timer was started (S20), and if it is determined that the time has elapsed, voice feedback is generated (S20). S19).

  In this flowchart, the path to return to the initial state for detecting the shape gesture is omitted, but this can be configured to return after a certain period of time or when a specific gesture is performed.

  <Sound feedback> It has been stated that it is effective to feed back the current state with voice when in doubt, etc. In addition to this, it is more effective to feed back with sound when an operation is executed. . In the conventional operation panel with buttons, the operator knows that he / she touched the button, knows whether or not the button has been pressed properly, and the volume changes. It is possible to know that the operation was actually performed by actually feeling an effect such as a song change.

  However, when the operation is performed by gesture, there is a problem that the operation feeling is poor because no one touches it. The same thing can be done, for example, in a computer.

  For example, when starting an application by double-clicking the desktop icon with the mouse, it usually takes time to load the program from a storage device such as a hard disk and display it on the screen. Does not start instantly. Therefore, after double-clicking, it is uncertain whether or not the operation was successful until the startup screen is actually displayed.

  In many computers and software, in order to deal with this problem, when the program is started by double-clicking, the display of the mouse cursor is immediately changed to indicate that the program is being started. For example, the arrow-shaped cursor changes to an hourglass shape, indicating that it is preparing for activation. This fact shows that some feedback is important.

  On the other hand, considering that the driver operates the in-vehicle device while driving, it is not appropriate to perform feedback by a change on the screen such as changing the shape of the cursor in this way. This is because attention must be diverted from the front to confirm feedback. Returning feedback by sound can be confirmed without the driver's attention being distracted from the front, and is particularly effective for operation input devices using gestures such as the present invention. Since feedback by voice takes time, it feels troublesome when asked every time, but since feedback by sound is instantaneously performed, there is no annoyance, but there is an effect that it is possible to confirm that the operation is completed and to be relieved.

  Furthermore, if the sound is changed depending on the type of gesture, it can also be determined whether or not the correct gesture has been performed. For example, when a “view” gesture is presented, a “beep” sound is fed back. When an “air conditioner” gesture is presented, a “pu” sound is fed back. When an “audio” gesture is presented, a “pop” sound is fed back. Then, even if you intend to select the “audio” operation, if you hear a beep, you can know that the operation was not performed correctly.

  In practice, users may rarely remember clearly that the “audio” and “pop” sounds are related, but they always hear the “pop” sound for the “audio” gesture. If you have a different sound, you will feel uncomfortable. Therefore, it is possible to know that “it was not done correctly” without knowing exactly what went wrong.

  In addition, it has been stated that when performing shape recognition, it is desirable to perform shape recognition processing only when a stationary state is detected. When performing such processing, a sound is generated when a stationary shape is found, and a sound is generated again according to the gesture when a gesture is detected. For example, when a stationary state is detected and shape recognition is started, a beep is sounded, and when the shape detection is successful, any of the above-mentioned beeps, pops, or pops sounds and detection fails. Then you will hear a “boo” sound. This tells you that recognition has failed, and lets you know when a gesture has been recognized without intention.

  Note that whether or not it is better to sound in two stages in this way depends on how long the recognition process takes. For example, if the recognition starts when the stationary state is met for 0.2 seconds and the recognition process ends in 0.1 seconds, the gesture is detected 0.3 seconds after the operator presents the shape. In this case, two-stage sound presentation may not be necessary. On the other hand, if recognition is started when the stationary state is met for 0.2 seconds and the recognition process is completed in 1 second, it is not known whether a gesture has been detected until 1.2 seconds have passed since the operator presented the shape. In this case, two-stage sound presentation is effective.

  Expressions such as “pip”, “puff”, and “po” described in this example merely indicate that the type of sound is different, and there is no limitation as long as the sound is short. However, it is desirable that the sounds produced by the three shape gestures differ to a certain extent so that the difference can be easily recognized. For example, the sound when recognition fails is relatively unsuccessful in terms of habit and culture, such as “boo”. It is desirable to use a sound that is easy to connect to.

  <Other Gestures> In this embodiment, the first gesture detection unit detects a shape gesture, and the second gesture detection unit detects a direction gesture. However, the embodiment is not limited to this, and two groups of gestures belonging to different categories may be selected from all types of gestures, and the operation mode selection and parameter adjustment may be performed using them.

  As a special example, the case where the first and second gesture detection units detect the same type of gesture is also included. For example, there is a method of first selecting four operation modes with a direction gesture and then changing parameters with the direction gesture.

In this case, when the same direction gesture is detected, the determination as to which gesture is to be controlled is determined by whether or not the current operation mode is selected.
If the operation mode is not selected, it is determined that the gesture is a gesture for selecting an operation. If the operation mode is already selected, it is determined that the gesture is a gesture for changing a parameter.

  In the above-described example, the direction gesture is determined only whether there is a gesture in either direction. Instead, it is possible to change the value continuously by moving the hand up, down, left and right. For example, in the above-described embodiment, each time an upward gesture is performed, the volume gradually increases, and when it is desired to greatly increase the upward gesture, the upward gesture must be presented many times. When this is moved up and down the hand, the volume changes accordingly. Then, when it is left stationary for a certain period of time or when movement in the left-right direction is detected, it is fixed at that volume. This method is also suitable for those who want to change parameters quickly.

  Moreover, although only the vertical and horizontal directions have been described, movement in the depth direction can also be used. When the image input device invented in Japanese Patent Laid-Open No. 10-177449 is used, the intensity of the reflected light depending on the object distance can be obtained as an image, so that the position in the depth direction can be obtained stably. Suitable for usage.

  <Rotation Gesture> As another example, a rotation gesture can be used instead of a direction gesture. That is, by presenting the shape of a hand having a knob and moving the hand by turning the knob, the amount of rotation of the movement is detected, and this is set as a parameter. The feeling of turning the tone adjustment knob can be realized.

  <Directional Gesture Input Method> Now, the direction gesture input method, which is an element of the above-described embodiment, will be described in detail. There is an important point in realizing the input of direction gestures. That is, gestures in the same direction can be repeatedly input, and it is important that they can be detected stably at that time. For example, in the hand gesture input device proposed in Japanese Patent Application No. 9-199647, input is performed by a hand crossing the front of the device.

  However, when such gestures are repeated in the same direction, malfunction may occur. That is, for example, when inputting “right” and “right” repeatedly, the hand must once return to the left side between two rightward gestures. At this time, this is erroneously recognized as a left direction gesture, and the intention to input “right” and “right” will be input as “right”, “left”, and “right”. To prevent this, when returning to the left, it is necessary to return the hand through a position that is not detected by the sensor. If such an operation has to be performed, it would be extremely tedious to repeatedly input a gesture in the same direction.

  <Moving, Returning, and Direction Input> FIG. 8 shows a direction gesture input method for solving this. This is a flowchart showing a process for stably acquiring a direction gesture.

  First, when the user puts his hand in front of the camera sensor and immediately detects motion information, it causes a malfunction. Therefore, first, the user puts his hand in front of the camera sensor and stops it to inform him that a direction gesture will be input. The apparatus detects that the object is stationary for a predetermined time within a predetermined area in the center of the screen (S23, S24), thereby determining that the user is about to input a direction gesture and starts detecting the direction gesture. .

  The locus of the hand position is tracked and it is detected that the hand has moved in a certain direction by a certain amount (S25, S26). The number of directions to be detected is, for example, four for gesture detection in four directions, and eight for diagonal directions including eight directions. Further, there are cases where the motion is detected not only in the vertical and horizontal directions but also in the depth direction.

  If it is detected that a certain amount of movement has been made in a certain direction, it is next monitored whether or not it has moved a certain amount in the opposite direction, that is, in the returning direction (S27). Then, when a certain amount of movement is made in the opposite direction (S28), and it is detected that there is still a fixed time (S29, S30), it is determined that there is a direction gesture (S31). The “stationary” state described here includes a time when the movement is small even if it is not strictly stationary.

  The good point of this method is that, when gestures in the same direction are continuously input, the returning operation during that time is not erroneously recognized as a gesture in the opposite direction. This is because the gesture always starts near the center, and the gesture ends by returning to the vicinity.

  In addition, when it is desired to continuously input direction gestures one after another, since the connection of each gesture is smooth, there is an advantage that it is easy for the operator to work.

  <Sound Feedback> Also, when inputting a direction gesture, it is effective to feed back a sound and inform the operator that the input has been accepted. In the case of FIG. 8, a sound is generated at the time when the detection of the direction gesture is completed in the final stationary state. In addition, if a different sound is generated depending on the input direction, the user can surely know that the correct input has been performed.

  <Feedback with two sounds> When sound is fed back, it may be more effective if sound is generated in two stages. In FIG. 8, when it is detected that a certain amount of movement has been made in any direction, the first sound is played, and after returning, the second sound is played when a direction gesture is detected.

  At this time, the sound that moves in either direction and the sound that is returned when a direction gesture is detected are changed. Then, even if the returning motion is mistakenly recognized as a motion that moves in one of the first directions, the user can immediately recognize it. The first sound may be different depending on the direction, but the returning sound should be the same in any case.

  For example, if the first sound of up, down, left, and right is “pa”, “pi”, “p”, “pe”, and the second sound is all “po”, the user will be given “ You will hear “Papo” “Pipo” “Pupo” “Pepo”. If the operation can be performed smoothly, each of these sounds like a sound effect, and there is no sense of incongruity for the user.

  <Cancel if you don't return> After moving a certain amount in either direction, or while trying to move, you may want to cancel if you notice a mistake or change your mind. If the above method is adopted, if it is desired to cancel, it can be canceled by moving the hand out of the field of view of the camera with the movement.

Alternatively, it can be canceled by stopping the movement on the way. In the flowchart of FIG. 8, the initial state is restored after a certain period of time.
<Returning gesture> In the above example, only the direction gesture has been described, but in general, when information is presented by a position trajectory, a series of gestures “starting from an area and ending at that area” is performed. This is very suitable for continuous input. For example, the gestures as shown in FIG. 9 are all so that not only directions but also various gestures can be input.

In FIG. 9, the arrows indicate the locus of the position.
<Excluding movements when returning due to a difference in distance> With the above-described configuration, it is possible to present continuous direction gestures while preventing the movement when returning from causing a malfunction. However, in some cases, you may still want to make a direction gesture by moving in one direction. When doing such a return movement, it is often returned with a slight pull.

  For example, if you think about stroking the desk twice in the right direction, first boil the surface of the desk once, boil it in the right direction, lift your hand slightly from the desk, return to the left side, and boil it in the right direction again. Usually done. This “slightly lifted state” is a “slightly pulled state” when considered in the air.

  By the way, in the image input apparatus described in JP-A-10-177449, it is possible to acquire relative distance information of the hand. Therefore, it can be detected that the movement to return to the left is performed in a “slightly pulled state” than the movement to the right. A configuration for this is shown in FIG.

  The image input from the image input unit 32 is input to the motion detection unit 34 and the distance detection unit 33, and the motion information 39 and the distance information 38 are obtained respectively. Based on these pieces of information, the determination unit 35 determines whether the motion information input from the motion detection unit 34 is a motion input or a return operation.

  More specifically, immediately after a direction gesture in a certain direction is detected, if a movement in the opposite direction is detected, and the distance information at that time is far from the previous direction gesture, this is an operation for returning. Judge that there is. If it is determined to be a direction gesture, information on the direction gesture is transmitted to the operation execution unit 36, and feedback is generated by sound as necessary. If it is determined that the movement is a return movement, nothing is transmitted to the operation execution unit and no sound feedback is generated.

  With regard to sound feedback, if the sound is changed between right and left so that the return does not generate a sound, the user can immediately notice if the return movement is mistakenly detected as a direction gesture. .

  <Determining Second Movement as Return Action> Even if an image sensor that cannot acquire distance information is used, such processing is not necessarily impossible. In the case of an image sensor that can only acquire motion information, it is possible to determine what the return motion is by looking at the time series of motion.

  For example, if there is an input of “right”, “left”, “right”, “left”, and “right”, this is input three times in the right direction, and it is determined that the two left gestures are return operations. However, with this alone, you cannot change to a left gesture on the way. It can be determined by making good use of the time interval between each movement, the so-called “ma”.

For example, if there are inputs “Right”, “Left”, “Right” (between), “Left”, “Right”, and “Left”, input “Left” and “Left” after inputting “Right” and “Right”. It can be determined that
<Following Menu Hierarchy> So far, gestures for presenting shapes and gestures for inputting directions have been mainly described, but it is possible to follow the menu hierarchy using these gestures. Often, an interface for operating a device having a large number of functions often divides the functions into categories, has a hierarchical structure, and reaches the functions by tracing from the top.

  For example, a computer often uses a method of selecting menus on the screen in order while moving the cursor with a mouse. For in-vehicle devices such as car navigation systems, use the cross button on the touch panel or remote control in order. In many cases, the menu is selected.

  When selecting a menu, it is an effective method to input a direction gesture without using a direction key or to present a shape gesture instead of pressing a function key. FIG. 11 shows a configuration diagram thereof. The image input from the image input unit 40 is processed by the gesture detection unit 41, and when a gesture is presented, it is detected. The menu control unit 42 determines how the menu is traced based on the detected gesture, and causes the menu display unit 43 to display a new menu.

  When a gesture is detected, the feedback unit 44 feeds back to the user with sound or voice. As an advantage at this time, since the operation is a gesture, it is easy to memorize the operation until reaching a certain function. For example, if “Goo”, “Cheap”, and “Par” are presented using a shape gesture, the temperature setting of the air conditioner is set. It is not necessary for the user to remember all of these, and it is only necessary to memorize only frequently used items in the time series of gestures.

  Actually, the time series of gestures is still a gesture, and in this respect, it can be said that this interface associates all functions with gestures on a one-to-one basis. At this time, if the same shape continues, it is difficult to indicate that it continues twice. Therefore, when “par” is issued and the next menu layer is moved, it is convenient that there is no menu presenting “par” in that layer. That is, it is preferable to configure the menu gesture so that the same gesture is not presented twice regardless of which function is selected.

  <Tracing the Menu Hierarchy with Directional Gestures> The method of following the menu hierarchy with presentation of shape gestures has been described, but the same is possible with the use of direction gestures. In the case of using the direction gesture, it is relatively desirable to use the four-direction gesture in consideration of the ease of movement of the human hand. However, the four-way gesture has a drawback that only four menu options can be displayed. There are several ways to solve this.

  <Repetitive Input> For example, different movement options may be selected when moving “right once” and when moving “right twice”. This is suitable for a direction gesture in a series of operations in which the hand is moved in any direction and returned to the original position in the direction gesture input method described above.

  For example, when moving a certain amount in the right direction and moving to the original position and stopping, “Right” is input. However, without moving still, it moves to the right again, and when returning to the original position and stopping, “Right 2” "Is entered.

  If this is repeated, infinite choices can be selected basically, but in practice it is reasonable to limit to two or three repetitions. This is because mouse clicks are usually single-click and double-click, and triple clicks are realized on some professional computers. On the other hand, double click is naturally used even by general users, and there is no problem with continuous input twice.

  When the menu hierarchy is traced by the direction gesture, for example, a menu screen as shown in FIG. 12 can be considered. Since each menu is arranged in the direction of movement, this configuration is easy to see.

  Further, when allowing up to two consecutive inputs, a menu screen as shown in FIG. The inner menu is selected by one direction input, and the outer menu is selected by two direction menus. If three consecutive inputs are allowed, the numeric keypad can also be input with a direction gesture (FIG. 13 (b)).

  <Shape + direction> When inputting a direction gesture, there is also a method of changing the shape of the hand and selecting an option by a combination of the shape and the direction. For example, if the direction is obtained by moving the center of gravity of the entire hand, the direction can be presented regardless of the shape of the hand. At this time, different options can be selected depending on whether the finger is moved to the left with one finger extended and the finger moved to the left with the two fingers extended. Although it is troublesome to repeatedly input the direction, it is relatively easy to increase the options when combining the shape and direction.

  <Start instruction button> Detecting a gesture requires a certain amount of processing cost, and there is always a risk that an unexpected action is picked up as a gesture. These problems do not occur if a gesture presentation button is always pressed when a gesture is presented.

  <Selection by two-dimensional pointing> By mounting an input interface using such a gesture on, for example, an in-vehicle device, a safe operation during driving can be realized. However, these operations include those that are not operated first during driving, and it is not always necessary to fix the line of sight forward when the passenger is operating.

  Therefore, an operation that can be rationally performed while viewing the screen may be realized as such. Instructing by direction has the merit of being able to operate without directing the line of sight, but when selecting from many options, as described above, it is possible to present a repeated direction gesture or use shape and direction together. There was a need to do. In a state where the line of sight can be directed to the screen, a method of selecting using the position of the point is also good.

  At that time, the selection decision input becomes a problem. For example, when selecting an icon on the screen using a mouse, the mouse cursor is moved to an icon and then clicked. When the same thing is realized using hand gestures, how to make a decision input corresponding to a click becomes a problem. It is conceivable to change the shape of the hand to “grab” from par to goo, or to “push” to handle the button and use it as a decision input.

  However, when such an operation is performed, position information fluctuates in the course of the operation, and an adjacent icon is often selected. This is because the operation of “moving the position” and the operation of “changing the shape” or “pushing back and forth” are not independent. This may be difficult for a person to control them independently, or as a characteristic of the device, it may be difficult to separate them.

  In addition, the selection trigger can be set in a stationary state for a certain period of time or more, but it may take a time to rest or may become stationary while in doubt. Realize more stable operation by using two movements that are easy to move independently of human movement and that can be sensed independently for “selection” and “decision”. be able to.

  This will be described with reference to FIG. The image input from the image input unit 46 is transmitted to the first motion detection unit 47 and the second motion detection unit 49, and each detection unit detects independent motion information as described above. The option or value selection unit 48 determines what is currently selected according to the output of the first motion detection unit 47. This result is typically fed back to the operator, such as by displaying it in reverse on a display unit (not shown).

  The determination operation determination unit 50 analyzes the output of the second motion detection unit 49 and determines whether the determination operation has been performed. The operation input processing unit 51 determines what has been selected and determined using the selection or value selection unit 48 and the output of the determination operation determination unit 50, and sends it to the operation execution unit 52 that actually performs processing.

  <Selection and Determination by Two Movements> More specifically, for example, a method of combining left and right movements and up and down movements. Select menus that are arranged side by side in the horizontal direction. The currently selected menu is highlighted, for example, so that it can be understood what is selected. In this state, this is a temporarily selected stage, and options are determined by moving up and down from here.

  In the example above, “select” in the left-right direction and “decision” in the up-down direction has a reason. In general, since a person can indicate the position more finely in the left-right direction, the left-right direction is set as the “selection” direction. Since the vertical movement is only a trigger input, there is no problem even if it cannot be controlled so finely. However, this is a general theory, and the reverse may occur depending on the operation conditions (body posture, etc.), but in any case, it is important to set the direction of fine control to the direction of “selection”.

  For example, FIG. 15A shows an example of a menu screen when selection is made by this method. As shown by an arrow in the figure, selection is made by horizontal movement, and determination is made by vertical movement. . In FIG. 6B, since there are many options, icons for scrolling left and right are arranged. If this is selected, the menu scrolls one after another.

  <Use up and down movement separately> Also, in the above example, the up and down directions are both “decision”, but the menu is made up of two levels and the up and down menus are selected respectively. You can also. This allows you to display twice as many options at once.

In this case, the provisional selection state (for example, reverse display) is selected at the same time by selecting the upper and lower two menus at the same time and inputting “enter”.
For example, FIG. 16 shows a menu screen example for performing this selection method. For example, Ibaraki Prefecture and Yamanashi Prefecture are selected, and if moved up, Ibaraki Prefecture is selected, and if moved down, Yamanashi Prefecture is selected and determined. Here, the “other area” and “return” icons at the end are selected regardless of whether they are moved up or down.

  <Moving up and down> In the examples so far, if the hand is moved up or down and selected, the hand moves up or down, which is not suitable for continuous menu operations. It is easy to make continuous menu selections by moving a certain amount up or down and “deciding” when returning to the original location. Just as in the previous example, it is the same as moving to the right and recognizing it as a “right” gesture when returning to the center.

  <Horizontal, vertical, horizontal and continuous> When selecting characters from a 50-sound table, there are many options to be displayed at a time. In such a case, first select “Line” such as “A Line”, “K Line”, and “S” Line in the horizontal movement, then select one of the five characters in the vertical movement, and then in the horizontal movement You can also choose. Also, once you select a line with a horizontal movement, when you select a character with the next vertical movement, even if it moves slightly in the horizontal direction, you will select a different line again. Must not.

  In this case, once the horizontal direction is determined, it is preferable to perform processing such that the horizontal movement is not detected until the next vertical direction is determined. Of course, in the case of a 50-note table, the cursor can be simply displayed at the top, bottom, left, and right positions for direct selection. In this case, it is determined by the movement in the depth direction or the static information. Furthermore, it is also possible to use depth information, such as changing hiragana and katakana when the depth is changed.

  It is also possible to specify a cloud point, semi-cloud point, etc. in the form of a finger when pointing, which is more efficient than inputting the cloud point etc. independently. For example, if you select with two fingers, it is a muddy point.

  <Mounting position of sensor> When an in-vehicle operation input device is configured using an image input sensor proposed in Japanese Patent Laid-Open No. 10-177449, the installation position of the sensor greatly depends on usability and performance.

  If it is most important that the driver can operate safely while driving, the camera should be placed in a place where the driver can operate with the hand on the handle or with little hand off the handle. Good. For example, it is an example to arrange in the back side of the handle so as to face the hand holding the handle.

  In addition, the normal operation is not performed when turning the curve, but it can be assumed that the operation is performed when the handle is slightly turned, such as a gentle curve. Then, it can be directly attached to the handle on the back side of the handle. Then, irrespective of the state of the handle, there is no change in the positional relationship (distance, orientation) between the hand and the camera, and stable detection performance can be obtained.

  In these cases, since the image input often includes parts such as a handle, processing is performed so as not to be affected by these. At this time, when processing an image, it is easier to obtain the vertical and horizontal movements of the image than to detect the diagonal movements. It is important to set the orientation.

  In addition, when operating not only the driver but also the passenger seat, it is desirable to arrange them at an intermediate position between them. In this case, it is necessary to arrange the driver at a position where the driver can easily reach his / her hand. Further, when performing a gesture operation, particularly when a position is indicated, a stable operation can be performed by fixing a part of an arm such as an elbow or a wrist or placing it on a table or the like.

  Therefore, it is important to place a table on which an elbow or wrist can be placed at an appropriate place. This also serves to determine the approximate location of the operating hand and is important for ensuring stable performance. For example, a shape like 53 in FIG. Here, the camera 54 is arranged in an oblique direction, and a depression is provided so that the hand does not approach the camera so much.

  Further, since the image input device invented in Japanese Patent Laid-Open No. 10-177449 has its own light emitting means, it can capture an object shape regardless of the surrounding brightness. However, if there is extremely bright outside light, such as direct sunlight, it can be affected. In particular, when a strong external light strikes a portion illuminated by the light emitting means of the image input device, the image is likely to be deteriorated.

  Therefore, disposing the sensor in a positional relationship such that strong external light does not strike the surface of the hand being imaged by the image input device has an effect of improving performance and reducing the cost of the optical BPF to be used.

  For example, FIG. 17 (b) shows a cross-sectional view of the shape of FIG. 17 (a), but the palm 55 faces downward, so that strong external light is less likely to strike here. It is also effective to have means for shielding outside light to such an extent that the ease of operation is not impaired.

  <Recognizing Driver and Passenger Seat> Some advanced information devices such as car navigation are dangerous if the driver concentrates on the operation, and there are operations that should not be performed while driving. For example, an operation of registering a destination is very complicated and cannot be performed during driving. Therefore, normally, the input of the vehicle speed sensor is used, and these operations cannot be performed during traveling.

  However, although it is not inherently dangerous for the passenger seat to operate even while traveling, it cannot be operated at present. If it is possible to detect whether the person who is currently performing the operation is a driver or a passenger seat, it is possible to prohibit the operation of only the driver. If an image sensor is used to capture the image of the entire operating person, or if not the entire image, up to the upper arm level of the operating hand, it will extend from either the driver seat or the passenger seat. Whether it can be judged.

  FIG. 18 is a block diagram of an apparatus for realizing this. The operator determination unit 58 uses the input image from the image input sensor 57 to determine whether the operator is a driver or a passenger. The result is sent to the operation input processing unit 60. If the function cannot be operated, the execution is rejected and the operator is informed that the operation cannot be performed.

  <Combination with Voice Recognition> Similar to the input by gesture described in the present invention, there is an input device using voice recognition as a device that allows the driver to operate safely while driving. In this method, voice generated by the driver is detected by a microphone, the voice data is analyzed, a user's intention is determined, and the operation is linked. A device using voice recognition has the merit that it can be used by anyone because it is easy to learn commands and can speak, but there is a problem that there are many misrecognitions that must always be spoken when operating. By combining the gesture input device of the present invention and voice recognition, a more manageable system can be constructed.

  For example, since the selection of a function can be performed immediately without following the hierarchy when using speech, speech recognition is used, and parameter input is performed using an intuitive gesture. For example, after “volume”, the volume is dynamically adjusted by moving the hand left and right (or up and down or in the depth direction). When I thought it was just right, I decided to say “OK”.

  Alternatively, voice input can be used as a decision input when a position instruction is given. Furthermore, a plurality of executable operations can be instructed by voice input on the object selected by the gesture.

  For example, a restaurant displayed on a map can be selected by a gesture, and voice commands such as “telephone”, “information”, and “route” can be issued. When you say “telephone”, you call the restaurant, when you say “information”, information about the restaurant is displayed, and when you say “route”, the route search to that restaurant is started.

  <Contact plate> Although it has already been stated that more stable operation can be performed by placing an elbow or wrist on a stand, etc., it is also significant to prepare a stand on which the hand or finger to be detected can be placed. There is. Such a configuration is shown in FIG. Here, an image input device described in JP-A-10-177449 is arranged as an image sensor.

  A transparent or translucent plate 56 is arranged between the operator's hands. If anti-reflection processing is applied to this, unnecessary reflection is eliminated and a stable image can be obtained. Alternatively, the plate may have an optical bandpass characteristic that allows only the wavelength of light from the light emitting means provided in the image input device to pass therethrough. In the case of a translucent plate, the amount of light passing therethrough changes, so that the operation is basically the same as when there is no plate except that it affects the sensitivity. The effect by having a board is as follows.

  First, the operator always operates by touching or not touching the plate, so that the distance between the hand and the sensor camera is kept almost constant, and there is an advantage that stable detection is possible. In addition, this is particularly noticeable when the position is indicated, but it is easier for a person to rest on a board than to rest his finger in the air.

  Further, when inputting a direction gesture, a difference in distance can be surely expressed by touching the board slightly when moving in the indicated direction and not touching when returning. In addition, since the detection distance when touching the plate is fairly stable and fits within a small width, it can be determined by a distance value (actually, distance information converted from the amount of reflected light). .

  Thus, for example, the shape gesture can be expressed by the number of fingers touching the board with one finger and touching the board with two fingers. Further, since the shape of the entire hand is also detected, it is possible to detect which finger is touched, and the number of shape gestures can be easily increased.

  Further, in this figure, the position of the camera is grasping the hand from the diagonal direction of the finger of the hand, but this position is not limited to this, for example, it may be directly under the hand or from the diagonal direction of the wrist. Good. However, since the shape of the fingertip is often a little important, it is possible to perform stable detection by imaging from this direction.

  <Shape detection> If it is intended to operate with only the touched shape, it can have a different structure. As an image sensor, a CCD camera is arranged, and there is a diffusion plate that diffuses light from the outside. Normally, external light is incident on the camera through the diffuser, so the camera always captures a dimly bright image. When the operator places his / her hand or finger on the diffuser plate, light does not reach from that portion, so an image is taken as a black portion.

  By analyzing this image, it is possible to detect information indicating how many fingers are touching or in which direction the touched finger has moved. Alternatively, if it is not touched but is very close, it is captured as a gray close to black, which can also be used. When the surroundings are dark, external light cannot be used. In this case, the same operation can be performed by providing an auxiliary light source outside.

  Further, the processing in the embodiment of the present invention can be realized by a computer-executable program, and this program can be realized as a computer-readable storage medium.

  <Recording medium> The storage medium in the present invention includes a magnetic disk, floppy (R), hard disk, optical disk (CD-ROM, CD-R, DVD, etc.), magneto-optical disk (MO, etc.), semiconductor memory, etc. As long as the storage medium can store the program and is readable by the computer, the storage format may be any form.

  Further, an OS (operation system) operating on the computer based on an instruction of a program installed in the computer from the storage medium, database management software, MW (middleware) such as a network, and the like for realizing the present embodiment A part of each process may be executed.

  Furthermore, the storage medium in the present invention is not limited to a medium independent of a computer, but also includes a storage medium in which a program transmitted via a LAN or the Internet is downloaded and stored or temporarily stored.

  Further, the number of storage media is not limited to one, and the processing in the present embodiment is executed from a plurality of media, and the configuration of the media may be any configuration included in the storage media in the present invention.

  The computer according to the present invention executes each process according to the present embodiment based on a program stored in a storage medium. The computer includes a single device such as a personal computer, and a system in which a plurality of devices are connected to a network. Any configuration may be used.

  The computer in the present invention is not limited to a personal computer, but includes a processing unit, a microcomputer, and the like included in an information processing device, and is a generic term for devices and devices capable of realizing the functions of the present invention by a program. .

  <Summary> According to the first invention, operations are performed using two different types of gestures among gestures such as the shape and movement of a human body or a part of the human body, and the operations are performed using the first type of gestures. It is possible to realize an operation input device that can select a mode and change parameters without using the operation mode with the second type of gesture.

  According to the second aspect of the present invention, it is possible to realize an operation input device that is extremely natural for a user, in which an operation mode is selected by a shape gesture of presenting a shape, and a parameter is changed by a direction gesture indicating a direction.

  According to the third invention, one shape gesture is associated with one or a plurality of similar operation modes, and when the shape gesture detecting means continuously detects the same shape gesture, the shape is detected. Since multiple operation modes associated with gestures can be selected in order, when there are multiple similar operation modes, all of them are assigned different gestures without taking a heavy burden on the user. Many similar operation modes can be selected.

  According to the fourth invention, the shape gesture detection means starts detecting the shape when detecting that the movement of the human body or a part of the human body is stationary or small in movement for a certain time or more. It is possible to prevent an unintended operator's movement from being erroneously recognized as a shape gesture. In addition, since the shape recognition process is performed only when stationary, the processing cost can be reduced.

  According to the fifth aspect, the display means displays the selectable operation mode and the image representing the shape gesture for selecting the same, or the changeable parameter and the image representing the direction for changing the same. Therefore, when the operator forgets the gesture, the user can know the type of the gesture without having to bother opening the instruction manual.

  According to the sixth aspect of the invention, information such as a currently selectable operation mode or a changeable parameter is presented by voice, so that the operator can know the current operation status while keeping his eyes on the front.

According to the seventh aspect, since voice presentation is performed when gesture presentation by the operator is delayed during the operation, there is no inconvenience that voice presentation is always performed.
According to the eighth invention, some or all of the fact that the shape gesture detection means has started detection, the detection has been completed, the detection has failed, and the direction gesture detection means has detected a direction. Since it has sound presenting means for presenting to the operator with sound when it occurs, it is possible to know whether or not the gesture made by the user is correctly recognized without looking at the screen.

  According to the ninth aspect, since different sounds are presented depending on the type of operation or the type of detected direction, it is known when a gesture different from the direction and shape to be input has been detected. be able to.

  According to the tenth aspect of the present invention, there is provided a method for detecting a direction gesture by analyzing a locus of position information corresponding to a position of a person's body or a part thereof, a person's body, or a person's body. First, when the object moves in one of a plurality of predetermined directions by a predetermined range, and then moves to the opposite side by a predetermined range. By using the direction gesture detection method, which is detected as a direction gesture in the direction of movement, when a gesture in the same direction is repeatedly presented, the "return" action that occurs between the gestures in the opposite direction and It is possible to prevent erroneous recognition. Moreover, since it always returns to the same place and a gesture is complete | finished, it is suitable for repeating a direction gesture.

  According to the eleventh aspect of the present invention, there is provided a method of detecting a direction gesture by analyzing a locus of position information corresponding to a position of a person's body or part thereof, a person's possession, or a person's body. Then, after the object stays stationary or in a state where there is little movement for a predetermined range within a predetermined range, it is determined in any one of a plurality of predetermined directions. As a directional gesture in the direction of the first movement when moving by the amount of the range and then moving to the opposite side by the amount of the determined range and staying still or less moving again for the determined range of time By using a direction gesture detection method characterized by detection, it is possible to prevent an unintended movement of the operator from being erroneously detected as a direction gesture.

  According to the twelfth invention, when the direction gesture is detected, the sound presenting means for presenting the detection by sound is provided, so that it can be confirmed that the direction gesture is correctly detected.

  According to the thirteenth aspect, since a different sound is presented depending on the direction of the detected gesture, it can be noticed when a direction different from the direction to be input is erroneously detected.

  According to the fourteenth aspect of the present invention, when one moves in the first direction, it presents one sound, moves back to the opposite side, and presents another sound when detected as a direction gesture. It is possible to reliably perform movement and return movement.

  According to the fifteenth aspect, the first sound presents a different sound depending on the direction of movement, and the second sound presents the same sound regardless of the direction gesture, so that it is always the same when the direction gesture is completed. I hear something. Therefore, when the situation in which the initial movement and the return movement are detected in reverse, it can be immediately noticed.

  According to the sixteenth aspect of the present invention, there is provided a method for detecting a gesture by analyzing a locus of position information corresponding to the position of a person's body or a part thereof, a thing held by a person, or a thing worn by a person. Thus, a gesture detection method is used, which is characterized by discriminating between an action that a person intentionally moves and an action that occurs unintentionally. As a result, when performing gestures in the air, an action that is effective for judgment by identifying an action that is intentionally performed by a person and an unintended action that occurs between the intended actions Can only be taken out.

  According to the seventeenth aspect of the present invention, there is provided a method for detecting a direction gesture by analyzing a locus of position information corresponding to a position of a person's body or a part thereof, a person's possession, or a person's body. A direction gesture having means for detecting movement in any one of a plurality of predetermined directions, and means for removing a "return" action that occurs between successive same direction gestures By using the detection method, it is possible to prevent erroneous detection of a “return” operation that becomes a problem when a direction gesture is indicated by movement in one direction.

  According to the eighteenth aspect of the present invention, there is means for detecting the distance of the moving object from the sensor, and when a movement in which the distance in the opposite direction is far is detected during a continuous movement in one direction, it is “return”. Therefore, when the operator performs the “return” operation, the “return” operation can be removed by moving the hand away from the camera sensor.

  According to the nineteenth aspect of the present invention, there is provided means for measuring a time interval between the movements, the movements in the opposite direction are detected during the continuous movements in a certain direction, and a predetermined time is set between them. If there is only a time interval that is less than or equal to the specified length, it is determined that the movement during this time is a `` return '' movement and is removed, so when presenting the same direction gesture, operate so that there is no time interval. When a reverse gesture is presented, it is possible to perform a stable operation without the “return” operation by temporarily stopping the gesture.

  According to the twentieth invention, different sounds are presented to the detected direction gestures depending on the directions, so that it is possible to know when the direction gesture is recognized without removing the “return” operation. .

  According to the twenty-first aspect, there is a method for detecting a gesture of position information by analyzing a locus of position information corresponding to the position of a person's body or part thereof, what a person has, or what a person is wearing. Thus, a trajectory in which the start position and the end position are close to each other is detected as a gesture, which is suitable for repeatedly inputting various gestures.

According to the twenty-second aspect, an operation input device using the direction gesture detection method as described above can be realized.
According to the twenty-third invention, a shape gesture detecting means capable of detecting the shape of a human body or a part thereof, a menu display means for displaying a menu, and a menu corresponding to the detected gesture are displayed. An input device having a menu selection means for selecting, wherein a gesture is associated with a menu so that a shape gesture of the same shape is not presented when hierarchically following the menu. By realizing the operation input device, it is possible to perform menu selection across layers by combining gestures as if one gesture was presented.

  According to the twenty-fourth invention, the direction gesture detecting means capable of detecting the direction of movement of a human body or a part thereof, the menu display means for displaying a menu, and the detected gesture are associated with each other. An input device having menu selection means for selecting a menu, and when direction gestures are continuously presented, a direction menu is simply used by selecting a different menu from that presented once. Sometimes many menus can be displayed.

  According to the twenty-fifth aspect of the present invention, the shape gesture detecting means capable of detecting the shape of the human body or part thereof, and the direction gesture detection capable of detecting the direction of movement of the human body or part thereof. An input device having a menu selection means for selecting a menu in association with a detected gesture and a menu selection means for selecting a menu in association with the detected gesture, and a detected shape gesture and a shape gesture detected at the same time In combination with, a menu to be selected is determined, so that a large number of menus can be displayed even when simply using a direction gesture.

  According to the twenty-sixth invention, using the information on the position and movement of a person's body or a part thereof, what a person has, or what a person wears, one of the items arranged is selected, or A method for setting a value, which can be easily controlled independently by a person, or can be easily detected independently by a detection means. The operation input method is characterized in that selection is made using the other and determination is made using the other. Thus, when the screen can be viewed, items or values can be input quickly. Since these two types of movements are highly independent, it is possible to prevent the occurrence of detection errors by affecting the operation of each other.

  According to the twenty-seventh aspect, the item or value is selected by moving in one direction, and the determination is made by presenting the movement in a different direction. Therefore, the selection operation can be performed using only the movement information. Since the operation corresponding to the click need not be indicated by a change in shape, the operation can be performed more stably.

  According to the twenty-eighth aspect, the direction in which the person can control the position more finely or the direction in which the detection means can detect the position more accurately is used as the selection direction. It is possible to provide a device that is easy for a person to operate.

  According to the twenty-ninth invention, when presenting a determination direction gesture, it is possible to display and select more items by selecting and determining different items or values according to two opposite directions. It becomes possible.

  According to the thirtieth aspect, when presenting the decision direction gesture, the decision direction is moved when it moves in the direction of the decision by the amount of the predetermined range and returns again by the amount of the predetermined range. Therefore, when this item selection is repeated, it can be operated efficiently.

According to the thirty-first aspect, an operation input device using the operation input method of the twenty-sixth to thirtieth aspects can be realized.
According to the thirty-second aspect, by providing a place for placing a part of the arm such as an elbow or a wrist, the operation can be performed more stably and easily.
According to the thirty-third invention, the first to ninth, twelfth to fifteenth, twentieth, twenty-second to twenty-fifth, thirty-first invention for detecting the position and shape of the target object using optical means. This is an operation input device that is devised in direction and arrangement so that strong external light does not strike the object surface on the image input device side. Stable performance can be exhibited even by using optical means that are easily affected by the above.

  According to a thirty-fourth invention, the operation input device according to the first to ninth, twelfth to fifteenth, twentieth, twenty-second to twenty-fifth, and thirty-first inventions used in an in-vehicle device, the driver Since the operation input device can be realized so that it can be operated with little or no release from the handle, the driver can operate more safely during driving.

  According to a thirty-fifth aspect, in the operation input device according to the first to ninth, twelfth to fifteenth, twentieth, twenty-second to twenty-fifth, and thirty-first inventions, An operation input device is used, characterized in that it has an operator determination means for determining whether the user belongs to, and has means for changing the operations that can be performed according to the category to which the operator belongs. As a result, the operation can be permitted or prohibited depending on conditions such as the state and position of the user.

  According to a thirty-sixth aspect of the invention, the operation input device according to the first to ninth, twelfth to fifteenth, twentieth, twenty-second to twenty-fifth, and thirty-first aspects of an operation input device used in an in-vehicle device. It has an operator determination means that determines whether the driver is a driver or the other, and if the operator is a driver, has a means to control that some operations cannot be performed under certain conditions Therefore, even if an operation that should not be operated by the driver during traveling is possible, it can be configured so that it can be operated by a passenger.

  According to a thirty-seventh aspect of the invention, the operation input device according to the first to ninth, twelfth to fifteenth, twentieth, twenty-second to twenty-fifth and thirty-first aspects of the invention, and further capable of giving instructions by voice. Since it has a means, it is possible to realize an environment that is easier to operate by using a simple input method by voice and a gesture input suitable for parameter input.

  According to the thirty-eighth aspect, the operation input device according to the first to ninth, twelfth to fifteenth, twentieth, twenty-second to twenty-fifth, and thirty-first aspects of the invention further prevents movement unintended by the user, By using an operation input device characterized by having stabilization means for stabilizing the movement of the hand or body, a more stable operation can be realized.

  According to a thirty-ninth aspect of the present invention, the stabilizing means is a contact plate for a user to operate by touching a hand or body at the beginning or end of an operation or during the operation. By using the operation input device, the operator can input the position more stably by touching the board, and can feel assured that the operation is performed by touching. Further, since the distance is kept constant, more stable detection performance can be realized.

  According to the fortieth aspect, the contact shape detecting means capable of detecting the shape of the part touched by the hand, the shape gesture detecting means for detecting the contact shape as a shape gesture, and the direction gesture detection for detecting the direction of movement. An operation mode selection unit that selects one of a plurality of operation modes in which a specific operation can be performed according to the detection result of the shape gesture detection unit, and the operation according to the detection result of the direction gesture detection unit. By configuring an operation input device having parameter changing means for changing parameters that can be changed in the mode, it is possible to realize an operation input environment that operates only when it is operated with certainty. With this configuration, a stable operation can be realized even with a normal CCD camera.

  According to a forty-first aspect of the invention, the operation input device of the first to ninth, twelfth to fifteenth, twentieth, twenty-second to twenty-fifth, and thirty-first aspects of the invention is provided. By using an operation input device that operates synchronously and has means for imaging only the object reflected light of the light emitted from the light emitting means, the shape of the object can be obtained stably and at high speed. Therefore, gesture input can be realized at a low processing cost.

According to the forty-second invention, a vehicle including the operation input device of the first to ninth, twelfth to fifteenth, twentieth, twenty-second to twenty-fifth, and thirty-first to forty-first inventions can be realized.
According to the forty-third invention, shape / movement information of a human body, a part thereof, or an object such as an object is detected, and one of a plurality of operation modes is selected based on the detection result, and then the same By using an operation input method characterized by detecting different types of shape / motion information of the target object and using the detection results to change the changeable parameters in the operation mode. A device that can be operated with a simple feeling can be configured.

  According to the forty-fourth invention, a storage medium storing a program for executing the gesture input method or the operation input method of the tenth, eleventh, sixteenth to nineteenth, twenty-first, twenty-sixth to thirty-third, or forty-third invention Can construct an operation input environment as described in the embodiment in an apparatus on which a computer is mounted.

1 is an overall configuration diagram for realizing the present invention. It is a more concrete whole block diagram which implement | achieves this invention. It is an example of the shape gesture used by this invention. It is a transition diagram of the operation using the shape gesture and the direction gesture used in the present invention. It is an example of a display of the gesture icon used by this invention. It is a block diagram for performing the audio | voice feedback used by this invention. The flowchart of the process which performs audio | voice feedback when the operation used by this invention has stagnated. It is a flowchart which shows the detection method of the direction gesture used by this invention. It is an example of the gesture with which the position of the beginning and the end used by this invention is near. It is a structural example which detects the distance used by this invention, and removes a return operation | movement. The structural example of the apparatus which can trace a menu hierarchy with the gesture used by this invention. It is an example of a screen at the time of selecting a menu with a direction gesture used in the present invention. It is an example of a screen which selects a different menu by repetition of the direction gesture used by this invention. It is a block diagram of the apparatus which performs selection and determination by two types of independent motion information used by this invention. It is an example of the screen of the apparatus which performs selection and determination with two independent types of motion information used in the present invention. It is the example of a screen in the structure which further divided the movement which performs a decision, and increased the number of choices. It is an example of the external appearance of the apparatus used by this invention. It is a block diagram of the apparatus which determines an operator and gives permission / non-permission of operation adaptively.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... 1st gesture detection part 2 ... 2nd gesture detection part 3 ... Operation type selection part 4 ... Parameter change part 5 ... Operation input process part 6 ... Operation execution part 7 ... Shape gesture detection part 8 ... Direction gesture detection part 9 ... Gesture icon 13 ... Audio feedback unit 32 ... Image input unit 33 ... Distance detection unit 34 ... Motion detection unit 47 ... First motion detection unit 48 ... Selection unit 49 ... Second motion detection unit 50 ... Determination operation determination Unit 53 ... Stand 54 on which a part of the arm is placed ... Camera 56 ... Contact plate 58 ... Operator determination unit

Claims (4)

A method of detecting a direction of the object by analyzing a locus of position information of the object to be detected,
After the position of the object stays within a predetermined area for a predetermined time or in a state of little movement, the object moves by a predetermined distance in a predetermined direction, and moves by a predetermined distance in a direction opposite to the predetermined direction, A direction detection method, wherein the direction of movement of the object is detected as a movement in the predetermined direction when it remains in the predetermined area again for a predetermined time or in a state of little movement. A method of detecting a direction of the object by analyzing a locus of position information of the object to be detected,
Within the predetermined region that occurs during a continuous movement in the same direction as the detected direction by detecting whether the position of the object has moved in any direction from within the predetermined region among a plurality of predetermined directions A direction detection method, wherein the direction of a plurality of movements of the object is detected by removing the movement back to step (b).   When the distance to the object is detected, and during a continuous movement in a predetermined direction, a movement in a direction opposite to the predetermined direction detects a movement farther than the movement in the predetermined direction The direction detection method according to claim 2, wherein the direction detection method removes the movement as a return movement.   Measuring a time interval between the movements of the object, and removing the movement that occurs during successive movements in a predetermined direction as a movement that returns when the time interval is equal to or less than a predetermined length. The direction detection method according to claim 2.

Images