JP2015132992A - Gesture input apparatus, gesture input method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve operability of a screen when a user performs gesture input.SOLUTION: A gesture input apparatus to be used for inputting information by gesture includes: a total delay time calculation section which calculates the total delay time due to gesture input, on the basis of at least one of delay time due to the amount of jiggle of a body section of a user who performs gesture, and delay time due to processing of a device in accordance with the gesture or communication; and a screen display section which displays an object on the basis of the total delay time due to the gesture input.

Description

  The present invention relates to a gesture input device, a gesture input method, and a program.

  There are devices that operate in accordance with operations on a GUI (Graphical User Interface) screen by a user and other UIs (see, for example, Patent Documents 1 and 2). Further, there is a demand for development of a UI that can input instructions more intuitively and in a natural sense. For example, as a user interface when operating away from the screen, development of a UI using a space gesture for performing a screen operation by a user's gesture or hand gesture has been proposed (see, for example, Patent Document 3). In the UI based on the space gesture, for example, the input device acquires the position of a part of the body such as a hand by a sensor, and displays a cursor at a position corresponding to the position acquired by the sensor on a remote screen. Operate objects such as icons on the screen with the cursor.

  At that time, a method has been proposed in which an object displayed on the screen is changed in accordance with the hand shake amount measured in advance. For example, when the amount of blur is small, an object on the screen is displayed in a small size to improve the listability on the screen. On the other hand, when the amount of shake is large, an operation error with respect to a user's gesture input is likely to occur if an object on the screen is displayed small. For this reason, when the amount of shake is large, an object on the screen is displayed in a large size in consideration of the operability of the screen.

JP 2010-134912 A JP-A-8-16353 International Publication No. 2012/005005

I. Scott MacKenzie and Colin Ware. 1993. Lag as a determinant of human performance in interactive systems.In Proceedings of the INTERACT’93 and CHI’93 Conference onHuman Factors in Computing Systems (CHI’93). ACM, pp.488-493

  However, there are cases where the amount of shake alone is not sufficient as information for enhancing the operability of the screen when the user inputs a gesture. For example, the shake amount itself varies depending on the accuracy of the sensor used. The accuracy of the sensor varies depending on the performance of the sensor used and the positional relationship between the user and the sensor. For example, if the position of the user and the sensor is close, the accuracy of the sensor is high, and if the position is far, the accuracy of the sensor is low.

  Further, for example, the position where the gesture is sensed is often smoothed by a low-pass filter or the like, and processing for eliminating or reducing the blurring of the cursor displayed at the position corresponding to the gesture is often performed. For this reason, the amount of blur used when inputting a gesture is not always the same as the actual amount of blur. In addition, a delay based on the smoothing process may affect a screen operation error due to a gesture input.

  Therefore, an object of one aspect is to improve the operability of a screen when a user inputs a gesture.

  In one proposal, a gesture input device that performs input using a gesture, a delay time based on a shake amount of a body part of a user performing the gesture, and a delay time based on processing or communication of a device corresponding to the gesture There is provided a gesture input device comprising: a total delay time calculation unit that calculates a total delay time of an input from a gesture from at least one; and a screen display unit that displays an object based on the total delay time of the input from the gesture. The

  According to one aspect, the operability of the screen in the user's gesture input can be improved.

It is the figure which showed the outline | summary of the gesture input device concerning one Embodiment. It is the figure which showed an example of the function structure of the gesture input device concerning one Embodiment. It is a figure for demonstrating the delay time concerning one Embodiment. It is the figure which showed an example of distribution of the position of the cursor concerning one Embodiment. It is the flowchart which showed an example of the delay amount calculation process which concerns on one Embodiment. It is the flowchart which showed an example of the screen display process which concerns on one Embodiment. It is the flowchart which showed an example of the calculation process of the operation difficulty which concerns on one Embodiment. It is the figure which showed an example of the screen display at the time of changing the delay time and distance (positional relationship) of a user and a screen which concern on one Embodiment. It is the figure which showed an example of the hardware constitutions of the gesture input device which concerns on one Embodiment.

  Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, the duplicate description is abbreviate | omitted by attaching | subjecting the same code | symbol.

[Outline of gesture input device]
First, an outline of a gesture input device 1 according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a diagram illustrating an outline of a gesture input device 1 according to an embodiment. Examples of the gesture input device 1 include a terminal having a screen such as a PC or a TV. The gesture input device 1 is a device that can operate a screen of a PC, a TV, or the like by a user operation such as gesture or hand gesture. The user's operation is hereinafter referred to as “gesture”.

  Note that the body part of the user who performs the gesture may be a part or all of the user's body. The gesture may be a movement or direction of a hand, arm, foot, torso, head, line of sight, or the like. The gesture may be a mouth movement or a voice.

  The gesture input device 1 includes a camera 10 and a display 20 and realizes a screen operation by a gesture, that is, gesture input, by software operating on the camera 10 and the display 20. The software part may be realized by hardware that performs an equivalent function.

  The gesture input device 1 does not depend on a specific hardware mechanism. For example, the camera 10 only needs to be able to acquire the position of a part of the user's body. For example, the camera 10 may be a combination of a sensor such as a distance sensor, a monocular camera, and a stereo camera and an object tracking device. A user may wear a terminal capable of acquiring a position using a gyro sensor, an acceleration sensor, an ultrasonic wave, or the like instead of the camera 10.

  The display 20 may be anything that can display a screen, such as a PC monitor, TV, projector, HMD (Head Mounted Display).

  In the gesture input device 1, for example, the position of the hand of the user U is detected by the camera 10. Based on the detected hand position, the cursor 21a is displayed at a position corresponding to the position of the hand on the screen 21 that is away. Then, a GUI operation such as selection of an icon on the screen 21 is performed by the displayed cursor 21a. In this manner, the user U can operate a remote screen by a gesture performed by the user U.

  The gesture input device 1 according to the present embodiment is based on the processing of sensor data acquired by the camera 10 as well as the amount of shaking of the body part of the user who makes the gesture (hereinafter, mainly “hand” is taken as an example). Considering the delay time, a screen with good operability for gesture input is displayed. In the present embodiment, the amount of shake that affects operability is “delay time due to smoothing the shake and sufficiently reducing the amount of shake”. As a result, the gesture input device 1 according to the present embodiment displays a screen that takes into account the difficulty in use due to the delay in cursor movement, based on the delay time due to the blur amount and the delay time due to the performance of the device. Below, the gesture input device 1 which concerns on this embodiment is demonstrated in detail.

(Functional configuration)
First, an example of a functional configuration of the gesture input device 1 according to an embodiment will be described. FIG. 2 is a diagram illustrating an example of a functional configuration of the gesture input device 1 according to the embodiment.

(Delay calculation function)
The gesture input device 1 has a function of calculating a delay amount. In order to realize this function, the gesture input device 1 calculates the accuracy of the cursor based on the accuracy of the sensor and the shake of the user's body, and smoothes the position of the cursor so as to reduce the shake sufficiently. The smoothing strength of is calculated. Then, a delay time until the user's gesture is reflected in the cursor operation on the screen is calculated from the calculated smoothing intensity. The delay time calculated here is the “delay time based on the shake amount of the body part of the user performing the gesture”, and the “shake amount” is observed when the user's body shakes and the user's body is observed. The accuracy of the sensor when performing the detection, that is, the error of the detected value when the position is detected by the sensor (the difference between the actual body position and the detected body position) is included. This delay time is a delay time mainly caused by a phase shift that occurs during the smoothing process. The gesture input device 1 also calculates a delay time based on the time required for the internal processing of the sensor and the performance of the device such as the screen display device. The gesture input device 1 calculates the total delay time based on the delay time due to the smoothing of the blur amount and the delay time due to the device performance.

(Screen display function)
The gesture input device 1 has a function of displaying a screen based on the calculated total delay time. In order to realize this function, the gesture input device 1 calculates the operation difficulty level of the object on the screen based on the total delay time and the distribution of the initial position of the cursor. The operation difficulty level indicates the difficulty level of the screen operation. The delay time until the gesture is reflected in the cursor operation on the screen and the operation difficulty level have a correlation that the operation difficulty level increases as the delay time increases. Therefore, the operation difficulty level indicating how much delay the operation becomes difficult can be calculated from the delay time.

  The gesture input device 1 calculates display conditions that allow objects on the screen to be visually recognized based on the positional relationship between the user and the screen. The display condition includes a condition that the object is displayed larger when the distance between the user and the screen is long, and the object is displayed small when the distance between the user and the screen is short. The gesture input device 1 displays an object on the screen based on the calculated operation difficulty level and display conditions. The gesture input device 1 displays an object having a size that makes the operation difficulty level equal to or less than a certain value so that the screen operation by the gesture does not become too difficult. At this time, the listability of the screen decreases as the size of the object increases. Therefore, the generated screen displays an object having an operation difficulty level of a certain value or less and a size that does not unnecessarily deteriorate the listability of the screen.

  In the present embodiment, the delay amount calculation function mainly includes a position acquisition unit 31, a position accumulation unit 32, an accuracy calculation unit 33, a smoothing strength calculation unit 34, a smooth delay time calculation unit 35, a device delay time calculation unit 36, And each function of the total delay time calculation unit 37.

  The position acquisition unit 31 acquires the position of the user's hand and the position of the pointing device, and calculates the position of the cursor on the screen. In the hand coordinate system, the normal direction of the screen of the display device is the z axis, and the direction away from the screen is positive. The horizontal direction in the plane of the screen is the x axis, and the vertical direction is the y axis. In this coordinate system, the position acquisition unit 31 appropriately acquires the coordinates (xh, yh, zh) of the user's hand and calculates the cursor coordinates (x, y) on the screen according to the acquired position of the hand. In the coordinate system of the cursor, the horizontal direction in the plane of the screen is the x axis (right direction is positive), and the vertical direction is y axis (down direction is positive). Formula (1) is given as an example of a calculation formula for calculating the coordinate p of the cursor from the coordinates of the hand.

  Here, ax, bx, ay, and by are constants of real numbers, and are values determined experimentally from the resolution of the screen.

  The position accumulation unit 32 accumulates the cursor coordinates p (x, y) calculated by the position acquisition unit 31 at regular intervals. The accumulated cursor coordinates are used when calculating the moving speed and direction of the cursor. For example, the fixed time is 30 times per second. The position storage unit 32 may not only store the coordinates of the cursor but also discard the coordinates of the cursor. For example, by discarding the coordinates of the cursor accumulated before a certain time, it is possible to prevent the cursor accumulation of a certain amount or more from being accumulated in the position accumulation unit 32.

  The accuracy calculation unit 33 calculates the accuracy of the cursor position using the coordinate string of the cursor having a certain length accumulated in the position accumulation unit 32. Specifically, the accuracy calculation unit 33 measures the shake amount and calculates the accuracy of the cursor position from the shake amount. The “shake amount” can be defined as a standard deviation when the hand is stationary and the position is acquired a certain number of times by the sensor.

As an example of a method for calculating the accuracy of the cursor position, first, the accuracy calculation unit 33 notifies the user that the accuracy is to be calculated. The user stops the cursor as much as possible in response to the notification. When a certain period of time elapses while still being stationary and the cursor position is sufficiently accumulated in the position accumulation unit 32, the accuracy calculation unit 33 calculates the accuracy of the cursor position according to Equation (2). The accuracy of the cursor position is calculated as variance as shown in Equation (2). The accuracy calculation unit 33 notifies the user that the accuracy of the cursor position has been calculated. Here, V (X) and V (Y) are standard deviations, m is the length of the coordinate sequence used for calculating the standard deviation,
And are the average of the x and y coordinates.

  As described above, the accuracy calculation unit 33 measures the shake amount of the user's hand and calculates the accuracy of the cursor position based on the shake amount. In addition, when the length on the image acquired with the camera 10 is converted into the length of the real world and handled, the blur amount is proportional to the distance. Therefore, the distance between the sensor and the user and the amount of shake are in a proportional relationship. When this proportional relationship is known, the accuracy calculation unit 33 does not need to remeasure when the distance between the user and the sensor changes, and based on the known relationship between the distance between the sensor and the user and the amount of shake, the cursor What is necessary is just to calculate the precision of position.

  The smoothing intensity calculation unit 34 calculates the intensity when the cursor position accumulated for a predetermined time is smoothed so that the fluctuation of the cursor position becomes sufficiently small. The method of calculating the smoothing intensity by the smoothing intensity calculator 34 differs depending on the function used for smoothing. For example, when smoothing with a moving average, the smoothing intensity calculator 34 calculates the number of observation points used for averaging. The number of observation points is an example of the smoothing strength.

  Further, for example, when smoothing with an exponential smoothing moving average, the smoothing strength calculating unit 34 calculates a coefficient for smoothing with an exponential smoothing moving average. Here, the value of the coefficient is an example of the strength of smoothing.

  More specifically, for example, when using a simple moving average like Formula (3), the smoothing intensity | strength calculation part 34 can calculate dispersion | distribution after smoothing by Formula (4). The calculated variance after smoothing indicates the accuracy of the cursor position.

  Here, x (t) is a coordinate string before smoothing (input coordinate string), Xs (t) is a coordinate string after smoothing, n is the length of the coordinate string used for moving average (input point used for averaging) ), V (Xs) is the variance after smoothing. However, although only the x coordinate is described in the equations (3) and (4), the same applies to the y coordinate.

  The smoothing strength calculation unit 34 is a condition that the variance after smoothing calculated by the equation (4) is sufficiently small (the variance V (Xs) after smoothing <threshold value Tv; Tv is a threshold value indicating sufficiently small variance) ) Solve for the length n of the coordinate sequence below. The smoothing strength calculation unit 34 may calculate the smoothing strength by regarding n (the length of the coordinate sequence used for the moving average) as the smoothing strength.

  Further, for example, when an exponential smoothing moving average such as Expression (5) is used, the smoothing intensity calculation unit 34 can calculate the variance after smoothing according to Expression (6). The calculated variance after smoothing indicates the accuracy of the cursor position.

  According to this, as in the case of a simple moving average, the smoothing strength calculator 34 may calculate the smoothing strength by regarding the value of the coefficient α as the smoothing strength. That is, with respect to the coefficient α under the condition that the variance after smoothing calculated by Expression (6) is sufficiently small (variance V (Xs) after smoothing <threshold T; T is a threshold indicating sufficiently small variance). solve. Here, the coefficient α is a smoothing constant, and the smaller the value, the higher the smoothing strength.

  Also, smoothing strength can be calculated by applying the same concept to smoothing methods other than these. That is, the smoothing intensity calculating unit 34 may determine parameters related to smoothing so that the amount of blurring (for example, variance) of the cursor after smoothing becomes sufficiently small. The smoothing strength calculator 34 may calculate the smoothing strength by regarding the parameter at that time as the smoothing strength.

  As shown in FIG. 3, the delay time mainly includes a delay time due to equipment and a delay time based on smoothing. The delay time by the device shown in (a) of FIG. 3 is a delay time based on the processing or communication of the device according to the gesture. Examples of the delay time by the device include a processing time inside the sensor and a communication time when information is transferred from the sensor to a device such as a server. Other examples of the delay time by the device include processing time such as rendering for displaying the transferred image on the screen, time required for calculating the UI screen for displaying the processed image on the screen, and the like. It is done. The movement trajectory of the cursor based on the delay time by the device has a simple delay indicated by a broken line with respect to the original movement trajectory of the cursor indicated by the solid line in the graph of FIG.

  The device delay time calculation unit 36 calculates the delay time due to the device shown in FIG. The device delay time calculation unit 36 calculates the delay time based on the performance of the sensor and the display device of the screen. The delay time calculated by the device delay time calculation unit 36 can be acquired by measuring the processing time of the sensor or information processing device in advance. In addition, the device delay time calculation unit 36 measures the processing time of the device to be used in advance, creates a correspondence table between the device performance and the processing time, and performs the performance of the specific device used based on the correspondence table. You may acquire processing time from.

  In addition, the device delay time calculation unit 36 can calculate the delay time by the device from, for example, a specification that defines the device performance. In addition, the device delay time calculation unit 36 determines the time from when the user performs an operation with a gesture until the operation is reflected on the screen in a state where the gesture input device 1 does not perform smoothing, separately from the sensor. It may be measured from an image taken with a camera and calculated as a delay time by the device.

  On the other hand, the delay time based on the smoothing shown in FIG. 3B is a delay depending on the frequency (speed). An example of the delay is not a simple delay as shown by the broken line with respect to the original movement locus of the cursor shown by the solid line in the graph of (b). For example, the smooth delay time calculation unit 35 calculates a delay time based on the smoothing shown in FIG. The smoothing delay time calculation unit 35 can use the time constant of the smoothing filter as a delay time based on smoothing. The specific calculation method by the smoothing delay time calculating unit 35 differs depending on how smoothing and smoothing strength are calculated. The calculation of the time constant of the smoothing filter when smoothing with a simple moving average or the calculation of the time constant of the smoothing filter when smoothing with an exponential smoothing moving average will be specifically described below.

  For example, in the case of smoothing with a simple moving average, the smoothing delay time calculation unit 35 can calculate the time constant τ using Equation (7). Here, the time constant τ is a delay time, T is a time interval (sampling period) in which the position accumulating unit 32 accumulates positions, and e is a natural logarithm base.

  When the exponential smoothing moving average is used, the smooth delay time calculation unit 35 can calculate the delay time (time constant) τ by the equation (8). Here, α (0 ≦ α ≦ 1) is a smoothing constant, time constant τ is a delay time, and T is a time interval (sampling period) in which the position accumulating unit 32 accumulates positions. The smaller α is, the smoother it is.

  The total delay time calculation unit 37 calculates the total delay time based on the delay time calculated by the device delay time calculation unit 36 and the delay time calculated by the smooth delay time calculation unit 35. For example, the total delay time calculation unit 37 may use the sum of the smooth delay time and the device delay time as the total delay time. The total delay time calculation unit 37 may use either the smooth delay time or the device delay time as the delay time.

  The smooth delay time is an example of a delay time based on the amount of blurring of the body part of the user who performs the gesture. The device delay time is an example of a delay time based on device processing or communication according to a gesture.

(Screen display function)
In the present embodiment, the screen display function is mainly realized by the operation difficulty level calculation unit 41, the display condition calculation unit 42, and the screen display unit 43. The operation difficulty level calculation unit 41 calculates the operation difficulty level of the object on the screen based on the total delay time and the distribution method of the initial position of the cursor. The object is displayed on the screen and includes images and text such as selectable figures and icons. Pointing refers to an operation of placing a cursor on an object.

  The operation difficulty level calculation unit 41 calculates the time required for pointing an object as the operation difficulty level. The object operation difficulty level is an amount that increases with respect to the distance between the initial position of the cursor and the object, decreases with respect to the size of the object, and increases with respect to the total delay time.

  The operation difficulty level calculation unit 41 can calculate the operation difficulty level based on the following equation. Expressions (9) and (10) are shown below as an example of calculating the operation difficulty.

  Here, a, b, and c are constants and are values determined experimentally. MT is a pointing time, ID is a degree of difficulty, A is the distance from the initial position of the cursor to the object, W is the size of the object, and Lag is the delay time. The initial position of the cursor is determined using a distribution of the initial position of the cursor described later.

  The pointing time MT calculated by Expression (9) is an example of the operation difficulty level indicating the difficulty level for the screen operation.

  In Expressions (9) and (10), the distance A from the initial cursor position to the object is calculated for each cursor position with respect to the object that is a display candidate. The operation difficulty level calculation unit 41 calculates a distance A for each of the positions of the cursor assumed from the distribution of the initial position of the cursor for each of the objects that are candidates for display. For example, the size W of the object may be assumed to be 5 cm, 10 cm, and 15 cm. Therefore, the operation difficulty level calculation unit 41 calculates the pointing time MT using the distance A from the initial position of the cursor to the object for each object having a different size and for each cursor position.

  Further, as another example of calculating the operation difficulty level for calculating the operation difficulty level, Expression (11), Expression (12), and Expression (13) are shown below.

  Here, a, b, c, and d are constants and are values determined experimentally. MT is the pointing time, A is the distance from the initial position of the cursor to the object, W is the size of the object, and Lag is the delay time. Φ is a cumulative normal distribution with an average of 0 and a variance of 1, N is a normal distribution, and F is a function that satisfies equation (13) for any B> 0. The operation difficulty level calculation unit 41 calculates the pointing time MT using the equation (11). The calculated pointing time MT is an example of the operation difficulty level.

(Initial cursor position)
The initial position of the cursor is generally unknown. This is because the process of the screen display unit 43 is executed before the actual screen display is performed, and therefore the user does not always operate the cursor when the operation difficulty level calculation unit 41 is executed. Therefore, the distance A from the initial position of the cursor to the object cannot be calculated.

  Therefore, the operation difficulty level calculation unit 41 assumes the distribution of the initial position of the cursor, and calculates the operation difficulty level at each cursor position according to the assumed distribution of the initial position of the cursor. The operation difficulty level calculation unit 41 may calculate an average value of the operation difficulty levels at the respective cursor positions and set the operation difficulty level. As an example of the distribution of the initial position of the cursor, there is a case where the existence probability of the cursor is uniform in the screen 21, as shown in FIG. As another example of the distribution of the initial position of the cursor, as shown in FIG. 4B, the cursor existence probability is centered on a point O corresponding to the position of the shoulder of the user U in the screen 21. In this case, a spindle-shaped distribution is used.

  The display condition calculation unit 42 calculates a display condition in which an object on the screen can be visually recognized based on the size of the screen and the positional relationship between the user and the screen. The display condition calculation unit 42 sets a threshold condition for the size of the object. The display condition calculation unit 42 increases the object size threshold Ts with respect to the screen size and sets the value to increase with respect to the distance between the user and the screen. According to this, when the distance from the user to the screen is far, the display becomes large, and when it approaches, the display is small. Therefore, even if the distance between the user and the screen changes, it is possible to provide a screen display that is easy for the user to view. . For example, the threshold value of the object size when the distance between the user and the screen is the distance d1 is Ts = t1. Further, the threshold value of the object size when the distance between the user and the screen is a distance d2 larger than the distance d1 is Ts = t2. At this time, the threshold value t2 may be set to a value larger than the threshold value t1 (t2> t1). Here, t1, t2, d1, and d2 are values determined by the developer. The threshold value Ts may be stored in a storage area (not shown), for example.

  The screen display unit 43 generates a screen on which an object is displayed based on the operation difficulty level calculated by the operation difficulty level calculation unit 41 and the display conditions calculated by the display condition calculation unit 42. As an example, the screen display unit 43 adjusts the size of the object on the screen to be generated so that the operation difficulty is equal to or less than a predetermined operation difficulty threshold Tm. The size of the object may be adjusted to a size that allows as many objects as possible to be displayed so that the listability of the display on the screen increases within a range where the operation difficulty is sufficiently small (that is, the threshold value Tm or less). The threshold value Tm may be stored in a storage area (not shown), for example.

  For example, a case is assumed where the object size candidates are s1, s2, and s3 (s1 <s2 <s3). The operation difficulty level calculation unit 41 calculates operation difficulty levels m1, m2, and m3 for the object sizes s1, s2, and s3 (s1 <s2 <s3). The screen display unit 43 is an object whose operation difficulty level (m1, m2, m3) is equal to or less than a certain threshold Tm among candidates having object sizes of s1, s2, and s3 based on the operation difficulty levels m1, m2, and m3. The object with the smallest size is adopted. If m1> Tm, m2 <Tm, and m3 <Tm, the screen display unit 43 employs an object candidate s2 that is smaller than the candidate s3. Furthermore, in consideration of display conditions, it is sufficient that the object size candidate s2 is larger than the size threshold Ts. If the candidate s2 is smaller than the size threshold Ts, the screen display unit 43 adopts the candidate s3 if the object candidate s3 larger than the candidate s2 is larger than the size threshold Ts. The screen display unit 43 displays the object with the adopted candidate size.

[Gesture input processing]
Next, an example of gesture input processing performed by the gesture input device 1 according to the present embodiment will be described with reference to FIGS. 5 and 6. Hereinafter, the gesture input process according to the present embodiment will be described by dividing it into a delay amount calculation process based on the gesture input and a screen display process corresponding to the gesture input. FIG. 5 is a flowchart illustrating an example of a delay amount calculation process according to an embodiment. FIG. 6 is a flowchart illustrating an example of a screen display process according to an embodiment. Each process will be described in the order of the delay amount calculation process in FIG. 5 and the screen display process in FIG. 6, so that the process from when the user performs a gesture until the screen corresponding to the gesture is displayed will be described.

(Delay calculation processing)
The position acquisition unit 31 calculates an operation position (cursor position) on the screen from the position of a part of the body such as a pointing device or a user's hand. The position accumulating unit 32 accumulates the cursor position acquired by the position acquiring unit 31 at predetermined time intervals. The position accumulating unit 32 determines whether the cursor position has been accumulated for a predetermined time (step S10). The position accumulation unit 32 repeats the process of step S10 until the cursor position is accumulated for a predetermined time.

  When the cursor position is accumulated for a predetermined time, the accuracy calculating unit 33 calculates the accuracy of the cursor position based on the fluctuation of the cursor position (step S12). The movement of the cursor position is generated based on, for example, the accuracy of the sensor and the shake (hand shake) of the user's body.

  Next, based on the accuracy of the cursor position calculated by the accuracy calculation unit 33, the smoothing intensity calculation unit 34 is an intensity for smoothing the cursor position so that the blur of the cursor position is sufficiently reduced. Is calculated (step S14). Next, the smooth delay time calculation unit 35 performs a delay time by smoothing based on the smoothing strength calculated by the smoothing strength calculation unit 34 (delay time based on the amount of shake; hereinafter, also referred to as “smooth delay time”). Calculate (step S16).

  Next, the device delay time calculation unit 36 calculates a delay time based on the performance of the sensor or device (delay time based on the device; hereinafter, also referred to as “device delay time”) (step S18). The total delay time calculation unit 37 calculates the total delay time by adding the smooth delay time and the device delay time (step S20).

(Screen display processing)
Next, the screen display process of FIG. 6 will be described. The display condition calculation unit 42 determines whether the user's position (such as the position of the user's hand) has been acquired from the camera 10 from the position acquisition unit 31 (step S30), and performs the process of step S30 until the user's position is acquired. repeat.

  When the position of the user is acquired, the display condition calculation unit 42 calculates a display condition that allows the object on the screen to be visually recognized based on the size of the screen and the positional relationship between the user and the screen (step S32).

  The screen display unit 43 generates an object that is a display candidate for a screen that satisfies the display conditions (step S34). Next, the operation difficulty level calculation unit 41 performs a process of calculating the operation difficulty level of the candidate object (step S36).

  The operation difficulty level calculation process called in step S36 will be described with reference to FIG. FIG. 7 is a flowchart illustrating details of the operation difficulty level calculation processing according to the embodiment.

  First, the operation difficulty level calculation unit 41 calculates one of the cursor positions according to the distribution of the initial position of the cursor (step S40). Next, the operation difficulty level calculation unit 41 calculates the distance A from the initial position of the cursor to the object (step S42). Next, the operation difficulty level calculation unit 41 calculates the operation difficulty level (pointing time MT) using, for example, Expression (9) and Expression (10) (Step S44). The operation difficulty level can also be calculated by another method such as equation (13). Next, the operation difficulty level calculation unit 41 determines whether the operation difficulty levels for all cursor positions have been calculated (step S46). If the operation difficulty level for all the cursor positions has not been calculated, the process returns to step S40, and the processes of steps S40 to S46 are performed.

  On the other hand, when the operation difficulty levels for all cursor positions have been calculated, the operation difficulty level calculation unit 41 calculates, for example, an average value of the operation difficulty levels as a representative value of the operation difficulty levels (step S48), and the process returns to FIG. The process of step S38 is performed.

  The screen display unit 43 determines whether the operation difficulty level is sufficiently small (step S38). When it is determined that the operation difficulty level is greater than the predetermined threshold value, the process returns to step S34, and the processes of steps S34 to S38 are repeated until the operation difficulty level becomes equal to or less than the predetermined threshold value.

  For example, if it is determined in step S38 that the operation difficulty level is larger than a predetermined threshold, the size of the display candidate object of the screen generated in step S34 is set larger than the size of the object set as the previous candidate. For example, when the size of the object selected as the previous candidate is 5 cm, the operation difficulty level calculation unit 41 increases the size of the object set as the current candidate to 10 cm, and substitutes it in W of Expression (10). From the above, the operation difficulty level may be calculated. Even in this case, if it is determined in step S38 that the operation difficulty level is greater than the predetermined threshold value, the operation difficulty level calculation unit 41 increases the size of the candidate object to 15 cm and substitutes it into W in Expression (10). Then, the operation difficulty level may be calculated from Equation (9).

  When it is determined in step S38 that the operation difficulty level is sufficiently small, the screen display unit 43 displays the generated screen (step S39), and ends this process.

  Heretofore, the gesture input process according to the present embodiment has been described. The gesture input device 1 according to this embodiment adjusts and displays the size of the object on the screen based on the delay time and the display condition.

  FIG. 8 shows an example of a screen display according to an embodiment. In FIG. 8, the horizontal axis represents the delay time, and the vertical axis represents the distance between the user and the screen. The operation difficulty level is calculated based on the delay time. The object on the screen shown in FIG. 8 is an object whose operation difficulty is equal to or less than a certain threshold value and has the smallest object size. Thereby, operability and listability can be made compatible.

  As an example, when the delay time is small and the distance is short, a small object is displayed. For example, when the delay time is medium and the distance is medium, an object having a medium size is displayed. For example, when the delay time is large and the distance is long, a large object is displayed.

  When the amount of shake is large, an operation error with respect to a user's gesture input is likely to occur if an object on the screen is displayed small. On the other hand, according to the gesture input device 1 according to the present embodiment described above, the shake amount is calculated as the delay time, and the size of the object on the screen is adjusted and displayed based on the delay time. For example, when the amount of blur is large, the delay time is large, so that the object is displayed large in order to suppress an operation error with respect to the gesture input. Thereby, the operativity of the screen in a user's gesture input can be improved.

  In particular, when the distribution of cursor positions is not uniform, the value of the operation difficulty varies depending on the position of the object, so that the size of the object allowed on the same screen may differ. In that case, the largest object size among the allowable object sizes for the position of each object may be adopted. Thereby, user operability can be improved.

(Hardware configuration example)
Finally, a hardware configuration example of the gesture input device 1 will be briefly described. FIG. 9 is a diagram illustrating a hardware configuration example of the gesture input device 1 according to the present embodiment. The gesture input device 1 includes an input device 101, a display device 102, an external I / F 103, a RAM (Random Access Memory) 104, a ROM (Read Only Memory) 105, a CPU (Central Processing Unit) 106, a communication I / F 107, and an HDD ( (Hard Disk Drive) 108 are connected to each other via a bus B.

  The input device 101 includes a camera 10, a keyboard, a mouse, and the like, and is used to input each operation to the gesture input device 1. The display device 102 includes a display 20, performs icon operation on the screen in accordance with a user's gesture input, and displays the result.

  The communication I / F 107 is an interface that connects the gesture input device 1 to a network. Thereby, the gesture input device 1 can communicate with other devices via the communication I / F 107.

  The HDD 108 is a non-volatile storage device that stores programs and data. The stored programs and data include OS (Operating System) that is basic software for controlling the entire apparatus, and application software that provides various functions on the OS. Further, the HDD 108 performs the accuracy calculation process, the smoothing intensity calculation process, the smooth delay time calculation process, the device delay time calculation process, the total delay time calculation process, the operation difficulty level calculation process, the display condition calculation process, and the screen display process. Stores a program executed by the CPU 106.

  The external I / F 103 is an interface with an external device. The external device includes a recording medium 103a. The gesture input device 1 can read and / or write to the recording medium 103 a via the external I / F 103. The recording medium 103 a includes a CD (Compact Disk), a DVD (Digital Versatile Disk), an SD memory card (SD Memory card), a USB memory (Universal Serial Bus memory), and the like.

  The ROM 105 is a non-volatile semiconductor memory (storage device), and stores programs and data such as BIOS (Basic Input / Output System), OS settings, and network settings that are executed at startup. The RAM 104 is a volatile semiconductor memory (storage device) that temporarily stores programs and data. The CPU 106 is an arithmetic device that realizes control and mounting functions of the entire apparatus by reading programs and data from the storage device (for example, “HDD” and “ROM”) onto the RAM and executing processing.

  Accuracy calculation unit 33, smoothing strength calculation unit 34, smooth delay time calculation unit 35, device delay time calculation unit 36, total delay time calculation unit 37, operation difficulty level calculation unit 41, display condition calculation unit 42, screen display unit 43, Each unit of the screen display unit 43 is realized by processing executed by the CPU 106 by a program installed in the HDD 108.

  The position acquisition unit 31 can be realized using the input device 101. The position accumulation unit 32 can be realized by using, for example, a storage device connected to the RAM 104, the HDD 108, or the gesture input device 1 via a network.

  As described above, the gesture input device, the gesture input method, and the program have been described in the above embodiment. However, the present invention is not limited to the above embodiment, and various modifications and improvements can be made within the scope of the present invention. In addition, when there are a plurality of the above-described embodiments and modifications, they can be combined within a consistent range.

  For example, the gesture input device 1 according to the present embodiment calculates the total delay time from the delay time based on the shake amount and the delay time based on device processing or communication. However, the present invention is not limited to this, and the gesture input device according to the present invention may use, for example, a delay time based on a shake amount as a total delay time, or a delay time based on device processing or communication according to a gesture as a total delay time. It is good. If only the delay time based on the amount of shake or only the delay time based on device processing or communication is used, it can be determined that an operation error for the gesture input is likely to occur if the delay time is large. Therefore, even when any one of the above delay times is set as the total delay time, if the total delay time is large, the object is displayed in a large size, thereby improving the operability of the screen for the user's gesture input. .

  The gesture input device 1 according to the present embodiment displays an object based on the total delay time and the positional relationship between the user and the screen. However, the present invention is not limited to this, and the gesture input device according to the present invention may display an object based on the total delay time. In this case, the size of the object on the screen is adjusted based on the delay time regardless of the positional relationship between the user and the screen.

  As an example of this case, a small object is displayed when the delay time is small, as shown at the top of FIG. When the delay time is medium, a medium-sized object is displayed. When the delay time is large, a large object is displayed.

Regarding the above description, the following items are further disclosed.
(Appendix 1)
A gesture input device for inputting by gesture,
The total delay time for calculating the total delay time of the input by the gesture from at least one of the delay time based on the shake amount of the body part of the user performing the gesture and the delay time based on the processing or communication of the device corresponding to the gesture A calculation unit;
A screen display unit for displaying an object based on a total delay time of input by the gesture;
A gesture input device.
(Appendix 2)
The total delay time calculation unit includes:
Calculating the total delay time from the delay time based on the amount of blur and the delay time based on the processing or communication of the device,
The gesture input device according to attachment 1.
(Appendix 3)
The screen display unit
Displaying an object based on the total delay time and the positional relationship between the user and the screen;
The gesture input device according to appendix 1 or 2.
(Appendix 4)
An operation difficulty calculating unit that calculates an operation difficulty indicating the difficulty of the screen operation from the total delay time;
The screen display unit
Displaying an object of a screen whose operation difficulty is equal to or less than a predetermined threshold;
The gesture input device according to any one of appendices 1 to 3.
(Appendix 5)
The operation difficulty calculation unit
Calculating the distribution of the initial position of the cursor based on the position of the body part of the user performing the gesture, and calculating the operation difficulty according to the calculated distribution of the initial position of the cursor,
The gesture input device according to attachment 4.
(Appendix 6)
A smooth delay time calculation unit that calculates a delay time based on the amount of blur based on a smoothing intensity for smoothing a position of a cursor acquired according to the gesture;
The gesture input device according to any one of appendices 1 to 5.
(Appendix 7)
A gesture input method for inputting by gesture,
Calculating the total delay time of the input by the gesture from at least one of the delay time based on the shake amount of the body part of the user performing the gesture and the delay time based on the processing or communication of the device according to the gesture,
Displaying an object based on a total delay time of input by the gesture;
A gesture input method in which processing is executed by a computer.
(Appendix 8)
Calculating the total delay time from the delay time based on the amount of blur and the delay time based on the processing or communication of the device,
The gesture input method according to attachment 7.
(Appendix 9)
Displaying an object based on the total delay time and the positional relationship between the user and the screen;
The gesture input method according to appendix 7 or 8.
(Appendix 10)
An operation difficulty level indicating the difficulty level of the screen operation is calculated from the total delay time,
Displaying an object of a screen whose operation difficulty is equal to or less than a predetermined threshold;
The gesture input method according to any one of appendices 7 to 9.
(Appendix 11)
Calculating the distribution of the initial position of the cursor based on the position of the body part of the user performing the gesture, and calculating the operation difficulty according to the calculated distribution of the initial position of the cursor,
The gesture input method according to attachment 10.
(Appendix 12)
A delay time based on the amount of blur is calculated based on a smoothing strength for smoothing a cursor position acquired according to the gesture;
The gesture input method according to any one of appendices 7 to 11.
(Appendix 13)
A program that operates the screen by inputting gestures,
Calculating the total delay time of the input by the gesture from at least one of the delay time based on the shake amount of the body part of the user performing the gesture and the delay time based on the processing or communication of the device according to the gesture,
Displaying an object based on a total delay time of input by the gesture;
A program that causes a computer to execute processing.
(Appendix 14)
Calculating the total delay time from the delay time based on the amount of blur and the delay time based on the processing or communication of the device,
The program according to attachment 13.
(Appendix 15)
Displaying an object based on the total delay time and the positional relationship between the user and the screen;
The program according to appendix 13 or 14.
(Appendix 16)
An operation difficulty level indicating the difficulty level of the screen operation is calculated from the total delay time,
Displaying an object of a screen whose operation difficulty is equal to or less than a predetermined threshold;
The program according to any one of appendices 13 to 15.
(Appendix 17)
Calculating the distribution of the initial position of the cursor based on the position of the body part of the user performing the gesture, and calculating the operation difficulty according to the calculated distribution of the initial position of the cursor,
The program according to appendix 16.
(Appendix 18)
A delay time based on the amount of blur is calculated based on a smoothing strength for smoothing a cursor position acquired according to the gesture;
The program according to any one of appendices 13 to 17.

1: Gesture input device 10: Camera 20: Display 31: Position acquisition unit 32: Position accumulation unit 33: Accuracy calculation unit 34: Smoothing enhancement calculation unit 35: Smoothing total delay time calculation unit 36: Device delay time calculation unit 37 : Total delay time calculation unit 41: Operation difficulty calculation unit 42: Display condition calculation unit 43: Screen display unit

Claims (8)

A gesture input device for inputting by gesture,
The total delay time for calculating the total delay time of the input by the gesture from at least one of the delay time based on the shake amount of the body part of the user performing the gesture and the delay time based on the processing or communication of the device corresponding to the gesture A calculation unit;
A screen display unit for displaying an object based on a total delay time of input by the gesture;
A gesture input device. The total delay time calculation unit includes:
Calculating the total delay time from the delay time based on the amount of blur and the delay time based on the processing or communication of the device,
The gesture input device according to claim 1. The screen display unit
Displaying an object based on the total delay time and the positional relationship between the user and the screen;
The gesture input device according to claim 1 or 2. An operation difficulty calculating unit that calculates an operation difficulty indicating the difficulty of the screen operation from the total delay time;
The screen display unit
Displaying an object of a screen whose operation difficulty is equal to or less than a predetermined threshold;
The gesture input device according to any one of claims 1 to 3. The operation difficulty calculation unit
Calculating the distribution of the initial position of the cursor based on the position of the body part of the user performing the gesture, and calculating the operation difficulty according to the calculated distribution of the initial position of the cursor,
The gesture input device according to claim 4. A smooth delay time calculation unit that calculates a delay time based on the amount of blur based on a smoothing intensity for smoothing a position of a cursor acquired according to the gesture;
The gesture input device according to any one of claims 1 to 5. A gesture input method for inputting by gesture,
Calculating the total delay time of the input by the gesture from at least one of the delay time based on the shake amount of the body part of the user performing the gesture and the delay time based on the processing or communication of the device according to the gesture,
Displaying an object based on a total delay time of input by the gesture;
A gesture input method in which processing is executed by a computer. A program that operates the screen by inputting gestures,
Calculating the total delay time of the input by the gesture from at least one of the delay time based on the shake amount of the body part of the user performing the gesture and the delay time based on the processing or communication of the device according to the gesture,
Displaying an object based on a total delay time of input by the gesture;
A program that causes a computer to execute processing.