JP2010079848A - Item selection method and apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method for smoothly performing an item selection operation which is an important operation that is frequently performed in an information apparatus operation in a case where a user does not operate an information component in the apparatus directly, for example, the user is away from the apparatus and a remote controller is not used. <P>SOLUTION: As a basic system for item selection, the item selection method uses a system in which an item selection is made by designating whether a user changes or determines a candidate 103 for a selection item 102 displayed by the apparatus. A continuous operation that is expressed by a cyclic chain of operation states is selected to indicate the intention of change, and an operation with fewer confusions with the above operation is selected to indicate the intention of determination. These operations are discriminated by an operation recognition means having a function to discriminate operation states. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

The present invention relates to an input device, an input system, a graphical user interface, and a graphical user interface operation method.

An item selection operation for selecting an item from an item set consisting of a plurality of items is an important operation that frequently occurs in the use of information equipment. A typical example is to select a command by handing the menu in the PC operation, but the operation of selecting a channel in TV operation, the operation of selecting a frame from a video clip in image editing, The operation for setting to a certain value can be seen as an example of an operation for selecting an item from this item set.
In general, the item selection operation is roughly divided into a direct pointing type and a method called a candidate change-determined type here.
In the direct pointing type item selection method, a user can select an item to be selected by directly touching an item to be selected from an item group displayed on a graphic display or the like with a pointing device such as a mouse.
In the candidate change-deterministic item selection method, button selection is performed in a framework in which items that are candidates for selection are highlighted and displayed separately from other items, or only candidates for selection are displayed. The selection operation is realized by repeating the operation of changing the selection candidate by the jog dial operation and pressing the confirm button when the item to be selected is selected as a candidate.
Both have advantages and disadvantages. Generally speaking, if the display device has enough space to display the item set at a time and there is a mechanism that can instantly indicate the display area of each item, the direct pointing type is advantageous. Candidate change-deterministic is advantageous.
The present invention provides a method and apparatus for selecting a desired item from an item set in an environment in which interface parts installed in the apparatus cannot be directly operated, such as when the apparatus is operated without using a remote control at a location remote from the apparatus. It is realized.
The following technologies have been developed for the above purpose.

  For example, Itoy Play released from Sony Computer Entertainment is equipped with the following interface as an item selection method by direct pointing. A plurality of items are displayed on the system display device, and an image of the user acquired by the camera is superimposed and displayed on the same display device as the user's alternation. The user moves his / her body so that his / her hand projected on the display device exactly overlaps an item to be selected. The system determines that the item is selected when the item position overlaps the user's hand for a certain period of time. This allows the user to select an item without touching the interface component.

  In "Clap & Gesture Recognition TV" announced by Victor at CEATEC in 2007, the icon is selected by overlapping the fingertip part on the icon on the display device, and it is determined by bending the fingertip in the same way as Aitoi Play. The function is implemented.

  All of the above are direct pointing types. Candidate change-definitive type is not found despite the useful scenes mentioned above.

The problems addressed by the present invention are important operations that occur frequently when operating information equipment in an environment where the interface parts installed in the device cannot be directly operated, such as when the user is away from the device and cannot use the remote control. It is to provide a method for smoothly performing an item selection operation.
Eye-toy play and applause sound & gesture recognition TV guides how the user can select items by displaying the interpretation of the user's operation status inside the system on the display device. is there. At this time, the user is required to move his / her own hand displayed on the remote screen to a designated part on the screen. However, it is actually not easy to align the hand of the other person on the screen, and there is a problem that the user's fatigue for the operation is great.
In addition, as described above, there is no example in which the candidate change-deterministic item selection method is realized under the condition that the interface parts installed in the device cannot be directly operated. It is easily conceivable to realize it by a method of performing self-alignment on the display device.
For example, a candidate change-deterministic item selection can be realized by providing a change button on the display device so that a change request can be issued once when the surrogate is put on the change button. However, in this implementation, it is necessary to repeat the position alignment on the screen where even fatigue is high, in order to perform the change request continuously at high speed, and the feeling of fatigue becomes very large. .
Instead of repeatedly performing the change request operation repeatedly, it is easy to implement such that the change request is repeatedly output while the alter button is put together. However, in this case, there is a problem that it becomes difficult to control the speed at which the candidate is repeatedly changed, and it is difficult to realize a smooth operation feeling.
In addition, in order to alleviate the labor problem related to alignment, after applying a certain posture or motion to a change request or confirmation request, these postures / motions are recognized by pattern recognition technology and these requests are detected. It is possible to do. For example, a change request may be made while continuing a circular motion, and a confirmation request may be made by a hand motion. However, even in this case, in order to perform the change request continuously, it is necessary to perform the same operation by dividing it by a stationary operation or to use time information for continuing the operation. Similar to the method of aligning the alternations, it has the problem of speed control when iteratively changes candidates.

  An object of the present invention is to provide a method and an apparatus capable of smoothly selecting an item without considering the position where a user performs an operation and the accuracy of an operation to be performed in consideration of such a problem.

In the present invention, the candidate change-determined item selection method is adopted as the basic method for item selection, and the candidate change request required here is detected by combining the contents on the display screen or simply by using an isolated pattern. This is realized by introducing continuous motion pattern recognition means having a plurality of operation state identification functions instead of recognition means. Continuous motion is modeled by a cyclic connection of multiple motion states. For example, a motion of drawing a circle by hand toward the camera, a motion of repeatedly shaking hands back and forth or left and right, and a motion of turning the neck , Actions that can be repeated any number of times, such as repeatedly changing the pitch of the voice, or moving the center of gravity in a circle on the chair.
The difference between the pattern recognition of the continuous operation having a plurality of operation state identification functions here and the simple pattern recognition here can be compared as follows. If the simple pattern recognition is to recognize that the position of the hand is "up" or "the hand is drawing a circle", then "the hand moves up in a series of actions that draw a circle." Recognizing that “is present” corresponds to pattern recognition of a continuous operation having a plurality of operation state identification functions.
By adopting a configuration that outputs candidate change requests when a specific state in a specific continuous operation is detected, the candidate change requests can be made smoothly and continuously, and the number of change requests per time can be freely changed. it can. For example, in the action of drawing a circle, if you combine the two states of the hand up and the state down with the candidate change request, you can issue two change requests by drawing a circle once, If a circle is drawn many times, a change request can be issued according to the number of rotations. By changing the speed of drawing a circle, you can also change the speed of issuing change requests.
Since the identification of the specific operation state is incorporated in the continuous operation recognition means, the operation state is identified with reference to the context, so that the identification accuracy of the operation state is improved. For example, there are many other motion states that are similar to the “motion state in which a hand is up in a series of circle drawing operations”. When trying to identify this operating state without associating it with the identification of continuous motion, there are many occurrences of misidentification. By identifying the operation state based on the identification of the operation, such a spring can be suppressed.
In addition, even in similar operating states, it becomes possible to distinguish according to the context, and the behavior of the device can be enhanced. For example, it is possible to separately grasp a stationary motion that is stopped in a horizontal position after turning a hand quickly and other stationary motions, and the behavior corresponding to them can be changed.
Although the above-mentioned contrivance solves the problem of operation labor and the problem of motion accuracy and reduces the problem of motion position to some extent, it cannot be said that the solution of the motion position problem is sufficient. One way to solve this problem is to simply convert the feature value used when recognizing continuous motion into a position-independent amount. For example, the movement of the hand is not regarded as a time change of the hand position, but is regarded as a time change of the velocity vector of the hand. However, the reliability of the speed information is lower than the reliability of the position, and when such processing is performed, the identification rate generally decreases. For this reason, in pattern recognition, position-dependent information and position-independent information are generally used in a balanced combination with appropriate weights, and the position-dependent information itself is normalized by appropriate coordinate transformation to reduce position dependency. Has been tried. Here, how to determine the conversion parameters for this normalization in each problem is an important issue. In the present invention, first, the problem of the operation position is solved by determining a conversion parameter by applying an online adaptation technique in pattern identification. Online adaptation can be performed using a history of physical quantities that change according to actions performed by the user, or values obtained by performing certain operations on the history.

By the way, human movements have a correlation with the positions of different movements, so if you know the movement position of a certain movement, you can approximate the position of other movements such as candidate conversion movements. , Position-dependent information can be normalized easily. Here, the device has a function of accepting an operation start request, and the operation that makes this request is a simple operation that can detect the operation and specify the operation position. Make it possible. Providing a function that accepts an operation start request is originally necessary to improve the operability of the entire interface, but this contributes to the normalization function of position-dependent information. The value will be higher.
The above two methods perform normalization of position information without making the user aware of it, but it is also useful for the user to request normalization of position information and give necessary information. For this purpose, we have defined an action to request adjustment of coordinate transformation parameters from the user side, and introduced a framework that can normalize the position information of the whole action with the operation position of this action. Here too, it is necessary to use an operation that is easy to detect the operation and specify the operation position.

  By showing the user the information on the jog dial and its operating point and feeding back the user's operating state detected by the apparatus using this jog dial, the user can feel as if the jog dial is being operated. This improves the enjoyment of operation. In addition, by drawing an initial value of the user's operation on the jog dial before the user starts the operation, the user can be guided to start the operation from that state. At this time, the detection accuracy can be improved by increasing the expected value for starting the operation from the state given on the jog dial in the operation detecting means.

  The proposed candidate change-determined type item selection method proposed here is a useful method, but it does not reach the remote control in terms of accuracy. For this reason, it is desirable to use a remote controller together. At this time, if the usability of the remote controller is different from the candidate change-determined item selection method based on the continuous motion proposed in the present invention, it is expected that the user will not be used to the continuous motion. Therefore, although a contact type sensor such as a touch pad is used as the sensor, a remote controller adopting the candidate change-determined type item selection method by the continuous operation proposed here is introduced. This makes it possible to provide a consistent operational feeling when using the remote control and when not using it.

According to the present invention, a user can smoothly perform an operation of selecting an item at a position away from the apparatus without using a remote controller and without being aware of the position and accuracy of the operation.

  The relationship between partial means and effects is as follows.

The candidate change-confirmation method is adopted as the basic method of item selection, and the candidate change request is detected by introducing a continuous motion pattern recognition means having a plurality of operation state identification functions. It is possible to easily change the candidate smoothly while freely changing the frequency of outputting the change request. In addition, the identification accuracy of the operating state is improved. Furthermore, even in similar operating states, it becomes possible to distinguish according to the context, and the behavior of the device can be enhanced.
Applying online adaptation technology for pattern identification to dynamically change conversion parameters required for motion detection, and effectively using motion position information when detecting start request motion and motion adjustment request motion Resistance to position is improved.

Feedback using the jog dial improved the user's operational feeling and guided the user's movement. By guiding the motion, the motion detection accuracy can be increased.
By using a contact-type sensor such as a touchpad for the remote control used in conjunction with this method, and introducing the proposed candidate change-deterministic item selection method based on continuous motion, the remote control can be used consistently and when it is not used. You can give a feeling of operation and get used to the operation of the equipment quickly.

[Description of common operation]
FIG. 1 shows a block diagram 100 of a proposed candidate change-determined item selection device by continuous action.
The user performs an operation of selecting a desired item from the items in the item set 101 and outputting the selected item 102 by an operation performed without directly touching the interface component of the apparatus.
Here, the item set may be a finite set or an infinite set. For example, if any month of the year is selected, each month from January to December is an item set, and if February is selected, this is a selection item. For example, if the angle is determined, a real number from 0 to less than 360 becomes the item set, and if 10.5 degrees is selected, this becomes the selection item. The item set may be changed during a series of operations.
The apparatus has candidate items 103 to be selected as data. Candidate display means 104 is composed of a display device such as a display, and when a change in candidate 103 is detected, the candidate value is displayed on the display and shown to the user.
After viewing the candidate, the user requests the apparatus to change the state through operation 105. If the candidate displayed by the candidate display unit 104 is a desired candidate, a confirmation request operation is performed, and if not, a candidate change request operation is performed. Here, generally, the candidate change request operation is selected as a continuous operation that can be modeled by the transition of the cyclic shape of the operation state, and the confirmation request operation is selected as an operation that can be easily distinguished from the candidate change request operation. Examples of the combination of the requested request action and the candidate request action include a set of a motion that draws the straight line and a motion that draws an approximate circle on a plane that is substantially orthogonal to the camera, a set of motion that nods to the camera and a motion that turns the face, A group of movements of moving the hand horizontally and a movement of repeating hands vertically, a group of movements of sitting and sitting in a chair and moving weights back and forth, and a movement of moving weights in a rotating manner, a movement of speaking “confirmed” by voice, An operation that fluctuates the pitch of the voice can be considered.
Reference numeral 106 denotes a sensor for capturing motion data into the device. Typically, the motion sensor is a camera when the motion is a gesture, a microphone when the sound is used, and a pressure distribution sensor when the weight is moved. It is. Multiple operations may be targeted using a plurality of sensors.
In the request identification unit 107, the request determination unit 111 uses the motion and motion state detection results obtained through the feature amount calculation 108, the motion identification 109 including the continuous motion, and the motion state identification 110, as a candidate for the user request. A change request and a confirmation request are detected, and the result is output to the control means 112. Here, the change request is not necessarily determined in a single way. For example, a change request 1 is output when a clockwise hand movement is identified, and a change request 2 is output when a counterclockwise hand movement is identified. There may be multiple types of change requests. Since this part depends on the embodiment, it will be described in detail in Embodiment 1 and later.
The control means 112 receives the output of the request identification means and determines the next candidate. The next candidate is determined by creating a circular list based on the elements of the candidate set, following this list in the forward direction when a change request 1 is made, and tracing backward when the change request 2 is made. The next candidate may be determined by an arithmetic operation regardless of the list operation. For example, in the example of determining the angle, this is an example in which an intermediate value between the candidate and 360 is a new candidate when the change request is 1, and an intermediate value between the candidate and 0 is a new candidate when the change request is 2. When the candidate is determined, the control means stores it in 103 and presents it to the user again through the candidate display means 105. When the control unit 116 receives the confirmation request from the request identifying unit 107, the control unit 116 outputs the item set as the current candidate as the selection item 102.

Next, referring to FIG. 2, the requested operation 201 performed by the user is an operation of drawing an approximate straight line using a hand on a plane approximately orthogonal to the camera when requesting candidate determination, and requesting candidate change An example of the request identifying unit 200 (corresponding to the request identifying unit 107 in FIG. 1) will be described in detail by taking as an example the case where the operation at this time is an operation of drawing an approximate circle using a hand on a plane substantially orthogonal to the camera.
In the figure, reference numeral 202 denotes a camera which is a sensor for taking a hand gesture into the apparatus. For example, as the output 203, 30 frames per second of 8-bit gradation data for each pixel is output with a resolution of 400 pixels vertically and 640 pixels horizontally.
The feature amount calculation means 204 receives the sensor output 203 and calculates the feature amount of the image. For example, an optical flow is obtained for each pixel by a gradient method, and this is smoothed for each block of a certain size, and the largest vector among the vectors obtained by smoothing is (sx, sy), and the pixel giving it The position is obtained as (xx, xy). In the operation to be identified in this embodiment, the fastest moving hand is the hand, so the maximum value of the smoothed vector obtained here is the velocity vector of the hand motion, the coordinate value that gives the maximum value. Gives the hand movement position. A set of vectors thus obtained, X = (sx, sy, xx, xy) is determined as a feature quantity. At this time, the feature amount may be obtained by compressing information by simply performing principal component analysis of the optical flow. Finally, X = (kA + (1-k) I) X + kB is corrected to obtain the final feature amount. A and B are for parameter adjustment to be described later. By default, A = I (unit matrix) and B = 0. k is a parameter for avoiding abrupt correction.
The operation to be recognized is described as an operation model 205. The model is expressed by a state transition model as shown in FIG. The state transition is repeated while outputting the feature quantity calculated in 204 one vector at a time. The model can also give a rule deterministically the conditions that must be satisfied by the features that can be output for each state, and the state conditions that allow transitions for each transition. Both are explained according to the hidden Markov model.
Reference numeral 310 denotes an operation for a confirmation request. Here, an operation in which the straight line is drawn on a plane substantially orthogonal to the camera is expressed.
320 represents an operation for requesting a candidate change, and here represents an operation in which an approximate circle is drawn on a plane approximately orthogonal to the camera. One of the features of the present invention is to use an operation in which the operation expression is circular like 320 for requesting a candidate change.
Reference numeral 330 denotes a probability model called a garbage model, which expresses movements other than the two movements that appear during the operation, such as an action for bringing the hand to the movement start point.
A node represented by double circles 311, 321, and 331 in the figure is a start node, and represents the start of each operation on the model.
In the figure, the nodes 312, 322, and 332 represented by small thick circles are terminal nodes, and express that this node can be connected to the start end of another behavior model.
Circles such as 313, 323, and 333 in the figure correspond to operating states. It has an output probability distribution indicating the probability of outputting each feature amount when transitioning to that state.
For example, the state 1 in 310 representing the substantially linear motion corresponding to the confirmation request is set as representing the starting end motion in the substantially linear motion, and is set as representing the end motion of the state 2 colinear motion. If the linear motion is a lateral motion from left to right, state 1 has a relatively large sx, sy is close to 0, and xx has a small feature vector output probability, and state 2 has a relatively high sx. The output probability of a feature vector having a large sy, close to 0, and a large xx is set high.
Also, state 1 in 320 representing the approximate circle motion corresponding to the candidate change request is set to represent the motion state in which the hand is at the highest position in the clockwise circle motion, and when the hand is moved to the right at the position. The output probability of the feature quantity corresponding to the feature quantity having a relatively large sx component and xy component is increased. State 2 and state 3 are set to represent the operation state when the hand descends clockwise in order and state 5 is at the lowest position, and the feature value is output when the hand is moved to the left at that position. An output probability distribution that is likely to be generated is provided.

  Compared to 310 and 320, the state 1 of 330 representing the operation other than the above two operations has an output probability distribution with no bias.

Such a probability distribution can be automatically learned by providing learning data.
In the figure, reference numerals 314, 324, and 334 denote arcs that indicate the possibility of state transition, and have a state transition probability that represents the ease of transition from the arc start point to the arc end point.
The likelihood calculation unit 205 in FIG. 2 receives the feature amount output from the feature amount calculation unit, and calculates the probability that each state of each model in FIG. 3 is appropriate as the current operation state.
For example, the probability of being in the state Si at time t is expressed as α (i, t). If the transition probability from the state Si to the state Sj is b (i, j) and the probability of outputting the feature quantity X in the state j is a (j, X), the probability α (( j, t + 1) is
α (j, t + 1) = Σ_i α (i, t) × b (i, j) × α (j, t + 1)
Can be written. Using this method, the likelihood of all motion states of all motions is calculated.
Given the likelihood of each model, the action identification means 207 examines which action is most likely to be currently performed, and the action state identification means 208 is most likely to be in which action state. Whether it is high or not is sent to the request determination unit 209.
The request determination unit 209 has a relational rule between the status string and the request to be output. Upon receiving the outputs of 207 and 208, the request judgment unit 209 checks the status string according to this rule, and when a defined status string is found, Output a request to
For example, if the operation identification means has a high possibility of the confirmation request operation, and the operation state identification means outputs the status column 1-2, a confirmation request is output.
For example, if the operation identification means has a high possibility of the candidate change request operation and the operation state identification means outputs the status column 1-2 or 5-6, the candidate change request is output.
As described above, in this embodiment, the method of integrating the identification of the confirmation request and the identification of the candidate change request by the probability model has been described. However, the confirmation request is performed by the SVM (support vector machine), and the candidate change request is Likelihood calculations may be performed in different ways, such as with a probabilistic model. Further, different methods may be combined, such as identifying candidate change request operations using SVM and identifying operation states using a probability model.
The adjustment parameter calculation unit 211 corresponds to the correspondence between the recent operation state and the feature amount used to express it from the feature amount history 212 provided by the feature amount calculation unit and the operation state history 213 provided by the operation state identification unit. You can get a relationship. Using this, the behavior of the feature quantity calculation means 204 or the behavior model 206 is changed so that the likelihood becomes high. When changing the behavior of the feature quantity calculating means, when a linear operation X ′ = AX + B is applied to the feature X in the feature quantity history, the transformation parameter A is such that X ′ approaches the average value vector of the corresponding operation state. , B are obtained and given to the feature quantity calculating means. When the behavior model 206 is changed, when the average value parameter of the behavior model is converted by M ′ = AM + B, A and B are obtained so that the output probability of the feature quantity in the history of 212 is increased, and this is used. Change the parameters of the behavior model. Such optimization can be obtained by a simple linear operation.
An example in which a hand movement is adopted as an operation for requesting the start of an item selection operation and an operation for requesting adjustment will be described. Hand gestures have excellent properties such as easy detection of the movement position and little confusion with other movements.
The feature amount calculation unit 2 221 performs FFT on the luminance value of each pixel and outputs a block smoothed value. The motion position detection means 222 obtains a pixel area whose intensity in a preset frequency band is stronger than a preset value as a hand shake area candidate, and further obtains the center of gravity of this area as a hand shake position candidate. The request determination unit 2 223 determines that there has been a hand movement when the hand movement region candidate obtained by the operation position detection unit 222 is wider than a preset area. At this time, if it is before the operation of the device, a start request is output as if there was a start request. At the same time, the hand movement position candidate obtained by the movement position detection unit 222 is sent to the adjustment parameter calculation unit 211. If it is after the operation of the apparatus, it is determined as an adjustment request, and the hand movement position candidate obtained by the movement position detection means 222 is sent to the adjustment parameter calculation means 211.
The adjustment parameter calculation means determines that the candidate change operation is likely to start when the received operation position is Y, and Y = Z + B is established for the average value Z of the distribution of the initial state of the candidate change operation. Thus, B is obtained, and this -B is given to the feature quantity calculating means. Alternatively, when the behavior model is changed, all average values M are changed by M ′ = M + B in the behavior model.
In this way, the process in the start request and the process in the adjustment request may be shared. Further, here, the feature quantity calculating means and likelihood calculating means of the request identifying means 200 for the confirmation request / candidate change request, and the feature quantity calculating means and likelihood calculating means of the request identifying means 220 for the start request, However, these may be shared.
The internal image of the jog dial as shown in FIG. 4400 is prepared for the probability model of the operation of drawing a circle in FIG. Reference numeral 401 denotes a rotating body, and 402, 403, and the like are candidates for positions at which marks representing rotation angles are placed, and each number corresponds to an operation state. On the display, as shown at 410, a mark candidate position at 400 is selected in a circle like 411 representing a rotating body, and a mark is placed at a corresponding location 412 in 411.
Here, while the motion identification model in FIG. 2207 is detecting the circular motion, the expression is distinguished from the case where the circular motion is not detected, for example, by changing the color of the mark as in 421. You may put a hand-shaped icon on the mark.
In principle, in the probability model of FIG. 3320, the mark position corresponding to the operation state with the highest likelihood is selected and the mark is placed. When the operating state with a high likelihood moves, the mark moves accordingly, so that the user feels like operating the jog dial.
If the user presents the mark position before drawing the circle, he gives the user the impression that the jog dial's operating point is there. Tend to start from position. For example, if the mark is in the initial state, the user tends to start writing a circle from above, and if the mark is in the initial state, the user tends to start writing a circle from the bottom. Therefore, by biasing the state transition probability for the state corresponding to the mark position from the initial state in FIG. 3320, the entropy of the model can be lowered and the identification accuracy can be increased. The feedback of the internal state by the jog dial is also effective in guiding the user's operation.
Place the touchpad on the remote control so that a confirmation request can be made by a linear motion drawn on the pad, and a candidate change request can be made by a circular motion drawn on the pad. Because you can perform consistent operations with remote controls and gestures in the space, you can speed up the operation procedure.

Next, an embodiment will be described in which the confirmation request is expressed by an action of swinging the head vertically toward the camera, and the candidate change request is expressed by an action of turning the face.
In the figure, reference numeral 202 denotes a camera which is a sensor for capturing a face image.
The feature amount calculation means 204 receives the sensor output 203 and calculates a shape parameter of AAM (Active Appearance Model) as the feature amount of the face image.
Similar to the first embodiment, the operation to be recognized is represented by a state transition model as shown in FIG. Hereinafter, both likelihood calculation and request determination can be performed in the same manner as in the embodiment.

Next, a description will be given of an embodiment in which a confirmation request is expressed by an action of largely shaking a hand on a plane substantially orthogonal to the camera, and a candidate change request is expressed by an action of finely shaking the hand vertically.
The sensor 202 is a camera.
The feature amount calculating means embodiment 204 is a feature amount derived from the optical flow described in the first embodiment.
The operation model and the garbage model corresponding to the confirmation request operation are the same as those in the first embodiment.
The change request operation is expressed by a state transition model shown in FIG.
Reference numeral 500 denotes an operation for requesting a candidate change, and expresses an operation of waving a hand vertically and vertically on a plane approximately orthogonal to the camera.
State 1 in 500 is set to represent an operation state where the hand is at the highest position in the operation of shaking the hand vertically, and the xy component corresponding to the feature amount when the hand is moved near the position is relatively The output probability of a feature quantity that is large and close to 0 for both sx and sy increases. The state 2 corresponds to an action when the hand is moved from the top to the bottom, and xy is close to 0, and the output probability of the feature quantity having a negative value for sy is high. State 3 is an operation state in which the hand is in a lower position, and 4 indicates an operation state in which the hand is raised from the bottom to the top.
The likelihood calculation, operation identification, and operation state identification method are the same as those in the first embodiment.
The candidate change request output is performed when a transition of state 2-3 is detected and a transition of state 4-1 is detected in a state where the candidate change request operation is obtained as an identification result.

Next, on a plane that is roughly perpendicular to the camera, when moving the hand slowly and slowly, change the candidate every half of the circle, and when moving the circle quickly and quickly, fast forward and rotate small and fast An embodiment will be described in which a function is provided for continuing fast-forwarding until another operation is performed when a post-stationary operation is performed.
A camera is used as the sensor, and the feature amount calculation means embodiment is the feature amount derived from the optical flow described in the first embodiment.
The operation model and the garbage model corresponding to the confirmation request operation are the same as those in the first embodiment.
The change request operation to be recognized is expressed by a state transition model shown in FIG. In this way, the change request operation may be represented by a plurality of models.
Reference numeral 610 represents an operation of drawing a small circle quickly on a plane substantially orthogonal to the camera. It is assumed that the hands are on the top, right, bottom, and left in the order of states 11, 12, 13, and 14, respectively, by drawing a circle. State 15 is a state corresponding to a stationary operation. Thus, the model representing the change request operation may also have a state in which no ring is configured. In the output probability setting in each state from state 11 to state 14, it is assumed that the output of the feature quantity at rest is lowered. That is, except for the state 15, when drawing a small circle, the drawing operation cannot be stopped. With this structure, it is possible to represent the action of drawing a small circle many times and the action of drawing a small circle and then resting on the right side. The operation represented by 610 is set as a candidate change request operation 1.
Reference numeral 620 represents an operation of slowly drawing a large circle on a plane substantially orthogonal to the camera. Here, in setting the output probability in each state, the output of the feature quantity at rest is also given. That is, when drawing a large circle, the drawing operation can be stopped halfway. The operation represented by 620 is set as a candidate change request operation 2.
The difference between a small circle and a large circle appears as a difference in output probability. Compared to 620, 610 gives a higher probability with a feature vector having a relatively large velocity component, and becomes a model with less difference in output probability due to a difference in position component for each state.
The likelihood calculation, operation identification, and operation state identification method are the same as those in the first embodiment.
For example, when the candidate change request operation 1 is obtained as an identification result and the transition of the state 12-13 is detected, the request determination unit outputs the candidate change request 1, and the candidate change request operation 1 is also identified. In a state obtained as a result, candidate change request 2 is output when a predetermined number of transitions of states 15-15 are detected continuously. Further, candidate change request 3 is output when transition of state 22-23 is detected and transition of state 24-21 is detected in a state where candidate change request operation 2 is obtained as an identification result. To do.
When receiving the candidate change request 3, the control means determines the next candidate by stepping through the circulation list one by one, and when receiving the candidate change request 1, determines the next candidate by skipping the circulation list one by one. Upon receipt, a candidate change request including fast-forwarding is realized by combining large and small circular motions and stationary motions, such as by skipping two circular lists and determining the next candidate.

Next, make a request to start the operation by waving your hand finely and quickly on a plane that is approximately orthogonal to the camera. If the hand is stopped at the right side of the start request action position, the forward candidate is forward-forwarded while it is stopped. If you continue to fast forward candidates in the fast forward direction and wave your hand on the left side from the start request action position, the list will be sent in the reverse direction according to the number of shakes and the candidate will be changed. If you stop and continue fast-forwarding the candidate while you stop, and if you put your hand on the left and stop further, you continue to fast-forward the candidate in the reverse direction while stopping and decide the candidate by shaking your hand finely Examples will be described.
A sensor is used as the sensor, and the feature amount calculation means and the feature amount calculation means 2 embodiment are derived from the optical flow-derived feature amount described in the first embodiment and from the FFT, respectively. The feature quantity derived from the FFT is used for the confirmation request operation. In this way, the feature quantities of the change request operation and the confirmation operation may be changed.
The change request operation to be recognized is expressed by a state transition model shown in FIG.
A model 710 expresses an operation when the hand is moving on the right side of the start request operation. States 11, 12, 13, and 14 are for representing an operation of waving at the right side of the start request operation, 11 is a state in which the hand is on the left side in the hand movement operation, and 12 is a hand from left to right , 13 indicates a state where the hand is on the right side, and 14 indicates an operation of returning the hand from right to left. State 15 is the state where the hand stops when it comes to the right at 14, 16 is the action of moving the hand further from the state where the hand is stopped at 15, and 17 is the action corresponding to 16, bringing the hand to the right By the way, the state in which the hand is stopped, 18 indicates the operation of returning the hand from the state in which the hand is stopped at 17 to the left.
A model 720 expresses an operation when the hand is moving on the left side of the start request operation. States 21, 22, 23, and 24 are for representing a hand-waving operation on the left side of the start requesting operation, 21 is a state in which the hand is on the right side in the hand-shaking operation, and 22 is a hand from the right to the left. , 23 indicates a state where the hand is on the left side, and 24 indicates an operation of returning the hand from left to right. State 25 is a state where the hand stops when it comes to the left at 24, 26 is an operation of moving the hand further from the state where the hand is stopped at 25, and 27 is an operation corresponding to 26 and brought the hand to the left By the way, the state in which the hand is stopped, 28 indicates the operation of returning the hand from the state in which the hand is stopped at 27 to the right.
Roughly speaking, the value of the position parameter xx of the feature quantity that increases the output probability is set to a value located from the left to the right in the order of 27> 25 = 23> 21 = 11> 13 = 15> 17. . In the states 15, 17, 25, and 27, the output probability of the feature quantity with the speed parameter sx being 0 is increased, and the state transition probability of the self-loop is also set to be high. In states 11, 13, 21, and 23, the value of the feature quantity sx that increases the output probability is set to be close to zero. For 13 and 23, the transition probability of the self-loop is lowered. In states 12, 16, 24, and 28, the value of sx is set to a value that represents the rightward movement, and in states 14, 18, 22, and 26, the value of sx is set to a value that represents the leftward movement.
Likelihood calculation, operation identification, and operation state identification method are the same as those in the first embodiment.
The determination process of the start request and the confirmation request is the same as that of the request determination unit 2 described in the first embodiment.
For example, the request determination means outputs candidate change request 1 when a transition of state 12-13 is detected in a state where the candidate change request operation is obtained as an identification result, and a certain number of transitions of state 15-15 occur. Candidate change request 2 is output when detected continuously, and candidate change request 3 is output when a certain number of transitions in states 17-17 are detected continuously. Also, a candidate change request 4 is output when a transition in the state 22-23 is detected, a candidate change request 5 is output when a certain number of transitions in the state 25-25 are continuously detected, and a state 27-27 The candidate change request 6 is output when a certain number of transitions are detected continuously.
When receiving the candidate change request 1, the control means determines the next candidate by stepping through the circulation list one by one in the forward direction, and upon receiving candidate change request 2, determines the next candidate by skipping the circulation list one by one in the forward direction. When the candidate change request 3 is received, the next list is determined by skipping the circulation list in the forward direction, and when the candidate change request 4 is received, the circulation list is processed one by one in the reverse direction to determine the next candidate. When the request 5 is received, the next candidate is determined by skipping the circulation list in the reverse direction to determine the next candidate. When the candidate change request 6 is received, the next candidate is determined by skipping the circulation list in the opposite direction. Candidate change request including fast-forwarding is realized by combining motion and stationary motion.

Candidate change by continuous operation-block diagram of deterministic item selection device Request identification block diagram Operation model example 1 Jog dial for internal status display Operation model example 2 Operation model example 3 Operation model example 4

Explanation of symbols

100 ... Candidate change by continuous operation-deterministic item selection device 101 ... Candidate set 102 ... Selection item 103 ... Candidate 104 ... Candidate display means 105 ... Input operation 106 ... Sensor 107 ... request identifying means 108 ... feature quantity calculating means 109 ... action identifying means 110 including continuous motions ... operation state identifying means 111 ... request judging means 112 ... control means 200 ..Request identifying means 201... User input operation 202... Sensor 203... Sensor output 204... Feature amount calculating means 205 .. Likelihood calculating means 206. Operation identification means 208 ... Operation state identification means 209 ... Request determination means 211 ... Adjustment parameter calculation means 212 ... Feature amount history 213 ... Operation state history 220 ... Request identification means 2
221... Feature amount calculation means 2
222... Operation position detecting means 223... Request determining means 2
310 ... Probability request action probability model 311 ... Confirmation request action model start node 312 ... Confirmation request action model end node 313 ... Confirmation request action model operation state node 314 ... Confirmation request Action model transition arc 320 ... Candidate change request action probability model 321 ... Candidate change request action model start node 322 ... Candidate change request action model end node 323 ... Candidate change request action model Operation state node 324 ... Transition arc 330 of candidate change request operation model ... Garbage probability model 331 ... Garbage model start node 332 ... Garbage model end node 333 ... Garbage model operation state Node 334 ... Garbage model transition arc 400 ... Inside the jog dial Image 401 ... rotary member 402 ... mark position candidate 1
403 ... Mark position candidate 2
410: Display example of jog dial when candidate change request is not identified 411 ... Rotating body 412 ... Mark 420 ... Display example 1 of jog dial when candidate change request is identified
421 ... Mark 430 ... Display example 2 of jog dial when candidate change request is identified
431 ... Mark 500 ... Probability model example 1 of candidate change request operation
600 ... Probabilistic model example 2 of candidate change request operation
710 ... Probability model example 3 of candidate change request action
720 ... Probability model example 4 of candidate change request operation.

Claims (11)

Candidate display means for displaying selection item candidates to the user, user request input means, request identification means for identifying whether the user request is a candidate change or confirmation, and control means, the control means includes the request When a candidate change request is received from the identification means, an item is selected and displayed as a new candidate and displayed on the candidate display means. When a confirmation request is received from the request identification means, the current candidate is confirmed as a selection item. A candidate change-deterministic item selection method for selecting an arbitrary item belonging to an item set, wherein the input means has a sensor for measuring a physical quantity that is changed by an operation performed by a user without directly touching an interface component, and the request means An identification unit analyzes the output of the sensor at each time interval to calculate a feature amount time series, and a unit for identifying a continuous operation using the feature amount , Item selection method characterized by having means for identifying the operating state constituting the continuous operation. The item selection method according to claim 1, wherein a time-series history storage unit of the feature amount calculated by the object feature amount calculation unit, and an operation state history storage unit obtained by the operation state identification unit; And an adjustment parameter calculation means for outputting an adjustment parameter for adjusting the behavior of the behavior model by using the feature quantity history and the operation state history. Selection method. 3. The item selection method according to claim 2, further comprising request identifying means 2 for receiving whether or not the user's requested action is a start request action in response to the output of the sensor, A feature amount calculating means 2 for analyzing the output of the sensor for each time interval to calculate a time series of the feature amount; and an operation position calculating means for calculating an operation position where the requested operation is performed from the feature amount. The start request output by the identification means is used as a trigger for starting an operation for item selection, and the operation position is used for calculation of the adjustment parameter by the adjustment parameter calculation means. Method. 3. The item selection method according to claim 2, further comprising request identifying means 3 for receiving whether the requested operation of the user is an adjustment requesting operation upon receiving the output of the sensor, wherein the request identifying means includes the request identifying means. It has a feature quantity calculation means 3 for calculating the time series of the feature quantity by analyzing the sensor output at each time interval, and an action position calculation means 2 for calculating the action position where the requested action is performed from the feature quantity. The item selection method, wherein the position is used for calculating the adjustment parameter in the adjustment parameter calculation means.   The internal state display means has a shape that represents a rotating body and a mark that represents the angle of rotation. The shape imitating a so-called jog dial is displayed using the internal state display means, and an estimated value related to the user's operating state is displayed. The item selection method according to claim 1, wherein the item is displayed using a shape imitating a jog dial. Candidate display means for displaying selection item candidates to the user, user request input means, request identification means for identifying whether the user request is a candidate change or confirmation, and control means, the control means includes the request When a candidate change request is received from the identification means, an item is selected and displayed as a new candidate and displayed on the candidate display means. When a confirmation request is received from the request identification means, the current candidate is confirmed as a selection item. A candidate change-deterministic type item selection device for selecting an arbitrary item belonging to an item set, wherein the input means has a sensor for measuring a physical quantity that is changed by an operation performed by a user without directly touching an interface component, and the request means An identification unit analyzes the output of the sensor at each time interval to calculate a feature amount time series, and a unit for identifying a continuous operation using the feature amount , Item selection device characterized by having means for identifying the operating state constituting the continuous operation. 7. The item selection device according to claim 6, wherein a storage unit for time series history of feature quantities calculated by the object feature quantity calculation unit, and a storage unit for operation state history obtained by the operation state identification unit; And an adjustment parameter calculation means for outputting an adjustment parameter for adjusting the behavior of the behavior model by using the feature quantity history and the operation state history. Selection device. 8. The item selection device according to claim 7, further comprising request identifying means 2 for identifying whether or not a user's requested action is a start request action in response to an output of the sensor, wherein the request identifying means 2 A feature quantity calculating means 2 for analyzing the sensor output at each time interval to calculate a time series of the feature quantity, and an action position calculating means for calculating the action position where the requested action has been performed from the feature quantity. The start request output by the means is used as a trigger for starting an operation for selecting an item, and the operation position is used for calculation of the adjustment parameter by the adjustment parameter calculation means. . 8. The item selection device according to claim 7, further comprising request identifying means 3 for receiving whether the user's requested operation is an adjustment request operation in response to an output of the sensor, wherein the request identifying means It has a feature quantity calculation means 3 for calculating the time series of the feature quantity by analyzing the sensor output at each time interval, and an action position calculation means 2 for calculating the action position where the requested action is performed from the feature quantity. The position is used for calculation of the adjustment parameter in the adjustment parameter calculation means.   A shape imitating a so-called jog dial having an internal state display means and a shape representing a rotating body and a mark representing the rotation angle is displayed using the internal state display means, and an estimated value related to the user's operating state is displayed. The item selection device according to claim 6, wherein the item selection device displays using a shape imitating a jog dial. The item selection device according to claim 10, further comprising a remote control device equipped with either a jog dial as an actual interface component or a jog dial as a virtual interface component configured using a touch pad or the like.

Images