JP2014085964A - Information processing method, information processing device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing method, an information processing device, and a program that are improved in a recognition rate of a gesture.SOLUTION: In an information processing method for photographing a user's palm to perform conversion into image data, detecting the user's palm, comparing a portion detected as the user's palm to analyze a motion of the palm, specifying a gesture predefined as a motion pattern of the palm, and performing processing predefined in association with an application being executed and the specified gesture, a gesture is specified in accordance with a movement direction of the portion detected as the palm within a predetermined speed range.

Description

  The present invention relates to an information processing method, an information processing apparatus, and a program capable of performing a gesture operation.

In recent years, information processing apparatuses having various functions have been developed due to technological progress, and a personal computer capable of performing a gesture operation can be cited (for example, see Patent Document 1).
The invention described in Patent Document 1 relates to “interface device and method thereof”. Specifically, “the gesture information display unit includes a list of gestures used for operation, a recognition result of the gesture recognition unit, and an operator's The image of the part that seems to be a hand is displayed, and the operator can operate while checking the screen without memorizing the gesture, and the gesture made by the operator can be corrected so that the gesture recognition unit can easily recognize it. Operability is improved. "

JP 2008-52590 A

  However, although the invention described in Patent Document 1 improves the operability by correcting the gesture performed by the operator, there is an individual difference in the speed of the gesture by the operator, and the improvement of the recognition rate is improved. There is room.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an information processing method, an information processing apparatus, and a program that improve the gesture recognition rate.

  In order to solve the above problem, the invention according to claim 1 shoots the palm of the user and converts it into image data, detects the palm of the user from the image data, and detects the palm of the user in a plurality of image data. Analyzing the palm movement by comparing the detected parts, identifying a predefined gesture as the palm movement pattern, and predefining the application being executed and the identified gesture In the information processing method for performing the process, the gesture is specified according to the moving direction of the part detected as a palm within a predetermined speed range.

  ADVANTAGE OF THE INVENTION According to this invention, provision of the information processing method, information processing apparatus, and program which improved the recognition rate of gesture is realizable.

(A) is the external appearance perspective view which shows the state in use of the notebook computer which concerns on one Embodiment, (b) is the external appearance perspective view seen from the user side of the notebook computer shown to (a). It is a block diagram which shows one Embodiment of the notebook personal computer as information processing apparatus. (A), (b) is explanatory drawing which shows an example of the gesture used for the notebook personal computer 100 shown in FIG. In this embodiment, a case where the user holds the right hand over the camera 103 will be described. It is an example of the flowchart for demonstrating operation | movement of the notebook computer shown in FIG. FIG. 3 is an example of a screen displayed when the gesture application is first activated in the notebook computer shown in FIG. 1. It is an example of the screen displayed when practicing gesture. It is an example of the screen displayed when gesture practice is complete | finished. It is an example of the screen displayed when a gesture function is started.

Next, the present embodiment will be described with reference to the drawings.
In this embodiment, the case where the information processing apparatus is a notebook personal computer will be described. However, the present invention is not limited to a notebook personal computer, and may be a desktop personal computer.

<Overview>
In this embodiment, “hovering” in which the movement of the mouse cursor is assigned to the movement of the palm in the camera image plane coordinates, and “flick”, “circle”, “wave” which are the movements of the palm in the camera image plane coordinates are also used. In order to realize gestures seamlessly, in addition to the movement of the palm, in addition to the movement of the palm, the starting point of gesture analysis (triggered when the palm is stationary for a certain period of time) and the movement speed (recognizable time range and Including the addition of (travel distance).

  In addition, in this embodiment, seamless realization of hovering and other gestures is possible only by one of the movement direction of the palm, the starting point of the vector analysis, and the movement speed, and the operation is performed by simultaneously determining three points. The improvement of the nature is aimed at. For example, if only the start point of vector analysis is added, hovering and other gestures can be quickly switched if the palm rest time for detecting the start point is shortened, but the malfunction rate at unintended stillness increases. By adding the moving speed here, it is possible to realize high operability with a short rest time and a low misrecognition rate.

<Configuration>
FIG. 1A is an external perspective view showing a state in which the notebook computer according to one embodiment is being used, and FIG. 1B is a view from the user side of the notebook computer shown in FIG. It is an external perspective view.
The laptop computer 100 recognizes a user's hand with a Web camera (hereinafter referred to as a camera) 103 in addition to an operation with a keyboard 101 and a mouse 102, and the recognized palm movement (also referred to as a gesture or hand gesture). Thus, a certain operation can be performed on the notebook personal computer 100.
The liquid crystal display device 104 is capable of displaying a user's face and hands and displaying an image for practicing gesture operation in addition to performing normal display.

FIG. 2 is a block diagram showing an embodiment of a notebook personal computer as an information processing apparatus.
A notebook computer 100 includes a keyboard 101, a mouse 102, a camera (Web camera) 103 as a photographing means, a display device 104, an optical reader 105, an HDD (Hard Disc Drive) 106, a control unit 107, and a RAM (Random Access Memory). / Write memory) 108, ROM (Read Only Memory) 109, and I / O (Input / Output) 110, but the present invention is not limited to this. Reference numeral 111 denotes a bus line.

  The keyboard 101 has normal input keys such as alphabetic keys in a QWERTY layout, numeric keys, numeric keys, space keys, enter keys, function keys, delete keys, and space keys.

  The mouse 102 has at least a left button and a right button, and may have a wheel button. In the figure, the mouse 102 is connected to the notebook computer 100 with a cable, but the present invention is not limited to this, and the mouse 102 may be connected to the notebook computer 100 wirelessly.

  The camera 103 is provided toward the user in the center of the upper side of the upper casing of the notebook computer 100, and for example, a CCD (Charge-Coupled Device) is used.

  For example, a liquid crystal display is used as the display device 104, but a plasma display may be used instead of the liquid crystal display.

  The optical reading device 105 is a device that reads an optical medium such as a DVD (Digital Versatile Disc), a CD (Compact Disc), or a CDR (CD Recordable).

  The HDD 106 is a storage device for storing large amounts of data such as character data, image data, and audio data as compared with the RAM 108. Instead of the HDD 106, an SSD (Solid State Drive) may be used.

  The control unit 107 has a function of comprehensively controlling the notebook personal computer 100 and is, for example, a circuit including a CPU (Central Processing Unit) of a microprocessor. The control unit 107 includes input control means 107a, training means 107b, analysis means 107c, interrupt control means 107d, disk control means 107e, display device control means 107f, and execution means 107g as software.

  The input control unit 107 a manages data from the keyboard 101, the mouse 102, the camera 103, and the optical reading device 105.

The training means 107b is configured by software, and forcibly practice gestures at the first activation of a gesture application (hereinafter referred to as a gesture application) in order to improve recognition accuracy for recognizing a user's palm movement as a gesture. It has a function to make it.
Here, the gesture application is an application in which processing corresponding to a gesture is defined in advance.
The gesture in the present embodiment means a user's palm movement within a range that can be captured by the camera 103 of the notebook computer 100 (also referred to as a camera image), and does not move the second arm or move the whole body.

The analysis unit 107c is configured by software, and has a function of analyzing image data from the camera 103 and recognizing the palm movement of the user as a gesture.
The interrupt control unit 107d controls interrupt processing that interrupts processing of another program during processing of a certain program.
The disk control unit 107e determines the type of the optical medium inserted into the optical reader 105, and manages the reading of character data, image data, and audio data.
The display device control unit 107f manages the image display of the display device 104.
The execution unit 107g executes an application.

For example, a flash memory is used as the RAM 108.
For example, a mask ROM is used as the ROM 109.
The I / O 109 manages data exchange between the notebook personal computer 100 and an external device including a USB (Universal Serial Bus) flash memory (not shown).
Note that the notebook computer 100 has a hardware configuration similar to that of an ordinary notebook computer, and may include a tuner for viewing digital terrestrial broadcasting and a modem for Internet connection.

FIGS. 3A and 3B are explanatory diagrams showing examples of gestures used in the notebook computer 100 shown in FIG. 1, and are views of the user's palm movement viewed from the user's face. In this embodiment, the case where the user holds the right hand over the camera 103 will be described, but the left hand may be used.
In the figure, a single-line arrow (→) represents a transition in hand shape or hand state, and a hollow arrow represents a vector.
In the embodiment, the notebook computer 100 is operated by 12 types of gestures, but the present invention is not limited to this.
FIG. 3A is a diagram showing a mouse cursor operation by the user's right hand, and FIG. 3B shows a case of moving as indicated by an arrow P1, but is not limited thereto.
As shown in FIG. 3 (b), the user's palm movements at predetermined time intervals are a plurality of (in the figure, five, but not limited) vectors V1 ←, V2 ←, V3 ←, V4 ←, V5 ←. Can be disassembled. The movement of the palm is analyzed from the change of these vectors V1 ←, V2 ←, V3 ←, V4 ←, V5 ←, and it is determined whether a predetermined command or a cursor operation is performed. Whether it is a command or not can be determined by the direction and speed. Whether or not the palm movement is a cursor operation can be determined by the immediately preceding command. That is, when the moving speed is larger than a predetermined value, it is determined that the command corresponds to the direction of the vector, and when the moving speed is lower than the predetermined value, it is determined that the cursor is operated in the direction of the vector.

  Here, “click” corresponds to the left click of the mouse, and “drag and drop” corresponds to the drag and drop of the mouse. Also, right flick is "forward", left flick is "back", upper flick is "up scroll", lower flick is "down scroll", clockwise circle is "enlarge", counterclockwise circle is "reduced" “,” “Pull” corresponds to activation of a portable information terminal (not shown), and “wave” corresponds to a command operation for returning to the start screen.

<Operation>
FIG. 4 is an example of a flowchart for explaining the operation of the notebook computer shown in FIG. FIG. 5 is an example of a screen displayed when the gesture application is first activated in the notebook computer shown in FIG. 1, FIG. 6 is an example of a screen displayed when practicing gesture, and FIG. FIG. 8 is an example of a screen displayed when the gesture practice is finished, and FIG. 8 is an example of a screen displayed when the gesture function is activated.

The main body of the operation of the flowchart shown in FIG. 4 is the control unit 107 shown in FIG.
The control unit 107 determines whether or not the user's palm is stationary for T _{* 1} time (step S1). When the control unit 107 determines that the palm is stationary (step S1 / Yes), the palm moves under a specific condition. It is determined whether there is (*) (step S2).
When the control unit 107 determines that there is movement of the palm (step S2 / Yes), the control unit 107 determines whether or not the elapsed time after detecting the stillness of the palm is within T _{* 2} (step S3), and the elapsed time T _{* 2.} If it is determined that it is within (step S3 / Yes), it is recognized as the corresponding gesture (step S4).
When the control unit 107 determines that the palm is not stationary for T _{* 1} time (step S1 / No), or determines that there is no palm movement (*) under the specific condition (step S2 / No), the control unit 107 If it is determined that the elapsed time after the stationary detection is not within T _{* 2} (No in step S3), the process returns to step S1.

Here, the up / down / left / right flick, circle, wave, and mouse circle operations will be described in detail again.
<Up / down / left / right flick>
Direction of palm movement: Up / Down / Left / Right Direction Gesture analysis start point: Point where the palm has stopped at T _f1 Movement speed: Movement over a certain distance (D _f ) within a certain time (T _f2 ) * T _f1 , T _f2 , Df: Arbitrary values When the palm of the hand is T _f1 stationary, the analysis of the gesture is started from that stationary point.
If the movement of the palm equal to or greater than D _f is detected in a straight line in either the top, bottom, left or right direction within T _f2 after the start of analysis, it is recognized as a flick in the movement direction.

<Circle>
Movement direction of the palm: locus of circle Gesture analysis start point: point where the palm is _Tc1 stationary Movement speed: circular movement of a certain radius ( _Dc ) or more within a certain time ( _Tc2 ) * _Tc1 , _Tc2 , D _c : Arbitrary value When the palm of the hand is T _c1 , the analysis of the gesture is started from that point.
If the movement of a circular palm with a radius Dc or more is detected within T _c2 after the start of analysis, it is recognized as a circle.

<Wave>
Movement direction of palm: Reciprocating left and right Gesture analysis start point: Point where palm is stationary T _W1 Movement speed: Within a certain time (T _w2 ), a certain width (D _w ) or more and a certain number of times (N _w ) * T _w1 , T _w2 , N _w : Arbitrary values When the palm of the palm stops at T _w1 , the analysis of the gesture is started from that stationary point.
After analysis start, within T _w2, the movement of the hand alternately left and right directions above Nw at D _w or more swing width of wave upon detecting recognize.

<Mouse cursor operation>
A palm movement in the camera plane direction that does not satisfy the above condition is recognized as a mouse cursor operation.

When the gesture application is activated, the menu screen shown in FIG. 5 is displayed.
In FIG. 5, the gesture function setting screen as a menu screen is an example of a note. “Hand gestures allow the user to operate the information processing device with only a palm gesture without using a remote control or a mouse. "Is displayed. The operation of the information processing apparatus means the above-described cursor operation or command operation.

  On the left side of the screen shown in FIG. 5, a plurality of buttons (four in the figure, but not limited) B1 to B4 are displayed. Of these, only the “practice” and “cancel” buttons B1 and B4 are active. This is to force the user to practice when the gesture app is first activated. Next to the “Practice” button B1, for example, the text information “I can practice each gesture method. I recommend you practice to use hand gestures comfortably.” Is displayed. . In order to use hand gestures comfortably, it is necessary to improve the recognition rate of gestures, which encourages users to practice. The “Cancel” button B4 is a button for ending the displayed screen.

In the screen shown in FIG. 6, a model image of the gesture is displayed on the left side, and a monitor window is displayed on the right side. A guide icon is displayed on the upper side of the monitor window. The positional relationship between the model video, the monitor window, and the guide icon is not limited to this. In the present embodiment, the guide icon is displayed. However, the present invention is not limited to this, and the guide icon may not be displayed.
A frame for displaying that the palm position of the user is extracted is displayed on the monitor window.

  Examples of the model image include a slide display image that displays a moving image of an instructor or an animation moving image or a still image of an instructor or an animation still image one frame at a time. At the same time as the model image is displayed, a voice explaining the gesture may be generated, or an explanatory text may be displayed in the model image.

  The monitor window displays an image including the user's hand taken by the camera 103. However, if displayed as it is, the instructor's palm movement as a model image will be reversed and the user may be confused. It is designed to display the same movement as the instructor by flipping left and right. A rectangle is displayed around the user's palm in the monitor window. This is for displaying the fact that the palm is extracted in the image processing of the notebook personal computer 100, thereby urging the user to position the palm within the screen.

For example, on the upper side of the model image, press “>” to go to the next gesture practice. To return to the previous gesture practice, press “<”. "Is displayed. When the user selects “>” or “<” according to this display, a model image of the gesture to be practiced is displayed.
A close radio button is displayed on the lower right side of the monitor window.

  When the user completes the gesture practice, clicking the end button by operating the keyboard 101 or the mouse 102 ends the hand gesture screen. Set the gesture function.

A gesture function setting screen shown in FIG. 7 is displayed. The difference between the screen shown in FIG. 7 and the screen shown in FIG. 5 is that the buttons B2 and B3 are active. The button B2 is a button for using the gesture function, and the button B3 is a button for ending the gesture function.
That is, when the gesture practice is completed, the gesture function can be used.

When the user clicks the button B2 with the keyboard 101 or the mouse 102, the gesture input operation starts and the screen shown in FIG. 8 is displayed. In the center of the screen, for example, the character information “gesture function activated” is displayed for a few seconds. In the lower right corner of the screen, a monitor window having a size of about one-tenth of the screen is displayed. A gesture icon is displayed at the top of the screen. These are images for user confirmation.
Here, the icons are displayed not on the center of the screen but on the periphery. This is in order not to get in the way when displaying, for example, a digital terrestrial television broadcast or internet broadcast screen on the screen.
The control unit 107 performs gesture application recognition based on the moving direction and moving speed of the palm of the user, and performs an operation corresponding to the gesture (for example, an operation such as clicking or flicking). When ending the gesture input, it is ended by clicking an end button in the screen (for example, a close button at the lower right of the screen).

Here, the screen shown in FIG. 7 corresponds to a selection screen for selecting whether or not to display a gesture practice screen when the user performs a gesture operation for the second time or later.
That is, when the user selects display of the practice screen, the practice screen is displayed, and when not selected, the practice screen is omitted and FIG. 8 is displayed as a production screen for gesture operation.

<Effect>
As described above, according to the present embodiment, since the gesture is recognized based on the moving direction and the moving speed of the palm of the user, it is possible to provide an information processing method, an information processing apparatus, and a program with an improved gesture recognition rate. be able to.

<Program>
The image processing apparatus according to the present invention described above is realized by a program that causes a computer to execute processing. Examples of the computer include general-purpose computers such as personal computers and workstations, but the present invention is not limited to this. Therefore, as an example, a case where the present invention is realized by a program will be described below.

For example,
In the computer of the information processing device,
A procedure in which the execution means executes the application;
The procedure by which the photographing means shoots the palm of the user and converts it to image data,
Analyzing means detects the palm of the user from the image data, analyzes the palm movement by comparing the parts detected as the palm of the user in a plurality of image data, and generates a gesture predefined as a palm movement pattern. Steps to identify,
A procedure in which the control means performs a predefined process in association with the application being executed by the execution means and the identified gesture;
A program for executing
A procedure for the analysis means to identify a gesture according to a moving direction of a portion detected as a palm within a predetermined speed range;
A program that executes

Thus, the information processing apparatus according to the present invention can be realized anywhere as long as there is a computer environment capable of executing the program.
<Storage medium>

Such a program may be stored in a computer-readable storage medium.
Here, examples of the storage medium include a computer-readable storage medium such as a CD-ROM (Compact Disc Read Only Memory), a flexible disk (FD), and a CD-R (CD Recordable), a flash memory, a RAM, and a ROM. And semiconductor memories such as FeRAM (ferroelectric memory) and HDDs.

  The above-described embodiment shows an example of a preferred embodiment of the present invention, and the present invention is not limited thereto, and various modifications can be made without departing from the scope of the invention. is there.

100 Notebook PC 101 Keyboard 102 Mouse 103 Camera (Web Camera)
104 Liquid crystal display device 105 Optical reader 106 HDD
107 control section 107a input control means 107b training means 107c analysis means 107d interrupt control means 107e disk control means 107f display device control means 107g execution means 108 RAM
109 ROM
110 I / O
111 bus line

Claims (4)

The user's palm is photographed and converted into image data, the user's palm is detected from the image data, and the operation of the palm is analyzed by comparing portions detected as the user's palm in a plurality of image data. In the information processing method for identifying a gesture defined in advance as the palm movement pattern, and performing a predefined process in association with the application being executed and the identified gesture,
An information processing method characterized by identifying a gesture according to a moving direction of a portion detected as a palm within a predetermined speed range.   The information processing method according to claim 1, wherein the photographed image of the user and a model image by the instructor are displayed on the same screen. An execution means for executing the application;
Photographing means for photographing a user's palm and converting it into image data;
The user's palm is detected from the image data, the movement of the palm is analyzed by comparing portions detected as the user's palm in a plurality of image data, and a gesture defined in advance as the palm movement pattern is analyzed. Analysis means for identifying
In an information processing apparatus comprising: an application executed by the execution unit; and a control unit that performs a predefined process in association with the specified gesture,
The information processing apparatus is characterized in that the analysis unit specifies a gesture according to a moving direction of a portion detected as a palm within a predetermined speed range. In the computer of the information processing device,
A procedure in which the execution means executes the application;
The procedure by which the photographing means shoots the palm of the user and converts it to image data,
Analyzing means detects the palm of the user from the image data, analyzes the operation of the palm by comparing portions detected as the palm of the user in a plurality of image data, and previously as an operation pattern of the palm Procedures to identify defined gestures,
A procedure in which the control means performs a predefined process in association with the application being executed by the execution means and the specified gesture;
A program for executing
A procedure in which the analyzing means specifies a gesture according to a moving direction of a portion detected as a palm within a predetermined speed range;
A program characterized by having executed.

Images