JP5831883B2 - Information processing apparatus, information processing method, and program - Google Patents

Description

  The present invention relates to an information processing apparatus capable of performing a gesture operation, an information processing method, and a program.

In recent years, information processing apparatuses having various functions have been developed due to technological progress, and a personal computer capable of performing a gesture operation can be cited (for example, see Patent Document 1).
The invention described in Patent Document 1 relates to an “interface device and method thereof”. Specifically, “a gesture information display unit in an interface device for operating a device such as a television or a computer has a gesture used for the operation. List of gestures, recognition results of the gesture recognition unit, and an image of the part that seems to be an operator's hand, so that the operator can operate while checking the screen without memorizing the gesture, and in the gesture recognition unit It is possible to correct the gesture performed by the operator so that it can be easily recognized, and the operability is improved.

JP 2008-52590 A

  However, since the invention described in Patent Document 1 operates while confirming the screen each time without memorizing the gesture, it takes time from the start of the gesture operation to recognition, and the gesture operation is performed while confirming the screen. There is room for improvement.

  Accordingly, an object of the present invention is to provide an information processing apparatus, an information processing method, and a program that improve the recognition rate of gesture motion.

In order to solve the above-mentioned problem, the invention described in claim 1 includes an execution unit that executes an application, a display unit that displays an execution result of the application, and a shooting unit that captures a palm of a user and converts the image into image data. analyzes the operation of the palm on the basis of a plurality of images data from the imaging means, and analyzing means for identifying a predefined gesture as an operation pattern of the palm, and applications running in said execution means And an information processing apparatus including a control unit configured to perform a predetermined process in association with the specified gesture, when a gesture is specified by the analysis unit, display corresponding to the specified gesture is performed. further comprising a display means for displaying a rectangle to enclose the palm of the user with And features.

  ADVANTAGE OF THE INVENTION According to this invention, provision of the information processing apparatus, the information processing method, and program which aimed at the improvement of the recognition rate of gesture operation | movement is realizable.

(A) is the external appearance perspective view which shows the state in use of the notebook computer which concerns on one Embodiment, (b) is the external appearance perspective view seen from the user side of the notebook computer shown to (a). It is a block diagram which shows one Embodiment of the notebook personal computer as information processing apparatus. (A)-(n) is explanatory drawing which shows an example of the gesture used for the notebook personal computer 100 shown in FIG. It is an example of the flowchart for demonstrating operation | movement of the notebook computer shown in FIG. FIG. 3 is an example of a screen displayed when the gesture application is first activated in the notebook computer shown in FIG. 1. It is an example of the screen in the case of gesture recognition success displayed when performing gesture practice. It is an example of the screen in the case of gesture recognition failure displayed when performing gesture practice. It is an example of the screen displayed when gesture practice is complete | finished. It is an example of the screen displayed when a gesture function is started.

Next, the present embodiment will be described with reference to the drawings.
In this embodiment, the case where the information processing apparatus is a notebook personal computer will be described. However, the present invention is not limited to a notebook personal computer, and may be a desktop personal computer.

<Configuration>
FIG. 1A is an external perspective view showing a state in which the notebook computer according to one embodiment is being used, and FIG. 1B is a view from the user side of the notebook computer shown in FIG. It is an external perspective view.
The laptop computer 100 recognizes the palm of the user with a Web camera (hereinafter referred to as a camera) 103 in addition to operations with the keyboard 101 and the mouse 102, and recognizes the palm movement (also referred to as a gesture or hand gesture). Thus, a certain operation can be performed on the notebook personal computer 100.
The liquid crystal display device 104 is capable of displaying a user's face and hands and displaying an image for practicing gesture operation in addition to performing normal display.

FIG. 2 is a block diagram showing an embodiment of a notebook personal computer as an information processing apparatus.
The notebook computer 100 is similar to a normal notebook computer in terms of hardware, and includes a keyboard 101, a mouse 102, a camera (Web camera) 103 as a photographing means, a display device 104, an optical reading device 105, an HDD (Hard Disc Drive). ) 106, control unit 107, RAM (Random Access Memory) 108, ROM (Read Only Memory) 109, and I / O (Input / Output) 110. However, the present invention is not limited to this, and a tuner for viewing digital terrestrial television broadcasting and a modem for connecting to the Internet may be provided. Reference numeral 111 denotes a bus line.

  The keyboard 101 has normal input keys such as alphabetic keys in a QWERTY layout, numeric keys, numeric keys, space keys, enter keys, function keys, delete keys, and space keys.

  The mouse 102 has at least a left button and a right button, and may have a wheel button. In the figure, the mouse 102 is connected to the notebook computer 100 with a cable, but the present invention is not limited to this, and the mouse 102 may be connected to the notebook computer 100 wirelessly.

  The camera 103 is provided toward the user in the center of the upper side of the upper casing of the notebook computer 100, and for example, a CCD (Charge-Coupled Device) is used.

  The display device 104 has a function of performing display corresponding to the specified gesture or display under the control of the control unit when the gesture is specified by the analyzing unit 107c. For example, a liquid crystal display is used, but the display device 104 is replaced with a liquid crystal display. Alternatively, a plasma display may be used.

  The optical reading device 105 is a device that reads an optical medium such as a DVD (Digital Versatile Disc), a CD (Compact Disc), or a CDR (CD Recordable).

  The HDD 106 is a storage device for storing large amounts of data such as character data, image data, and audio data as compared with the RAM 108. Instead of the HDD 106, an SSD (Solid State Drive) may be used.

  The control unit 107 has a function of comprehensively controlling the notebook personal computer 100 and is, for example, a circuit including a CPU (Central Processing Unit) of a microprocessor. The control unit 107 includes input control means 107a, training means 107b, analysis means 107c, interrupt control means 107d, disk control means 107e, display device control means 107f, and execution means 107g as software.

  The input control unit 107 a manages data from the keyboard 101, the mouse 102, the camera 103, and the optical reading device 105.

The training means 107b is configured by software, and forcibly practice gestures at the first activation of a gesture application (hereinafter referred to as a gesture application) in order to improve recognition accuracy for recognizing a user's palm movement as a gesture. It has a function to make it.
The gesture in the present embodiment means a palm action of the user in a range that can be taken by the camera 103 of the notebook computer 100 (also referred to as a camera image).

The analysis unit 107c is configured by software, and has a function of analyzing image data from the camera 103 and recognizing the palm movement of the user as a gesture.
The interrupt control unit 107d controls interrupt processing that interrupts processing of another program during processing of a certain program.
The disk control unit 107e determines the type of the optical medium inserted into the optical reader 105, and manages the reading of character data, image data, and audio data.
The display device control unit 107f manages the image display of the display device 104.
The execution unit 107g executes an application.

For example, a flash memory is used as the RAM 108.
For example, a mask ROM is used as the ROM 109.
The I / O 109 manages data exchange between the notebook personal computer 100 and an external device including a USB (Universal Serial Bus) flash memory (not shown).

3A to 3N are diagrams showing examples of gestures used in the notebook computer 100 shown in FIG. 1, and are views of the user's own hand movement as seen from the user's viewpoint. In this embodiment, the case where the user holds the right hand over the camera 103 will be described, but the left hand may be used.
In the figure, a single-line arrow (→) represents a transition of a hand shape or a hand state, and a hollow arrow represents a movement of the palm.
In the present embodiment, the notebook computer 100 is operated by 12 types of gestures, but the present invention is not limited to this.
FIG. 3A is a diagram showing a mouse cursor operation by the user's right hand, and shows a case of moving as indicated by a double arrow, but is not limited thereto.
In FIG. 3B, a left-click gesture operation is performed by opening and closing the palm like “par → goo → par”.
3C to 3E, a gesture operation of dragging and dropping is performed by moving the palm like “par → goo → moving with goo → par”.
In FIG. 3 (f), a gesture operation of a right flick is performed by moving more quickly than clicking in the arrow direction (right direction) with the palm spread. Here, “flick” is originally a kind of operation on the touch panel, which is to move a finger so as to lightly touch the screen, but includes quickly moving the palm.

  In FIG. 3 (g), a gesture operation of a left flick is performed by moving more quickly than clicking in the arrow direction (left direction) with the palm spread.

  In FIG. 3 (h), an upward flick gesture is performed by moving the finger in the direction of the arrow (upward) more quickly than clicking with the palm extended.

  In FIG. 3 (i), a gesture operation of a lower flick is made by moving the hand in the direction of the arrow (downward) more quickly than clicking with the palm expanded.

  FIG. 3J shows a clockwise circle gesture operation by drawing a clockwise circle (diameter of about 10 cm to 20 cm) about the size of the display device 104 of the notebook personal computer 100 with the palm spread.

  FIG. 3K shows a counterclockwise circle gesture operation by drawing a counterclockwise circle (diameter of about 10 cm to 20 cm) of the size of the display device 104 of the notebook personal computer 100 with the palm spread. .

  In FIG. 3L, a push gesture operation is performed by pushing toward the display device 104 of the notebook personal computer 100 with the palm spread.

  FIG. 3 (m) shows a pull gesture operation by pushing and pointing away from the display device 104 of the notebook personal computer 100 with the palms extended.

  In FIG. 3 (n), a wave gesture operation is performed by swinging left and right toward the display device 104 of the notebook computer 100 with the palm spread. This gesture movement is also called a wave.

  Here, “click” corresponds to the left click of the mouse, and “drag and drop” corresponds to the drag and drop of the mouse. Also, right flick is "forward", left flick is "back", upper flick is "up scroll", lower flick is "down scroll", clockwise circle is "enlarge", counterclockwise circle is "reduced" “,” “Pull” corresponds to activation of a portable information terminal (not shown), and “wave” corresponds to a command operation for returning to the start screen.

<Operation>
FIG. 4 is an example of a flowchart for explaining the operation of the notebook computer shown in FIG. FIG. 5 is an example of a screen displayed when the gesture app is first activated in the notebook computer shown in FIG. FIG. 6 is an example of a screen in the case of successful gesture recognition displayed when performing a gesture practice, and FIG. 7 is an example of a screen in the case of a gesture recognition failure displayed when performing a gesture practice. FIG. 8 is an example of a screen that is displayed when the gesture practice is finished, and FIG. 9 is an example of a screen that is displayed when the gesture function is activated.

The main body of the operation of the flowchart shown in FIG. 4 is the control unit 107 shown in FIG.
The control unit 107 determines whether or not the gesture app has been activated (step S1). If the gesture app is not activated, the control unit 107 waits (step S1 / No). If the gesture app is activated, the control unit 107 is illustrated in FIG. Display the menu screen.
In FIG. 5, the gesture function setting screen as a menu screen is an example of a note. “Hand gestures allow the user to operate the information processing device with only a palm gesture without using a remote control or a mouse. "Is displayed. The operation of the information processing apparatus means the above-described cursor operation or command operation.

  On the left side of the screen shown in FIG. 5, a plurality of buttons (four in the figure, but not limited) B1 to B4 are displayed. Of these, only the “practice” and “cancel” buttons B1 and B4 are active. This is to force the user to practice when the gesture app is first activated. Next to the “Practice” button B1, for example, the text information “I can practice each gesture method. I recommend you practice to use hand gestures comfortably” is displayed. . In order to use hand gestures comfortably, it is necessary to improve the recognition rate of gestures, which encourages users to practice. The “Cancel” button B4 is a button for ending the displayed screen.

It is determined whether or not the first page of the practice screen is displayed when the user presses the button B1 with the mouse 102 or the keyboard 101 (step S2).
When the control unit 107 determines that the user has pressed the button B1 (step S2 / Yes), the control unit 107 displays an image including the user's hand taken by the camera 103 as a hand gesture practice screen as shown in FIG. . You may give an explanation by a voice and a character with the display of a practice screen. At this time, the control unit 107 analyzes the palm movement of the user (step S3).
It is determined whether or not the recognition of the gesture is successful (step S4). When the recognition is successful (step S4 / Yes), a recognition success icon is displayed.

Here, on the screen shown in FIG. 6, a model image of the gesture is displayed on the left side, and a monitor window is displayed on the right side. A guide icon is displayed on the upper side of the monitor window. The positional relationship between the model video, the monitor window, and the guide icon is not limited to this. In this embodiment, the guide icon is displayed. However, the present invention is not limited to this, and the guide icon may not be displayed.
A frame for displaying that the palm position of the user is extracted is displayed on the monitor window.

  Examples of the model image include a slide display image that displays a moving image of an instructor or an animation moving image or a still image of an instructor or an animation still image one frame at a time. At the same time as the model image is displayed, a voice explaining the gesture may be generated, or an explanatory text may be displayed in the model image.

  The monitor window displays an image including the user's hand taken by the camera 103. However, if displayed as it is, the instructor's palm movement as a model image will be reversed and the user may be confused. It is designed to display the same movement as the instructor by flipping left and right. A rectangle is displayed around the user's palm in the monitor window. This is for displaying the fact that the palm is extracted in the image processing of the notebook personal computer 100, thereby urging the user to position the palm within the screen.

On the upper side of the model image, for example, “The guide icon on the right side will shine when recognized. It was recognized. The image instructor may be smiling. In this case, the corresponding icon among the guide icons displayed on the right side of the description of the recognition success is brightened (or the icon may blink). This glowing icon is an icon for successful recognition.
Below the recognition result, press “>” to go to the next gesture practice. To return to the previous gesture practice, press “<”. "Is displayed. When the user selects “>” or “<” according to this display, a model image of the gesture to be practiced is displayed.
A close button is displayed on the lower right side of the monitor window.

  When gesture recognition fails (step S4 / No), a recognition failure image is displayed (step S12).

In FIG. 7 as an image display of recognition failure, for example, the explanation text on the upper side of the model screen is, for example, “The guide icon on the right side shines when it is recognized. It was not recognized. Try it again. Is displayed. " In this case, the guide icon does not light up or flash, and the instructor's smile may be a straight face.
The user can move the palm again to meet the instructor's movement and cause the notebook computer 100 to analyze it.
When the recognition success icon is displayed (step S5), the control unit 107 determines whether or not to end the practice (step S6). When the practice is not finished, that is, when the gesture practice is continued (step S6 / No), the practice screen of the gesture selected by the user is displayed (step S11), and the process returns to step S3.

  When the end button is clicked by operating the keyboard 101 or the mouse 102 (step S6 / Yes), the hand gesture screen is ended and the process proceeds to step S7.

When the processing proceeds to step S7, the gesture function setting screen shown in FIG. 8 is displayed. The difference between the screen shown in FIG. 8 and the screen shown in FIG. 5 is that the buttons B2 and B3 are active. The button B2 is a button for using the gesture function, and the button B3 is a button for ending the gesture function.
That is, when the gesture practice is completed, the gesture function can be used.

  When the user clicks the button B2 with the keyboard 101 or the mouse 102, the gesture input operation starts (step S7), and the screen shown in FIG. 9 is displayed. In the center of the screen, for example, the character information “gesture function activated” is displayed for a few seconds. In the lower right corner of the screen, a monitor window having a size of about one-tenth of the screen is displayed. A gesture icon is displayed at the top of the screen. As with the practice screen, the guide icon becomes brighter (or may blink) when the gesture is recognized, and gives feedback to the user that the gesture has been recognized.

Here, the icons are displayed not on the center of the screen but on the periphery. This is in order not to get in the way when displaying, for example, a digital terrestrial television broadcast or internet broadcast screen on the screen.
The control unit 107 analyzes the user's palm movement (step S8), and performs an action corresponding to the gesture (for example, an action such as clicking or flicking) (step S9). When ending the gesture input, it is ended by clicking an end button (for example, a close button at the lower right of the screen) in the screen (step S10).

Here, the screen shown in FIG. 8 corresponds to a selection screen for selecting whether or not to display a gesture practice screen when the user performs a gesture operation for the second time or later.
That is, when the user selects display of the practice screen, the practice screen is displayed, and when not selected, the practice screen is omitted and FIG. 9 is displayed as a production screen for gesture operation.

<Effect>
As described above, according to the present embodiment, since the icon display of the gesture recognition result is performed, the recognition rate of the gesture motion can be improved.

<Program>
The information processing apparatus according to the present invention described above is realized by a program that causes a computer to execute processing. Examples of the computer include general-purpose computers such as personal computers and workstations, but the present invention is not limited to this. Therefore, as an example, a case where the present invention is realized by a program will be described below.

For example,
In the computer of the information processing device,
A procedure in which the execution means executes the application;
A procedure in which the display means displays the execution result of the application;
A procedure for the photographing means to photograph the user's palm and convert it to image data,
Analyzing means, analyzing the palm movement based on a plurality of image data from the imaging means, to specify a gesture predefined as the palm movement pattern,
A procedure in which the control means performs a predefined process in association with the application being executed by the execution means and the specified gesture;
In a program that executes
A procedure in which the display means performs display corresponding to the specified gesture when the gesture is specified;
A program that executes

Thus, the information processing apparatus according to the present invention can be realized anywhere as long as there is a computer environment capable of executing the program.
Such a program may be stored in a computer-readable storage medium.

<Storage medium>
Here, examples of the storage medium include a computer-readable storage medium such as a CD-ROM (Compact Disc Read Only Memory), a flexible disk (FD), and a CD-R (CD Recordable), a flash memory, and a RAM (Random). Examples include semiconductor memories such as Access Memory (ROM), Read Only Memory (ROM), and FeRAM (ferroelectric memory), and HDD (Hard Disc Drive).

The above-described embodiment shows an example of a preferred embodiment of the present invention, and the present invention is not limited thereto, and various modifications can be made without departing from the scope of the invention. is there.
For example, in the above-described embodiment, the case where the gesture is practiced has been described. However, the present invention is not limited to this and may be applied to the case where the gesture is input.

100 Notebook PC 101 Keyboard 102 Mouse 103 Web Camera (Camera)
104 Liquid crystal display device 105 Optical reader 106 HDD
107 control section 107a input control means 107b training means 107c analysis means 107d interruption means 107e disk control means 107f display device control means 107g execution means 108 RAM
109 ROM
110 I / O
111 bus line

Claims (6)

An execution means for executing the application;
Display means for displaying an execution result of the application;
Photographing means for photographing a user's palm and converting it into image data;
Analysis means for analyzing the operation of the palm on the basis of a plurality of images data, I identify a predefined gesture as an operation pattern of the palm from the imaging means,
In an information processing apparatus comprising: an application executed by the execution unit; and a control unit that performs a predefined process in association with the specified gesture,
An information processing apparatus, further comprising display means for displaying a rectangle corresponding to the specified gesture and displaying a rectangle so as to surround the palm of the user when a gesture is specified by the analysis means.   The information processing apparatus according to claim 1, wherein the control unit performs display corresponding to a process defined in advance in association with the application being executed by the execution unit and the specified gesture.   The information processing apparatus according to claim 1, wherein the control unit displays a guide icon.   The control means is characterized in that the display of the guide icon corresponding to the specified gesture among the guide icons displayed corresponding to the defined gesture is displayed differently from the display of the other guide icons. The information processing apparatus according to any one of claims 1 to 3. The user's palm is photographed and converted into image data, the palm movement is analyzed based on a plurality of image data, a gesture predefined as the palm movement pattern is identified, and the application being executed and the In an information processing method for performing a predefined process in association with an identified gesture,
An information processing method, wherein when a gesture is specified, a display corresponding to the specified gesture is performed and a rectangle is displayed so as to surround the palm of the user . In the computer of the information processing device,
A procedure in which the execution means executes the application;
A display means for displaying an execution result of the application;
A procedure for the photographing means to photograph the user's palm and convert it to image data,
Analyzing means, analyzing the palm movement based on a plurality of image data from the imaging means, to specify a gesture predefined as the palm movement pattern,
A procedure in which the control means performs a predefined process in association with the application being executed by the execution means and the specified gesture;
In a program that executes
A step of displaying a rectangle so as to surround the palm of the user while the display means performs display corresponding to the specified gesture when a gesture is specified;
A program characterized by having executed.

Images