JP6232694B2 - Information processing apparatus, control method thereof, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, a control method thereof, and a program.

  2. Description of the Related Art Conventionally, when inputting character information such as e-mail by a mobile phone, there is a technique for selecting characters by mainly replacing a numeric keypad with hiragana input buttons and pressing each button a plurality of times. Since inputting and selecting characters using this technique requires a plurality of pressing instructions, there is a problem that it takes more time than inputting from a keyboard or the like.

  On the other hand, in a mobile terminal, a technique is disclosed in which a finger image acquired by a camera is monitored, and character input or function operation is performed based on the monitored finger image (for example, Patent Document 1). The technique described in Patent Document 1 determines a character on which a cursor is positioned on a display screen as an input character when a finger approaches the mobile terminal as a result of monitoring.

JP-A-9-185456

  However, in the technique disclosed in Patent Document 1, in order to press a plurality of types of buttons, the cursor movement operation and the press instruction must be repeated in order.

  In addition, since the keyboard is arranged on a plane, the keyboard may not be fully displayed depending on the size of the terminal screen.

The present invention, while Check the virtual key to move the hierarchy of the pointer without information input by the virtual key contacted by the pointer between hierarchy belongs virtual key moves, the bottom of the virtual key and to provide a mechanism that allows to the information input by the virtual keys by a simple operation of moving upward as contrary to the virtual key without penetrating.

An information processing apparatus according to the present invention is an information processing apparatus that includes a display unit, includes a display unit, and enables information input using a virtual key having a position and thickness defined in a three-dimensional space. Detecting a hierarchy to which the virtual key belongs among a plurality of hierarchies defined in a three-dimensional space, correspondence storage means for storing the virtual key in association with the position of the virtual key, and an indicator for operating the virtual key The position specifying means for specifying the position of the indicator in the three-dimensional space, and the indicator of the position specified by the position specifying means in the state where the indicator exists without touching the virtual key. virtual key, is switched to the virtual key of the destination layer in accordance with the movement between the layer of the indicator, display key display controlling so as to display a display key is a key to be displayed on the display unit And control means, wherein the indicator is whether moves upward as contrary to the virtual key without penetrating the bottom of the virtual keys after contact from above the virtual keys, specified by the position specifying means Determination means for determining based on the position of the indicator that has been made and the position of the virtual key stored in the correspondence storage means, and the indicator passes through the bottom of the virtual key that is in contact with the virtual key. When moving to another level below, the information is not input by the virtual key that has been touched, and the determination means penetrates the bottom of the virtual key after the indicator touches the virtual key. If it is determined to have moved upward as contrary to the virtual key without includes determining means determines to perform the input of information by the virtual key, is possible to input in the determining means it is determined Information And the display key display control means moves the indicator through the bottom of the touching virtual key to a level below the virtual key. case, the control to said Rukoto order to display the virtual key of the destination of the lower hierarchy as display key.

According to the present invention, while Check the virtual key to move the hierarchy of the pointer without information input by the virtual key contacted by the pointer between hierarchy belongs virtual key moves, the virtual key It is possible to provide a mechanism that makes it possible to input information using a virtual key with a simple operation of moving upwards against the virtual key without penetrating the bottom .

ADVANTAGE OF THE INVENTION According to this invention, the mechanism which enables the information input by a virtual key by simple operation can be provided , confirming the virtual key of the hierarchy in which a pointer exists according to the movement of a pointer .

It is a figure which shows an example of the mode of a character input in embodiment of this invention. It is a figure which shows an example of the structure of the hardware block diagram of information processing apparatus in embodiment of this invention. It is a figure which shows an example of a structure of the function block diagram of information processing apparatus in embodiment of this invention. It is a flowchart which shows the outline | summary of the character input process in embodiment of this invention. It is a figure which shows the example of arrangement | positioning of the virtual keyboard in the case of the hiragana-kanji conversion input system in embodiment of this invention. It is a figure which shows the example of arrangement | positioning of the virtual keyboard in the case of the Roman character input system in embodiment of this invention. It is a figure which shows the example of arrangement | positioning of the virtual keyboard containing the character conversion key in embodiment of this invention. It is a figure which shows an example of the specific method of arrangement | positioning on the plane of a virtual keyboard and the key of an input instruction | indication object in embodiment of this invention. It is a figure which shows an example of the mode of the side surface of a virtual keyboard in embodiment of this invention. It is a figure which shows the positional relationship of a virtual keyboard and a user's finger | toe in embodiment of this invention. It is a figure which shows an example of the identification method of the point for specifying the coordinate of a user's finger | toe in embodiment of this invention. It is a figure which shows an example of the X-axis / Y-axis group coordinate of a virtual key in embodiment of this invention. It is a figure which shows an example of a Z-axis group coordinate in embodiment of this invention. It is a figure which shows an example of the mode of a finger | toe position determination in embodiment of this invention. It is a figure which shows an example of the structure of the matching data of the area of the triangle formed in each virtual key and a user's fingertip in embodiment of this invention. It is a figure which shows an example of a structure of the virtual key setting data table according to the magnitude | size of the finger | toe in embodiment of this invention. It is a figure which shows an example of the mode of a response | compatibility of a virtual key and an area value in embodiment of this invention. It is a figure which shows an example of the mode of the display switching of a virtual keyboard in embodiment of this invention. It is a figure which shows an example of the mode of the display switching of the display screen at the time of input character confirmation in embodiment of this invention. It is a figure which shows embodiment using another housing | casing of the character input system of this invention. It is a figure which shows embodiment using another housing | casing of the character input system of this invention.

  Hereinafter, an example of an embodiment of the present invention will be described with reference to the drawings. First, with reference to FIG. 1, the state of character input in the embodiment of the present invention will be described. FIG. 1 is a diagram showing an example of character input according to the embodiment of the present invention.

  The information processing apparatus 100 includes a display 101 and an imaging device 102, and a display screen including a character input unit 103, a keyboard 105, a cursor 106, and the like is displayed on the display 101.

  The keyboard 105 displays the virtual keyboard 104 arranged in the virtual space by the information processing apparatus 100 on the display screen so that the user can identify it. Here, the display 101 also serves as a touch panel.

  The imaging device 102 captures an image of a user's finger (instruction medium) and detects its position and movement. The character input unit 103 is a character input unit through which a character is input (output) when the CPU 201 of the information processing apparatus 100 accepts a pressing instruction from the user to the virtual keyboard 104. The virtual keyboard here is not a keyboard displayed on a display screen like a software keyboard, but a keyboard that is virtually formed and arranged in real space and has no substance.

  The determination of which key of the virtual keyboard 104 is pressed is specified from the video acquired from the imaging device 102 and the position of the user's finger on the space. A cursor 106 is a cursor that indicates an operation state of a user's finger photographed by the imaging device 102.

  The cursor 106 is displayed on the display screen and moves and operates according to the movement of the user's finger. The user can check the state of the input operation to the virtual keyboard by looking at the keyboard 105 and the cursor 106 on the display screen. The above is the description of the state of character input in the embodiment of the present invention shown in FIG.

  Next, an example of the configuration of the hardware configuration diagram of the information processing apparatus in the embodiment of the present invention will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of a hardware configuration diagram of the information processing apparatus according to the embodiment of this invention.

  The CPU 201 comprehensively controls each device and controller connected to the system bus 204.

  Further, the ROM 203 or the external memory 213 includes a BIOS (Basic Input / Output System) or an operating system program (hereinafter referred to as an OS) which is a control program of the CPU 201, a character input system program of the present invention, and functions executed by each PC. Various programs necessary for realization are stored. The RAM 202 functions as a main memory, work area, and the like for the CPU 201.

  The CPU 201 implements various operations by loading a program necessary for execution of processing into the RAM 202 and executing the program.

  An input controller (input C) 205 controls input from a pointing device such as the imaging device 209, the touch panel 210, or a mouse (not shown).

  The video controller (VC) 206 controls display on a display device such as a CRT display (CRT) 211 and is connected to audio reproduction control. It is assumed that the subject of the invention is a tablet terminal such as a smartphone, but when applying this system as a controller to other devices, any display device may be used and is directly related to the present invention. There is nothing.

  The memory controller (MC) 207 is a hard disk (HD) or flexible disk that stores a boot program, browser software, various applications, font data, user files, editing files, various data, etc., or an adapter in a PC card slot via an adapter. Controls access to an external memory 213 such as a connected memory card.

  A communication I / F controller (communication I / FC) 208 is connected to and communicates with an external device via a network, and executes communication control processing in the network. For example, Internet communication using TCP / IP is possible.

  Note that the CPU 201 enables display on the CRT 211 by executing, for example, outline font rasterization processing on a display information area in the RAM 202. In addition, the CPU 201 enables a user instruction with a mouse cursor (not shown) of the CRT 211.

  A program to be described later for realizing the present invention is stored in the external memory 213 and is executed by the CPU 201 by being loaded into the RAM 202 as necessary. Furthermore, a definition file and various information tables used when executing the program are also stored in the external memory 213, and detailed description thereof will be described later. The sound reproducing device is a device that reproduces sound data stored in the external memory 213. The above is an example of the configuration of the hardware configuration diagram of the information processing apparatus in the embodiment of the present invention shown in FIG.

  Next, an example of the configuration of the functional configuration diagram of the information processing apparatus in the embodiment of the present invention will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a configuration of a functional configuration diagram of the information processing apparatus in the embodiment of the present invention.

  As shown in FIG. 3, the information processing apparatus 100 includes a data transmission / reception unit 301, an imaging control unit 302, a video analysis unit 303, a virtual keyboard control unit 304, a data calculation processing unit 305, an image display unit 306, and the like.

  For example, a portable information processing apparatus such as a smart device is assumed as a portable information processing apparatus according to the present invention, but the information processing apparatus is not limited to this form. For example, it may be a PC, a digital camera (imaging device), a home appliance remote control, a vehicle operation switch, or the like.

  The data transmission / reception unit 301 performs wireless access to the Internet (network 999) and acquires a program for executing the processing according to the present invention. When the program is stored in the information processing apparatus according to the present invention, the data transmission / reception unit 301 acquires the program from the information processing apparatus.

  The imaging control unit 302 is a control unit that controls the imaging device 109 in order to acquire a finger image captured by the imaging device 109. The image of the finger is used for processing to specify which character input is instructed by the user, which will be described later.

  The video analysis unit 303 analyzes the image acquired by the imaging control unit 302 and specifies the finger position. For example, the position of the finger in the space is specified using the photometric distance information (corresponding to the pointing medium position specifying means).

  The virtual keyboard control unit 304 is an arrangement control unit that arranges virtual keys horizontally and vertically with respect to the information processing apparatus 100 in a space. Note that, by accepting an instruction from the user on a setting screen such as a keyboard layout setting screen (not shown), the key interval can be adjusted horizontally and vertically, and the range in which the key reacts can be adjusted.

  The data operation processing unit 305 is a processing unit that compares data of the video analysis unit 303 and the virtual keyboard control unit 304 and identifies a character that has received an input instruction from the user.

  The image display unit 306 displays the keyboard placed in the virtual space by the virtual keyboard control unit 304, and the characters received from the user by the data calculation processing unit 305 are, for example, a search box or a character input It is a display unit that outputs and displays in a target area such as a column. The above is an example of the configuration of the functional configuration diagram of the information processing apparatus in the embodiment of the present invention shown in FIG.

  Next, the outline of the character input process in the embodiment of the present invention will be described with reference to FIG. FIG. 4 is a flowchart showing an outline of the character input process in the embodiment of the present invention. The processing described below is realized by the CPU 201 of the information processing apparatus 100 executing the program of the present invention stored in the external memory 213 using the RAM 202 of the information processing apparatus 100 as a work area.

  The CPU 201 of the information processing apparatus 100 accepts an instruction from the user and activates and displays a character input screen (steps S401 and S402). In other words, an area (search box, character input field, etc.) for outputting a character that has received an input, activation of the imaging device, placement of the virtual keyboard in the virtual space for receiving character input, and the virtual keyboard Display on the display screen.

  The instruction from the user here is, for example, a tap instruction from the user for an area such as a character input area. Since the example of the character input screen has been described in the description of FIG. 1, the description thereof is omitted here.

  A keyboard, Kana-Kanji conversion input key, alphabet conversion input key, etc. may be displayed on a part of the display screen.

  Here, with reference to FIGS. 5-7, an example of the mode of arrangement | positioning of the virtual keyboard in embodiment of this invention is demonstrated. 5 to 7 show how the virtual keyboard is arranged by the augmented reality method.

  The virtual keyboard is virtually arranged in a plurality of layers (hierarchies) in the front space of the imaging device, for example, like the virtual keyboard 104 in FIG. The keyboard used. For example, the layer refers to a hierarchy such as a hierarchy to which the input key of “A” sound belongs and a hierarchy to which the input key of “I” sound belongs in FIG.

  Here, it is assumed that the coordinates of the virtual keyboard (the coordinates of each virtual key) are set in advance, and the contents of the settings (coordinate information) are stored in a storage area such as the external memory 213 of the information processing apparatus 100. It is assumed that

  In addition, for each virtual key, it is assumed that the area of the fingertip (a triangle 801 in FIG. 8 and a triangle 1101 in FIG. 11 to be described later) associated with each virtual key is stored in association with each other. It is assumed that the association information is stored in a storage area such as the external memory 213 of the information processing apparatus 100.

  It is assumed that the coordinates of the arrangement of the virtual keyboard can be set and changed separately from a setting screen (not shown). Details of the arrangement and usage of the virtual keyboard will be described later with reference to FIG.

  FIG. 5 is a diagram showing an arrangement example of the virtual keyboard in the case of the hiragana-kanji conversion input method in the embodiment of the present invention. FIG. 6 is a diagram showing an arrangement example of the virtual keyboard in the case of the Roman character input method in the embodiment of the present invention. FIG. 7 is a diagram illustrating an arrangement example of a virtual keyboard including character conversion keys in the embodiment of the present invention.

  In addition to the input character key, a key for changing the selected character and converting it to Kanji and a key for changing the character type are formed as virtual keys in the space. The above is an explanation of an example of how the virtual keyboard is arranged according to the augmented reality method in FIGS.

  Returning to the description of FIG. The CPU 201 of the information processing apparatus 100 determines whether or not a character input method switching instruction has been received (step S403). The character input method here refers to an input method based on a virtual keyboard input method according to an embodiment of the present invention, a character input method based on pressing a keyboard on a touch panel in a conventional smart device, and a character input based on pressing a keyboard installed on the smart device body. An input method such as a method.

  Here, it is assumed that the input method when the character input screen is activated in step S401 is a character input method using a virtual keyboard. Therefore, in step S403, whether or not the character input method using the virtual keyboard has been changed to, for example, a character input method by pressing a keyboard on a touch panel in a conventional smart device or a character input method by pressing a keyboard installed on the smart device body. It shall be determined.

  For example, in the character input method change screen (such as a pop-up screen) that is displayed by receiving a long press (continuous press) on the character input field from the user, It is executed by accepting an instruction to select a character input method to be changed.

  If it is determined that a character input method switching instruction has been received (YES in step S403), the process proceeds to step S404, and the display screen display is shifted to a screen for performing character input in another character input method. To do. For example, the display of the virtual keyboard is erased from the display screen, and the display of the display screen is shifted so as to display a conventional keyboard for performing character input in response to a touch panel operation.

  If it is determined that a character input method switching instruction has not been received (NO in step S403), the process proceeds to step S405.

  The CPU 201 of the information processing apparatus 100 captures (continuously captures) the operation of the user's finger. That is, in order to specify which key (virtual key) in the virtual keyboard is pressed, the user's finger that operates to press the virtual key is photographed. The continuous shooting here is, for example, moving image shooting (step S405).

  The CPU 201 of the information processing apparatus 100 specifies the position of the finger from the image of the finger obtained by the continuous shooting (corresponding to step S406 / instruction medium position specifying means). For example, the fingertip area in the moving image (image) is specified (corresponding to the area specifying means), and the finger position in the real space is determined from the information on the size of the area and the position of the finger in the moving image (image). Specify the position (coordinates). Note that the distance between the finger and the imaging device is measured using the photometric distance information, and the finger position in the real space is specified together with the finger position information in the moving image (image). Also good.

  Here, with reference to FIGS. 8 to 10, the positional relationship between the virtual keyboard and the user's finger in the embodiment of the present invention will be described.

  FIG. 8 is a diagram illustrating an example of a virtual keyboard layout on the plane and an input instruction target identification method according to an embodiment of the present invention.

  The coordinates on the plane of the virtual keyboard are defined by the X axis and the Y axis. The CPU 201 of the information processing apparatus 100 identifies the finger from the video imaged by the imaging device, and uses the tip of the finger as a vertex and the other two vertices.

A regular triangle (a triangle 1101 to be described later) inscribed in the outline of the finger is formed on the belly of the finger, and points on the triangle (for example, the center point / point 802 and the point 1103 of the triangle) are defined by the defined X axis. , It is determined at which coordinate on the Y-axis, and it is determined which key can be pressed on the plane coordinate.

For example, the captured video is analyzed and digitized to measure the finger position in space. As an example of the image abstraction method at this time,
The comparison of the area of the equilateral triangle is applied to the distance measurement of the finger from the imaging device.

  Here, with reference to FIG. 11, an example of a point specifying method for specifying the coordinates of the user's finger in the embodiment of the present invention will be described. In other words, an example of a triangle formed on the belly of the finger and a point on the triangle will be described. FIG. 11 is a diagram illustrating an example of a point specifying method for specifying the coordinates of the user's finger in the embodiment of the present invention.

  The triangle 1101 is formed starting from three points on the belly outline of the finger 1102. The starting point may be defined anywhere as long as it is on the user's finger. A point 1103 is a point for specifying the position of the user's finger (coordinates on the X and Y axes) used to determine which key is pressed. The above is the description of an example of the point specifying method for specifying the coordinates of the user's finger in the embodiment of the present invention shown in FIG.

  Returning to the description of FIG. For example, as shown in FIG. 8, the key “NA” is “X = 3, Y = 6”, “X = 4, Y = 6”, “X = 3, Y = 5”, “X = 4”. , Y = 5 ″ is set inside the area indicated by the four points, the CPU 201 of the information processing apparatus 100 indicates that the point 802 (point 1103) on the triangle 801 (triangle 1101) described above is 4 described above. When it is determined that the object is inside the area indicated by the point and a reverse operation (operation in which the user's finger leaves the imaging device) is detected, which is an operation of releasing the finger from the touched key, “NA” is input. Is identified as having been accepted. Also, when it is determined that the finger is in the “na” key area (that is, when it is determined that the point on the triangle is included in the “na” area), The character “NA” is displayed in the character input section.

  The inversion operation indicates, for example, an operation in which the user's finger is separated from the imaging device. That is, for example, an operation in which the finger approaches the imaging device, presses the key, and then lifts the finger away from the key. The CPU 201 of the information processing apparatus 100 determines the input candidate character as an input character by detecting the inversion operation.

  In other words, as in the case of general keyboard press acceptance and character input processing, after detecting a movement in which the user's finger is moved away from the virtual key and pressed toward you (the direction opposite to the touched surface) after pressing the virtual key, A character that has been specified as an input candidate character immediately before the action to be pulled forward is specified as an input character and output. Here, the operation of releasing the finger from the virtual key toward the near side (upward) (the operation of moving the finger from the position on the space corresponding to the virtual key) has been described as the reversal operation. When the CPU 201 of 100 detects the movement of the finger in the horizontal direction, the downward direction, and the diagonal direction, it may be determined that the movement of the finger is a reversal operation.

  In addition, when a certain period of time elapses while the user's finger is in contact with the virtual key, characters corresponding to the key that the user's finger is in contact with are continuously output (characters are detected). You may make it output / display in the area | region for input. The predetermined time here is set in advance, and information on the setting is stored in a storage area such as the external memory 213 of the information processing apparatus 100. The above is the description of FIG.

  FIG. 9 is a diagram illustrating an example of a side view of the virtual keyboard in the embodiment of the present invention. A virtual keyboard (virtual key group) is arranged in a range that can be imaged by the imaging device 209 of the information processing device 100.

  The position and coordinates on the space where the virtual keyboard is placed can be freely set and changed by receiving a virtual keyboard coordinate setting instruction from the user on a virtual keyboard coordinate setting screen (not shown), for example. And Details of the arrangement of the virtual keyboard on the Z coordinate and the positional relationship between the virtual keyboard and the user's finger will be described later with reference to FIG. The above is the description of an example of the state of the side surface of the virtual keyboard in the embodiment of the present invention shown in FIG.

  FIG. 10 is a diagram showing the positional relationship between the virtual keyboard and the user's finger in the embodiment of the present invention. As shown in FIG. 10, the virtual keyboard 1001 is arranged and set in a plurality of layers at predetermined intervals on the Z coordinate (Z axis).

  In determining which key has been pressed, the CPU 201 of the information processing apparatus 100, for example, from the home position area between the first hierarchy (conversion key hierarchy) and the second hierarchy (the "A" sound hierarchy), It is specified whether or not the user's finger has approached the imaging apparatus, and how close or apart the user's finger has been. That is, the position of the finger on the Z axis is determined from the size of the area of the triangle on the abdomen of the finger, and it is determined which key is pressed.

  Here, it is assumed that the degree of finger approach (coordinates on the Z-axis of the finger) is specified using the above-described information on the area size of the triangle specified from the video acquired by the imaging device.

  Note that the degree of approach of the finger may be specified from a photometric distance specified from an image acquired by the imaging apparatus, or both the size of the triangle area and the information of the photometric distance may be determined. You may make it specify using.

  The home position area 1002 is an area where the user's finger does not touch any key. For example, when specifying and setting coordinates on the virtual keyboard space based on an instruction received on a virtual keyboard coordinate setting screen (not shown), the setting is performed based on the position (initial position) of the user's finger. Is possible.

  At this time, the position of the user's finger and the distance from the imaging device (distance from the information processing device 100) are specified from the area (size) of the triangle specified from the video acquired by the imaging device. .

  The home position area is set according to the position of the finger (coordinates of the finger on the X, Y, and Z axes). It is assumed that when the finger is present at which coordinate, the coordinate at which the home position area is set is set in advance as setting information for the arrangement of the virtual keyboard.

  Here, the plane coordinates (coordinates on the X and Y axes) of the center point of the triangle virtually formed on the belly of the finger are the centers (coordinates of the centers on the X and Y axes) of the virtual keyboard. The coordinates of the home position area are set so as to be the center of the home position area (the positions of the center points on the X, Y, and Z axes in the home position area), and the coordinates of the virtual keyboard are set. Made. That is, initial positions on the X and Y axes are set. Here, the relationship (positional relationship) between the home position area and the coordinates of each virtual key is set in advance as a fixed value.

  That is, the position on the Z axis where the point in the triangle on the imaged finger exists is on the Z axis on the bottom surface of the key belonging to the first layer and on the Z axis on the top surface of the key belonging to the second layer. The coordinates on the Z axis of the virtual keyboard are set so as to be located at the midpoint (center point) of the distance between the position and the position.

  The coordinate on the Z-axis of the finger where the point on the triangle is located (initial position on the Z-axis), the coordinate on the Z-axis of the bottom surface of the key belonging to the first layer, and the top surface of the key belonging to the second layer The distance between the coordinates on the Z axis is the same. The above is the description of the positional relationship between the virtual keyboard and the user's finger in the embodiment of the present invention shown in FIG.

  Next, an example of the X-axis / Y-axis group coordinates of the virtual key in the embodiment of the present invention will be described with reference to FIG. FIG. 12 is a diagram illustrating an example of the X-axis / Y-axis group coordinates of the virtual key in the embodiment of the present invention.

  As shown in FIG. 12, the X-axis / Y-axis group coordinate 1200 includes an X-axis coordinate 1201, a Y-axis coordinate 1202, a group 1203, and the like of each virtual key.

  X-axis coordinates 1201 and Y-axis coordinates 1202 indicate the coordinates of the X-axis and Y-axis of each virtual key, respectively. The group 1203 defines a group indicating which virtual key is arranged and set in the coordinates on the X axis and the Y axis indicated by the X axis coordinate 1201 and the Y axis coordinate 1202.

  That is, for example, in the coordinates of X-axis = X3-X4 and Y-axis = Y5-Y6, a virtual key “Nani, N, Ne, No, down arrow” is arranged according to the coordinates of the Z axis. Which indicates that. The above is an example of the X-axis / Y-axis group coordinates of the virtual key in the embodiment of the present invention shown in FIG.

  Next, an example of Z-axis group coordinates in the embodiment of the present invention will be described with reference to FIG. FIG. 13 is a diagram illustrating an example of Z-axis group coordinates in the embodiment of the present invention.

  As shown in FIG. 13, the Z-axis group coordinate 1300 is composed of a hierarchy 1301, a Z-axis coordinate 1302, an input target key 1303, and the like. It is assumed that the Z-axis group coordinates 1300 are stored in a storage area such as the external memo 213 of the information processing apparatus 100 (corresponding to a storage function in claims).

  A hierarchy 1301 indicates a hierarchy to which the Z-axis group of each virtual key belongs (each hierarchy shown in FIG. 10). The Z-axis coordinate 1302 indicates the coordinate on the Z-axis of the virtual key of each Z-axis group.

  An input target key 1303 indicates a virtual key belonging to each Z-axis group. That is, it is a group of virtual keys that are set on the same coordinates on the X and Y axes and set on different coordinates on the Z axis.

  The thickness of the virtual key is set to the Z-axis coordinate 1302, and for example, the virtual key can be pressed while moving through the space of the thickness of the virtual key even when the finger is lowered from the top to the bottom. Judge as a state. For example, if the position of the finger on the X / Y axis is on the virtual key “A” and the coordinate on the Z axis of the finger is Z5 to Z6 (in the Z axis coordinate 1302), the virtual keyboard “A” Is pressed, and the virtual key “A” is enlarged and displayed on the display screen.

  When the coordinate on the Z-axis of the finger is closer to the imaging device than Z6 (in the Z-axis coordinate 1302) and farther from the imaging device, the virtual keyboard of “Yes” row is displayed on the display screen. The above is an explanation of an example of the Z-axis group coordinates in the embodiment of the present invention shown in FIG.

  Next, with reference to FIG. 14, how the finger position is determined in the embodiment of the present invention will be described. FIG. 14 is a diagram illustrating an example of finger position determination according to the embodiment of the present invention.

  As shown in FIG. 14, when the distance to the imaging device 209 is L1, the area of the triangle is the area S1, and when the distance to the imaging device 209 is L2, the area of the triangle is the area S2. Shall. Here, it is assumed that S1 = the area of the triangle at the initial position measured when the coordinates of the virtual keyboard described above are set.

  In the external storage memory 213 of the information processing apparatus 100, which virtual key is in a state where it can be pressed when the triangular area is the above-described area (that is, the point on the user's finger). Is at which coordinate, which level of virtual key group is displayed on the display screen, and at which position on the display screen the cursor 106 in FIG. 1 is displayed). Assume that information is stored associating data of each virtual key and the area of the triangle (see FIG. 15).

  In addition, a virtual key is set according to the size of the user's finger at the initial position (the area of the triangle by calculation measurement) acquired when the home position is set (when the coordinates of the virtual keyboard are set). Assume that the data table is stored in the external storage memory 213 of the information processing apparatus 100.

  Here, with reference to FIG. 16, an example of the configuration of the virtual key setting data table corresponding to the size of the finger in the embodiment of the present invention will be described. FIG. 16 is a diagram illustrating an example of a configuration of a virtual key setting data table corresponding to the size of a finger in the embodiment of the present invention.

  In the example of the virtual key setting data table 1600 shown in FIG. 16, the virtual key ID 1601, the key size (key X-axis length 1602, key Y-axis length 1603) and thickness according to the size of the user's finger. (Key Z-axis length 1604) and the like are set and recorded.

  A virtual key corresponding to an area 1605 having a size closest to the area of the triangle is determined by determining which area 1605 is the area of the triangle of the user's fingertip acquired when the CPU 201 of the information processing apparatus 100 sets the home position. Are arranged in space.

  For example, when the area of a triangle set on the belly of a finger is 150, a virtual keyboard having a virtual key ID 1601 = KM having an area value closest to 150 is placed in the space And The above is an example of the configuration of the virtual key setting data table corresponding to the size of the finger in the embodiment of the present invention shown in FIG.

  FIG. 15 is a diagram illustrating an example of a configuration of the association data of the area of the triangle formed on each virtual key and the user's fingertip in the embodiment of the present invention. The association data is stored in a storage area such as the external memory 213 of the information processing apparatus 100.

  The association data 1500 is an area value calculated for each of the 50 sound virtual keys 1501 based on the area of the triangle of the user's fingertip acquired and specified by the imaging device when the home position is set (at the time of initial position determination). 1502 are set.

  The calculation process uses existing technology based on information on the position of the imaging apparatus, the distance between each virtual key and the initial position from the imaging apparatus, and the area of the triangle corresponding to the initial position. Shall be calculated. Note that information on the photometric distance from the imaging device is used for each virtual key from the imaging device and the distance between the initial positions. The above is an example of the configuration of the association data of the area of the triangle formed on each virtual key and the fingertip of the user in the embodiment of the present invention in FIG.

  FIG. 17 is a diagram illustrating an example of a state of correspondence between virtual keys and area values in the embodiment of the present invention. Here, a state of correspondence between each virtual key of the “A” row and the area of the triangle of the fingertip, which is set by the association data 1500 in FIG. 15 is shown.

  The symbol a is ah. Lee, yes. C D. Oh, oh. As the image changes from “A” to “O”, the area of the equilateral triangle inscribed in the finger increases. There is also a movement of the center point of the equilateral triangle.

  In the description of the present embodiment, as shown in FIG. 15, association data of each virtual key and area value is stored, and which association data is used to determine which key has been pressed. However, as shown in FIG. 17, for example, the case virtual key and the image information of the finger may be stored in association with each other, and it may be determined which key is pressed.

  For example, the CPU 201 of the information processing apparatus 100 has coordinates X and Y on the finger space positioned on the virtual keys of the “A” row (X1-X2 and Y7-Y8 shown in FIG. 12). The area S1 shown in FIG. 14 (the size of the finger acquired from the imaging device) is smaller than the area of the triangle stored in association with the virtual key “A” (the matching area = 165 shown in FIG. 15) (or In the following case, a virtual keyboard (key group / virtual keys of the hierarchy H3 shown in FIG. 13) belonging to the same hierarchy as the virtual key “A” is displayed, and the cursor 106 is moved over the virtual key “A”. To display.

  Further, the coordinates X and Y on the finger space are located on the virtual key of the “A” row (X1-X2 and Y7-Y8 shown in FIG. 12), and the area S2 shown in FIG. It is larger than (or more than) the area of the triangle that is stored in association with the virtual key “a” (applicable area = 165 shown in FIG. 15), and is stored in association with the virtual key “i”. A virtual keyboard (key group / hierarchy H4 shown in FIG. 13) belonging to the same hierarchy as the virtual key of “Yes” when the area is smaller than the triangular area (applicable area = 170 shown in FIG. 15) (or below). , And the cursor 106 is displayed on the virtual key “I”.

  That is, the CPU 201 of the information processing apparatus 100 switches the display of the virtual keyboard displayed on the display screen according to the information on the position of the user's finger on the X, Y, and Z axes.

Here, with reference to FIG. 18, a state of display switching of the virtual keyboard in the embodiment of the present invention will be described. FIG. 18 is a diagram illustrating an example of a state of display switching of the virtual keyboard in the embodiment of the present invention.
A character input area 1801 is an area in which a character whose input has been confirmed is input and displayed. The manner of switching the display screen when the input is confirmed will be described later with reference to FIG.

  The keyboard display unit 1802 (keyboard display unit 1806) is in accordance with the operation of the user's finger 1804 (in accordance with the size of the fingertip area acquired from the finger image acquired from the imaging device and the distance from the imaging device to the finger). It is a display unit that displays the state of the virtual keyboard and switches the display.

  A cursor 1803 is a cursor indicating the position of the user's finger with respect to the virtual keyboard. When it is detected that the user's finger touches the virtual key, the virtual key touched by the finger is enlarged and displayed (corresponding to input key identification display means), as indicated by an enlarged display key 1805.

  For example, when the user's finger is on the first level (home position), as shown in the keyboard display unit 1802, keys such as a Kana-Kanji conversion button are displayed on the display screen.

  In addition, when the user's finger is between the first and second layers, the display screen displays a sound key (a, ka, sa, ta, na, ha, ya, la, w) Is displayed. When the user's finger is between the second and third hierarchies, a sound key is displayed on the display screen.

  When the user's finger is between the third layer and the fourth layer, as shown in the keyboard display unit 1806, a key for a sound is displayed on the display screen.

  When the user's finger is between the fourth layer and the fifth layer, the sound key is displayed on the display screen. When the user's finger is between the fifth hierarchy and the sixth hierarchy, a sound key is displayed on the display screen. When the user's finger is below the sixth hierarchy, no key is displayed on the display screen.

  Also, the coordinate position of the finger acquired from the imaging device is located on the virtual key of the “ta” row (X1-X2 and Y5-Y6 shown in FIG. 12), and the fingertip triangle acquired from the imaging device is When the area matches the conforming area = 175 in FIG. 15, the display on the display screen of the “u” virtual key is enlarged from the other virtual keys as shown by the enlarged display key 1805, and the user's finger is The user can recognize that the user touches the virtual key "u" on the virtual keyboard. The above is the description of the virtual keyboard display switching in the embodiment of the present invention shown in FIG.

Next, with reference to FIG. 19, the display switching of the display screen at the time of input character confirmation in the embodiment of the present invention will be described. FIG. 19 is a diagram illustrating an example of the display switching of the display screen when the input character is confirmed in the embodiment of the present invention. The configuration of FIG. 19 is almost the same as that described with reference to FIG.
As shown in FIG. 19, when the user's finger touches the virtual key and the enlarged display key 1902 is displayed, the character (input character) that is still input is unconfirmed (undecided). Input characters are not displayed in area 1901.
The CPU 201 of the information processing apparatus 100 detects that the user's finger has moved away from the touched virtual key upward (toward a shallower layer / for example, from the fourth layer to the third layer). In this case, that is, when the reversal operation is detected, the input character is determined. Then, the display of the enlarged display key is returned to the normal display as indicated by the key 1903, and the input character is input and displayed in the character input area as indicated by the character input area 1904. The above is the description of the display switching of the display screen at the time of input character confirmation in the embodiment of the present invention of FIG.

  Returning to the description of FIG. In this example, “when the area of the triangle of the fingertip acquired from the imaging device matches 175” is set, but the thickness (Z-axis length 1604 of the key shown in FIG. 16) is set for each virtual key. Therefore, while the user's finger passes through the space specified by the thickness of the virtual key (until it touches the bottom surface of the virtual key), the virtual key is enlarged compared to other virtual keys. May be displayed. The above is the description of the finger position determination in the embodiment of the present invention shown in FIG.

  Returning to the description of FIG. The CPU 201 of the information processing apparatus 100 displays a virtual keyboard to be displayed on the display screen using the information on the area of the triangle acquired from the imaging apparatus and the point information on the triangle (step S407).

  Then, the contact between the virtual key and the finger is detected (step S408). That is, by using information on the position on the X / Y / Z axis where the point on the triangle exists and the area of the triangle stored in association with the area of the triangle and the virtual key, the user's A virtual key touched by a finger and the contact are detected (corresponding to a touch key determination unit / virtual key specifying unit). At this time, the touched virtual key is enlarged and displayed so that the user can recognize which virtual key is being pressed (described later in the description of step S409 / FIGS. 18 and 19).

  The CPU 201 of the information processing apparatus 100 determines whether the reversal operation is detected within a predetermined time after the contact detection (whether the operation of releasing the finger from the virtual key is detected) (corresponding to step S410 / detection means). If it is determined that the reversal operation has not been detected within the predetermined time (NO in step S410), the process returns to step S405. The predetermined time here is assumed to be preset and stored in a storage area such as the external memory 213 of the information processing apparatus 100.

  If it is determined that a reversal operation has been detected within a predetermined time (corresponding to YES / detection means in step S410), the virtual key character (information corresponding to the virtual key) that detected contact before the reversal operation was detected. Then, it is confirmed as an input character (corresponding to step S411 / input information determining means), an input conversion candidate is displayed on the display screen, and an input conversion operation (conversion candidate selection operation instruction) is accepted (step S412).

  The CPU 201 of the information processing apparatus 100 determines whether or not a character input end instruction has been received (step S413). If it is determined that a character input end instruction has not been received (NO in step S413), the process proceeds to step S405. If it is determined that an instruction to end character input has been received (YES in step S413), the character input process ends.

  An instruction to end character input is, for example, an instruction to press a search button or an instruction to end search processing when an input operation of a search character string in a search box is accepted. In addition, when a character input operation in the document creation software is received, it is an operation instruction such as termination of the software. The above is the description of the outline of the character input process in the embodiment of the present invention shown in FIG.

  Hereinafter, when a character is input using a character input method using the character input method according to the embodiment of the present invention and a small keyboard displayed on a display such as a numeric keypad or a smart device in a mobile phone terminal. Is compared with the number of input characters (only hiragana character input is counted without Kanji conversion).

  In all input methods, enter the sentence “I will happily thank you for your time.” To do.

  In the case of numeric keypad hiragana character input in a mobile phone terminal, the input character is changed by receiving the same key press a plurality of times. For example, “I” is input when the “A” key is pressed twice, and “U” is input when the key is pressed three times. Hereinafter, for example, the key pressed to input the character “I” and the number of times it was pressed are expressed as “Ai”, and the key pressed to input the character “U” and the number of times it was pressed are expressed as “ "It's like that." In the case of “Ai”, the type of the pressed key is the “A” key that is the first character of “Ai”, and the number of times the information processing apparatus 100 has accepted the pressing of the key is two. In the case of “Yes”, the type of the pressed key is the “A” key that is the first character of “AY”, and the number of times that the information processing apparatus 100 has received the key press is three times. To do.

  For example, when “masashi” is entered, it is assumed that a key press “ma, sashi, masashi,” is accepted. In this case, the number of times the key is pressed is 8 times. Also, for example, when accepting a character input of “O”, by accepting the pressing of another key after pressing the “A” key, the change order of the characters to be input is reversed, so that “A” is not “Aioo”. Can be entered.

  In order to enter “I ’ll be happy and happy” on my mobile device, I ’d like to say “I ’m going to do it.” , Sasose, Ai, Aoi, ⇒, Ai, Nano, Kako, Tato, ⇒, Tato, Ao, Yayo, Raro, Kako, Hahi, Ma, Ah, Sashi, A, Kokeki , “, Masa, masashi, wa” key press acceptance processing is required, and the number of key press acceptances is 64 times.

  In addition, when a small keyboard (Romaji input) similar to a PC is displayed on the display of the smartphone and a key press is accepted, “JI, KA, MA, SU, MA, SU, GO, SE, I, E, I , NO, KO, TO, TO, O, YO, RO, KO, BI, MO, U, SHI, A, GE, MA, SU, etc. "is required. The number of times is 50.

  In addition, when a key press or flick operation is accepted by flick input (Hiragana-Kanji conversion) on a smartphone (here, one flick is counted as one press acceptance process), Sashi, Ma, Sasu, Kako, ', let, Ai, ⇒, Ae, ⇒, Ai, Nano, Kako, Tato, ⇒, Tato, Ao, Yayo, Raro, Kako, Hahi,', Momo, The key press reception process of “Ah, Sashi, Ah, Kake,“ Ma, Sasu,-”is required, and the number of key presses received is 58 times.

  When the character input method using the virtual keyboard in the embodiment of the present invention is used, “Shi,”, “ka,”, “ma,”, “s”, “se,”, “e,”, “e,”, “no,” ,,,,,,,,,,, H,,,,,,,,,,,,, The number of times is 32, and the sentence can be completed with a smaller number of times of pressing acceptance than when other input methods are used.

  As described above, according to the present invention, it is possible to provide an information processing apparatus, a control method, and a program thereof that can assist information input with a simple operation.

  Further, the area on the plane for the key arrangement is reduced, and the user can input characters without repeatedly pressing the character input key.

  Hereinafter, with reference to FIG. 20 and FIG. 21, an embodiment using another casing of the character input system of the present invention will be described.

  FIG. 20 shows an example in which the imaging apparatus 2002 is attached to a pointing device 2001 that can be connected to a PC with a cable 2003. When the virtual keyboard is arranged in the space on the imaging device according to the above-described embodiment of the present invention and a virtual key pressing instruction is received from the user, the input character is displayed on the PC display screen. Display above.

  In this case, for example, the CPU mounted on the PC acquires the video from the imaging device 2002 installed on the pointing device 2001, analyzes it, specifies the key that has been pressed, and specifies the character to be input. Suppose that it is displayed on a display screen of a PC (not shown).

  That is, among the processes of steps S401 to S413 illustrated in FIG. 4, the CPU of the PC performs the imaging process of step S405 using the imaging device installed on the pointing device, and the CPU of the PC performs other processes. Processing is performed using software and hardware installed in the PC.

  This eliminates the need for an actual keyboard and enables character input without moving the hand from the pointing device 2001 onto the keyboard. The above is the description of the embodiment using another casing of the character input system of the present invention shown in FIG.

  FIG. 21 shows an example in which the present invention is implemented using, for example, a ring-type device 2101 on which an imaging device 2102 is mounted, which is mounted on a ring finger mainly for photographing the movement of the thumb.

  For example, a portable terminal having a display screen display function that is not equipped with an imaging device and a ring-type device 2101 are connected by communication, and character input is realized without using an existing keyboard as described above. In this case, for example, the CPU mounted on the portable terminal acquires the video of the user's finger from the imaging device 2102 installed in the ring type device 2101, analyzes it, specifies the key that has been pressed, and inputs it A character is specified and displayed on the display screen.

  That is, among the processes of steps S401 to S413 shown in FIG. 4, the CPU of the portable terminal performs the imaging process of step S405 using the imaging apparatus 2102 installed on the ring-type apparatus 2101, and performs other processes. The CPU of the terminal performs processing using software and hardware installed in the portable terminal. The above is the description of the embodiment using another casing of the character input system of the present invention shown in FIG.

  The embodiment in the case where the imaging device and the device having a display function are separate cases has been briefly described above.

  In the embodiment of the present invention, when the character to be input is confirmed, the character that has been confirmed to be input is displayed in the character input area on the display screen. Corresponding sound may be uttered from a sound reproduction device (not shown) mounted on the information processing apparatus 100. By doing so, even in an information processing apparatus that does not have a display screen, it is possible to make the user recognize characters that are input using the present invention.

  Thereby, it is possible to make the input character recognized even for a user who has difficulty in visually recognizing the character displayed on the display screen, such as a person with a visual ginger.

  As described above, according to the present invention, it is possible to provide an information processing apparatus, a control method, and a program thereof that can assist information input with a simple operation.

  In the above-described embodiment, the position of the finger in the space is specified using information on the area of the user's fingertip. For example, a sensor for detecting the position of the object is provided in the information processing apparatus 100. It may be mounted and the position of the finger in the space may be specified using the function of the sensor.

  Also, a recording medium that records a program that realizes the functions of the above-described embodiments is supplied to a system or apparatus, and a computer (or CPU or MPU) of the system or apparatus reads and executes the program stored in the recording medium. It goes without saying that the object of the present invention can also be achieved.

  In this case, the program itself read from the recording medium realizes the novel function of the present invention, and the recording medium storing the program constitutes the present invention.

  As a recording medium for supplying the program, for example, a flexible disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD-R, DVD-ROM, magnetic tape, nonvolatile memory card, ROM, EEPROM, silicon A disk or the like can be used.

  Further, by executing the program read by each device, not only the functions of the above-described embodiments are realized, but also an OS (operating system) running on the computer is actually executed based on the instructions of the program. It goes without saying that a case where the function of the above-described embodiment is realized by performing a part or all of the process and the process is included.

  Furthermore, after the program read from the recording medium is written to the memory provided in the function expansion board inserted into the computer or the function expansion unit connected to the computer, the function expansion board is based on the instructions of the program code. It goes without saying that the case where the CPU or the like provided in the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  Further, for example, as described above, the present invention may be applied to a system including a plurality of devices or an apparatus including a single device. Needless to say, the present invention can be applied to a case where the present invention is achieved by supplying a program to a system or apparatus. In this case, by reading a recording medium storing a program for achieving the present invention into the system or apparatus, the system or apparatus can enjoy the effects of the present invention.

  Furthermore, by downloading and reading a program for achieving the present invention from a server, database, etc. on a network by a communication program, the system or apparatus can enjoy the effects of the present invention.

  In addition, all the structures which combined each embodiment mentioned above and its modification are also included in this invention.

  In addition, each process realized by the above-described software can be realized as firmware or hardware configuration, and each process can be realized as each means. The technical scope of the present invention is realized by such firmware or hardware configuration. Is also included.

DESCRIPTION OF SYMBOLS 100 Information processing apparatus 101 Display 102 Imaging apparatus 103 Character input part 104 Virtual keyboard 105 Keyboard 106 Cursor 201 CPU
202 RAM
203 ROM
204 System Bus 205 Input Controller 206 Video Controller 207 Memory Controller 208 Communication / IF Controller 209 Imaging Device 210 Touch Panel 211 CRT
212 Audio playback device 213 External memory 999 network

Claims (7)

An information processing apparatus comprising a display unit and enabling information input by a virtual key having a position and thickness defined in a three-dimensional space ,
Correspondence storage means for storing the virtual key in association with a hierarchy to which the virtual key belongs among a plurality of hierarchies defined in a three-dimensional space, and a position of the virtual key in the hierarchy;
A position specifying means for specifying a position of the pointer in the three-dimensional space by detecting a pointer for operating the virtual key by a detection unit;
The virtual key of the hierarchy existing in a state where the indicator at the position specified by the position specifying means does not touch the virtual key is switched to the virtual key of the destination hierarchy according to the movement of the indicator between the hierarchies. Display key display control means for controlling to display as a display key which is a key to be displayed on the display unit;
The pointer is, whether moving in the upward direction as contrary to the virtual key without penetrating the bottom of the virtual keys after contact from above the virtual keys, the position the indication specified in specifying means Determination means for determining based on the position of the body and the position of the virtual key stored in the correspondence storage means;
When the indicator moves through the bottom of the touching virtual key to another level below the virtual key, the determination means does not input information using the touched virtual key. , if the pointer is determined to have moved upward as contrary to the virtual key without penetrating the bottom of the virtual keys after contact to the virtual key, for inputting information by the virtual key and determining means for determining that,
Input means for performing control to input information determined to be input by the determination means to the input unit;
Equipped with a,
When the indicator moves through the bottom of the touching virtual key to the lower layer of the virtual key, the display key display control means displays the virtual key of the lower layer of the moving destination. the information processing apparatus according to claim that you controlled to display as a key. The correspondence storage means stores a hierarchy to which a plurality of virtual keys belong,
The display key display control means is configured to select the plurality of virtual keys of the hierarchy existing in a state where the indicator at the position specified by the position specifying means does not contact the virtual key from the plurality of virtual keys. 2. The display key according to claim 1, wherein the display key is controlled so as to be switched in accordance with movement of the indicator between the layers so that a user can select a virtual key with which the indicator is brought into contact. Information processing device. The display key display control means, the information processing apparatus according to claim 1 or 2, characterized in that an identifiable control to be displayed on the display unit the position of the pointer specified by the position specifying means . The information according to any one of claims 1 to 3 , wherein the display key display control means displays the virtual key touched by the indicator on the display unit as the display key so that the virtual key can be identified. Processing equipment. Information storage means for storing the area of the indicator and information for specifying which virtual key the indicator of the area has contacted;
An area specifying means for specifying an area of the surface of the indicator, which is used for specifying that the indicator has contacted the virtual key, from an image obtained by imaging the indicator;
A contact key specification that specifies which virtual key the indicator has touched using the area size information specified by the area specification means and the information stored in the information storage means Means,
The information processing apparatus according to any one of claims 1 to 4 , further comprising: A display unit, enabling information input by a virtual key whose position and thickness are defined in a three-dimensional space , wherein the virtual key is a hierarchy to which the virtual key belongs among a plurality of hierarchies defined in the three-dimensional space; And a corresponding storage unit that stores the virtual key in association with the position of the virtual key, and a pointer that operates the virtual key is detected by the detection unit, thereby specifying the position of the pointer in the three-dimensional space. A method for controlling an information processing apparatus comprising a position specifying unit,
The virtual key of the hierarchy existing in a state where the indicator at the position specified by the position specifying means does not touch the virtual key is switched to the virtual key of the destination hierarchy according to the movement of the indicator between the hierarchies. A display key display control step for controlling to display as a display key which is a key to be displayed on the display unit ;
The pointer is, whether moving in the upward direction as contrary to the virtual key without penetrating the bottom of the virtual keys after contact from above the virtual keys, the position the indication specified in specifying means A determination step of determining based on the position of the body and the position of the virtual key stored in the correspondence storage means;
When the indicator moves through the bottom of the touched virtual key and moves to another level below the virtual key, the information is not input by the touched virtual key, and the determination step , if the pointer is determined to have moved upward as contrary to the virtual key without penetrating the bottom of the virtual keys after contact to the virtual key, for inputting information by the virtual key and a determination step of determining that,
An input step for controlling to input information determined to be input in the determination step to the input unit;
Only including,
In the display key display control step, when the indicator moves through the bottom of the touching virtual key and moves to the lower layer of the virtual key, the virtual key of the lower layer of the movement destination is displayed. A control method for an information processing apparatus, wherein control is performed so as to display as a key . A display unit, enabling information input by a virtual key whose position and thickness are defined in a three-dimensional space , wherein the virtual key is a hierarchy to which the virtual key belongs among a plurality of hierarchies defined in the three-dimensional space; And a corresponding storage unit that stores the virtual key in association with the position of the virtual key, and a pointer that operates the virtual key is detected by the detection unit, thereby specifying the position of the pointer in the three-dimensional space. A program that can be executed by an information processing apparatus comprising:
The information processing apparatus;
The virtual key of the hierarchy existing in a state where the indicator at the position specified by the position specifying means does not touch the virtual key is switched to the virtual key of the destination hierarchy according to the movement of the indicator between the hierarchies. Display key display control means for controlling to display as a display key which is a key to be displayed on the display unit;
The pointer is, whether moving in the upward direction as contrary to the virtual key without penetrating the bottom of the virtual keys after contact from above the virtual keys, the position the indication specified in specifying means Determination means for determining based on the position of the body and the position of the virtual key stored in the correspondence storage means;
When the indicator moves through the bottom of the touching virtual key to another level below the virtual key, the determination means does not input information using the touched virtual key. , if the pointer is determined to have moved upward as contrary to the virtual key without penetrating the bottom of the virtual keys after contact to the virtual key, for inputting information by the virtual key and determining means for determining that,
Functioning as input means for performing control to input information determined to be input by the determination means to the input unit ;
When the indicator moves through the bottom of the touching virtual key to the lower layer of the virtual key, the display key display control means displays the virtual key of the lower layer of the moving destination. program of the information processing apparatus characterized that you controlled to display as a key.

Images