JP2017162126A - Input system, input method, control program and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an input system which has improved operation ability relative to prior ones, and which achieves various kinds of input operations.SOLUTION: An input system comprises: a projector 207 for displaying an UI component on an operation plane; a touch panel 330 for detecting a touch position on which a user touched the operation plane by a finger; and a distance image sensor part 208 for acquiring a shape of a hand comprising a finger which touches the operation plane. The input system recognizes a hand area in which the hand is projected to the operation plane based on the shape of the hand, and detects a touch operation to the UI component based on the touch position and a display position of the UI component. The input system determines the operation content to the UI component according to the hand area and the touch operation.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an input system that detects the operation content of a UI (User Interface) component being displayed. UI parts refer to screen parts and functional parts.

  There is an input system in which UI parts are displayed by a display device such as a projector or a large liquid crystal display, and this is operated using a touch panel which is a position detection device capable of detecting a plurality of touch positions. When the user touches a UI component displayed on the display device and the touch position is detected by the touch panel, processing according to the UI component and the touch position is performed. In such an input system, it is assumed that a plurality of users simultaneously operate UI parts. Therefore, when the input system detects a plurality of touch positions, it is necessary to determine whether one user is performing multi-touch or whether a plurality of users are performing single touches. Patent Document 1 determines whether a single user is performing multi-touch or a plurality of users are performing single touch based on the finger shape that touches the display surface during a touch operation, and displays the contents. Disclose the technology to change.

JP 2014-16795 A

  When the operation content is determined based on the finger shape of the portion that is in contact with the display surface during the touch operation as in the past, it is difficult to determine the posture of the user's hand during the touch operation. Therefore, it is not possible to perform processing such as changing the operation content in accordance with the posture of the hand (for example, holding the hand or opening the hand). By discriminating the posture of the hand, a wider variety of input operations than before can be performed.

  In order to solve the above problems, it is a main object of the present invention to provide an input system that has improved operability as compared with the prior art and is capable of various input operations.

  An input system according to the present invention includes display means for displaying a UI component on an operation plane, position detection means for detecting a touch position where a user touches the operation plane with a finger, and a finger touching the operation plane. Image acquisition means for acquiring the shape of the hand, gesture recognition means for recognizing a hand region obtained by projecting the hand onto the operation plane from the shape of the hand, the touch position detected by the position detection means, and the display means Depending on the position of the UI component displayed by the event detection means for detecting a touch operation on the UI component, the hand region recognized by the gesture recognition means, and the touch operation detected by the event detection means And determining means for determining the operation content of the UI component.

  According to the present invention, not only the touch operation to the UI component but also the shape of the hand at the time of touch is used for the determination of the operation content. .

1 is an overall configuration diagram of an input system. (A)-(c) is explanatory drawing of a camera scanner. The hardware configuration example figure of a controller part. The structural example figure of a functional module group. The flowchart of the process which a distance image acquisition part performs. (A)-(c) is explanatory drawing of the measurement principle of the distance image by a pattern projection system. The flowchart of the process of a gesture recognition part. (A)-(d) is explanatory drawing of the outline | summary of a fingertip detection process. The flowchart of the process of a touchscreen event detection part. The flowchart of the control process of a touch panel event. (A), (b) is explanatory drawing of the notification from a main control part to a user interface part. (A)-(c) is a schematic diagram in case a touch panel event detection part detects a multi-touch event. The flowchart of the control process of a touch panel event. The schematic diagram which shows the relationship between a hand area | region and a touch point. The flowchart of the control process of a touch panel event. The schematic diagram which shows the relationship between a hand area | region and a touch point. The flowchart of the control process of a touch panel event. The schematic diagram which shows operation by a user.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the present invention according to the claims, and all combinations of features described in the embodiments are not necessarily essential to the solution means of the present invention. .

[First Embodiment]
FIG. 1 is an overall configuration diagram of an input system according to this embodiment. This input system includes a camera scanner 101, a host computer 102, and a printer 103, which are communicably connected via a network 104 such as Ethernet.
This input system can execute a scan function for reading an image from the camera scanner 101 and a print function for outputting scan data by the printer 103 according to an instruction from the host computer 102. Further, it is possible to execute a scan function and a print function by direct instructions to the camera scanner 101 without using the host computer 102.

<Configuration of camera scanner>
FIG. 2 is an explanatory diagram of the camera scanner 101. An overall configuration diagram of the camera scanner 101 is shown in FIG. As illustrated, the camera scanner 101 includes a controller unit 201, a camera unit 202, an arm unit 203, a short focus projector 207, and a distance image sensor unit 208.
The controller unit 201, the camera unit 202, the projector 207, and the distance image sensor unit 208 are connected by an arm unit 203. The arm part 203 can be bent and stretched using a joint. The distance image sensor unit 208 is a sensor that measures the distance to an object (document 206 in the illustrated example) in a non-contact manner, and includes a lens that points to the object. The camera scanner 101 is installed on the operation plane 204. The lenses of the camera unit 202 and the distance image sensor unit 208 are directed in the direction of the operation plane 204, and can read an image in the reading area 205 surrounded by a broken line. In the illustrated example, the document 206 is placed in the reading area 205. Therefore, the camera scanner 101 can read the document 206.

  A turntable 209 is also provided on the operation plane 204. The turntable 209 can be rotated by an instruction from the controller unit 201. That is, the angle between the object placed on the turntable 209 and the camera unit 202 can be arbitrarily changed. The camera unit 202 may capture an image with a single resolution, but it is preferable that the camera unit 202 can capture a high-resolution image and a low-resolution image.

  A touch panel 330 is provided in the operation plane 204. The touch panel 330 is a position detection device that senses information on a touched position and outputs it as an information signal when the user touches it with a hand or applies pressure with a stylus pen. Although not shown in FIG. 2, the camera scanner 101 may further include a speaker. Furthermore, various sensor devices such as a human sensor, an illuminance sensor, and an acceleration sensor for collecting surrounding environmental information may be included.

  FIG. 2B is an explanatory diagram of a coordinate system in the camera scanner 101. In the camera scanner 101, coordinate systems such as a camera coordinate system, a distance image sensor coordinate system, and a projector coordinate system are defined for each hardware device. In these coordinate systems, an image plane captured by the RGB camera 363 of the camera unit 202 and the distance image sensor unit 208 or an image plane projected by the projector 207 is an XY plane. The direction orthogonal to the image plane is defined as the Z direction. Further, in order to be able to handle the three-dimensional data of these independent coordinate systems in a unified manner, an orthogonal coordinate system in which a plane including the operation plane 204 is an XY plane and a direction perpendicular to the XY plane is a Z axis. Define

  As an example of transforming the coordinate system, FIG. 2C illustrates a rectangular coordinate system, a space expressed using a camera coordinate system centered on the camera unit 202, and an image plane captured by the camera unit 202. Show the relationship. The three-dimensional point P [X, Y, Z] in the orthogonal coordinate system can be converted to the three-dimensional point Pc [Xc, Yc, Zc] in the camera coordinate system by the equation (1).

  Here, Rc and tc are configured by external parameters determined by the posture (rotation) and position (translation) of the camera unit 202 with respect to the orthogonal coordinate system. Rc is called a 3 × 3 rotation matrix, and tc is called a translation vector. Conversely, a three-dimensional point defined in the camera coordinate system can be converted into an orthogonal coordinate system by equation (2).

  The two-dimensional camera image plane photographed by the camera unit 202 is obtained by converting the three-dimensional information in the three-dimensional space into two-dimensional information by the camera unit 202. That is, the three-dimensional point Pc [Xc, Yc, Zc] on the camera coordinate system is converted by perspective projection conversion to the two-dimensional coordinate pc [xp, yp] on the camera image plane according to the equation (3). Can do.

  Here, A is a 3 × 3 matrix called an internal parameter of the camera and expressed by a focal length and an image center.

As described above, by using the expressions (1) and (3), the three-dimensional point group represented by the orthogonal coordinate system is converted into the three-dimensional point group coordinates and the camera image plane in the camera coordinate system. be able to.
It is assumed that the internal parameters of each hardware device and the position and orientation (external parameters) with respect to the orthogonal coordinate system are calibrated in advance by a known calibration method. Hereinafter, when there is no particular notice and it is expressed as a three-dimensional point group, it represents three-dimensional data in an orthogonal coordinate system.

<Configuration example of controller unit>
FIG. 3 is a diagram illustrating a hardware configuration example of the controller unit 201 which is the main body of the camera scanner 101. The controller unit 201 is configured by connecting various functional components to the system bus 301. The functional components include a computer having a CPU 302, a RAM 303, a ROM 304, and an HDD 305 as basic components. A CPU (Central Processing Unit) 302 controls the overall operation of the controller unit 201 by executing a control program for the camera scanner 101. That is, the control program is for operating the computer as the controller unit 201 or the camera scanner 101.
A RAM (Random Access Memory) 303 is a volatile rewritable storage medium and functions as a work area of the CPU 302. A ROM (Read Only Memory) 304 is a non-rewritable storage medium, and stores a startup program for the CPU 302. The HDD 305 is a hard disk drive (HDD) having a larger capacity than the RAM 303. The HDD 305 stores a control program for the camera scanner 101 executed by the controller unit 201.

  Functional components connected to the system bus 301 include a network I / F (I / F is an abbreviation of an interface, the same applies hereinafter) 306, an image processor 307, and a camera I / F 308. Furthermore, a display controller 309, a serial I / F 310, an audio controller 311 and a USB (Universal Serial Bus) controller 312 are also included in the functional components.

When the camera scanner 101 is activated by turning on the power or the like, the CPU 302 executes the activation program stored in the ROM 304. This activation program is for reading a control program stored in the HDD 305 and developing it on the RAM 303. When executing the startup program, the CPU 302 executes the control program developed on the RAM 303 and performs control.
The CPU 302 also stores data used for the operation by the control program on the RAM 303 to read / write. Various settings necessary for operation by the control program and image data generated by camera input can be further stored on the HDD 305 and read / written by the CPU 302. The CPU 302 communicates with other devices on the network 104 via the network I / F 306.

The image processor 307 reads the image data stored in the RAM 303, performs necessary image processing, and writes the processing result back to the RAM 303. Note that image processing executed by the image processor 307 includes rotation, scaling, color conversion, and the like.
The camera I / F 308 is connected to the camera unit 202 and the distance image sensor unit 208, and acquires image data from the camera unit 202 and distance image data from the distance image sensor unit 208 in accordance with an instruction from the CPU 302, and writes the acquired image data into the RAM 303. In addition, a control command from the CPU 302 is transmitted to the camera unit 202 and the distance image sensor unit 208 to set the camera unit 202 and the distance image sensor unit 208. The distance image sensor unit 208 includes an infrared pattern projection unit 361 that projects an infrared pattern image, an RGB camera 363 that captures a visible image, and an infrared camera 362 that captures an infrared image.

  The controller unit 201 can further include at least one of a display controller 309, a serial I / F 310, an audio controller 311, and a USB controller 312. A display controller 309 controls display of an image on the display in accordance with an instruction from the CPU 302. The display controller 309 is connected to the short-focus projector 207 and the touch panel 330.

  The serial I / F 310 inputs and outputs serial signals. In this example, the serial I / F 310 is connected to the turntable 209. Then, the CPU 302 transmits the rotation start / end and rotation angle instructions to the turntable 209. The serial I / F 310 is also connected to the touch panel 330. The CPU 302 acquires the coordinates (position) pressed via the serial I / F 310 when the touch panel 330 is pressed. Further, the CPU 302 determines whether or not the touch panel 330 is connected via the serial I / F 310.

  The audio controller 311 is connected to the speaker 340. The audio controller 311 converts audio data into an analog audio signal in accordance with an instruction from the CPU 302 and outputs audio through the speaker 340. The USB controller 312 controls an external USB device in accordance with an instruction from the CPU 302. In this example, the USB controller 312 is connected to an external memory 350 such as a USB memory or an SD card, and reads / writes data from / to the external memory 350.

<Functional configuration example of camera scanner>
FIG. 4 is an exemplary diagram of a configuration of a functional module group 401 formed in the controller unit 201 when the CPU 302 executes a control program. The control program is stored in the HDD 305 as described above, and is expanded on the RAM 303 when the CPU 302 is activated. Of the functional module group 401, the main control unit 402 is the center of control and controls the controller unit 201.

  The image acquisition unit 416 is a module that performs image input processing, and includes a camera image acquisition unit 407 and a distance image acquisition unit 408. The camera image acquisition unit 407 acquires an image output from the camera unit 202 via the camera I / F 308 and stores it in the RAM 303. The distance image acquisition unit 408 acquires the distance image output from the distance image sensor unit 208 via the camera I / F 308 and stores it in the RAM 303. Details of the processing of the distance image acquisition unit 408 will be described later.

The recognition processing unit 417 detects and recognizes the movement of an object on the operation plane 204 from images (image data) acquired by the camera image acquisition unit 407 and the distance image acquisition unit 408. For this purpose, the recognition processing unit 417 includes a gesture recognition unit 409, an object detection unit 410, and a touch panel event detection unit 420.
The gesture recognition unit 409 continuously acquires images on the operation plane 204 from the image acquisition unit 416, and recognizes a gesture such as touch from the acquired images. When the gesture recognition unit 409 recognizes the gesture, the gesture recognition unit 409 notifies the main control unit 402. Details of the processing of the gesture recognition unit 409 will be described later.
The object detection unit 410 acquires an image obtained by imaging the operation plane 204 from the image acquisition unit 416, and detects when the object is placed on the operation plane 204, when the object is placed and stopped, or when the object is removed. Perform the process.
When the user touches the touch panel 330 with an indicator such as a finger or a pen, the touch panel event detection unit 420 acquires touch coordinates representing a position where the indicator contacts via the serial I / F 310. The touch panel event detection unit 420 notifies the main control unit 402 of the acquired touch coordinate information as a touch panel event. In the following description, a finger is used as an indicator.

  The image processing unit 418 performs processing for analyzing images acquired from the camera unit 202 and the distance image sensor unit 208 by the image processing processor 307. The gesture recognition unit 409, the object detection unit 410, and the touch panel event detection unit 420 perform processing using the processing result of the image processing unit 418.

  The user interface unit 403 receives a request from the main control unit 402 and generates a UI component such as a message or a button. The user interface unit 403 requests the display unit 406 to display the generated UI component. The display unit 406 displays the requested UI component by the projector 207 via the display controller 309. Since the projector 207 is installed toward the operation plane 204, it is possible to project UI parts on the operation plane 204. Further, the user interface unit 403 receives a gesture operation such as a touch recognized by the gesture recognition unit 409, a touch operation detected by the touch panel event detection unit 420, and coordinates thereof via the main control unit 402. The user interface unit 403 determines the operation content (such as a pressed button) by associating the content of the screen (operation screen) being displayed with the operation coordinates. The main control unit 402 receives a user operation by acquiring the operation content from the user interface unit 403.

  The network communication unit 404 communicates with other devices on the network 104 by TCP / IP via the network I / F 306. The data management unit 405 stores and manages various data including work data generated during the operation of the functional module group 401 in a predetermined area on the HDD 305.

<Distance Image Sensor and Distance Image Acquisition Unit>
The operation of the distance image acquisition unit 408 will be described with reference to FIGS. FIG. 5 is a flowchart of processing executed by the distance image acquisition unit 408. FIGS. 6A to 6C are explanatory diagrams of the measurement principle of the distance image by the pattern projection method. The distance image sensor unit 208 is a pattern image type distance image sensor using infrared rays.

  Referring to FIG. 5, when starting the processing, the distance image acquisition unit 408 projects a three-dimensional shape measurement pattern 522 using infrared rays onto the object 521 using an infrared pattern projection unit 361 as shown in FIG. (S501). The distance image acquisition unit 408 acquires an RGB camera image 523 obtained by photographing the object 521 using the RGB camera 363 and an infrared camera image 524 obtained by photographing the three-dimensional shape measurement pattern 522 projected in S501 using the infrared camera 362. (S502).

  Note that the infrared camera 362 and the RGB camera 363 have different installation positions. That is, as shown in FIG. 6B, the shooting areas of the RGB camera image 523 and the infrared camera image 524 that are shot are different. Therefore, the distance image acquisition unit 408 matches the infrared camera image 524 to the coordinate system of the RGB camera image 523 using coordinate system conversion from the coordinate system of the infrared camera 362 to the coordinate system of the RGB camera 363 (S503). It is assumed that the relative positions of the infrared camera 362 and the RGB camera 363 and their internal parameters are known by a prior calibration process.

  Thereafter, as shown in FIG. 6C, the distance image acquisition unit 408 extracts a corresponding point between the three-dimensional shape measurement pattern 522 and the infrared camera image 524 that has undergone coordinate transformation in S503 (S504). . For example, one point on the infrared camera image 524 is searched from the three-dimensional shape measurement pattern 522, and association is performed when the same point is detected. Alternatively, a pattern around the pixels of the infrared camera image 524 may be searched from the three-dimensional shape measurement pattern 522 and associated with a portion having the highest similarity.

  When the corresponding points are extracted, the distance image acquisition unit 408 calculates the distance from the infrared camera 362 by performing calculation using the principle of triangulation with the straight line connecting the infrared pattern projection unit 361 and the infrared camera 362 as the base line 525. Calculate (S505). For the pixels that can be associated in the process of S504, the distance from the infrared camera 362 is calculated and stored as a pixel value. For pixels that could not be associated, an invalid value is stored as the portion where the distance could not be measured. The distance image acquisition unit 408 generates a distance image in which each pixel has a distance value by performing this operation on all the pixels of the infrared camera image 524 that have undergone coordinate conversion in S503.

  Thereafter, the distance image acquisition unit 408 generates a distance image having four values of R, G, B, and distance for each pixel by storing the RGB values of the RGB camera image 523 in each pixel of the distance image ( S506). The distance image acquired here is based on the distance image sensor coordinate system defined by the RGB camera 363 of the distance image sensor unit 208. Therefore, as shown in FIG. 2B, the distance image acquisition unit 408 converts the distance data obtained as the distance image sensor coordinate system into a three-dimensional point group in the orthogonal coordinate system (S507). In the following description, unless otherwise specified, the term “three-dimensional point group” refers to a three-dimensional point group in an orthogonal coordinate system.

  In the first embodiment, an infrared pattern projection method is adopted as the distance image sensor unit 208, but a distance image sensor unit of another method can also be used. For example, a stereo system that performs stereo stereoscopic vision with two RGB cameras or a TOF (Time of Flight) system that measures distance by detecting the flight time of laser light may be used.

<Gesture recognition unit>
Details of the processing of the gesture recognition unit 409 will be described with reference to the flowchart of FIG. In FIG. 7, when the gesture recognition unit 409 starts the process, the gesture recognition unit 409 performs an initialization process (S601). That is, the gesture recognition unit 409 acquires one frame of the distance image from the distance image acquisition unit 408. The gesture recognizing unit 409 recognizes the plane of the operation plane 204 as an initial state because no object is placed on the operation plane 204 when the operation starts. That is, the widest plane is extracted from the acquired distance image, its position and normal vector (hereinafter referred to as plane parameters of the operation plane 204) are calculated, and stored in the RAM 303.

  When the initialization process is completed, the gesture recognition unit 409 acquires a three-dimensional point group of an object existing on the operation plane 204 (S602). Specifically, the gesture recognition unit 409 acquires one frame of the distance image and the three-dimensional point group from the distance image acquisition unit 408 (S621). In addition, the gesture recognizing unit 409 removes the point group on the plane including the operation plane 204 from the acquired three-dimensional point group using the plane parameter of the operation plane 204 (S622). Thereafter, the gesture recognizing unit 409 performs a hand shape and fingertip detection process of detecting the shape and fingertip of the user's hand from the acquired three-dimensional point group (S603). More specifically, the process of S631-S634 is performed.

  Here, an outline of the fingertip detection process in S631 to S634 will be described with reference to FIG. The gesture recognizing unit 409 extracts a skin-colored three-dimensional point group that is higher than a predetermined height from the plane including the operation plane 204 from the three-dimensional point group acquired in S602. Thereby, the three-dimensional point group 661 of the hand shown in FIG. 8A is extracted (S631). Thereafter, the gesture recognition unit 409 generates a two-dimensional image 662 obtained by projecting the extracted three-dimensional point group 661 of the hand onto the plane of the operation plane 204, and detects the outer shape of the hand (S632). Then, hand center-of-gravity coordinates are calculated from the detected outer shape of the hand. The projection may be performed by projecting the coordinates of the point group using the plane parameters of the operation plane 204. Further, as shown in FIG. 8B, by extracting only the value of the xy coordinates from the projected three-dimensional point group, a two-dimensional image viewed from the z-axis direction is taken as a hand region 663 that occupies the operation plane 204. Can be handled. At this time, it is stored which of the coordinates of the hand region 663 projected by each point of the three-dimensional point group of the hand onto the plane of the operation plane 204 is stored.

Thereafter, as shown in FIG. 8C, the gesture recognizing unit 409 calculates the curvature of the outer shape at each point on the detected outer shape of the hand, and calculates a point where the calculated curvature is smaller than a predetermined value. It is detected as a fingertip (S633). FIG. 8C is a diagram schematically showing a method for detecting a fingertip from the curvature of the outer shape. In this figure, a part of a point 664 representing the outer shape of the two-dimensional image (hand region 663) projected on the plane of the operation plane 204 is shown.
Here, it is considered that a circle is drawn so as to include five adjacent points among the points 664 representing the outer shape. Circles 665 and 667 are examples. Such a circle is drawn in order with respect to all the points of the outer shape, and its diameter (for example, 666, 668) is smaller than a predetermined value (curvature is small), and is used as a fingertip. In this example, five adjacent points are used, but the number is not limited. In addition, the curvature is used here, but the fingertip may be detected by performing elliptic fitting on the outer shape.

  When the fingertip is detected, the gesture recognition unit 409 calculates the number of detected fingertips and the coordinates of each fingertip (S634). Since the gesture recognition unit 409 stores the correspondence between each point of the two-dimensional image projected on the operation plane 204 and each point of the three-dimensional point group of the hand, the gesture recognition unit 409 can obtain the three-dimensional coordinates of each fingertip. .

  In this example, the method for detecting a fingertip from an image projected from a three-dimensional point group onto a two-dimensional image has been described, but the image to be detected by the fingertip is not limited to this. For example, a hand region may be extracted from a background difference of a distance image or a skin color region of an RGB image, and a fingertip in the hand region may be detected by a method similar to the above (external curvature calculation or the like). In this case, since the coordinates of the detected fingertip are coordinates on a two-dimensional image such as an RGB image or a distance image, it is necessary to convert the coordinate information into the three-dimensional coordinates of the orthogonal coordinate system using the distance information of the distance image at that coordinate. is there. At this time, the center of the curvature circle used when detecting the fingertip may be used as the fingertip point, instead of the point on the outer shape that becomes the fingertip point.

  After the fingertip detection process, the gesture recognition unit 409 performs an arm detection process (S606). FIG. 8D is a diagram schematically illustrating the arm detection process. The gesture recognizing unit 409 extracts a three-dimensional point group connected to the three-dimensional point group 661 of the hand region extracted in S631, and detects the three-dimensional point group 669 of the arm (S661). The gesture recognition unit 409 calculates an arm region 670 obtained by projecting the detected arm three-dimensional point group 669 onto the plane of the operation plane 204. The gesture recognizing unit 409 calculates the coordinates of the point where the edge of the operation plane 204 and the arm region 670 intersect as the entry point 671 of the arm operation plane 204 (S662).

  The gesture recognition unit 409 notifies the main control unit 402 of the coordinates (position information) of the fingertip calculated in S634 and the coordinates (position information) of the intrusion point 671 calculated in S662 (S607). After the notification, the gesture recognition unit 409 returns to the process of S602 and repeats the gesture recognition process. In the above embodiment, the gesture recognition with one finger has been described. However, a plurality of fingers, a plurality of hands, an arm, or the entire body may be set as a target for gesture recognition.

<Touch panel event detection unit>
The process of the touch panel event detection unit 420 will be described with reference to the flowchart of FIG. When the user touches the touch panel 330 with a finger or the like, the touch panel event detection unit 420 acquires touch coordinates that are position information indicating a position where the finger or the like has touched from the serial I / F 310.

  When the process is started, the touch panel event detection unit 420 confirms whether or not the touch coordinates have been acquired (S701). Until the touch coordinates are acquired, the touch panel event detection unit 420 repeats the touch coordinate acquisition confirmation (S701: N). When the touch coordinates are acquired (S701: Y), the touch panel event detection unit 420 notifies the main control unit 402 of the acquired touch coordinates as a touch panel event (S702). After that, the touch panel event detection unit 420 returns to the processing of S701 and confirms acquisition of touch coordinates. By repeating such processing, the touch panel event detection unit 420 continues to notify the main control unit 402 of touch panel events.

<Main control unit>
The main control unit 402 performs processing according to the touch panel event notified from the touch panel event detection unit 420. FIG. 10 is a flowchart of touch panel event control processing by the main control unit 402.

  When starting the processing, the main control unit 402 instructs the gesture recognition unit 409 to start processing (S801). In response to this instruction, the gesture recognition unit 409 performs the above-described processing of FIG. In addition, the main control unit 402 instructs the touch panel event detection unit 420 to start processing (S802). The touch panel event detection unit 420 performs the above-described processing of FIG. 9 in response to this instruction.

  The main control unit 402 confirms whether or not there is a touch event notification from the gesture recognition unit 409 (S803). The main control unit 402 confirms the presence / absence of a touch event notification until notified (S803: N). When there is a touch event notification (S803: Y), the main control unit 402 performs a single touch process (S804).

  In the single touch process, the main control unit 402 first acquires the number of fingers detected by the gesture recognition unit 409 (S811). The main control unit 402 determines whether the number of fingers detected by the gesture recognition unit 409 is 1 (S812). When the number of detected fingers is one (S812: Y), the main control unit 402 notifies the user interface unit 403 of a pointing posture touch event (S813). When the number of detected fingers is not one (S812: N), the main control unit 402 notifies the user interface unit 403 of a touch event that is not a pointing posture (S814). The main control unit 402 that has notified the user interface unit 403 returns to S803 and waits for a touch event notification from the gesture recognition unit 409.

  FIG. 11 is an explanatory diagram of notification from the main control unit 402 to the user interface unit 403.

FIG. 11A is an explanatory diagram of processing for notifying the user interface unit 403 of the pointing posture touch event in S813. The user interface unit 403 projects a UI component 801 on the operation plane 204. A touch point 803 is a finger contact position on the operation plane 204 detected by the touch panel event detection unit 420. A hand region 802 is a user hand region recognized by the gesture recognition unit 409. The gesture recognition unit 409 detects one fingertip 804 by the process of S633 in FIG.
Thus, since the gesture recognition unit 409 detects one fingertip, the main control unit 402 notifies the user interface unit 403 that the touch is in the pointing posture. The user interface unit 403 can use not only that the UI component 801 has been touched at the touch point 803 but also that the UI component 801 has been touched with the pointing posture, and therefore can perform processing according to them.

FIG. 11B is an explanatory diagram of processing for notifying the user interface unit 403 of a touch event that is not the pointing posture in S814. A touch point 805 is a finger contact position on the operation plane 204 detected by the touch panel event detection unit 420. A hand region 806 is a hand region recognized by the gesture recognition unit 409. The gesture recognizing unit 409 detects five fingertips 807 to 811 by the process of S633 in FIG.
Thus, since the gesture recognizing unit 409 detects one or more fingertips, the main control unit 402 notifies the user interface unit 403 that the touch is in a posture other than the pointing posture. The user interface unit 403 can use not only that the UI component 801 is touched at the touch point 805 but also that the UI component 801 is touched in a posture that is not a pointing posture, and can execute processing according to them. it can.

  As described above, the user interface unit 403 can determine whether or not the user is in the pointing posture when the UI component 801 is touched. Different processes can be performed depending on the contents. For example, a line drawing process is performed on the UI component 801 when it is determined that a touch operation is performed with a pointing posture, and an internal image of the UI component 801 is displayed when it is determined that a touch operation is performed with a posture that is not a pointing posture. It is possible to perform another process such as scrolling. Each process is registered in advance in the user interface unit 403 for each UI component displayed by the main control unit 402.

  In addition to the touch point detected by the touch panel event detection unit 420, the input system as described above improves the user operability in order to use the posture of the hand detected by the gesture recognition unit 409 for UI component operation. A wide variety of input operations are possible. In this embodiment, the example in which the operation to the UI component is switched according to the pointing posture when the user touches has been described. However, the posture of the hand detected by the gesture recognition unit 409 is not limited to this. For example, the UI component operation may be switched between a posture with two fingers and a posture with one finger. In addition to determining the hand posture based on the number of detected fingertips, the gesture recognizing unit 409 calculates a hand skeleton model from the three-dimensional point cloud of the hand, and determines the hand posture from the calculated skeleton model. Also good.

[Second Embodiment]
In 1st Embodiment, the process in case the touch panel event detection part 420 detects the single touch in which a user touches only one point on the operation plane 204 was demonstrated. In the second embodiment, a process when the touch panel event detection unit 420 detects multi-touch in which the user touches two points on the operation plane 204 will be described. Since the configuration of the input system is the same as that of the first embodiment, description thereof is omitted.

  FIG. 12 is a schematic diagram when the touch panel event detection unit 420 detects a multi-touch event.

  FIG. 12A shows a case where the user performs multi-touch of two points with one hand. The touch panel event detection unit 420 detects touch points 903 and 904. The gesture recognition unit 409 detects the hand region 901 and the arm entry point 902. When there are two touch points and one hand area, the main control unit 402 determines that one user is a one-hand multi-touch.

  FIG. 12B shows a case where the user performs multi-touch of two points with both hands. The touch panel event detection unit 420 detects touch points 913 and 916. The gesture recognition unit 409 detects hand areas 911 and 914 and arm entry points 912 and 915. When there are two touch points, two hand areas, and two arm entry points are on the same side of the operation plane 204, the main control unit 402 allows one user to place both hands on the same side of the operation plane 204. Judge that they are putting their hands from above. Accordingly, the main control unit 402 determines that the touch point 913 and the touch point 916 are two-hand multi-touch by one user.

  FIG. 12C shows a case where two users touch one point at a time. The touch panel event detection unit 420 detects touch points 923 and 926. The gesture recognition unit 409 detects the hand regions 921 and 941 and the arm entry points 922 and 925. When there are two touch points, two hand areas, and two arm intrusion points on different sides of the operation plane 204, the main control unit 402 allows two users to move from different directions on the operation plane 204, respectively. Judge that you are putting your hand. Thereby, the main control unit 402 determines that the touch point 923 and the touch point 926 are two single touches by two users.

  FIG. 13 is a flowchart of touch panel event control processing by the main control unit 402 during multi-touch. Processes similar to those in the flowchart of FIG. 10 are given the same step numbers.

When starting the processing, the main control unit 402 instructs the gesture recognition unit 409 to start processing (S801). The main control unit 402 instructs the touch panel event detection unit 420 to start processing (S802). The main control unit 402 waits for notification of a touch event from the gesture recognition unit 409 (S803).
When the notification of the touch event is acquired (S803: Y), the main control unit 402 determines whether or not the number of touch points is “1” (S901). When the number of touch points is “1” (S901: Y), the main control unit 402 performs the single touch process of FIG. 10 (S804).

  When the number of touch points is greater than “1” (S901: N), the main control unit 402 performs multi-touch processing (S902). In the multi-touch process, the main control unit 402 first acquires the number of hand regions detected from the gesture recognition unit 409 (S911). The main control unit 402 determines whether or not the acquired number of hand regions is “1” (S912).

  When the number of hand regions is “1” (S912: Y), the main control unit 402 determines that the multi-touch is performed by one user as shown in FIG. The main control unit 402 notifies the user interface unit 403 of the one-handed multi-touch event (S916). When the number of hand areas is 2 or more (S912: N), the main control unit 402 determines whether the entry point of the hand area into the operation plane 204 is on the same side of the operation plane 204, that is, the arm from the same direction. It is determined whether or not the user has entered (S913).

  When the arm has entered from the same direction (S913: Y), the main control unit 402 determines that the event is a two-handed multi-touch event of one user as shown in FIG. The main control unit 402 notifies the user interface unit 403 of the two-hand multi-touch event (S914). If the arm has entered from a different direction (S913: N), the main control unit 402 determines that the single touch event is performed by each of the two users as shown in FIG. The main control unit 402 notifies the user interface unit 403 of a single touch event by each of the two users (S915). The main control unit 402 that has notified the user interface unit 403 returns to S803 and waits for a touch event notification from the gesture recognition unit 409.

Through the processing of the main control unit 402 as described above, the user interface unit 403 can determine which of the following operations is performed when the touch panel event detection unit 420 detects two touch events. .
-Two single touches by two users-One-handed multi-touch-Two-handed multi-touch by one user

  The user interface unit 403 can execute different processes according to the above three determination results. For example, the user interface unit 403 performs enlargement / reduction of a UI component in one-hand multi-touch, rotation of a UI component in two-hand multi-touch, and drawing of a locus of a touch point on the UI component in two single touches. Each processing is registered in advance in the user interface unit 403 for each UI component displayed by the main control unit 402, as in the first embodiment.

  Thus, the input system determines the number of operating users by using not only the touch points detected by the touch panel event detection unit 420 but also the number of hand regions and entry points detected by the gesture recognition unit 409. Is possible. Therefore, a plurality of users can perform single touch and multi-touch at the same time, so that operability is improved and a variety of input operations can be performed as compared with the prior art.

[Third Embodiment]
In the second embodiment, the maximum number of users is two, and the maximum number of touch points on the operation plane 204 is two. In the third embodiment, an appropriate touch event is executed even when there are two or more touch points by associating the touch points detected by the touch panel event detection unit 420 with the hand regions detected by the gesture recognition unit 409. Processing will be described. Since the configuration of the input system is the same as that of the first embodiment, description thereof is omitted.

  FIG. 14 is a schematic diagram illustrating a relationship between a hand region and a touch point. The hand region 1001 is associated with one touch point 1003 and enters the operation plane 204 at the entry point 1002. The hand region 1001 is determined as a single touch because there is no other hand region that enters the operation plane 204 from the same direction and there is one touch point. The hand region 1004 is associated with the two touch points 1006 and 1007 and enters the operation plane 204 at the entry point 1005. The hand region 1004 is determined as one-hand multi-touch because there is no other hand region that enters the operation plane 204 from the same direction and there are two touch points.

  The hand region 1008 is associated with one touch point 1010 and enters the operation plane 204 at the entry point 1009. The hand region 1011 is associated with one touch point 1013 and enters the operation plane 204 at the entry point 1012. The entry point 1009 in the hand area 1008 and the entry point 1012 in the hand area 1011 are on the same side of the operation plane 204. Therefore, it is determined that the hand area 1008 and the hand area 1011 have entered from the same direction. Further, since the total number of touch points of the hand area 1008 and the hand area 1011 is “2”, the hand area 1008 and the hand area 1011 are determined to be two-hand multi-touch.

  FIG. 15 is a flowchart of touch panel event control processing by the main control unit 402 that performs touch type determination and touch event determination processing described with reference to FIG. 14. Processes similar to those in the flowchart of FIG. 10 are given the same step numbers.

  When starting the processing, the main control unit 402 instructs the gesture recognition unit 409 to start processing (S801). The main control unit 402 instructs the touch panel event detection unit 420 to start processing (S802). The main control unit 402 waits for notification of a touch event from the gesture recognition unit 409 (S803).

  When the notification of the touch event is acquired (S803: Y), the main control unit 402 calculates which hand region is included in each touch point notified by the touch event, and includes the included hand region and the touch point. Are associated (S1001). In addition, since the coordinates of the touch point are detected by the touch panel 330 and the hand region is detected by the distance image sensor unit 208, a coordinate shift due to a difference in detection method may occur. For this reason, the touch point may not be included in any hand region. In this case, the main control unit 402 associates the touch point with the nearest hand region.

  The main control unit 402 performs the subsequent loop processing for all hand regions (S1002, S1003). First, the main control unit 402 determines whether or not there is an associated touch point for one hand region that is a target of the loop processing (S1011). If there is no associated touch point (S1011: N), the main control unit 402 performs a loop process for the next hand region. When there is an associated touch point (S1011: Y), the main control unit 402 performs a touch event determination process (S1021).

  In the touch event determination process, the main control unit 402 first enters the operation plane 204 from the same direction as the presence or absence of a hand area having an intrusion point on the same side as the side of the entry point of the hand area to be processed. The presence or absence of an intruding hand region is confirmed (S1012). Further, the main control unit 402 determines whether a touch point is associated with a hand region that enters the operation plane 204 from the same direction. When there is a hand region that enters the operation plane 204 from the same direction and is associated with a touch point (S1012: Y), the main control unit 402 performs a multi-handed operation using both hands as in the hand region 1008 and the hand region 1011 in FIG. Judged to be a touch. The main control unit 402 notifies the user interface unit 403 of the two-hand multi-touch event (S1016).

  When there is no hand region that enters the operation plane 204 from the same direction and is associated with a touch point (S1012: N), the main control unit 402 determines whether there is one touch point associated with the hand region to be processed. Is determined (S1013). When there is one touch point (S1013: Y), the main control unit 402 determines that the touch is single touch. The main control unit 402 notifies the user interface unit 403 of a single touch event (S1014). When there are more touch points than 1 (S1013: N), the main control unit 402 determines that the event is a multi-touch event with one hand. The main control unit 402 notifies the user interface unit 403 of the one-hand multi-touch event (S1015).

  The main control unit 402 that has notified the touch event to the user interface unit 403 confirms whether there is an unprocessed hand region (S1003). When there is an unprocessed hand area, the main control unit 402 repeats the processes after S1002 until the processes for all the hand areas are completed. When the processes for all the hand regions are completed, the main control unit 402 returns to S803 and waits for a touch event notification from the gesture recognition unit 409.

  Through the processing of the main control unit 402 as described above, the user interface unit 403 can separately acquire notifications of single touch events, one-hand multi-touch events, and two-hand multi-touch events. Furthermore, as described above, by associating the hand region and the touch point, even when these touch operations occur simultaneously as shown in FIG. 14, the user interface unit 403 can acquire the corresponding touch event. it can. Thereby, it becomes possible for a plurality of users to simultaneously operate UI components projected by the user interface unit 403 on the operation plane 204 from different directions. Therefore, the operability of a plurality of users is improved, and a wider variety of input operations than before are possible.

[Fourth Embodiment]
In the second and third embodiments, when there are a plurality of hand areas touching on the operation plane 204, two-hand multi-touch is performed depending on whether the hand area has entered the operation plane 204 from the same direction, or It is determined whether it is a single-handed single touch. In the fourth embodiment, discrimination between both-hand multi-touch and one-hand single touch is performed based on whether the hand region is the right hand or the left hand. This is performed by the main control unit 402 calculating the shape of the hand region, and thereby determining whether the hand region is the right hand or the left hand. Since the configuration of the input system is the same as that of the first embodiment, description thereof is omitted.

  FIG. 16 is a schematic diagram illustrating a relationship between a hand region and a touch point. The hand region 1101 is associated with one touch point 1102 and enters the operation plane 204 at the entry point 1103. The main control unit 402 determines whether the touch point 1102 is a one-hand single touch touch point or a two-hand multi-touch touch point as follows. First, the main control unit 402 determines that the hand region 1101 is the right hand from the shape of the hand region 1101. Then, the intruding hand region is searched from the left side of the intrusion point 1103. When the hand region 1101 is the left hand, the hand region intruding from the right side of the intrusion point 1103 is searched. In FIG. 16, a hand region 1104 invading from the intrusion point 1106 is found. The main control unit 402 determines that the hand region 1104 is the right hand from the shape of the found hand region 1104. Since the hand region 1101 is the right hand and the left hand region 1104 is also the right hand, the touch point 1102 of the hand region 1101 is determined to be a single touch with one hand.

  Whether the touch point 1105 of the hand region 1104 is a one-hand single touch touch point or a two-hand multi-touch touch point is also determined in the same manner as the hand region 1101. That is, it is performed by determining that the hand region 1104 is the right hand and searching for the intruding hand region from the left side. Here, not only the hand region that has entered from the same side 1110 of the operation plane 204 but also the hand region that has entered from the side 1111 adjacent to the left side of the side where the entry point 1106 exists, An intruding hand area 1107 can be found. If the hand region 1104 is the left hand, the intruding hand region is searched from the right side 1112. Subsequently, it is determined that the hand region 1107 is the left hand from the shape of the hand region 1107 found on the left side. Since the hand region 1104 is the right hand and the left hand region 1107 is the left hand, the touch point 1105 of the hand region 1104 is determined to be a two-hand multi-touch touch point.

  FIG. 17 is a flowchart of touch panel event control processing by the main control unit 402, which performs the touch type determination and touch event determination processing described in FIG. The process of FIG. 17 is replaced with the process of S1021 in the flowchart of FIG. That is, before the flowchart of FIG. 17, the processing of S801 to S1011 of FIG. 15 is performed.

  The main control unit 402 determines whether the hand region is the right hand or the left hand from the shape of the hand region that is the target of the touch event determination (S1101). The main control unit 402 searches for a hand region having an entry point on the left side if the hand region to be determined is the right hand and on the right side if the hand region is on the left hand (S1102). The hand area to be searched includes a hand area where the intrusion point has the same side of the operation plane 204, and a left side of the operation plane 204 if the touch event determination target hand area is the right hand, and a touch event determination target hand area If it is the left hand, it is a hand region having an entry point on the right side of the operation plane 204.

  The main control unit 402 finds an adjacent hand region and determines whether there is a touch point in the hand region (S1103). When there is an adjacent hand region and there is a touch point in the hand region (S1103: Y), the main control unit 402 determines whether the hand region is a right hand or a left hand based on the shape of the adjacent hand region. Is discriminated (S1104). The main control unit 402 determines whether or not the hand region for touch event determination and the adjacent hand region are opposite hands, that is, one is a right hand and the other is a left hand (S1105). When the hand region for the touch event determination target and the adjacent hand region are opposite hands (S1105: Y), the main control unit 402 determines that the event is a two-handed multi-touch event. The main control unit 402 notifies the user interface unit 403 of the two-hand multi-touch event (S1106).

  When there is no adjacent hand region or there is no touch point in the hand region even if there is an adjacent hand region (S1103: N), the main control unit 402 has one touch point associated with the hand region to be touch event determined. It is determined whether or not there is one (S1107). In addition, the main control unit 402 also determines the touch point associated with the touch event determination target hand region even when the hand region of the touch event determination target and the adjacent hand region are not opposite hands (S1105: N). It is determined whether or not there is one (S1107). The case where the hand region for touch event determination and the adjacent hand region are not the opposite hand is a case where both the right hand or both the left hand are used.

  When there is one touch point (S1107: Y), the main control unit 402 determines that it is a single touch event. The main control unit 402 notifies the user interface unit 403 of the single touch event (S1108). When there are a plurality of touch points (S1107: N), the main control unit 402 determines that the event is a one-handed multi-touch event. The main control unit 402 notifies the user interface unit 403 of a one-handed multi-touch event. The main control unit 402 that has notified the touch event to the user interface unit 403 ends the touch event determination process.

  Through the processing of the main control unit 402 as described above, the user interface unit 403 can separately acquire notifications of single touch events, one-hand multi-touch events, and two-hand multi-touch events. Further, by associating the hand region and the touch point as described above, even when these touch operations occur simultaneously as shown in FIG. 14, the user interface unit 403 can acquire the corresponding touch event. it can. Thereby, it becomes possible for a plurality of users to simultaneously operate UI components projected by the user interface unit 403 on the operation plane 204 from different directions. Therefore, the operability of a plurality of users is improved, and a wider variety of input operations than before are possible. Furthermore, as shown in the hand region 1104 and the hand region 1107 in FIG. 16, it is possible to perform two-hand multi-touch across the two sides of the operation plane 204.

[Fifth Embodiment]
In the first to fourth embodiments, the distance image sensor unit 208 is arranged to read the inside of the reading area 205, and the input system discriminates the operation using the hand area on the operation plane 204. In the fifth embodiment, the distance image sensor unit 208 acquires a distance image of an object around the operation plane 204 by setting the angle of view to be a wider angle and being arranged at a higher position. Thereby, not only the three-dimensional point group of the hand on the operation plane 204 but also a three-dimensional point group representing the shape of the arm and the human body around the operation plane 204 can be acquired. The input system can determine the human body to which the hand region is connected using the acquired three-dimensional point cloud. Since the configuration of the input system is the same as that of the first embodiment, description thereof is omitted.

  FIG. 18 is a schematic diagram illustrating an operation by the user. In addition to the hand areas 1201, 1204, and 1206 on the operation plane 204, the distance image sensor unit 208 detects human bodies 1203 and 1208.

The main control unit 402 determines that the human body 1203 to which the hand area 1201 is connected is not connected to the hand area in which a touch operation is performed in addition to the hand area 1201. Accordingly, the main control unit 402 determines that the touch point 1202 of the hand area 1201 is a single touch, and notifies the user interface unit 403 of a single touch event.
The main control unit 402 determines that the hand region 1206 having the touch point 1207 is connected to the human body 1208 to which the hand region 1204 is connected. Thus, the main control unit 402 determines that the touch point 1205 and the touch point 1207 are two-hand multi-touch touch points, and notifies the user interface unit 403 of the two-hand multi-touch event.

  Such a touch event determination process by the main control unit 402 is incorporated in the touch event determination process (S1202) of FIG. Accordingly, the user interface unit 403 can distinguish and acquire a single touch event, a one-hand multi-touch event, and a two-hand multi-touch event. With the processing as described above, it becomes possible for a plurality of users to operate from different directions at the same time, so that the operability is improved and a wider variety of input operations than before are possible.

(Other examples)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

  DESCRIPTION OF SYMBOLS 101 ... Camera scanner, 201 ... Controller part, 202 ... Camera part, 204 ... Operation plane, 207 ... Projector, 208 ... Distance image sensor part, 330 ... Touch panel, 402 ... Main control part, 409 ... Gesture recognition part, 420 ... Touch panel Event detector

Claims (12)

Display means for displaying UI parts on the operation plane;
Position detecting means for detecting a touch position where a user touches the operation plane with a finger;
Image acquisition means for acquiring a shape of a hand including a finger touching the operation plane;
Gesture recognition means for recognizing a hand region obtained by projecting the hand onto the operation plane from the shape of the hand;
Event detection means for detecting a touch operation on the UI component from the touch position detected by the position detection means and the position of the UI component displayed by the display means;
Determination means for determining operation content to the UI component according to the hand region recognized by the gesture recognition means and the touch operation detected by the event detection means,
Input system. The gesture recognition means detects the number of fingers from the recognized hand region,
The determination unit determines an operation content to the UI component according to the number of fingers detected by the gesture recognition unit and the touch operation detected by the event detection unit.
The input system according to claim 1. When the position detection unit detects one touch position, the determination unit has different operation contents depending on whether the number of fingers detected by the gesture recognition unit is one or not. It is characterized by making a judgment,
The input system according to claim 2. The image acquisition means acquires a shape of an arm connected to the hand,
The gesture recognition means detects an entry point where the arm enters the operation plane from the shape of the arm,
The determination means determines the operation content according to the number of the touch positions detected by the position detection means, the number of hand regions recognized by the gesture recognition means, the number of intrusion points, and the position of the intrusion points. It is characterized by
The input system according to claim 2. When the position detecting unit detects two touch positions and the number of the arms whose shape is recognized by the gesture recognizing unit is 1, the determining unit is a one-hand multi-touch by one user. It is characterized by judging,
The input system according to claim 4. When the position detecting unit detects the two touch positions and the number of the arms whose shape is recognized by the gesture recognizing unit is 2, the determination unit determines that the positions of the two entry points are the operation planes. Depending on whether or not they are on the same side, the operation content is determined differently.
The input system according to claim 4 or 5. An association means for associating the touch position detected by the position detection means with a hand region of a hand including a finger touching the touch position;
The determination means determines the operation content according to the number of the touch positions associated with the hand region and the position of the entry point of the arm connected to the hand,
The input system according to any one of claims 4 to 6. The associating means associates each touch position with a hand area when the position detecting means detects a plurality of the touch positions;
The determination unit determines the operation content according to whether the hand recognized by the gesture recognition unit is a right hand or a left hand and the position of the intrusion point.
The input system according to claim 7. The image acquisition means acquires a shape of a human body to which the arm connects,
The position detection means determines the operation content according to a hand shape of a hand connected to the human body,
The input system according to claim 1. An image for acquiring a shape of a hand including a display means for displaying a UI component on the operation plane, a position detection means for detecting a touch position where the user touches the operation plane with a finger, and a finger touching the operation plane. And a recognition method for recognizing a hand region obtained by projecting the hand onto the operation plane from the shape of the hand,
From the touch position detected by the position detection unit and the position of the UI component displayed by the display unit, a touch operation to the UI component is detected,
According to the hand region and the touch operation, the operation content to the UI component is determined,
input method. An image for acquiring a shape of a hand including a display means for displaying a UI component on the operation plane, a position detection means for detecting a touch position where the user touches the operation plane with a finger, and a finger touching the operation plane. An acquisition means, and a computer,
Gesture recognition means for recognizing a hand region obtained by projecting the hand onto the operation plane from the shape of the hand;
Event detection means for detecting a touch operation on the UI component from the touch position detected by the position detection means and the position of the UI component displayed by the display means;
A determination unit that determines an operation content to the UI component according to the hand region recognized by the gesture recognition unit and the touch operation detected by the event detection unit;
Control program to function as   A computer-readable storage medium storing the control program according to claim 11.