JP5675196B2 - Information processing apparatus and control method thereof - Google Patents

Description

  The present invention is a technique for improving user operability with respect to an object displayed on a screen.

  An apparatus has been proposed in which an electronic image, document, or the like (hereinafter referred to as an object) is displayed on a large table display or wall display, and a user operates the displayed object (Patent Document 1). . This device is used for meetings, training, etc., and by performing operations such as enlargement / reduction, movement, editing, grouping, etc. on the objects displayed by attendees, it is possible to organize ideas, activate discussions, and improve efficiency. Facilitate.

  In the vicinity of the display of this apparatus, a touch panel, a digitizer, a distance sensor, and the like are incorporated, and an intuitive operation such as a touch operation or a gesture operation can be performed on an object displayed on the screen.

JP 2006-40271 A

  However, when operating by two operation methods including gesture operation, there exists a subject that gesture operation and other operation methods (for example, touch operation) interfere, and a user's operativity falls. The assumed use cases are described below.

  An operation (for example, selection operation) of an object displayed on the screen is performed. As the operation method, both a gesture operation and another operation method (for example, a touch operation) are used. In the gesture operation, an object is selected based on the distance between the operator's hand and the screen. Specifically, when this distance is short, a nearby object is selected with reference to the operator, and when it is far, a far object is selected with reference to the operator. The touch operation selects an object by directly touching a display object that the user wants to select. When operating with these two operation methods, when an object is selected by a touch operation, if there is an object near the object to be selected, the nearby object is selected by the gesture operation immediately before touching. This is not limited to a touch operation, and is similar to a case where an operation is performed with a mouse, a pen, or the like. This is a problem that occurs when a gesture operation that is a spatial operation and another operation method are used.

  The present invention has been made in consideration of the above-described problems. When an object displayed on the display unit is operated by two operation methods including a gesture operation, interference between the two operations is suppressed, and a user operation is performed. The purpose is to improve sex.

In order to solve this problem, an information processing apparatus according to the present invention is an information processing apparatus capable of recognizing input by a plurality of types of operations including a gesture operation, and is displayed on a position and a display surface of an operator who performs the operation. An object distance acquisition unit that acquires a distance from a display position of a plurality of objects, an operation input distance acquisition unit that acquires a distance between a position of an operator who performs the operation and an operation input position of the operator, of objects, the object distance said acquired by the object distance obtaining means becomes larger than the distance obtained by the operation input distance obtaining means, a gesture operation target setting unit for setting as an object operable by the gesture operation It is characterized by having.

  According to the present invention, when an object displayed on the display surface is operated by a plurality of operation methods including a gesture operation, interference between the plurality of gesture operations and other operation methods can be suppressed, and the operability of the user can be improved. .

FIG. 2 is a hardware configuration diagram of the information processing apparatus 100. FIG. 2 is a functional configuration diagram of the information processing apparatus 100. FIG. 5 is a processing flowchart of the information processing apparatus 100. The detailed process flowchart of S403. FIG. 5 is a schematic diagram of an object selection operation in the first embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, the Example demonstrated below shows an example at the time of implementing this invention concretely, and is not restricted to this.

  FIG. 1 is a schematic diagram of a system 1001 in the present embodiment. FIG. 1A is a view from the top, and FIG. 1B is a view from the side. As shown in FIG. 1, a system 1001 is a system for organizing and discussing ideas using a table-type display 101 by a user. The user selects and operates the objects (102a to 102e) displayed on the display 101 serving as a display unit by a touch operation using a finger or a hand or a gesture operation.

  In this embodiment, an object is projected onto the display 101 using the projector 104 that can project downward. The display 101 has a touch panel incorporated therein. The touch panel uses a resistance film method, a capacitance method, or the like. In this embodiment, an example in which the operation is performed by two operation methods of a gesture operation and a touch operation is shown. In other words, any gesture operation that is a spatial operation and other operations that perform an operation by contact may be used.

  The image sensor 103 is used as a sensor for recognizing a gesture operation on the object. Then, the image captured from the image sensor 103 is analyzed, and the position of the operator and the positions of hands and fingers used for operation input are calculated. Each of these devices is connected to the information processing apparatus 100 via a network or USB, and functions in cooperation.

  In this embodiment, an example is shown in which one display 101 and one image sensor 103 are used. However, the present invention is not limited to this, and a plurality of displays may be used. In addition, instead of a projection display device such as the projector 104, a display 101 having a different display method such as liquid crystal, plasma, organic EL, or inorganic EL may be used. Moreover, the installation method of the display 101 is not limited to the table type, and may be installed on a floor surface, a wall surface, or the like. In addition, the position calculation process may be performed using an infrared sensor or an ultrasonic sensor without being limited to the image sensor 103.

  FIG. 2 is a hardware configuration diagram of the information processing apparatus 100 in the present embodiment. The information processing apparatus 100 includes a CPU 200 (Central Processing Unit), a RAM 201 (Random Access Memory), a ROM 202 (Read Only Memory), and a bus 203. Further, the display I / F 204, the storage I / F 205, the storage 206, and the input I / F 207 are configured. Hereinafter, the hardware configuration of the information processing apparatus 100 will be described in detail.

  A CPU 200 (Central Processing Unit) executes a control program stored in a ROM 202 (Read Only Memory) and / or a storage 206 to control each device.

  A RAM 201 (Random Access Memory) has a work area for the CPU 200, a data save area during error processing, a control program load area, and the like.

  The ROM 202 stores an operating system (hereinafter referred to as OS), processing programs according to the present invention, device drivers, and the like.

  The bus 203 includes an address bus, a data bus, and a control bus, and mediates various communications.

  The display I / F 204 converts a display screen created by the information processing apparatus 100 into a signal that can be displayed by the display 101.

  The storage I / F 205 mutually converts the video data format that can be processed by the information processing apparatus 100 and the video data format stored in the storage 206.

  The input I / F 207 receives image information from the image sensor 103 as an input signal, and converts it into information that can be processed by the information processing apparatus 100.

  The storage 206 is a large-capacity information storage device represented by a hard disk drive device, and records objects to be displayed on the information processing apparatus 100. The storage 206 stores computerized multimedia data. Specific data types are video, still image, audio, document, and the like. These data are preferably provided with metadata such as creation date / time, edit date / time, and group. The display order of data is determined using these metadata. On the other hand, when metadata is not assigned, the display order may be determined by the name of the data file. The storage 206 may be mounted inside the information processing apparatus 100 or may be connected via various interfaces such as Ethernet (registered trademark), USB, and memory card I / F. At that time, a plurality of storages 206 may be connected.

  FIG. 3 is a functional configuration diagram of the information processing apparatus 100 in the present embodiment. Hereinafter, each unit shown in FIG. 3 will be described. Each functional unit can be realized in various forms by a combination of hardware and software.

  The holding unit 301 corresponds to the RAM 201, and reads and holds data to be displayed and related metadata from the storage 206.

  The display control unit 302 generates a display screen from the data held in the holding unit 301 and outputs the display screen to the display 101. In the display, various data read from the storage are displayed on the display 101 as objects.

  The object position acquisition unit 303 acquires position information on the screen of the object displayed on the display 101 from the holding unit 301. The position information is, for example, position coordinates of the center point of the object in the screen. Here, it is not necessary to limit to the center point, and it may be a center of gravity point, a point on the outer shape, or an arbitrary representative point. Further, the coordinates are composed of, for example, two vertical and horizontal axes on the display surface, and an axis in the height direction may be used.

  The operator position acquisition unit 304 performs image processing on the image captured from the image sensor 103 and acquires the position of the operator near the display. In the image processing, for example, the position of a person is detected using a general human body detection algorithm. Alternatively, the position of the face is detected using a general face detection algorithm. Then, the position of the operator is calculated based on the detected face position. These algorithms are stored in the ROM 202 and executed by the CPU 200.

  The operation input position acquisition unit 305 performs image processing on the image captured from the image sensor 103 and acquires the position of the operator's arm, hand, or finger. In the image processing, for example, a position of an arm, a hand, or a finger is detected by using a general object detection algorithm. This algorithm is stored in the ROM 202 and executed by the CPU 200.

  The object distance calculation unit 310 calculates the distance between the object display position and the operator position from the object display position acquired by the object position acquisition unit 303 and the operator position acquired by the operator position acquisition unit 304. Is calculated. In addition, it is preferable to detect only the distance in the plane parallel to the screen of the display. That is, the distance in the height direction with respect to the screen is not considered, and only the distance in the vertical and horizontal directions (parallel) is calculated with respect to the screen. Therefore, the calculation cost can be reduced by limiting the components of the distance to be calculated.

  The operation input distance calculation unit 311 operates from the position of the operation input such as the operator's hand or finger acquired by the operation input position acquisition unit 305 and the position of the operator acquired by the operator position acquisition unit 304. The distance between the position of the operator and the position of the operator is calculated. As with the object distance calculation unit 310, it is preferable to calculate only the distance in the vertical and horizontal directions (parallel planes) with respect to the display screen.

  The gesture operation target setting unit 306 sets an object whose distance calculated by the object distance calculation unit 310 is larger than the distance calculated by the operation input distance calculation unit 311 as the operation target of the gesture operation. That is, an object farther than the position of the operation input such as a hand or a finger with respect to the operator is an operation target of the gesture operation.

  The gesture operation recognition unit 307 performs a selection operation using a gesture on the object selected by the gesture operation target setting unit 306. The method of selecting an object by gesture selects an object based on the distance between the operator's hand and the display screen. Specifically, when selecting a distant object, the operator moves his hand away from the screen, and when selecting a nearby object, the operator moves his hand closer to the screen. Information necessary for the above processing is acquired from the object position acquisition unit 303, the operator position acquisition unit 304, and the operation input position acquisition unit 305. Further, the objects to be selected may be limited by using the direction in which the operator holds his / her hand during the operation. Specifically, the direction is calculated as an angle from the operator position and the operation input position, and an object existing within a range of ± 5 ° of the calculated angle is a target of selection processing.

  The touch operation recognition unit 308 recognizes an operator's touch operation detected by the touch panel, and selects an object based on the recognized operation. Specifically, the operator touches an object displayed on the display 101. The touched place is preferably on the displayed object, but it may be near the object. When the vicinity is touched, the object closest to the touched point is selected.

  The object processing unit 309 performs predetermined processing on the object selected by the touch operation recognition unit 308 and the gesture operation recognition unit 307 and displays the object on the display 101. When the object is displayed without being processed, the object processing unit 309 displays the object selected by the touch operation recognition unit 308 and the gesture operation recognition unit 307 on the display 101 without operating.

  FIG. 4 is a processing flowchart of the information processing apparatus 100 in the present embodiment. When the operator selects an object selection process from an operation menu (not shown) of the information processing apparatus 100, the above process is started. Then, the process is repeated until an operation menu other than the object selection process is selected.

  In step S <b> 401, the touch operation recognition unit 308 determines whether there is an operator's touch operation on the touch panel. If there is a touch operation, the process proceeds to S402, and if there is no touch operation, the process proceeds to S403.

  In S402, the touch operation recognition unit 308 selects an object based on the operator's touch operation.

  In step S <b> 403, the gesture operation target setting unit 306 sets an object to be a gesture operation target. Details will be described with reference to the process flowchart of FIG.

  In S404, the gesture operation recognizing unit 307 selects an object based on the gesture operation from the objects that are the operation target in S403.

  FIG. 5A is a detailed process flowchart of S403 in the present embodiment.

  In step S <b> 501, the operator position acquisition unit 304 acquires the operator's position.

  In step S502, the object position acquisition unit 303 acquires the display position of the object displayed on the display 101.

  In step S503, the object distance calculation unit 310 calculates the distance between the position of the operator and the display position of the object.

  In step S <b> 504, the operation input position acquisition unit 305 acquires the position of the operation input for performing the gesture operation.

  In step S505, the operation input distance calculation unit 311 calculates the distance between the position of the operator and the position of the operation input.

  In S506, the gesture operation target setting unit 306 sets an object whose distance calculated in S503 is larger than the distance calculated in S505 as the operation target of the gesture operation. That is, an object farther than the position of the hand with respect to the position of the operator is set as a gesture operation target.

  As the processing order, S504 and S505 may be performed before S502 and S503. If parallel processing is possible, S502 and S503, and S504 and S505 may be performed in parallel.

  FIG. 6 is a schematic diagram of an object selection operation in the present embodiment. FIGS. 6A, 6B, and 6C are views of the same process as viewed from a plurality of viewpoints. 6A is a view from the top, FIG. 6B is a view from the side, and FIG. 6C associates the object, the hand performing the gesture operation, and the distance from the screen. FIG.

  In FIG. 6A, objects A, B, and C are displayed on a table-type display. Then, the operator selects an arbitrary object from these objects through a gesture operation.

  In FIG. 6B, objects A, B, and C are displayed on the screen in the order from the operator as a reference. The operator is trying to select the object B, and the position of the hand performing the selection processing gesture is between the objects A and B. Therefore, the objects B and C are the operation targets for the gesture operation, and the object A is not the operation target.

  FIG. 6C illustrates a method of selecting the objects B and C that are the operation targets of the gesture operation with the gesture. In the internal processing, the object to be selected is associated with the distance from the screen and the hand performing the gesture. First, the movement range of the hand performing the gesture is set to be within 1 m from the screen. Next, a range less than 0.5 m from the screen is associated with the object B, and 0.5 m or more is associated with the object C. Therefore, the object B is selected when the operator brings his hand close to the screen, and the object C is selected when the operator moves away.

  From the above, it is possible to suppress interference between the touch operation and the gesture operation by using the information processing apparatus of the present invention. Therefore, user operability can be improved.

  In the present embodiment, as in the first embodiment, an embodiment in which the information processing apparatus 100 of the present invention is used will be described. The difference from the first embodiment is that at least one change in the display position of the object, the position of the operator, and the position of the operation input is detected. Then, when a position change is detected, processing after distance calculation is performed.

  FIG. 3B is a functional configuration diagram of the information processing apparatus 100 in the present embodiment. A position change detection unit 312 is added to the configuration of the information processing apparatus 100 according to the first embodiment. Hereinafter, only a different part from FIG. 3 (A) of Example 1 is demonstrated.

  The position change detection unit 312 detects at least one change in the display position of the object, the position of the operator, and the position of the operation input. For example, each position when the previous process is executed is stored in the holding unit 301, and is detected as a position change when the newly acquired position moves by a predetermined threshold or more. This position change detection result is notified to the object distance calculation unit 310 and the operation input distance calculation unit 311.

  The object distance calculation unit 310 and the operation input distance calculation unit 311 calculate a distance when a position change is detected by the position change detection unit 312. The method for calculating the distance is the same as in the first embodiment.

  FIG. 5B is a detailed flowchart of S403 in the present embodiment. S507 is added to the part different from FIG. 5A of the first embodiment.

  If a position change is detected by the position change detection unit 312 in S507, the process proceeds to S501. If not detected, the process is terminated.

  Based on the above, changes in the display position of the object, the position of the operator, and the position of the operation input are detected, and when the position change is detected, the above processing is executed at an appropriate time by performing processing after the distance calculation. can do. Therefore, the calculation cost can be reduced while maintaining smooth operability.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (7)

An information processing apparatus capable of recognizing input by a plurality of types of operations including a gesture operation, and acquiring an distance between a position of an operator who performs the operation and display positions of a plurality of objects displayed on a display surface Distance acquisition means;
Operation input distance acquisition means for acquiring a distance between the position of the operator performing the operation and the operation input position of the operator;
Among the plurality of objects, an object distance obtained by the object distance obtaining means becomes greater than the distance obtained by the operation input distance obtaining means, the gesture operation target setting to be set as an object operable by the gesture operation An information processing apparatus comprising means. Furthermore, it has a position change detecting means for detecting a change in at least one of the display position of the object, the position of the operator, or the operation input position.
The information processing apparatus according to claim 1, wherein when the change in position is detected by the position change detection unit, the object distance acquisition unit and the operation input distance acquisition unit acquire the distance.   The display device according to claim 1, wherein the object distance acquisition unit and the operation input distance acquisition unit acquire a distance in a plane parallel to the display surface.   The touch operation recognition means for recognizing a touch operation on an object that is not set as an object operable by the gesture operation by the gesture operation target setting means among the plurality of objects. 4. The display device according to any one of 3. A method for controlling an information processing apparatus capable of recognizing input by a plurality of types of operations including gesture operations,
An object distance acquisition means for acquiring a distance between the position of the operator who performs the operation and the display positions of the plurality of objects displayed on the display surface;
An operation input distance obtaining unit obtains a distance between the position of the operator who performs the operation and the operation input position of the operator,
Gesture operation target setting means, wherein among the plurality of objects, an object distance obtained by the object distance obtaining step becomes larger than the distance obtained by the operation input distance obtaining step, the object can be operated in the gesture operation A control method for an information processing apparatus, comprising: a gesture operation target setting step that is set as   A program that causes a computer to function as each unit of the information processing apparatus according to claim 1 by being read and executed by the computer.   A computer-readable storage medium storing the program according to claim 6.

Images