JP5627314B2 - Information processing device - Google Patents

Description

The present invention relates to a technique for improving operability for a display object.

  Attention has been focused on systems that support organizing ideas and activating discussion by displaying digitized documents on a large table display or wall display and making the displayed documents interactively operable. In such a system, a touch panel, a digitizer, and the like are incorporated in the display, and an intuitive object operation such as a touch operation on a document (object) displayed on the screen is possible. In general, an operation using a touch panel is easily used by a user who is unfamiliar with information equipment, and is used in many information equipment such as mobile terminals and copiers.

  These touch panel type information processing devices are required to allow anyone to easily select a target object by touch operation, but if the displayed object is small, there is a high possibility of selecting the wrong object. . Similarly, when the interval between adjacent objects is narrow, the adjacent object is erroneously selected by touch.

  In order to solve such a problem, Patent Literature 1 detects that the fingertip of the operator has approached, obtains the distance between the fingertip and the display screen, and the display object that the user wants to select as the distance decreases. A process of displaying a large image so that it is easy to select is performed.

JP 2009-259110 A

  However, when an object displayed on the table is selected by touching, in Patent Document 1, if the display object is displayed in an overlapping manner, it is troublesome to select the object by a touch operation, which reduces the operability. End up.

  In addition, when an object displayed on the table is selected by touching, in Patent Document 1, it is not possible to select if the object to be selected is displayed at a distance that the operator cannot reach. For example, in FIG. 1A, the operator A cannot select by directly touching the displayed objects 103d and 103e because the hand is out of reach.

  The present invention has been made in consideration of the above problems, and provides an information processing apparatus capable of easily selecting an object even when the objects are overlapped or displayed in a place out of reach. For the purpose.

In order to solve the above problems, an information processing apparatus of the present invention provides:
An information processing apparatus that selects an object from a plurality of objects displayed by a display unit, the acquisition unit acquiring an operation direction by an operation object , and the acquisition unit acquired by the acquisition unit among the plurality of objects The order in which each object displayed in the operation direction is limited to selection candidates, and the distance from the operation object to the surface on which the plurality of objects are displayed is determined based on the display position of each object that is the selection candidate And an object selecting unit that selects an object corresponding to the plurality of objects from the plurality of objects.

  According to the present invention, it is possible to easily select an object that is difficult to select because it is displayed overlappingly, or an object that is displayed far away and cannot be selected.

The system schematic diagram of information processing apparatus. The hardware block diagram of information processing apparatus. The functional block diagram of information processing apparatus. Explanatory drawing of an object selection process. The main flowchart which shows the process of information processing apparatus. The flowchart of an initial screen display process. The flowchart of an object selection process. The figure which shows the specific example of 2nd Embodiment. The figure which shows the remote operation distance calculation method. Explanatory drawing of the object selection order in 2nd Embodiment. FIG. 9 is an overview diagram of an information processing apparatus according to a third embodiment. The figure which shows the table of matching. The figure which shows the positional infomation acquisition method of an object and an object.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, embodiment described below shows an example at the time of implementing this invention concretely, and is not restricted to this.

[First Embodiment]
FIG. 1A is a system outline diagram in this embodiment. This is a system for interactively organizing and discussing ideas using a table-type display 101 and a wall-type display 102. Participants A and B select and operate the objects (103a to 103e) displayed on the displays (101 and 102), which are display devices, by touch operation or gesture operation using fingers or hands.

  FIG. 1B illustrates an example of a system 1001 that projects an object using the projector 104 on both the table-type display 101 and the wall-type display 102 using the projector 104 that can project in both the downward and lateral directions. . Further, the distance sensor 105 is used as a sensor for recognizing a gesture operation on the object. Each of these devices is connected to the information processing apparatus 100 via a network or USB, and functions in cooperation. Although FIG. 1B illustrates an example in which two display devices and one distance sensor are used, the present invention is not limited to this. There may be only one display device, not a projection display device such as the projector 104, but a large display incorporating a touch panel as shown in FIG. Good. Also, a plurality of proximity sensors and ultrasonic sensors may be used as sensors that can calculate distance information. It is also possible to calculate distance information from the analysis result of the image.

  FIG. 2 is a hardware configuration diagram of the information processing apparatus 100 in the present embodiment. The information processing apparatus 100 includes a CPU 200 (Central Processing Unit), a RAM (Random Access Memory) 201, a ROM (Read Only Memory) 202, and a bus 203. Further, the display I / F 204, the storage I / F 205, the storage 206, and the input I / F 207 are included. However, the present invention is not limited to these interfaces. An information processing apparatus may be realized using an interface such as a touch panel I / F or a memory card I / F.

  Hereinafter, details of the hardware configuration of the information processing apparatus 100 will be described.

  The CPU 200 controls the entire information processing apparatus 100 using the RAM 201, the ROM 202, and the bus 203. Although not shown here, an operating system (hereinafter referred to as OS), each processing program according to the present invention, a device driver, and the like are stored in the ROM 202, temporarily stored in the RAM 201, and appropriately executed by the CPU 200. Here, the OS, each processing program, and the like may be stored in the storage 206, in which case they are appropriately read into the RAM 201 when the power is turned on, and are activated by the CPU 200.

  The display I / F 204 converts a display screen created in the information processing apparatus 100 into a signal that can be processed by the display 101. The storage I / F 205 mutually converts a video data format that can be used inside the information processing apparatus 100 and a video data format to be stored in the storage 206. The input I / F 207 receives distance information from the distance sensor 105 as an input signal, and converts the information into information that can be processed by the information processing apparatus 100.

  The storage 206 is a large-capacity information storage device typified by a hard disk drive device, and a large number of objects to be displayed on the information processing apparatus 100 are recorded in the storage 206. The storage 206 itself may be mounted inside the information processing apparatus 100 or may be connected via various interfaces such as Ethernet (registered trademark), USB, and memory card I / F. At that time, a plurality of storages 206 may be connected. In addition, unless otherwise indicated below, what was demonstrated using the other figure attaches | subjects the same code | symbol, and abbreviate | omits the description.

  FIG. 3A is a functional block diagram of the information processing apparatus 100 in the present embodiment. The information processing apparatus 100 includes a distance acquisition unit 300, a holding unit 301, an object acquisition unit 302, a display control unit 303, and a selection unit 304. Each of these functional blocks can be realized in various forms by a combination of hardware and software.

  In the present embodiment, data (objects) stored in the storage 206 are mainly multimedia data, such as video, still images, audio data, and various electronic documents. These data are hereinafter referred to as objects. These objects are given meta information necessary for determining the order of the objects, such as creation date / time information, editing date / time information, and group information. However, the present embodiment is not limited to such meta information. For example, when there is no meta information for determining the order of the objects, the order of the objects may be determined by sorting the objects by the name of each object after reading the objects.

  When the information processing apparatus 100 is activated, the object acquisition unit 302 reads an object to be displayed and meta information from the storage 206. The read object and meta information are held in the RAM 201. At this time, if the read data is too large for the RAM 201, the read data may be subdivided and processed sequentially. Alternatively, it may be saved in the storage 206 as swap data that can be read immediately.

  The holding unit 301 determines the order of the objects from the meta information of the objects held in the RAM 201 and holds them. Here, the order of the objects is the order of the display layers to which the objects belong on the display screen.

  The display control unit 303 generates a display screen from the objects held in the RAM 201 and outputs the display screen to the display 101. An example of a display screen created by the display control unit 303 is shown by 400 in FIG. In FIG. 4, 401 a, 401 b, and 401 c indicate objects displayed and output on the display 101.

  The information processing apparatus 100 is operated by performing a gesture operation (remote operation instruction) using an object such as a hand or a finger. When the distance acquisition unit 300 detects an object while the object is displayed on the display device (display 101), the object selection process is started. The distance acquisition unit 300 detects an object and calculates a remote operation distance from the object to the display screen. Then, the calculated remote operation distance is input to the selection unit 304. Below, it demonstrates using the hand which is an example of an object.

  The selection unit 304 selects an object as an operation target from the order of the objects held by the holding unit 301 and the remote operation distance acquired by the distance acquisition unit 300.

  The outline of the object selection process of this embodiment will be described with reference to FIG.

  Reference numeral 400 in FIG. 4 shows an example of a display screen on which objects are displayed in this embodiment, and a plurality of objects 401a, 401b, 401c are displayed on one screen.

  In general, in order to individually select a plurality of objects displayed on a display, it is necessary to manage the objects with one display layer. A state in which the object is managed for each display layer is shown in (401x, 401y, 401z) in FIG.

  Also in this embodiment, each object is managed by the display layer. The holding unit 301 determines and holds the order of the objects based on the order of the layers to which the objects belong.

  The display control unit 303 manages objects in each display layer, generates one display screen by superimposing the display layers in the order of display layers, and outputs the display screen to the display. The display screen generated in this way is the display screen 400 of FIG.

  While the display screen 400 is displayed, gesture input is performed using the object 403. The gesture input in the present embodiment is an operation of moving the hand in the vertical direction with respect to the display screen 400. The gesture input is detected by the distance acquisition unit 300, and the remote operation distance is calculated. In the present embodiment, a gesture operation is detected by a distance sensor 105 attached above the display screen 400 and the object 403, and a remote operation distance is calculated.

  A specific method for calculating the remote operation distance in the present embodiment is shown in FIG. Here, a description will be given using a hand which is an example of an object. First, the distance sensor 105 obtains the vertical distance Dh1 from the distance sensor 105 to the display screen 101 of the display 101. Further, the back of the hand performing the gesture operation is recognized, and the distance sensor 105 acquires the vertical distance Dh2 from the distance sensor 105 to the back of the hand. The remote operation distance x calculated in the present embodiment is the vertical distance between the display screen and the hand, and can be calculated by x = Dh1-Dh2. In the present embodiment, the remote operation distance is calculated using the distance sensor, but this is not a limitation. A proximity sensor or an ultrasonic sensor may be installed on the display screen 400, and the remote operation distance may be calculated from the sensor information.

  Next, the selection unit 304 selects an object from the calculated remote operation distance according to the order of the objects held by the holding unit 301.

  In the present embodiment, the order of objects is based on the order of display layers. Therefore, as shown in FIG. 4, the order of the objects is the display layer order Object1 (401a), Object2 (401b), and Object3 (401c). At this time, if the remote operation distance x between the object 403 on which the gesture operation is performed and the display screen 400 is 0 <x <= a, Object3 is selected as an operation target, and if a <x <= b, Object2 is selected. Select as an operation target. Similarly, when the remote operation distance x is b <x <= c, Object3 is selected as an operation target. When x = 0 and c <x, the object is not selected for gesture operation and no object is selected.

  Since the object order is selected by associating the object order (display layer order in this embodiment) with the distance, the object can be easily selected even when the display objects are displayed in an overlapping manner.

  In addition, as in the present embodiment, it is possible to match an object having a higher display layer with a long distance (mapping) to match the sense of the user who operates, and to select the object more intuitively.

  Further, as indicated by 404a, 404b, and 404c in FIG. 4, the display control unit 303 highlights the selected object on the display, and an indicator that indicates the correspondence between the distance and the selected object is indicated by 405a, 405b, and 405c. Display feedback. By performing such feedback display, the user can know in advance how to perform gesture input next to select an object to be selected, thereby realizing a user interface (hereinafter referred to as UI) that is easier to use.

  In the present embodiment, the range in which the hand is moved up and down (operable distance) is set to 60 cm in consideration of ease of user operation, and the range is divided into three to be boundaries a, b, and c. However, this is not the case. The range in which the user can operate the hand up and down varies depending on the state of the user operating the object (standing or sitting). Therefore, the range in which the hand is moved up and down may be dynamically changed by recognizing the state of the operator. Note that the operable distance is stored in advance in the ROM 202 of the information processing apparatus 100.

  Further, the range in which the hand is moved up and down and the number of divisions (number of boundaries) vary depending on the total number of objects to be operated. For example, when there are six operation target objects, the distance may be associated with the object by dividing 60 cm into six. Conversely, the resolution of the up / down operation of the hand may be fixed to 2 cm, and the range in which the hand is moved up / down may be determined according to the number of operation target objects. As described above, the minimum value Min and the maximum value Max of the range in which the hand is moved up and down can be determined by various methods.

  FIG. 5A is a flowchart showing the flow of processing of the information processing apparatus 100 in the present embodiment. Details of the data display process and the object selection process in this embodiment will be described with reference to this flowchart.

  When the information processing apparatus 100 starts the data display process, the object acquisition unit 302 first acquires meta information used to determine the objects to be displayed and the order, and stores them in the RAM 201. Next, the order of the objects is held from the meta information acquired by the holding unit 301, and this information is also held in the RAM 201 (S10). In the case of the first embodiment, the order of objects is determined based on the order of display layers.

  When the order of the objects is determined, an initial screen display process (S11) is executed.

  FIG. 6 is a process flow diagram of the initial screen display process (S11). In the initial screen display process, a display screen is created from the object display layer order and the objects held in the RAM 201. The display control unit 303 creates a display screen by overlapping the display layers of the objects in order from the lowest (S20). Then, the created screen is displayed and output on the display 101 (S21). At this time, an example of an initial screen displayed on the display 101 is shown as 400 in FIG. With the above processing, the initial screen display processing (S11) is completed, and the result is returned to the caller.

  When the initial screen display process (S10) is completed, the distance sensor 105 detects a remote operation (S12). When the remote operation is not detected (S12, N), nothing is performed and the process proceeds to S12 again to enter a remote operation waiting loop. When a remote operation is detected (S12, Y), the remote operation distance (the vertical distance from the display screen to an object such as a hand) is acquired from the information of the distance sensor 105 and stored in the RAM 201. When the acquisition of the remote operation distance is completed, the object selection process is started (S14).

  FIG. 7A is a process flow diagram of the object selection process (S14) of the first embodiment. When the object selection process is started, the selection unit 304 reads the display layer order of the objects from the RAM 201. Then, the distance between the object and the distance is associated based on the display layer order (S30). For the association, a table showing the relationship between the object and the distance (operational distance) is created based on the order of the display layers. As an example of the table, a table as shown in FIG.

  Further, the selection unit 304 reads the remote operation distance from the RAM 201, and selects an object as an operation target based on the association result (table). Further, the display control unit 303 creates a display screen that highlights the selected object (S31), and outputs it to the screen (S32), thereby providing feedback to the user. Examples of a screen that displays the selected object as feedback are shown in 404a (select Object 3), 404b (select Object 2), and 404c (select Object 1) in FIG. With the above processing, the object selection processing (S14) is completed, and the result is returned to the calling source processing.

  When the object selection process (S14) ends, it is determined whether or not to end the data display (S15). If the data display is not finished (S15, N), the process proceeds to S12 again. When the data display is to be ended (S15, Y), the information processing apparatus 100 ends the data display process.

  As described above, as described in the present embodiment, an arbitrary object can be selected by a simple gesture operation by associating an object with a distance (an operable distance) based on the order of display layers. Further, even when objects are displayed so as to overlap each other, it is possible to easily select and operate the objects.

  In the present embodiment, an object can be selected more intuitively (to match the user's feeling of operation) by associating an object whose display layer order is higher with a long distance.

  In the present embodiment, the order of the objects is determined based on the display layer to which the objects belong on the display screen. For example, using the information on the date and time when the object was created, The order of the objects may be determined. In addition, the order of objects can be determined according to the use of the application. For example, the objects having related meta information are grouped and the order is determined by sorting within each group.

[Second Embodiment]
Next, a second embodiment of the present invention will be described in detail with reference to the drawings. Here, a schematic diagram of the system, a hardware configuration diagram, and an initial screen display processing flowchart are the same as those in the first embodiment described with reference to FIGS. 1, 2, and 6. Also in the second embodiment, unless otherwise specified, the same reference numerals are given to those described with reference to other drawings, and the description thereof is omitted.

  The difference from the first embodiment is that a position information acquisition unit 305 that acquires an object performing gesture operation and position information of the object is added to the functional block.

  FIG. 3B shows a functional block diagram according to the second embodiment. Other than the position information acquisition unit 305 is the same as that of the first embodiment described with reference to FIG.

  The position information acquisition unit 305 acquires the positions of the object and the display object that perform the gesture operation from the display position information of the object that can be acquired from the display control unit 303 and the position information of the object that can be acquired from the distance sensor 105. In the second embodiment, as the position information, the remote operation direction of the object to be remotely operated and the distance between the object and the object displayed in the direction are acquired. Further, only the objects located in the remote operation direction of the acquired object are selected, and the holding unit 301 determines and holds the order of the objects to be selected.

  FIG. 5B is a flowchart illustrating a processing flow of the information processing apparatus 100 according to the second embodiment. The processing is the same as that of the first embodiment described with reference to FIG. 5A except for the processing S16 for acquiring the position information of the object and the object and the object selection processing S17. In S16, the remote operation direction and the distance to the object in that direction are acquired, and the information is stored in the RAM 201.

  FIG. 13 shows a method for acquiring object and object position information in the present embodiment. Here, description will be made using a hand 403 which is an example of an object. First, the remote operation distance x is the same as that of the first embodiment described with reference to FIG. In this embodiment, in addition to the remote operation distance x, the center distance of the hand 403 and the horizontal distance y to the center position of the object 103a are also calculated. In order to calculate the horizontal distance y, the projector 104 and the distance sensor 105 must be accurately calibrated as a precondition. When the calibration is correctly performed, the center of gravity position p1 of the hand on the display screen is obtained from the information of the distance sensor. Further, the center of gravity position p2 of the object 103a is obtained from the display information of the object 103a. From this information, the horizontal distance between the hand and the object can be calculated by y = | p1-p2 |.

  FIG. 7B shows a process flow of the object selection process (S17) in the second embodiment.

  When the object selection process is started, the selection unit 304 reads the position information acquired in S16. Then, the object is filtered so that only the object in the remote operation direction is selected (S33). Next, a distant object is associated with a long distance (S34).

  In the present embodiment, a distant object (an object with a large horizontal distance y) is associated with a long distance, and if the objects overlap, the display layer is positioned at the upper level, as in the first embodiment. Match things to long distances. A specific example is shown in FIG. Assume that the object B and the object C in FIG. In addition to the position information of the hand and the object, the order of the objects is determined in consideration of the overlapping order of the objects. In the case of the present embodiment, in order to associate a long distance with an object that is far away and has a higher overlapping order, in the example of FIG. 10, the object selection order is E, D, B, C, A. An example of the table is shown in FIG.

  Furthermore, the selection unit 304 selects one object as an operation target based on the association result (table). Further, the display control unit 303 creates a display screen that highlights the selected object (S31), and outputs it to the screen (S32).

  The object selection process (S17) is completed by the above process, and the result is returned to the caller process. The processing after the end of the object selection processing (S17) is the same as that in the first embodiment described with reference to FIG.

  Next, an example in which an object within a reachable range is selected by a touch operation, and an object within a reachable range is selected by a gesture operation.

  FIG. 8 shows an example of an object selection operation when a large number of objects are displayed. This example considers the case where ideas such as brainstorming are performed using the information processing apparatus 100 of the present embodiment. In this case, the idea is described in the object. In this example, a plurality of objects are displayed without overlapping.

  Assuming that the objects to be displayed are 10 cm in length and width, at least 300 objects can be displayed on the display screen 500 of about 100 inches (200 cm × 150 cm). Of these, objects within the reachable range 502 (assumed to be within 60 cm) are excluded from the remote operation target objects as those selected by the touch operation. As shown in FIG. 8, when the angle resolution is 10 degrees, the position information acquisition unit 305 can acquire directions in 18 steps. Further, if the range of gesture operation in the vertical direction with respect to the display screen is 60 cm and the resolution is 5 cm, the selection unit 304 can associate the distance and the object to be operated in 12 steps.

  In the case of the above assumption, the direction of the area 503 in FIG. 8 can display the most objects, and the number thereof is eleven. Here, as described above, since it is possible to associate an object with a distance in 12 steps, it is possible to select 11 objects displayed in an unreachable area by a simple gesture operation of up and down movement of the hand. It becomes.

  As described above, as described in the present embodiment, it is possible to easily select an object by acquiring position information of the object and associating the distance with the object based on the distance to the object in an arbitrary direction. it can. That is, even when a large number of objects are displayed, the objects can be selected with a simple gesture operation.

[Third Embodiment]
Next, a third embodiment of the present invention will be described in detail with reference to the drawings. Here, the hardware configuration diagram and the processing flow diagram are the same as those in the first embodiment described with reference to FIGS. 2, 5A, 6, and 7A. Also in the third embodiment, unless otherwise specified, the same reference numerals are given to those described with reference to other drawings, and the description thereof is omitted.

  The difference from the first embodiment is that the information processing apparatus is an integrated mobile device. FIG. 11 shows an overview of the information processing apparatus 106 according to the third embodiment.

  The information processing apparatus 106 according to the third embodiment includes a small display 107 and one or a plurality of proximity sensors 108 capable of calculating a remote operation distance. Reference numerals 109a, 109b, and 109c are objects displayed on the small display 107.

  Similarly to the case shown in the first embodiment, the small information processing apparatus 106 shown in the third embodiment can select an arbitrary object by a simple gesture operation. By using this, an operation for easily selecting an object that is difficult to select because it is displayed on a small display can be realized even on a mobile device.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

Claims (10)

An information processing apparatus that selects an object from a plurality of objects displayed by a display unit,
Acquisition means for acquiring the operation direction by the operation object;
Of the plurality of objects, each object displayed in the operation direction acquired by the acquisition unit is limited to a selection candidate, and a distance from the operation object to a surface on which the plurality of objects are displayed is the selection An information processing apparatus comprising: an object selection unit that selects an object corresponding to an order determined based on a display position of each object that is a candidate from the plurality of objects. Further comprising distance acquisition means for acquiring a distance from the operation object to a surface on which the plurality of objects are displayed;
The object selection means associates each object with a distance operable by the operation object based on an order determined based on a display position of each object that is the selection candidate, and the association and the distance The information processing apparatus according to claim 1, wherein an acquisition unit selects an object based on the acquired distance. The order of the objects is the order of the display layers to which the objects belong on the display screen,
The information according to claim 2, wherein the object selection unit associates an object whose display layer is higher in order with a long distance, and selects the object based on the association and the acquired distance. Processing equipment. Further, the object selection means associates an object whose display position is far from the operation object with a long distance, and selects the object based on the distance acquired by the association and the distance acquisition means. The information processing apparatus according to claim 2 or 3. Further, the object selecting means excludes the object displayed in the touchable range of the operation object from the selection candidates on the surface on which the plurality of objects are displayed among the objects that are the selection candidates, The information processing apparatus according to claim 2, wherein the association is performed. 6. The object selecting unit according to claim 5, wherein the object selecting unit selects the object when it is detected that an object displayed in a touchable range of the operation object is touched. Information processing device.   The display unit according to any one of claims 1 to 6, further comprising a display control unit that causes the display unit to display an indicator that highlights an object selected by the object selection unit and indicates correspondence between the selected object and a distance. The information processing apparatus according to any one of claims. By executing read into the computer, the computer program for causing to function as each unit of the information processing apparatus according to any one of claims 1 to 6. A computer-readable storage medium storing the computer program according to claim 8 . An information processing apparatus control method for selecting an object from a plurality of objects displayed by a display unit,
An acquisition step of acquiring the operation direction by the operation object by the acquisition means;
By the object selection means, each object displayed in the operation direction acquired in the acquisition step among the plurality of objects is limited to a selection candidate, and from the operation object to the surface on which the plurality of objects are displayed. A method for controlling an information processing apparatus, comprising: an object selection step of selecting an object corresponding to an order determined based on a display position of each object that is a selection candidate from the plurality of objects.

Images