JP2012032945A - Information processor and control method of the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily group a display object into a group at a desired group level by simple gesture operation.SOLUTION: An object selection unit 300 selects an object from among plural objects displayed on a display 101. A distance acquisition unit 305 acquires a remote control distance from a body to a screen of the display. A holding unit 309 holds association between the remote control distance and a group level. The object is grouped and displayed on the basis of the held association, the remote control distance acquired by the distance acquisition unit 305 and the object selected by the object selection unit 300.

Description

  The present invention relates to a technique for improving operability for a display object.

  Attention has been focused on systems that support organizing ideas and activating discussion by displaying digitized documents on a large table display or wall display and making the displayed documents interactively operable. In such a system, a touch panel, a digitizer, and the like are incorporated in the display, and an intuitive object operation such as a touch operation on a document (hereinafter referred to as an object) displayed on the screen is possible. In general, an operation using a touch panel is easily used by a user who is unfamiliar with information equipment, and is used in many information equipment such as mobile terminals and copiers.

  In these touch panel type information processing apparatuses, anyone can easily select a target object by a touch operation, but the operability decreases in an operation involving operations on a plurality of objects such as grouping of objects (hereinafter, grouping).

  In general, when performing grouping by touch operation, it is possible to select and group objects one by one, or draw a figure such as a circle using handwritten gestures to enclose the objects, and group the enclosed objects. It is done. However, selecting and grouping objects one by one is cumbersome. In addition, if many objects are displayed and an attempt is made to perform grouping by surrounding the objects, there is a high possibility that unrelated objects will be grouped.

  There is another means for automatically grouping based on information added to an object. The grouping in this case is generally cumbersome because a display switching operation is required every time grouping is performed because the entire screen is switched to a grouping display in accordance with an action such as pressing a button. Further, since the entire screen is switched to the grouping display, it is not possible to perform operations such as grouping only the object of interest.

  In order to solve such a problem, Patent Literature 1 performs grouping display processing for only an object of interest by pressing a button displayed at an arbitrary position on the display screen.

JP 2008-299818 A

  However, in Patent Document 1, it is necessary to display and press a button for display switching. In addition, it is difficult to group and display the represented object at a group level selected from among several group levels. If grouping is to be performed at a plurality of group levels using the technique of Patent Document 1, a cumbersome operation such as displaying a plurality of buttons (use of a useless display area) or pressing a button a plurality of times is required.

  The present invention has been made in consideration of the above-described problems, and an object of the present invention is to easily group display objects into groups of a desired group level by a simple gesture operation.

  In order to solve the above-described problems, an information processing apparatus according to the present invention includes a selection unit that selects an object from a plurality of objects displayed on a display unit, and a remote operation distance from the object to the display screen of the display unit. Based on the acquired distance acquisition means, the holding means for holding the correspondence between the remote operation distance and the group level, and the obtained remote operation distance, the correspondence and the selected object, the objects are grouped. It has a display control means to display.

  According to the present invention, display objects can be easily grouped into a desired group level group by a simple gesture operation.

The system schematic diagram of information processing apparatus. The hardware block diagram of information processing apparatus. The functional block diagram of information processing apparatus. Explanatory drawing of a grouping process. The main flowchart which shows the process of information processing apparatus. The flowchart of an initial screen display process. The flowchart of a group level selection process. The figure which shows the specific example of the object displayed on a screen (UI screen example). The figure which shows the remote operation distance calculation method. The figure which shows the table of matching.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In addition, embodiment described below shows an example at the time of implementing this invention concretely, and is not restricted to this.

[First Embodiment]
FIG. 1A is a system outline diagram in this embodiment. This is a system for interactively organizing and discussing ideas using a table type display (hereinafter referred to as display 101) and a wall surface type display 102. Participants A and B can select and operate the objects (103a to 103e) displayed on the displays (101 and 102), which are display units, by touch operation or gesture operation using fingers or hands.

  FIG. 1B illustrates an example of a system 1001 that projects an object on both the display 101 and the wall surface display 102 using a projector 104 that can project in both a downward direction and a lateral direction. Further, the distance sensor 105 is used as a sensor for recognizing a gesture operation on the object. Each of these devices is connected to the information processing apparatus 100 via a network or USB, and functions in cooperation. Although FIG. 1B illustrates an example in which two display units and one distance sensor are used, the present invention is not limited thereto. There may be only one display unit, not a projection display unit such as the projector 104, but a large display incorporating a touch panel as shown in FIG. Good. Also, a plurality of proximity sensors and ultrasonic sensors may be used as sensors that can calculate distance information. It is also possible to calculate distance information from the analysis result of the image.

  FIG. 2 is a hardware configuration diagram of the information processing apparatus 100 in the present embodiment. The information processing apparatus 100 includes a CPU (Central Processing Unit) 200, a RAM (Random Access Memory) 201, a ROM (Read Only Memory) 202, and a bus 203. Further, the display I / F 204, the storage I / F 205, the storage 206, and the input I / F 207 are included. However, the present invention is not limited to these interfaces. An information processing apparatus may be realized using an interface such as a touch panel I / F or a memory card I / F.

  Hereinafter, details of the hardware configuration of the information processing apparatus 100 will be described.

  The CPU 200 controls the entire information processing apparatus 100 using the RAM 201, the ROM 202, and the bus 203. Although not shown here, an operating system (hereinafter referred to as OS), each processing program according to the present invention, a device driver, and the like are stored in the ROM 202, temporarily stored in the RAM 201, and appropriately executed by the CPU 200. Here, the OS, each processing program, and the like may be stored in the storage 206, in which case they are appropriately read into the RAM 201 when the power is turned on, and are activated by the CPU 200.

  The display I / F 204 converts a display screen created in the information processing apparatus 100 into a signal that can be processed by the display 101. The storage I / F 205 mutually converts a data format that can be used inside the information processing apparatus 100 and a data format that is stored in the storage 206. The input I / F 207 receives distance information from the distance sensor 105 as an input signal, and converts the information into information that can be processed by the information processing apparatus 100.

  The storage 206 is a large-capacity information storage device represented by a hard disk drive device, and the storage 206 stores a large amount of object data to be displayed on the information processing apparatus 100. The storage 206 itself may be mounted in the information processing apparatus 100, or may be connected via various interfaces such as a USB and a memory card I / F. At that time, a plurality of storages 206 may be connected. In addition, unless otherwise indicated below, what was demonstrated using the other figure attaches | subjects the same code | symbol, and abbreviate | omits the description.

  FIG. 3A is a functional block diagram of the information processing apparatus 100 in the present embodiment. The information processing apparatus 100 includes an object selection unit 300, a group information generation unit 301, an object acquisition unit 302, a display control unit 303, a group selection unit 304, a distance acquisition unit 305, a gesture recognition unit 308, and a holding unit 309. Each of these functional blocks can be realized in various forms by a combination of hardware and software.

  In the present embodiment, data (objects) stored in the storage 206 are mainly multimedia data, such as video, still images, audio data, and various electronic documents. These data are hereinafter referred to as objects. It is assumed that additional information necessary for generating group information such as creation date / time information, edit date / time information, location information, and person information is given to these objects. However, this embodiment is not limited to these additional information. For example, when there is no additional information for generating group information, group information may be generated based on the name (alphabet) of each object after reading the object. Further, group information may be generated based on position information where the object is displayed.

  When the information processing apparatus 100 is activated, the object acquisition unit 302 reads an object to be displayed and additional information from the storage 206. The read object and additional information are held in the RAM 201. At this time, if the read data is too large for the RAM 201, the read data may be subdivided and processed sequentially. Alternatively, it may be saved in the storage 206 as swap data that can be read immediately.

  The group information generation unit 301 generates and holds a plurality of group level group information for each object from the additional information of the object held in the RAM 201. In this embodiment, group information is generated from date information added to an object, and there are three group levels: “day”, “month”, and “year”. The group information is generated for each object, and information indicating which group belongs to each group level is generated. For example, in the case of an object having date information of “July 30, 2009”, “July 30, 2009” in the day group, “July 2009” in the month group, “2009” in the year group Is generated.

  The display control unit 303 generates a display screen from the objects held in the RAM 201 and outputs the display screen to the display 101. An example of a display screen created by the display control unit 303 is shown by 400 in FIG. In FIG. 4, 401 a, 401 b, and 401 c indicate objects displayed and output on the display 101.

  The information processing apparatus 100 is operated by performing a gesture operation (remote operation instruction) using an object such as a hand or a finger. Below, it demonstrates using the hand which is an example of an object.

  When the distance acquisition unit 305 detects an object while the object is displayed on the display unit (display 101), the object selection unit 300 starts selecting the object. The object selection unit 300 selects an object based on the detected position of the object and the display position information of the object. The distance acquisition unit 305 detects an object and calculates a remote operation distance from the object to the display screen. Then, the calculated remote operation distance is input to the group selection unit 304. The holding unit 309 holds a table indicating the association between the group level and the operation distance.

  The group selection unit 304 selects a group level based on the group information generated by the group information generation unit 301, the remote operation distance acquired by the distance acquisition unit 305, and the table of the holding unit 309. The gesture recognition unit 308 uses a distance sensor to recognize whether the action performed by the hand that is the object is a predetermined gesture. In this embodiment, selection, grouping, group selection confirmation, and data display end are executed by recognizing a gesture.

  FIG. 4A is an explanatory diagram of grouping processing in the present embodiment. FIG. 5 is a flowchart showing data display processing of the information processing apparatus 100 in the present embodiment. Details of the data display processing and group level selection processing of this embodiment will be described with reference to FIG.

  When the information processing apparatus 100 starts the data display process, first, the object acquisition unit 302 acquires additional information used to generate the object to be displayed and group information from the storage 206, and stores the additional information in the RAM 201 (S10). Next, the group information generation unit 301 generates group information from the acquired additional information, and this group information is also stored in the RAM 201 (S11).

  In this embodiment, the group information is generated from the additional information added to the object acquired by the object acquisition unit 302. However, the present invention is not limited to this. For example, group information may be generated using additional information of all objects stored in the storage 206.

  When the group generation process is completed, an initial screen display process (S12) is executed.

  FIG. 6 is a process flow diagram of the initial screen display process (S12). In the initial screen display process, a display screen is created from the object and group information held in the RAM 201. The display control unit 303 creates a display screen by randomly arranging objects (S20), and displays and outputs the created screen on the display 101 (S21). At this time, an example of the initial screen displayed on the display 101 is shown as 500 in FIG. A total of 12 objects including the object 501 are displayed at random.

  The initial screen display process (S12) is completed by the above process, and the result is returned to the caller.

  While the screen example 500 in FIG. 8A is displayed, gesture input is performed using an object. Here, a description will be given using a hand (401 in FIG. 4) which is an example of an object. The gesture input in the present embodiment is to move the hand 401 on the display screen 400. An object selection gesture can be input by moving the hand 401 horizontally with respect to the display screen 400, and a grouping gesture can be input by moving the hand 401 vertically with respect to the display screen 400. The gesture input is detected and recognized by the distance acquisition unit 305, and the relative position and the remote operation distance (height) of the hand 401 on the display screen are calculated. In the present embodiment, gesture input is detected and recognized by the gesture recognition unit 308 using the distance sensor 105 attached above the display screen 400 and the hand 401. In the present embodiment, the gesture recognition unit 308 performs gesture recognition by analyzing the distance image every 20 frames.

  When the initial screen display process (S12) ends, the gesture recognition unit 308 recognizes the object selection gesture (S13). If the selection gesture has not been recognized (N in S13), nothing is performed and the process proceeds to S13 again to enter a loop for waiting for the selection gesture. When the selection gesture is recognized (S13, Y), the relative position of the hand 401 with respect to the display screen 400 is calculated from the information of the distance sensor 105, and the object displayed at the same position as the relative position of the hand 401 is selected. (S14). The selected object is indicated by reference numeral 402 in FIGS. The selected object is highlighted so that the user can recognize that it has been selected. The relative position of the hand 401 can be calculated if the projector 104 displaying the screen and the distance sensor 105 are correctly calibrated.

  When the object selection (S14) is completed, the distance acquisition unit 305 acquires the remote operation distance z1 (vertical distance between the display screen 400 and the hand 401) and the time t1 when the object is selected, and stores it in the RAM 201. (S15). When acquisition of the remote operation distance and time is completed, group level selection processing is started (S16).

  Here, a specific method for calculating the remote operation distance in the present embodiment is shown in FIG. First, the distance sensor 105 acquires the vertical distance Dh1 from the distance sensor 105 to the display screen 400. Further, the back of the hand performing the gesture operation is recognized, and the distance sensor 105 acquires the vertical distance Dh2 from the distance sensor 105 to the back of the hand. The remote operation distance z calculated in the present embodiment is a vertical distance between the display screen and the hand, and can be calculated by z = Dh1−Dh2. In the present embodiment, the remote operation distance is calculated using the distance sensor, but this is not a limitation. A proximity sensor or an ultrasonic sensor may be installed on the display screen 400, and the remote operation distance may be calculated from the sensor information.

  FIG. 7A is a process flow diagram of the group level selection process (S16) of the first embodiment. When the group level selection process is started, the gesture recognition unit 308 recognizes a grouping gesture (S30). If the grouping gesture is not recognized (S30, N), nothing is performed, the group level selection process (S16) is completed, and the result is returned to the caller. When the grouping gesture is recognized (S30, Y), the distance acquisition unit 305 acquires the remote operation distance z2 when the grouping gesture is recognized from the information of the distance sensor 105 (S31). The acquisition method is the same as that for z1 described with reference to FIG. When the distance acquisition unit 305 acquires z2, the distance acquisition unit 305 reads z1 from the RAM 201, and calculates a remote operation distance difference z = | z2-z1 | (S32). Then, the result is held in the RAM 201.

  The group selection unit 304 reads the group information stored in the RAM 201 and the difference z between the remote operation distances, and further selects the group level based on the table information indicating the correspondence between the group level and the remote operation distance created in advance. (S33). In this embodiment, it is assumed that the group level is higher in the order of day group, month group, and year group, and based on this group level order, a table indicating the correspondence between the group level and the operation distance is created and stored in the holding unit 309. It shall be held. An example of a table held in FIG.

  For example, when a grouping gesture is performed in the state of FIG. 4A and z is 0 <z <a, no group is selected. However, if z is a <= z <b, a day level group is selected. Similarly, when z is b <= z <c, a month group is selected, and when z> = c, a year group is selected. The selected group level is held in the RAM 201.

  With the above processing, the group level selection processing (S16) is completed, and the result is returned to the calling source processing.

  When the group level selection process (S16) ends, the group display is changed. The display control unit 303 reads the selected group level from the RAM 201, generates a display screen that groups objects belonging to the same group level as the selected object 402, and displays the screen (S17). For example, when the day group level is selected from the state of FIG. 4A, the objects 403a and 403b having the same day group group information as the selected object 402 are grouped around the object 402, and FIG. B). Furthermore, when the month group level is selected from this state, the objects 403c, 403d, and 403e having the same month group group information as the object 402 are grouped, and the state shown in FIG. Further, when the year group level is selected, the objects 403f and 403g having the same year group group information as the object 402 are grouped, and the state shown in FIG.

  When the group display change (S17) ends, the gesture recognition unit 308 determines whether to confirm the group selection. For example, the group selection is confirmed by determining whether the position of the hand 401 has moved for a predetermined time. If the group selection is not confirmed (S18, N), the process returns to S16 again. When group selection is confirmed (S18, Y), the gesture recognition unit 308 determines whether to end the data display (S19). If the data display is not finished (S19, N), the process proceeds to S13 again. When the data display is ended (S19, Y), the information processing apparatus 100 ends the data display process.

  As described above, as described in the present embodiment, by associating the difference z between the group level and the remote operation distance, the group level can be quickly changed by a simple gesture operation such as an up / down operation of the hand 401. In this embodiment, a group having a higher group level is associated with a long remote operation distance. This makes it possible to realize a grouping UI that is intuitive (matches the user's feeling of operation) and is easy to use.

  In the present embodiment, group information is generated in advance by the group information generation unit 301, and grouping is performed using the generated group information. Therefore, by generating group information in advance, there is an effect that the grouping process can be performed at high speed. However, grouping may be performed based on the additional information when the grouping process is performed without generating the group information. When the group information is not generated, it is not necessary to have the group information, so that the amount of data to be held can be reduced.

  In the present embodiment, in the initial screen display process, a screen is created by randomly arranging objects, but this is not restrictive. The screen may be created by arranging the objects in the order of the names from the upper left of the screen.

  In this embodiment, when a grouping gesture is performed, a group of multiple granularities is automatically created based on date information and grouping is performed. However, if there are multiple types of additional information, which additional information is used. It is more convenient if the user can select whether to perform grouping. In such a case, as shown in the screen example 500 of FIG. 8B, processing may be performed in which a balloon 503 for selecting additional information to be used is displayed at the same time as the object 502 is selected. In the case of FIG. 8B, there are tagA, tagB, and tagC as additional information that can be used, and represents a state in which tagA is focused. For example, it is assumed that group information is generated using date information for tagA, location information for tagB, and person information for tagC. After selecting additional information to be used in a grouping process in the same way as selecting an object, grouping is performed. Processing can be performed.

[Second Embodiment]
Next, a second embodiment of the present invention will be described in detail with reference to the drawings. Except for the flowchart of the group level selection process, it is the same as that of the first embodiment. Also in the second embodiment, unless otherwise specified, the same reference numerals are given to those described with reference to other drawings, and the description thereof is omitted. The second embodiment solves an erroneous change of the group level due to an inadvertent grouping gesture, for example, the group level is changed even when the hand is brought close to the display screen to touch the object. It is.

  FIG. 7B shows a process flow of the group level selection process (S16) in the second embodiment. Processing until the group level selection processing (S16) is started is the same as that in the first embodiment.

  When the group level selection process is started, the gesture recognition unit 308 recognizes a grouping gesture (S30). If the grouping gesture is not recognized (S30, N), nothing is performed, the group level selection process (S16) is completed, and the result is returned to the caller. When the grouping gesture is recognized (S30, Y), the distance acquisition unit 305 acquires time t2, and calculates the difference t = t2-t1 (S34). If t is not equal to or greater than a threshold value (for example, 1 second) (S35, N), the group level selection process (S16) is completed, and the result is returned to the caller. When t is equal to or greater than a threshold (for example, 1 second) (S35, Y), the distance acquisition unit 305 acquires the remote operation distance z2 at that time from the information of the distance sensor 105 (S31). When the distance acquisition unit 305 acquires z2, the distance acquisition unit 305 reads z1 from the RAM 201, and calculates a remote operation distance difference z = | z2-z1 | (S32). Then, the result is held in the RAM 201. It is assumed that the threshold value t is stored in advance in the ROM 202 of the information processing apparatus 100.

  The group selection unit 304 associates the group level with z in accordance with the group information stored in the RAM 201 and the remote control distance difference z, and selects the group level based on the correspondence result (S33). The selected group level is held in the RAM 201.

  With the above processing, the group level selection processing (S16) is completed, and the result is returned to the calling source processing. The processing after the group level change processing (S16) is completed is the same as in the first embodiment.

  As described above, by calculating the time t until the grouping gesture is recognized and filtering with the threshold value as described in the present embodiment, it is possible to prevent erroneous group level changes due to careless grouping gestures.

[Third Embodiment]
Next, a third embodiment of the present invention will be described in detail with reference to the drawings. In the second embodiment, the display change speed of the group performed by the display control unit 303 is constant at a system-defined speed. In the present embodiment, an example in which the group change display speed is made to correspond to the speed of the grouping gesture is shown. Also in the third embodiment, unless otherwise specified, the same reference numerals are given to those described with reference to other drawings, and the description thereof is omitted.

  FIG. 3B shows a functional block diagram according to the third embodiment. Except for the speed calculation unit 306, the second embodiment is the same as the second embodiment.

  The speed calculation unit 306 calculates the gesture speed z / t of the grouping gesture using the difference z in the remote operation distance and the difference t in time. The display control unit 307 controls the grouping display change speed using the gesture speed.

  FIG. 5B is a flowchart in the third embodiment. Except for S40 and S41, the second embodiment is the same as the second embodiment.

  In S40, the speed calculation unit 306 calculates the gesture speed z / t of the grouping gesture using the remote control distance difference z calculated in S32 and the time difference t calculated in S34. In S41, the display control unit 307 controls the group display change speed by using the input gesture speed. Specifically, the group display change speed is displayed in proportion to the gesture speed z / t.

  For example, it can be used when grouping display control is performed using animation. When the calculated gesture speed is fast, the grouping animation is accelerated, and when it is slow, the grouping animation is slowed. In this way, when the user wants to perform grouping while confirming the object to be grouped, the user can use a grouping gesture at a slow speed.

  As described above, the grouping UI that can reflect the user's intention can be realized by calculating the gesture speed and using it for display control, as described in the present embodiment.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described in detail with reference to the drawings. Except for the function of the object selection unit in the functional block diagram and the explanatory diagram of the group level selection process shown in FIGS. 4E to 4H, it is the same as in the first embodiment. Also in the fourth embodiment, unless otherwise specified, the same reference numerals are given to those described with reference to other drawings, and the description thereof is omitted.

  The fourth embodiment is a variation of the first embodiment. When the object selection unit 300 selects an object, one or more objects can be selected, which is different from the first embodiment. FIGS. 4E to 4H are explanatory diagrams of the group level selection process in the fourth embodiment.

  By selecting a plurality of objects, group information with a plurality of granularities can be generated using additional information common to the selected objects. As a result, AND grouping can be realized.

  FIG. 4E shows an example in which two thumbnails (objects) 404 in which A and B are shown are selected. The group information generation unit 301 uses additional information to make a group “only Mr. A and Mr. B are shown”, a group “some other people including Mr. A and B” are shown, “A The group information “All including Mr. and Mr. B” is generated. The generated group information is held in the RAM 201.

  Also in this embodiment, group granularity and z are associated according to the group information held in the RAM 201 and the remote control distance difference z, and the group granularity is selected based on the correspondence result. The association is performed by creating a table indicating the correspondence between the group level and the remote operation distance difference z. As an example of the table, a table as shown in FIG. 10B is created and held by the holding unit 309. For example, when a grouping gesture is performed in the state of FIG. 4E and z is 0 <z <a, no group is selected. However, if z is a <= z <b, the group “only A and B are shown” is selected. Similarly, when z is b <= z <c, a group “several people including A and B” is displayed. When z> = c, a group “all including A and B” is selected. Select each one. The selected group level is held in the RAM 201.

  The display control unit 303 reads the selected group level from the RAM 201, generates a display screen in which objects related to the selected object 404 are grouped, and outputs the display screen to the display 101. For example, when the group “only A and B are shown” is selected in the state of FIG. 4E, the objects 405a and 405b showing only A and B are grouped around the object 404, and FIG. It will be in the state of (F). Further, in this state, when the group “Several people including A and B” is selected, the objects 405c, 405d, and 405e including several people including A and B are grouped, and FIG. ) State. Similarly, when the group “all including A and B” is selected, all the objects 405f and 405g in which A and B are shown are grouped, and the state shown in FIG.

(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (5)

A selection means for selecting an object from a plurality of objects displayed on the display unit;
Distance acquisition means for acquiring a remote operation distance from an object to the display screen of the display unit;
Holding means for holding the correspondence between the remote operation distance and the group level;
An information processing apparatus comprising display control means for grouping and displaying objects based on the acquired remote operation distance, the association, and the selected object. And generating means for generating a plurality of group level group information from the additional information added to the object,
The said display control means groups and displays an object based on the acquired remote operation distance, the association, the selected object, and the group information. Information processing device. A selection step of selecting an object from a plurality of objects displayed on the display unit;
A distance acquisition step of acquiring a remote operation distance from an object to the display screen of the display unit;
A display control step of grouping and displaying objects based on the correspondence between the remote operation distance held in the holding unit and the group level and the acquired remote operation distance and the selected object; A method for controlling an information processing apparatus.   A program that causes a computer to function as each unit of the information processing apparatus according to claim 1 by being read and executed by a computer.   A computer-readable storage medium storing the program according to claim 4.

Images