WO2013011862A1

WO2013011862A1 - Information processing device, operation screen display method, control program, and recording medium

Info

Publication number: WO2013011862A1
Application number: PCT/JP2012/067525
Authority: WO
Inventors: 正義神原; 信子三浦; 敬一長谷川; 晋吾山下; 英樹西村; 陽柯
Original assignee: シャープ株式会社
Priority date: 2011-07-15
Filing date: 2012-07-10
Publication date: 2013-01-24
Also published as: JP2013025409A; JP5172997B2

Abstract

This information processing device, which is provided with an input unit and a display unit, achieves superior operability. A tablet terminal (100) is characterized by being provided with: a touch information generation unit (21) that acquires the trajectory along which an indicator pointing out a position on the screen of the display unit (12) has moved; an object specification unit (22) that specifies, as a selected object, an object of which at least a portion is included in a region encircled by the trajectory; a related item extraction unit (23) that extracts the item associated with the specified object as a related item by referring to a related information recording unit (42), which associates and records objects and items related to the objects; and an operation screen processing unit (24) that disposes the icons of the extracted related items along the contour line of a ring, and displays the icons at the display unit.

Description

Information processing apparatus, operation screen display method, control program, and recording medium

The present invention relates to a user interface technology of an information processing apparatus including an input unit and a display unit.

In recent years, so-called tablet terminals such as smartphones and tablet PCs are rapidly spreading. The tablet terminal has a flat outer shape and includes a touch panel as a display unit and an input unit. By touching the object displayed on the touch panel with a finger, a pen, or the like, the user can perform various operations on the tablet terminal body.

The tablet terminal can discriminate various contact operations on the user's screen by the touch panel, and can perform object display according to the contact operation. For example, for the touch action, various actions such as tapping (lightly tapping), flicking (flicking, picking), pinching (pinch with a finger), dragging an object displayed on the screen with a finger (pen) There is. The tablet terminal discriminates such various contact actions, and selects / moves objects, scrolls a list, enlarges / reduces an image, etc. according to the discrimination result. The tablet terminal realizes a more intuitive operation by the touch panel as described above, and is supported by many people.

For example, Patent Document 1 discloses a mobile communication terminal including a touch panel display unit. In the portable communication terminal of Patent Document 1, an object (URL, e-mail address, character string, image, etc.) can be selected by touching it with a finger (pen) or touching it so as to surround it. ing. When an object is selected by such an operation, the mobile communication terminal extracts a keyword from the selected object and accesses a related site.

Further, Patent Document 2 discloses a portable device having a touch panel display. The portable device of Patent Document 2 displays a through image (an image reflected on a camera, etc.) on a touch panel display, detects a specific target in the through image selected by touching the surrounding area, and reduces the specific target. The image can be displayed on the edge of the touch panel display as a release button.

Patent Document 3 discloses a website search system using a touch panel. When the keyword displayed in the keyword display area is touched by hand, the website search system of Patent Literature 3 accepts it as a search keyword and displays a first mother icon corresponding to the accepted keyword. Then, the website search system searches the website with a search engine according to the keyword, and displays a thumbnail image of the searched website around the first mother icon.

Patent Document 4 discloses an information processing apparatus including a display panel having a contact sensor. The information processing apparatus disclosed in Patent Document 4 detects rotation of the operating body (finger) by a predetermined angle or more while an object is selected, and displays operation items related to the object around the object.

Further, Patent Document 5 discloses an information processing apparatus including a touch panel unit. The information processing apparatus disclosed in Patent Document 5 acquires a trajectory of a user's touch position, specifies an object image selected by the trajectory, and moves the selected object image to a position corresponding to an end point of the trajectory.

On the other hand, regardless of the tablet terminal and regardless of the size of the device, in general information processing devices, user interface technology that displays a menu and accepts selection is very often used. According to this technology, the information processing apparatus displays a menu so that the user can select a desired item, and accepts the user's selection. Thus, the user can operate the apparatus by selecting a desired item. For example, Patent Documents 6 to 9 disclose information processing apparatuses that realize menu display for the purpose of improving user convenience and operability.

JP 2010-218322 A (published on September 30, 2010) JP 2010-182023 A (released on August 19, 2010) JP 2009-134738 A (released on June 18, 2009) JP 2011-13980 A (published January 20, 2011) JP 2006-244353 A (published on September 14, 2006) JP-A-8-305535 (released on November 22, 1996) Japanese Patent Laid-Open No. 10-307664 (published November 17, 1998) No. 11-507455 (announced on June 29, 1999) JP 2001-265475 A (published September 28, 2001)

The good operability of the tablet terminal is how to display the final result, which is the user's purpose, with a simple contact operation and a small number of operations, and a natural flow that does not contradict the user's intuition, It depends on whether to display the result based on the contact operation.

Such an improvement in operability is realized by appropriately grasping the user's purpose, the user's state, and the user's tendency. For example, the tablet terminal displays what the user wants to do now, what the user wants to do next, how the user is now operating, where the user is, and how the user moves. It is required to “see” the user's intentions from all points of view, such as whether or not it is natural.

The configurations of the devices described in Patent Documents 1 to 9 are not necessarily sufficient to detect the user's intention.

More specifically, Patent Document 1 discloses that an object is selected by an operation of surrounding the object, but it is disclosed that an item related to the object is extracted and displayed by the above operation. Not. Further, Patent Document 2 discloses that an icon corresponding to an object is displayed by an operation of enclosing the object. However, the object is selected by the above operation, and the object is displayed along with the selection. Extracting and displaying related items is not disclosed. Further, Patent Document 3 discloses that when an object is selected, the object is displayed, and a thumbnail related to the object is displayed around the object. Are not connected. Patent Document 4 discloses that an object is touched to be selected and an icon related to the object is displayed around the object. In order to display an icon around the object, the object is displayed. Apart from the selection, it is necessary to perform complicated operations (operations such as pressing the finger against the touch surface and twisting the finger angle) until the number of operations increases and the desired result (icon) is displayed. The operation becomes very complicated.

As a result, a series of processes such as object selection, determination, display of the result (related items), selection of the result, display of the final result, a simple contact operation, a small number of operations, and There is a problem that the device cannot be operated by an intuitive contact operation.

The above-mentioned operability problem is not limited to a small tablet terminal excellent in portability, but also an information processing apparatus of any size provided with a touch panel type display unit and input unit, as well as all types of forms not limited to a touch panel. This is a problem that commonly occurs in information processing apparatuses including a display unit and an input unit.

The present invention has been made in view of the above-described problems, and an object thereof is to realize excellent operability in an information processing apparatus including an input unit and a display unit.

In order to solve the above-described problem, the information processing apparatus of the present invention is surrounded by a trajectory acquisition unit that acquires a trajectory of movement of an indicator that indicates the position of the screen of the display unit, and a trajectory acquired by the trajectory acquisition unit. With reference to an object specifying means for specifying an object including at least a part in the selected area as the selected object, and a related information storage unit that stores the object and an item related to the object in association with each other, Related item extracting means for extracting items associated with the object specified by the object specifying means as related items, and icons of related items extracted by the related item extracting means are arranged side by side on the outline of the ring. And an operation screen processing means for displaying on the display section.

According to the above configuration, the trajectory of the movement (indicator movement) that the trajectory acquisition unit encloses is acquired, and based on this trajectory, the object specifying unit specifies the object selected by the user by the enclosing operation. Subsequently, the related item extracting unit extracts items related to the identified object. Since the related information storage unit stores the object and the item related to the object in association with each other, the related item extracted by the related item extracting unit is related to the object selected by the user. It is. Finally, the operation screen processing means arranges the icons of the extracted related items in a ring shape that is easily associated with the surrounding operation.

Thereby, the icons are arranged as described above, and the generated operation screen is presented to the user. As described above, the information processing apparatus according to the present invention uses a very natural and simple user action as an opportunity to specify an object, which is to “enclose” the object with an indicator (such as a pen or a finger). An operation screen in which icons of related items related to the selected object are arranged in the shape of a ring can be provided to the user.

Since the trajectory of “enclose” has a shape that encloses something, it can be said that the shape of the ring is similar to the trajectory of movement of the indicator obtained by the enclosing operation. Therefore, the result (an icon arranged in a ring shape) is likely to be associated with the above-described operation surrounded by the user.

In other words, it can be said that the transition from the event of "enclosing the object" to the event of "an operation screen with icons arranged in a circle" is a natural flow that does not contradict the user's intuition. .

In addition, the information processing apparatus of the present invention can preliminarily detect related items that the user will select next after selecting an object, and can display the related items in a selectable manner for the user. Specifically, according to the above configuration of the present invention, the icons arranged on the operation screen by the operation screen processing means are all related item icons extracted as items related to the object selected by the user. is there. In other words, after selecting the object, the related item that will be selected next is displayed with a circular icon, so the user can immediately specify the next desired icon from the icon ring. be able to.

Also, the menu list with related item icons arranged in a circle has the following advantages compared to the linear one-dimensional menu list. For example, in a one-dimensional menu list, for example, icons are arranged from top to bottom or from left to right. Therefore, priorities are unintentionally assigned to the icons according to the arrangement positions. On the other hand, according to the circular menu list, it is possible to treat all icons arranged in a circular manner on an equal basis.

From the above, the information processing apparatus according to the present invention can display the final result desired by the user in a natural flow that does not contradict the user's intuition, while having a simple operation and a small number of operations. As a result, there is an effect that it is possible to realize excellent operability in the information processing apparatus including the input unit and the display unit.

In the information processing apparatus of the present invention, it is preferable that the operation screen processing means determines the position and size of the ring so that an icon is arranged around the selected object.

According to the configuration described above, the information processing apparatus places an icon around the object in response to an extremely natural and simple user action of “enclosing” the object. Can be output.

The user can obtain, as a result, an operation screen in which icons of related items are arranged so as to surround the object that he / she has previously enclosed and selected. The positional relationship between these icons and the object matches the positional relationship between the object and the locus of movement of the indicator by the action previously performed by the user. Further, the movement trajectory of the indicator obtained by surrounding the object is similar to the shape of the ring in which the icon is arranged.

In other words, the transition from the event of “enclosing the object” to the event of “an operation screen with icons arranged around the object” is more natural than the user's intuition. It can be said that.

Furthermore, a menu list in which icons of related items are arranged in a circle around an object has the following advantages compared to a linear one-dimensional menu list. For example, in a one-dimensional menu list, for example, icons are arranged from top to bottom or from left to right. Therefore, priorities are unintentionally assigned to the icons according to the arrangement positions. On the other hand, according to the circular menu list, it is possible to treat all icons arranged in a circular manner on an equal basis. Furthermore, even if a one-dimensional menu list is displayed near the previously selected object, it is difficult to express the relationship between the object and each icon. In contrast, if a circular menu list is displayed around the previously selected object, there is a relationship between the previously selected (enclosed) object and the surrounding icons. This makes it possible for the user to recognize this naturally.

From the above, the information processing apparatus according to the present invention can display the final result desired by the user in a natural flow that does not contradict the user's intuition, while having a simple operation and a small number of operations. As a result, there is an effect that it is possible to realize excellent operability in the information processing apparatus including the touch panel.

In the information processing apparatus of the present invention, it is preferable that the operation screen processing means determines the trajectory acquired by the trajectory acquisition means, or a similar shape or an approximate shape thereof, as the shape of the ring.

According to the above configuration, the user performs an operation of freely surrounding the object with an arbitrary shape, and the trajectory at this time is held by the trajectory acquisition means. Then, when creating the operation screen, the operation screen processing means sets each icon so as to surround a predetermined area (or the object itself) on the contour line of the ring that is the same or similar to the locus obtained as described above. Deploy.

This makes it possible to output a result that icons are arranged in a ring shape in response to a very natural and simple user action in specifying the object “enclose” around the object. That is, the user can obtain, as a result, an operation screen in which icons of related items are arranged so as to surround a predetermined area (or the object itself).

Furthermore, the shape of the ring in which the icons are arranged is displayed on the operation screen in a state that matches or is similar to the movement trajectory of the operation body obtained by surrounding the object.

That is, when the user surrounds the object, an operation screen on which icons are arranged “as the user surrounded the object” is obtained. It can be said that the transition of this event is a more natural flow that does not contradict the user's intuition.

Further, since the icons are arranged in the shape as enclosed by the user, the user can obtain the operation screen in which the icons are arranged in the desired shape by surrounding the object in the desired shape. Thereby, the playability at the time of displaying an operation screen and operating information processing apparatus increases.

In addition, since the user can predict the arrangement of the icons, surround the object as he / she desires, and display the icons of related items, the operability is further improved.

In the information processing apparatus of the present invention, the operation screen processing means may individually determine the timing at which each icon is displayed.

According to the above-described configuration, it is possible to realize an operation screen to which animations for displaying individual icons at different timings as well as displaying all the icons of the extracted related items simultaneously.

Furthermore, in the information processing apparatus according to the present invention, the trajectory acquisition unit measures an elapsed time from the start of the movement of the indicator, and travel time indicating the elapsed time at least some of the points constituting the trajectory. The operation screen processing means sequentially displays each icon clockwise or counterclockwise according to the moving direction of the indicator determined from the trajectory and the moving time information. It is preferable.

According to the above configuration, the movement time information is associated with each point of the trajectory. With reference to this, regarding the enclosing operation, whether the object is surrounded in the movement direction, that is, clockwise, or counterclockwise. To see if the object is surrounded.

The operation screen processing means causes each icon to sequentially appear in the same direction in a circular manner in accordance with the actual movement (indicator moving direction) surrounding the user's object. Specifically, the operation screen processing means arranges icons one by one in a ring shape clockwise when the object is surrounded clockwise, and when the object is surrounded counterclockwise, The icons are arranged in a ring in a counterclockwise direction one by one.

As a result, a result (icon appearance pattern) with almost the same movement (the same movement as the direction of movement of the indicator) is obtained as when the user is surrounded. As a result, the operation screen can be provided in a natural flow that does not contradict the user's intuition.

Furthermore, in the information processing apparatus of the present invention, it is preferable that the operation screen processing means determines a position relatively the same as the position of the start point of the trajectory in the ring as a display position of the first icon.

According to the above configuration, the operation screen processing means determines the appearance start position on the outline of the ring, where the first icon appears, when the icons appear sequentially in the same movement direction as the indicator. It is determined based on the starting point of the movement trajectory.

In other words, the position that is relatively the same as the position of the start point of the locus in the ring that is the reference line for placing the icon is determined as the display position of the first icon, and from there, the same movement direction (clockwise) as the indicator , Or counterclockwise), the remaining icons appear sequentially.

This makes it possible to provide an operation screen with a more natural flow that is more reminiscent of the user's intuition and results are more reminiscent of the operation.

Furthermore, in the information processing apparatus of the present invention, the operation screen processing means may determine the timing for sequentially displaying the icons so as to correspond to the moving speed of the indicator that forms the locus.

According to the above configuration, the operation screen processing means sets each icon to a predetermined position in the ring when the icons sequentially appear in the same movement direction as the indicator and with the start position of the movement of the operation body matched. The timing of appearance is made to correspond to the moving speed of the indicator that formed the locus.

As a result, a result (icon appearance pattern) with the same movement at the same speed as the movement of the user and the speed of the movement is obtained, so the user's action and the displayed result The relevance can be further enhanced, and as a result, the operation screen can be provided in a natural flow that does not contradict the user's intuition.

Alternatively, in the information processing apparatus according to the present invention, the related information storage unit may include a plurality of related items as long as the related items can be processed continuously or simultaneously. Cooperation information indicating the presence or absence of cooperation is stored for each related item, and the operation screen processing means arranges icons of related items having cooperation among the related items extracted by the related item extracting means next to each other. You may arrange.

If a plurality of related items having linkage are selected at the same time, the processing is executed continuously or simultaneously. Therefore, it is preferable that icons of a plurality of related items to be linked are selected at the same time in order to make the number of operations as small as possible. Here, if the enclosing operation is used, all the icons can be enclosed in a single enclosing operation and selected simultaneously. In order to make it easy to enclose all the icons together in a single enclosing operation, even if the icons are arranged in a ring shape, the icons of a plurality of related items having cooperation are adjacent to each other. It is preferable that they are arranged side by side (in the case of three or more).

According to the above configuration, icons of related items having cooperation are arranged so as to be arranged next to each other. That is, when two or more icons are linked, they are arranged side by side on the outline of the ring in a continuous manner when there are three or more icons.

This makes it easy for the user to enclose all the icons together in a single enclosing operation. That is, the final result desired by the user can be displayed in a natural flow that does not contradict the user's intuition while being a simple operation and a small number of operations. As a result, there is an effect that it is possible to realize excellent operability in the information processing apparatus including the input unit and the display unit.

Alternatively, the information processing apparatus of the present invention further includes a position detection unit that acquires position information indicating the position of the own device, a direction detection unit that acquires direction information indicating the direction of the own device, and peripheral devices of the own device. Based on the communication unit that communicates and acquires the position information of the peripheral device, the position information of the own device acquired by the position detection unit, and the position information of the peripheral device acquired by the communication unit, Identifies the position of each peripheral device in relation to its own device by specifying the positional relationship with the peripheral device and specifying the orientation of its own device based on the direction information acquired by the direction detection unit. Device direction specifying means, and the operation screen processing means specifies the related items of the peripheral device so as to correspond to the direction in which the peripheral device exists, specified by the device direction specifying means. Position of the response icons may be determined.

According to the above configuration, the device direction specifying means can specify the direction in which each peripheral device exists with respect to the own device based on the following three pieces of information. That is, (1) position information indicating the position of the own device acquired by the position detecting unit, (2) direction information indicating the direction of the own device acquired by the direction detecting unit, and (3) acquiring by the communication unit. , Position information of each peripheral device.

Then, the operation screen processing means determines the arrangement position of the icon corresponding to the related item of the peripheral device so as to correspond to the direction in which the peripheral device exists, specified by the device direction specifying means.

As a result, when the icon of the peripheral device is displayed in a ring shape around the object, the positional relationship between the central object and each icon corresponds to the positional relationship between the tablet terminal 100 and each peripheral device. For this reason, the user can intuitively grasp the relevance between the user's own operation and the information processing that occurs as a result. For example, if the user wants to transfer a photo to the digital television 1, the user may drag the photo object in the direction in which the digital television 1 is actually located. Therefore, it is possible to provide an operation screen that can be operated in a natural flow that does not contradict the user's intuition. As a result, there is an effect that it is possible to realize excellent operability in the information processing apparatus including the input unit and the display unit.

Alternatively, in the information processing apparatus according to the present invention, the operation screen processing unit relates to an operation that acts on other devices other than the own device in the related information storage unit among the related items extracted by the related item extracting unit. You may arrange | position the icon of the related item with which the operation | movement attribute which shows that it is an item is linked | related above the selected said object.

According to the above configuration, the operation screen processing means can grasp the operation attribute of each extracted related item by referring to the related information storage unit. Specifically, the operation screen process can identify a related item that is an item related to an operation that acts on a device other than the own device based on the information of the operation attribute.

Then, when the extracted related item is an item related to an action that acts on other devices other than the own device, the operation screen processing means sets the icon of such a related item above the selected object. Decide to place in.

Thus, the icon related to the action to work on the other device is displayed at the upper part of the screen than the previously selected object.

In this way, when an icon related to an operation that works on another device is placed above the object, the user next wants the information processing apparatus to perform an “operation on other device (information processing)” In this case, an operation of dragging the object to the target icon, that is, a gesture of moving an object from one's place to the other side is performed.

“Working with other devices (information processing)” is more intuitively related to the gesture of moving an object away from your place than the gesture of bringing an object in front of you Deeper and more likely to be associated.

According to the above configuration, it is possible to further increase the relevance between the user's action and the resultant product, and as a result, it is possible to provide the operation screen in a natural flow that does not contradict the user's intuition.

Alternatively, in the information processing apparatus of the present invention, the trajectory acquisition means acquires the trajectory of the indicator that has occurred in a predetermined period until the indicator surrounding the object displayed on the display unit moves. When the operation screen processing means determines that the trajectory acquired during the predetermined period is biased toward a specific area on the screen of the display unit, the ring of the ring is arranged so that an icon is arranged in the specific area. The position may be determined.

According to the above configuration, the trajectory acquisition means acquires a trajectory not only for the surrounding motion but also for the motion that occurred in the past predetermined period. Subsequently, the operation screen processing means can grasp the position where the operation has occurred on the screen of the display unit in the past predetermined period based on the acquired trajectory. If it concentrates on the specific area | region of the screen of a display part, the deviation of a contact position can be detected by it.

It can be inferred that the contact position is biased in this way is that the information processing apparatus is used under a special usage situation in which only that area can be touched (or other areas are difficult to touch).

Therefore, the operation screen processing means determines the position of the ring so that the icon is arranged in the area where the bias is detected.

Thus, the icon is displayed in the area where the user can touch, and when the user next performs an operation of selecting the icon, the desired icon can be selected immediately from the touchable area.

The following is a more detailed description using specific examples. For example, when operating with one hand, the touch position tends to be biased toward the lower left area of the touch panel (when operating with the left hand) or the lower right area of the screen (when operating with the right hand). When the user is using the information processing apparatus in such a situation, there is a problem that the operation becomes complicated if an object or icon that requires a contact operation is displayed on the upper part of the screen or the lower part of the screen opposite to the hand. . This is because the user cannot touch the target object immediately and must perform an extra operation of rushing to a contactable area or switching to a two-handed operation.

Therefore, the information processing apparatus according to the present invention can solve the above-mentioned problem by observing the usage status of the user with the above-described configuration. In other words, the information processing apparatus of the present invention can be configured to detect the deviation of the contact position of the indicator and place the icon in a region where the user's indicator is estimated to arrive immediately.

As a result, when the user operates the information processing apparatus with one hand, the icon is displayed so as to fit within the lower left (or right) area of the screen of the touch panel. There is no need to drag icons, and a desired icon can be selected immediately.

Alternatively, in the information processing apparatus according to the present invention, the operation screen processing means may include an icon in a region surrounded by the locus of the indicator or a specific region including the enclosed region on the screen of the display unit. The position of the ring may be determined so as to be arranged.

According to the above configuration, the icon is arranged at or near the position surrounded by the indicator such as the finger by the user. It can be said that the position surrounded by the user is a contactable area for the user. Therefore, it is possible to reliably display an icon in the contactable area.

In the information processing apparatus of the present invention, the input unit and the display unit included in the information processing apparatus constitute a touch panel, and the trajectory acquisition unit acquires a trajectory of movement of the indicator that has moved on the touch panel. May be.

According to the above configuration, it is possible to increase the relevance between the contact operation performed by the user for selecting an object and the resultant product obtained according to the operation, and the operation is performed in a natural flow that does not contradict the user's intuition. A screen can be provided. As a result, there is an effect that it is possible to realize excellent operability in the information processing apparatus including the touch panel.

Alternatively, in the information processing apparatus of the present invention, the input unit included in the information processing apparatus inputs an instruction to move the cursor displayed on the display unit to the information processing apparatus, and the trajectory acquisition unit includes: The locus of movement of the cursor as the indicator may be acquired.

According to the above configuration, it is possible to increase the relevance between the input operation performed for the user to select an object by operating the input unit and the resultant product obtained according to the operation, and the user's intuition The operation screen can be provided in a natural flow that does not violate. As a result, there is an effect that it is possible to realize excellent operability in the information processing apparatus including the display unit and the input unit.

In order to solve the above-described problem, an operation screen display method of the present invention acquires a trajectory in which an indicator that indicates a position of a screen of a display unit included in the information processing apparatus moves in the operation screen display method in the information processing apparatus. A trajectory acquiring step, an object specifying step for specifying, as a selected object, an object at least part of which is included in the region surrounded by the trajectory acquired in the trajectory acquiring step, and the object and the object A related item extracting step of extracting an item associated with the object identified in the object identifying step as a related item with reference to a related information storage unit that associates and stores the item, and the related item extracting step Arrange the icons of the related items extracted in step by step on the outline of the ring. It is characterized in that it comprises an operation screen processing step of displaying on the display unit.

According to the above method, the final result desired by the user can be displayed in a natural flow that does not contradict the user's intuition while being a simple operation and a small number of operations. As a result, there is an effect that excellent operability can be realized in the information processing apparatus including the input unit and the display unit.

The information processing apparatus may be realized by a computer. In this case, an information processing apparatus control program for causing the information processing apparatus to be realized by the computer by causing the computer to operate as the above-described means, and A computer-readable recording medium on which is recorded also falls within the scope of the present invention.

In order to solve the above-described problem, an information processing apparatus according to the present invention provides a trajectory acquisition unit that acquires a trajectory in which an indicator that indicates a position of a screen of the display unit has moved in an information processing device including an input unit and a display unit. And an object specifying means for specifying an object at least partially included in the area surrounded by the trajectory acquired by the trajectory acquiring means as a selected object, and an object and an item related to the object A related item extracting unit that extracts an item associated with the object specified by the object specifying unit as a related item with reference to the related information storage unit to be stored, and a related item extracted by the related item extracting unit An operation screen process that arranges the icon of the item on the outline of the ring and displays it on the display unit. It is characterized in that it comprises a means.

In order to solve the above-described problem, the operation screen display method of the present invention is a trajectory in which an indicator indicating the position of the screen of the display unit moves in the operation screen display method in an information processing apparatus including an input unit and a display unit. A trajectory acquisition step for acquiring the object, an object specifying step for specifying, as the selected object, an object at least part of which is included in the region surrounded by the trajectory acquired in the trajectory acquisition step, and the object and the object A related item extracting step of extracting an item associated with the object identified in the object identifying step as a related item with reference to a related information storage unit that associates and stores an item associated with The related item icons extracted by the item extraction step are displayed on the outline of the ring. Arranged Te, it is characterized by including an operation screen processing step of displaying on the display unit.

Therefore, the final result desired by the user can be displayed in a natural flow that does not contradict the user's intuition while being a simple operation and a small number of operations. As a result, there is an effect that it is possible to realize excellent operability in the information processing apparatus including the input unit and the display unit.

It is a functional block diagram which shows the principal part structure of the tablet terminal in one Embodiment of this invention. It is a block diagram which shows the hardware constitutions of the tablet terminal in one Embodiment of this invention. It is a top view which shows the external appearance of the tablet terminal in one Embodiment of this invention. It is a figure explaining a mode when a user hold | grips and operates a tablet terminal, (a) is a figure explaining a mode that a tablet terminal is hold | gripped with one hand and is operated with the hand, (b) It is a figure explaining a mode that a tablet terminal is hold | gripped with one hand and is operated with the other hand. It is a figure explaining operation | movement of the object specific part of a tablet terminal, (a) is a figure which shows a mode that the user performed contact operation of "surrounding" an object in order to select the target object, (b) ) Is a diagram showing an example of contact information generated by the contact information generation unit in accordance with the contact operation shown in (a), and (c) is a display unit during the period from t0 to tn when contact is detected. It is a figure which shows an example of the map information of the displayed video frame. It is a figure which shows an example of the relevant information memorize | stored in the relevant information storage part of a tablet terminal. It is a figure which shows the specific example of the icon image memorize | stored in the icon memory | storage part of a tablet terminal. It is a figure explaining the processing content of the operation screen process part of a tablet terminal, (a) is a figure explaining an example of the display process of the object by an operation screen process part, (b) is by an operation screen process part. It is a figure which shows an example of the icon arrangement | positioning of a related item. It is a figure which shows the specific example of the operation screen obtained as a result of the operation screen production | generation process performed by the operation screen process part. It is a flowchart which shows the flow of the operation screen display process by a tablet terminal. It is a functional block diagram which shows the principal part structure of the tablet terminal in other embodiment of this invention. It is a figure which shows the specific example of the contact information memorize | stored in the contact information memory | storage part of a tablet terminal, (a) is an operation which "encloses" an object in arbitrary shapes in order for a user to select the target object. It is a figure which shows the mode performed, (b) is a figure which shows an example of the contact information which the contact information generation part produced | generated with the contact operation | movement shown to (a). It is a figure which shows an example of the icon arrangement | positioning of a related item performed by the operation screen process part containing a ring shape determination part. It is a flowchart which shows the flow of the operation screen display process by the tablet terminal in other embodiment of this invention. It is a functional block diagram which shows the principal part structure of the tablet terminal in further another embodiment of this invention. It is a figure which shows an example of the relevant information memorize | stored in the relevant information storage part of a tablet terminal. It is a figure explaining the procedure in which the icon arrangement | positioning determination part of an operation screen process part determines the arrangement | positioning of an icon, (a) is a table which shows the result that the icon arrangement | positioning determination part classified the related item (its icon) based on the attribute. (B) is a figure explaining the procedure in which the icon arrangement | positioning determination part arrange | positions the icon of each related item according to the classification | category of (a). It is a figure which shows the specific example of the operation screen obtained as a result of having arrange | positioned the icon according to the determination of the icon arrangement | positioning determination part by the operation screen process part. It is a figure which shows the other specific example of the operation screen obtained as a result of having arrange | positioned the icon according to the determination of the icon arrangement | positioning determination part by the operation screen process part. It is a functional block diagram which shows the principal part structure of the tablet terminal in further another embodiment of this invention. It is a figure which shows an example of a use environment when the user of a tablet terminal is using the tablet terminal in an indoor room. It is a figure which shows the specific example of the apparatus direction information which the apparatus direction specific | specification part of a tablet terminal produces | generates. It is a figure which shows the specific example of the relevant information regarding the peripheral device memorize | stored in the relevant information storage part of this embodiment. It is a figure which shows the specific example of the operation screen obtained as a result of the operation screen production | generation process which the operation screen process part performed. FIG. 25 is a diagram showing a specific example of an operation screen obtained following the operation screen shown in FIG. 24 as a result of the operation screen generation process executed by the operation screen processing unit. It is a figure which shows the modification of the icon display method of a related item. It is a figure which shows the modification of the icon display method of a related item. It is a figure which shows the modification of the icon display method of a related item, (a) shows the mode of the ring of the icon initially arrange | positioned small around the object, (b) expands the ring of an icon. A state in the middle is shown, and (c) shows a state in which a ring of icons is finally arranged largely in the center of the screen through (a) to (b). It is a figure which shows one modification of the icon display method of a related item, (a) is a figure which shows a specific example of contact information, (b), (c), (d), (e) It is a figure which shows a mode that a some icon is sequentially displayed at a fixed space | interval, (f) is a figure which shows a specific example of the operation screen finally obtained. It is a figure which shows one modification of the icon display method of a related item, (a) is a figure which shows a specific example of contact information, (b) is a figure which shows an example of the operation screen at the time of t0. (C) is a diagram showing an example of the operation screen at the time point ta, (d) is a diagram showing an example of the operation screen at the time point tb, and (e) is an operation at the time point tc. It is a figure which shows an example of a screen, (f) is a figure which shows an example of the operation screen at the time of tn. It is a figure which shows the modification of the icon arrangement pattern of a related item. It is a figure which shows the modification of the icon arrangement pattern of a related item. It is a figure explaining operation | movement of the tablet terminal of this invention which can show an operation screen according to a user's usage condition, (a) demonstrates an example of the condition where the user is operating with the left hand. (B) is a figure which shows the specific example of the contact information produced | generated with the contact operation | movement of (a). It is a figure which shows an example of an operation screen when an icon is arrange | positioned according to the ring shape determined by the ring shape determination part of the operation screen process part. (A) is a figure explaining a mode that a user rotates the ring of an icon by dragging on a contact possible area | region, (b) rotates a ring by said contact operation "drag". It is a figure which shows an example of the operation screen after arrangement | positioning of an icon is changed by having been performed. (A) is a figure which shows a mode that the user performed the contact operation of "enclosing" in order to select the several target object, (b) is a contact operation shown to (a) of the figure. It is a figure which shows the specific example of the operation screen produced | generated by the operation screen process part according to it. (A) is a figure which shows a mode that the user performed the contact operation of "surrounding" in order to select a plurality of different types of objects, and (b) is shown in (a) of the figure. It is a figure which shows the specific example of the operation screen produced | generated by the operation screen process part according to contact operation. It is a figure which shows the other example of the relevant information memorize | stored in the relevant information storage part.

Embodiment 1
An embodiment of the present invention will be described with reference to FIGS. 1 to 10 as follows.

In the embodiment described below, a case where the information processing apparatus of the present invention is applied to a tablet terminal will be described as an example. In the present embodiment, as an example, the tablet terminal is realized by a small smartphone that can be operated with one hand and is excellent in portability.

However, the information processing apparatus of the present invention is not limited to the above example, and the information processing apparatus of the present invention can be applied to an information processing apparatus of any size (for example, an electronic blackboard equipped with a large touch panel). Good.

[Hardware configuration of tablet terminal]
FIG. 2 is a block diagram illustrating a hardware configuration of the tablet terminal 100 according to the present embodiment. As shown in FIG. 2, the tablet terminal 100 includes at least a control unit 10, an input unit 11, a display unit 12, and a storage unit 19. Furthermore, the tablet terminal 100 may include an operation unit 13, an external interface 14, a communication unit 15, a wireless communication unit 16, an audio output unit 17, and an audio input unit 18 in order to realize inherent functions. .

When the tablet terminal 100 is a multi-function mobile communication terminal such as a smartphone, the tablet terminal 100 is omitted here. However, the tablet terminal 100 includes a call processing unit, an imaging unit (such as a lens / image sensor) that performs imaging, and a broadcast image. Other parts (such as a tuner / demodulation unit), GPS, and sensors (such as an acceleration sensor and an inclination sensor) may be included as well as various components that are typically included in a smartphone.

The input unit 11 is for inputting an instruction signal for the user to operate the tablet terminal 100 via the touch panel. The input unit 11 is a touch surface that accepts contact with an indicator (indicating the screen position of the display unit 12, here, for example, a finger or a pen), and contact / non-contact between the indicator and the touch surface. (Approach / non-approach) and a touch sensor for detecting the contact (approach) position. The touch sensor may be realized by any sensor as long as it can detect contact / non-contact between the indicator and the touch surface. For example, it is realized by a pressure sensor, a capacitance sensor, an optical sensor, or the like.

The display unit 12 displays an object to be processed by the tablet terminal 100 (any display object such as an icon) and a processing result, and displays an operation screen for the user to operate the tablet terminal 100 using a GUI (Graphical (User 表示 Interface) screen. The display unit 12 is realized by a display device such as an LCD (Liquid Crystal Display).

In this embodiment, the input unit 11 and the display unit 12 are integrally formed, and these constitute a touch panel. Therefore, in such an embodiment, an object to be moved (operated) to indicate a screen position, that is, an operation body (here, a finger or a pen) is simultaneously positioned on the screen of the display unit 12. It is also an indicator that indicates

For example, when the touch panel of the tablet terminal 100 of the present invention is realized by a projected capacitive touch panel, specifically, the touch sensor has a transparent electrode pattern in a matrix shape made of ITO (Indium Tin Oxide) or the like. It is formed on a transparent substrate such as glass or plastic. When an indicator (such as a user's finger or pen) touches or approaches the touch sensor, the electrostatic capacity of a plurality of transparent electrode patterns in the vicinity changes. Therefore, the control unit 10 can detect the position where the indicator is in contact or approached by detecting a change in the current or voltage of the transparent electrode pattern.

In the following, the term “contact” when “contact detection”, “contact operation”, “contact position”, etc. is not only the state in which the indicator and the touch surface are in complete contact (contact), It also includes a state in which the indicator and the touch surface are close (approaching) to the extent that the touch sensor can detect.

The operation unit 13 is for the user to directly input an instruction signal to the tablet terminal 100. For example, the operation unit 13 is realized by an appropriate input mechanism such as a button, switch, key, or jog dial. For example, the operation unit 13 is a switch for turning on / off the power of the tablet terminal 100.

The external interface 14 is an interface for connecting an external device to the tablet terminal 100. The external interface 14 is realized by, for example, but not limited to, a socket for inserting an external recording medium (memory card or the like), an HDMI (High Definition Multimedia Interface) terminal, a USB (Universal Serial Bus) terminal, or the like. The The control unit 10 of the tablet terminal 100 can exchange data with an external device via the external interface 14.

The communication unit 15 communicates with an external device via a communication network. The communication unit 15 is connected to various communication terminals via a communication network, and realizes data transmission / reception between the tablet terminal 100 and the communication terminal. Further, when the tablet terminal 100 is a mobile communication terminal such as a smartphone, the communication unit 15 transmits / receives voice call data, e-mail data, and the like to / from other devices via the mobile phone network. To do.

The wireless communication unit 16 communicates with an external device wirelessly. The wireless communication unit 16 is not particularly limited, and may implement any wireless communication means such as infrared communication such as IrDA and IrSS, Bluetooth communication, WiFi communication, and a non-contact type IC card. A plurality of means may be realized. The control unit 10 of the tablet terminal 100 can communicate with devices in the vicinity of the tablet terminal 100 via the wireless communication unit 16, and can exchange data with these devices.

The sound output unit 17 outputs sound data processed by the tablet terminal 100 as sound, and is realized by a speaker, a headphone terminal, headphones, and the like.

The voice input unit 18 receives voice input generated outside the tablet terminal 100, and is realized by a microphone or the like.

The storage unit 19 includes (1) a control program executed by the control unit 10 of the tablet terminal 100, (2) an OS program, and (3) an application program for the control unit 10 to execute various functions of the tablet terminal 100, And (4) storing various data read when the application program is executed. Alternatively, (5) the control unit 10 stores data used for calculation and calculation results in the course of executing various functions. For example, the above data (1) to (4) are stored in a non-volatile storage device such as a ROM (read only memory), a flash memory, an EPROM (Erasable ROM), an EEPROM (Electrically EPROM), an HDD (Hard Disc Drive). Remembered. For example, the data (5) is stored in a volatile storage device such as a RAM (Random Access Memory). Which data is to be stored in which storage device is appropriately determined based on the purpose of use, convenience, cost, physical restrictions, and the like of the tablet terminal 100.

The control unit 10 performs overall control of each unit included in the tablet terminal 100. The control unit 10 is realized by, for example, a CPU (central processing unit). The functions of the tablet terminal 100 are such that the CPU as the control unit 10 reads a program stored in a ROM or the like into a RAM or the like and executes the program. It is realized by doing. Various functions (particularly, the operation screen display function of the present invention) realized by the control unit 10 will be described later with reference to other drawings.

[Appearance of tablet terminal]
FIG. 3 is a plan view showing the appearance of the tablet terminal 100. As illustrated in FIG. 3, the tablet terminal 100 includes an input unit 11 and a display unit 12 as a touch panel. The tablet terminal 100 includes an operation unit 13, an external interface 14, a wireless communication unit 16, an audio output unit 17, an audio input unit 18, and the like, although these are not essential components. For example, when the wireless communication unit 16 is realized by infrared communication means, an infrared transmission / reception unit is provided as the wireless communication unit 16 on the side surface of the tablet terminal 100.

FIG. 4 is a diagram illustrating a state when the user holds and operates the tablet terminal 100. More specifically, FIG. 4A is a diagram illustrating a state in which the tablet terminal 100 is gripped with one hand and is operated with that hand, and FIG. It is a figure explaining a mode that it is hold | gripped with a hand and operated with the other hand.

In the present embodiment, the tablet terminal 100 is a palm-sized information processing apparatus that can be held with one hand. As shown in FIG. 4A, the tablet terminal 100 is held with the thumb of the hand while holding the tablet terminal 100 with one hand. The touch surface of the input unit 11 can be operated. For example, when there is an icon to be operated at a position where the thumb does not reach, the tablet terminal 100 draws the icon near the thumb by flicking, and surrounds or taps the icon with the thumb. The icon can be selected.

Further, as shown in FIG. 4B, the user may hold the tablet terminal 100 with one hand and operate the touch surface of the input unit 11 with the finger of the other hand. Alternatively, although not shown, the tablet terminal 100 may be horizontally long, hold both sides with both hands, and operate the touch surface of the input unit 11 with the thumbs of both hands.

[Tablet function]
Next, the functional configuration of the tablet terminal 100 will be described. FIG. 1 is a functional block diagram illustrating a main configuration of the tablet terminal 100 according to the present embodiment.

As shown in FIG. 1, the control unit 10 of the tablet terminal 100 according to the present embodiment includes at least a contact information generation unit 21, an object specification unit 22, and related functions as functional blocks for realizing the operation screen display function of the present invention. An item extraction unit 23 and an operation screen processing unit 24 are provided.

Each functional block of the control unit 10 described above includes a CPU (central processing unit) that stores a program stored in a non-volatile storage device realized by a ROM (read only memory) or the like (RAM (random access memory)). It can be realized by reading and executing the above.

In addition, the storage unit 19 is specifically a frame map storage unit 41 or a related unit as a storage unit for reading or writing data when the above-described units of the control unit 10 execute the operation screen display function. An information storage unit 42 and an icon storage unit 43 are included.

The contact information generation part 21 processes the signal output from the touch sensor of the input part 11, and produces | generates contact information. The contact information includes at least contact coordinate information indicating the coordinate position of the contact position of the indicator (for example, a finger). Thereby, each part of the control part 10 can acquire the locus | trajectory of the movement of the said indicator from the said contact information. In the present embodiment, the contact time information indicating the time when the contact has occurred (the movement time information indicating the movement time of the indicator) is further associated with each point constituting the trajectory as necessary. Also good.

More specifically, the contact information generation unit 21 from when the touch sensor of the input unit 11 detects contact between the touch surface and the indicator (in this embodiment, a finger) until it detects non-contact. Acquires a signal output from the touch sensor. This signal includes information indicating that “contact” has been detected and information indicating the contact position. Based on this signal, the contact information generation unit 21 generates contact coordinate information indicating the contact position in coordinates. Generate. Further, the contact information generation unit 21 measures the time from when contact is detected until it becomes non-contact, and associates the contact time information with the contact coordinate information. The contact information generation unit 21 may acquire and use absolute time information held by the clock unit mounted on the tablet terminal 100. However, in the present embodiment, the contact information generation unit 21 detects contact. Then, timing is started and relative contact time information is obtained. For example, the contact information generation unit 21 measures the elapsed time with the time point when the contact is first detected (t0) as 0.00 seconds, and continues the measurement until the time point when the contact is finally detected (tn). Thus, the relative contact time information corresponding to the contact position may be acquired.
The contact information generation unit 21 generates contact information by associating the obtained contact time information with the contact coordinate information. In the present embodiment, the generated contact information is supplied to the object specifying unit 22 and used by the object specifying unit 22.

The object specifying unit 22 specifies an object selected by the user's contact operation. The object specifying unit 22 compares the contact information generated by the contact information generating unit 21 with the map information of the video frame displayed on the display unit 12 while the contact is occurring. Thereby, the object specification part 22 can specify the object enclosed by contact operation from the objects currently displayed on the display part 12. FIG.

The frame map storage unit 41 stores map information of the video frame output to the display unit 12 at the time of contact. The map information is information indicating the layout of the video frame displayed on the touch panel. Specifically, the map information includes information for individually identifying each object displayed, and information on the shape, size, and display position of each object. That is, the map information is obtained by plotting each object corresponding to the coordinate system of the touch panel.

FIG. 5 is a diagram for explaining the operation of the object specifying unit 22. More specifically, FIG. 5A is a diagram showing that the user has performed a contact operation of “enclosing” an object in order to select the target object. FIG. 5B is a diagram illustrating an example of contact information generated by the contact information generation unit 21 in accordance with the contact operation illustrated in FIG. FIG. 5C is a diagram illustrating an example of map information of a video frame displayed on the display unit 12 during a period from t0 to tn in which contact is detected.

As shown in (a) of FIG. 5, it is assumed that the user performs a contact operation of “enclosing” one of the objects 80 (here, photographs) displayed on the touch panel of the tablet terminal 100. For example, it is assumed that the contact operation is performed in a period from t0 to tn so that the contact point passes the position of the broken line in FIG.

The object specifying unit 22 acquires contact information as shown in FIG. 5B from the contact information generating unit 21. In the present embodiment, the coordinate system of the contact information corresponds to the coordinate system of the touch panel of the tablet terminal 100, and has the leftmost upper end of the panel as the origin. In FIG. 5B, among the trajectories of the “enclose” contact movement of the user, the start point is indicated as t0 and the end point is indicated as tn. However, contact time information may also be associated with each point in between.

The object specifying unit 22 acquires the map information shown in FIG. 5C (that is, the layout of the video frame displayed on the display unit 12 during the period from t0 to tn) from the frame map storage unit 41. Then, the object specifying unit 22 compares the contact information with the map information, and the object specifying unit 22 completely or substantially overlaps the area enclosed by the trajectory of the user's finger obtained from the contact information or the circumscribed rectangle of the area. 80 is identified as the selected object. In the example shown in FIG. 5, the object specifying unit 22 specifies “Picture 1” in FIG. 5C as the selected object. The object specifying unit 22 supplies information on the specified object to the related item extracting unit 23.

The related item extracting unit 23 extracts related items related to the object specified by the object specifying unit 22, that is, the object selected by the user. When an object is selected, an item deeply related to the selected object is extracted by the related item extraction unit 23.

For example, when the object is data such as “photo”, the photo is transferred to “display”, “edit”, “send as an email”, “peripheral device (TV etc.)” It is assumed that operations such as “print” and “print” are executed. Therefore, an item corresponding to an “action” executed on an object that is an “action target” may be extracted as a related item of the object.

Or, if the object is data such as “photograph”, it is assumed that the photo is sent to a specific person. Therefore, an item corresponding to the relationship of “action partner” when an action is executed on an object that is an action target may be extracted as a related item.

Alternatively, if the object is an “album” or “folder” that includes multiple photos or other data, the user may want the photos or data contained in the object. In this way, items belonging to the lower layer of the object may be extracted as related items.

The related information storage unit 42 stores related information indicating the relationship between objects and items. FIG. 6 is a diagram illustrating an example of related information stored in the related information storage unit 42.

As shown in FIG. 6, the related information is information in which at least “related items” are associated with each “object”. The association information indicates the association between the object and the item by this association.

When an object is specified by the object specifying unit 22, the related item extracting unit 23 refers to related information stored in the related information storage unit 42 and extracts items related to the specified object as related items. To do.

For example, as shown in FIGS. 5A to 5C, it is assumed that the object specifying unit 22 specifies that the selected object is “Photo 1”. In this case, since “Photo 1” is classified as “Photo” as an object, the related item extraction unit 23 extracts a related item group 60 associated with the object “Photo” from the related information.

Information on the related items extracted by the related item extracting unit 23 is supplied to the operation screen processing unit 24. The extracted related items are displayed so as to be selectable (for example, as icons) as items related to the previously selected object.

Although not limited to this configuration, in this embodiment, as shown in FIG. 6, an icon may be assigned to each “related item”. For example, the icon “1: TV” is associated with the related item “display on television (transfer to television)” associated with the object “photo”. The icon “1: TV” is, for example, an icon on which an illustration of a TV or the like is drawn, and is preferably a picture reminiscent of “sending a photograph to the TV for display”. Other related items are also assigned icons with appropriate patterns that recall the contents of the related items.

Based on such related information, the related item extracting unit 23 may supply the operation screen processing unit 24 with icons (or icon identification information) corresponding to the extracted related items. Thereby, the operation screen processing unit 24 can proceed to display the icon specified by the related item extraction unit 23.

In the present embodiment, the related information may not hold the “operation attribute” and “condition” information shown in FIG. 6 for each related item. Related items “operation attribute” and “condition” will be described in another embodiment or modification of the present invention described later.

The operation screen processing unit 24 performs processing (operation screen generation processing) for generating an operation screen for displaying an object and a related item (its icon) related to the selected object in a selectable manner for the user. It is.

FIG. 7 is a diagram showing a specific example of the icon image stored in the icon storage unit 43. As shown in FIG. As shown in FIG. 7, in this embodiment, each icon image can be identified by icon identification information. For example, the icon identification information “1: TV” is associated with an icon image depicting a TV. In addition, although not shown, a portrait or an avatar image of the person may be used as an icon representing personal information such as an acquaintance who often makes a call.

The operation screen processing unit 24 reads the icon images assigned to the related items extracted by the related item extraction unit 23 from the icon storage unit 43, and performs operations so that these are displayed at an appropriate position and an appropriate timing. A screen is generated and output to the display unit 12 via a display control unit (not shown).

Specifically, in the present embodiment, the operation screen processing unit 24 has a function of displaying an icon of a related item related to the object selected by the touch action “enclose” around the selected object. ing.

FIG. 8 is a diagram for explaining processing contents of the operation screen processing unit 24. More specifically, FIG. 8A is a diagram illustrating an example of an object display process executed by the operation screen processing unit 24, and FIG. 8B is a diagram illustrating the operation screen processing unit 24. It is a figure which shows an example of the icon arrangement | positioning of the related item performed.

In the present embodiment, the operation screen processing unit 24 first arranges the object 80 selected by the previous “enclose” contact operation at the center as shown in FIG. Next, as illustrated in FIG. 8B, the operation screen processing unit 24 arranges the icons of the extracted related items evenly around the object 80 in a ring shape. FIG. 8B shows an example in which, when eight related items are extracted, the operation screen processing unit 24 uniformly arranges eight icons along the outline of the oval ring. . Here, regarding the shape of the reference “ring” indicating the arrangement position of the icon, the oval shape is merely an example, and there is no intention to limit the shape of the ring of the present invention. Further, the “ring” does not necessarily mean a shape formed by a curve. For example, the operation screen processing unit 24 may define the shape of the ring as a circle, square, rectangle, or other polygon, or may include a complicated shape, irregular shape, or non-geometric shape. A figure having an outline that separates the outside from the outside is defined as a ring. Further, “ring” does not necessarily mean a closed curve. Even if the starting point and the ending point of the contour line of the ring are not completely coincident with each other, a contour line that largely separates the inside and the outside may be defined as the ring. The operation screen processing unit 24 arranges icons on the contour lines of the rings having any shapes defined as described above.

In addition, the broken line which shows the outline of a ring shown in FIG.8 (b) is the shape of the ring which the tablet terminal 100 hold | maintains as information inside, and is not actually displayed on the display part 12. FIG. May be. Similarly, the broken line indicating the outline of the ring in each of the drawings shown below is not actually displayed on the display unit 12.

Note that the order in which the icons are arranged is not particularly limited in the present embodiment. For example, the icons may be arranged in a clockwise direction from the top of the object 80 in the order extracted by the related item extraction unit 23.

FIG. 9 is a diagram illustrating a specific example of the operation screen obtained as a result of the operation screen generation process executed by the operation screen processing unit 24. The example shown in FIG. 9 is a specific example of the operation screen obtained when the object 80 (object “photo 1”) is enclosed, as in FIGS. 5 and 8.

The related item extraction unit 23 refers to the related information shown in FIG. 6, and icon identification information associated with the object of the photograph, that is, “1: TV”, “2: Printer”, “3: Mail”, “4” : “Photo display”, “5: Information display”, “6: Palette”, “7: Trash”, and “8: Memory card” are extracted.

The operation screen processing unit 24 reads a corresponding icon image from the icon storage unit 43 as shown in FIG. 7 based on the icon identification information extracted by the related item extraction unit 23. Then, the selected object 80 is arranged at the center, and the read icon image is arranged around it.

In the above description, the operation screen processing unit 24 performs the process of placing the object 80 in the center, but this is not an essential structure. However, in the present embodiment, the operation screen processing unit 24 arranges the icons of the related items of the object 80 around the object 80, so that a space for arranging the icons around the object 80 is ensured as widely and evenly as possible. Therefore, it is preferable to arrange the object 80 in the center.

[Operation screen display flow]
Next, a processing flow when the tablet terminal 100 executes the operation screen display function will be described. FIG. 10 is a flowchart showing a flow of operation screen display processing by the tablet terminal 100.

When the input unit 11 detects that an indicator (such as a user's finger) has touched the touch surface of the touch panel (YES in S101), the contact information generation unit 21 starts the finger from that time (t = t0). Acquisition of the contact coordinate information indicating the contact position is started, and this is acquired over time (S102). This tracking of the contact position is continued until no contact is detected between the touch surface and the finger (NO in S103). When contact is not detected in the input unit 11 (YES in S103), the contact information generation unit 21 obtains the contact coordinate information and the contact time information acquired from t = t0 to this time (t = tn). Corresponding contact information is generated (S104).

The object specifying unit 22 compares the contact information generated in S104 (for example, (b) in FIG. 5) with the map information (for example, (c) in FIG. 5) stored in the frame map storage unit 41. Then, the object overlapping the area surrounded by the user is specified as the selected object (S105). In the example shown in FIG. 5C, the object 80 “Photo 1” is specified.

The related item extraction unit 23 refers to the related information (for example, FIG. 6) in the related information storage unit 42 based on the object specified in S105, and extracts the related item of the specified object (S106). Alternatively, identification information of icons assigned to related items may be extracted.

The operation screen processing unit 24 acquires the icon image of the related item extracted in S106 from the icon storage unit 43 (for example, FIG. 7). Then, the acquired icon image is arranged around the object specified in S105 to generate an operation screen (S107). At this time, the operation screen processing unit 24 arranges the object in the center and arranges each icon in a ring shape around the object.

The video signal of the operation screen generated as described above is output to the display unit 12. As shown in FIG. 9, the operation screen is displayed on the display unit 12 of the tablet terminal 100.

According to the above-described configuration and method of the present invention, the tablet terminal 100 displays an icon around the object in response to an extremely natural and simple user contact operation of “enclosing” the object. Can be output.

The user can obtain an operation screen in which icons of related items are arranged so as to surround the object as a result. The positional relationship between these icons and the object matches the positional relationship between the finger trajectory and the object by the contact operation previously performed by the user. In addition, the finger trajectory obtained by surrounding is similar to a ring shape in which icons are arranged.

In other words, the transition from the phenomenon of “contacting the object to“ enclose ”the object” to the phenomenon of “obtaining an operation screen with icons arranged around the object” is a natural transition that does not contradict the user's intuition. It can be said that there is.

In addition, the tablet terminal 100 can preliminarily detect related items that the user will select after selecting an object, and can display the related items in a selectable manner for the user. Specifically, according to the above configuration of the present invention, the icons displayed around the object are all related item icons extracted as items related to the object. That is, after the user surrounds and selects an object, the user can immediately designate “motion”, “motion target”, “action partner”, and the like related to the object from surrounding icons.

From the above, the tablet terminal 100 can display the final result desired by the user in a natural flow that does not contradict the user's intuition while having a simple contact operation and a small number of operations. As a result, the tablet terminal 100 including the touch panel has an effect that it is possible to realize excellent operability.

<< Embodiment 2 >>
Another embodiment of the information processing apparatus according to the present invention will be described below with reference to FIGS. For convenience of explanation, members having the same functions as those in the drawings described in the first embodiment are denoted by the same reference numerals, and description of the same contents as those in the first embodiment is omitted.

In the first embodiment described above, each icon is arranged along a predetermined ring shape. However, in this embodiment, the operation screen processing unit 24 moves the ring shape according to the user's contact operation. It is the structure determined automatically. Thereby, according to a user's intuition, the result recalled more intuitively from contact operation | movement can be output.

[Tablet function]
FIG. 11 is a functional block diagram illustrating a main configuration of the tablet terminal 100 according to the present embodiment.

The tablet terminal 100 according to the present embodiment has a configuration in which the control unit 10 further includes a ring shape determining unit 30 as a functional block, as compared with the tablet terminal 100 (FIG. 1) of the first embodiment. The storage unit 19 further includes a contact information storage unit 44.

Here, in the present embodiment, the control unit 10 of the tablet terminal 100 is not essential, but further, as necessary, as a functional block, a gesture determination unit 25, an icon ranking determination unit 31, an animation determination unit 32, and The icon arrangement determining unit 33 may be provided. Note that the icon rank determination unit 31, the animation determination unit 32, and the icon arrangement determination unit 33 will be described in a later embodiment or modification, and the description thereof is omitted in this embodiment.

The contact information storage unit 44 stores the contact information generated by the contact information generation unit 21. In the first embodiment, the contact information is temporarily stored in a storage unit (such as a cache) (not shown) so that the object specifying unit 22 can be used. In the present embodiment, the contact information is used so that each part of the operation screen processing unit 24 and the operation screen processing unit 24 can perform operation screen generation processing (including processing for displaying icons). It is stored in the storage unit 44. Whether or not the contact information storage unit 44 is realized by a non-volatile storage device, that is, whether or not the contact information is stored in a nonvolatile manner depends on the purpose of the operation screen display function executed by the operation screen processing unit 24 and the assumed use. It is determined as appropriate from the environment, the purpose of use of the tablet terminal 100 itself, convenience, cost, physical restrictions, and the like.

The ring shape determination unit 30 determines the ring shape when the operation screen processing unit 24 arranges icons around the object. In the first embodiment, the operation screen processing unit 24 is configured to arrange icons in a predetermined ring shape. For example, the icon is arranged on an elliptical outline of a predetermined position and a predetermined size. In the present embodiment, the ring shape determination unit 30 determines a ring shape for arranging icons based on the contact information stored in the contact information storage unit 44. That is, the ring shape determination unit 30 determines the ring shape according to the finger trajectory obtained from the user's contact operation “enclose”. The ring shape determination unit 30 can directly change the shape of the trajectory of the “enclose” operation into a ring shape for arranging icons. Also, the size of the ring can be determined based on the size of the area enclosed by the “enclose” operation. Further, the position of the ring can be determined based on the position of the enclosed region. The ring shape determination unit 30 may further determine the ring shape based on the map information stored in the frame map storage unit 41. That is, the size and position of the ring may be determined according to the display position and size of the enclosed object.

In accordance with the ring shape determined by the ring shape determination unit 30, the operation screen processing unit 24 arranges the extracted icons. The ring shape information determined by the ring shape determination unit 30 may further include ring size information and / or ring position information.

As described above, it is preferable that the control unit 10 of the tablet terminal 100 further includes the gesture determination unit 25. If the contact operation (gesture) performed on the input unit 11 is assumed to be other than “enclose”, it is necessary to determine whether the gesture is “enclose” or another gesture. .

The gesture determination unit 25 determines what gesture it is for the contact operation performed on the input unit 11. For example, the gesture determination unit 25 can determine gestures such as “tap”, “flick”, “pinch”, “drag”, and “enclose”. A known technique can be appropriately employed as an algorithm for determining a gesture.

The gesture determination unit 25 instructs each unit of the control unit 10 to execute processing corresponding to the determined gesture according to the determination result.

In this embodiment, when the gesture determination unit 25 determines that the detected contact action is a gesture of “enclose”, the contact information generation unit 21 sends the generated contact information to the contact information storage unit 44. It is preferable to instruct to store. As a result, the operation screen processing unit 24 can refer to all information (such as the position, size, locus, contact time, contact point movement timing, etc.) of the gesture “enclose”, and the contact operation can be performed. If the gesture is other than the “enclose” gesture, it is possible to avoid unnecessary writing to the contact information storage unit 44. However, the contact information generation unit 21 may be configured to write all contact information in the contact information storage unit 44 regardless of the determination result of the gesture determination unit 25.

FIG. 12 is a diagram illustrating a specific example of contact information stored in the contact information storage unit 44. More specifically, FIG. 12A is a diagram showing that the user has performed a contact operation of “enclosing” an object in an arbitrary shape in order to select a target object. FIG. 12B is a diagram illustrating an example of contact information generated by the contact information generation unit 21 in accordance with the contact operation illustrated in FIG.

As shown in (a) of FIG. 12, the user refers to one of the objects (here, photographs) displayed on the touch panel of the tablet terminal 100 in an arbitrary shape (for example, “enclose” in a heart shape). Assume that the contact operation is executed, for example, when the contact point is executed in the period from t0 to tn so that the contact point passes the position of the broken line in FIG.

The gesture determination unit 25 acquires contact information as illustrated in FIG. 12B from the contact information generation unit 21. In FIG. 12B, among the traces traced by the user, the start point is indicated as t0 and the end point is indicated as tn. However, contact time information may also be associated with each point in between.

The gesture determination unit 25 determines that this contact operation is a gesture of “enclose” based on the contact information shown in FIG. The gesture determination unit 25 instructs the contact information generation unit 21 to store the contact information illustrated in FIG. 12B in the contact information storage unit 44. Thereby, each part of the operation screen process part 24 can refer to the contact information shown by (b) of FIG. 12 memorize | stored in the contact information storage part 44, when performing the process which displays an icon. .

FIG. 13 is a diagram illustrating an example of icon display of related items executed by the operation screen processing unit 24 including the ring shape determining unit 30.

First, the operation screen processing unit 24 can place the selected object 80 (photo 1) in the center. Next, in the present embodiment, the ring shape determination unit 30 acquires the contact information stored in the contact information storage unit 44. Based on the movement trajectory of the finger tip (contact point) obtained from the contact information, the ring shape determination unit 30 determines a shape that is the same as or similar to the trajectory as the ring shape for arranging the icons. To do. In the present embodiment, as an example, the ring shape determination unit 30 determines to place the ring at the center and to place it as large as possible on the touch screen. As shown in FIGS. 12A and 12B, the object 80 is surrounded by a heart shape. Therefore, the ring shape determination unit 30 determines the ring shape 81 so that the similar shape of the heart-shaped locus is arranged in the center of the screen as shown by the broken line in FIG.

The operation screen processing unit 24 arranges icons on the outline of the ring shape 81 determined by the ring shape determination unit 30 as shown in FIG. The operation screen processing unit 24 may arrange the icons at equal intervals, or may arrange the icons at arbitrary positions on the contour line according to another rule.

The operation screen processing unit 24 may determine the approximate shape of the trajectory as the ring shape when the trajectory has an extremely complicated shape. By rounding a fine and distorted line of a locus with a straight line or a curve, the amount of information defining the shape of the ring can be reduced, and the processing load for arranging icons can be reduced.

[Operation screen display flow]
FIG. 14 is a flowchart showing a flow of operation screen display processing by the tablet terminal 100 in the present embodiment.

When the input unit 11 detects that an indicator (such as a user's finger) has touched the touch surface of the touch panel (YES in S201), the contact information generation unit 21 starts the finger from that time (t = t0). The acquisition of the contact coordinate information indicating the contact position is started and acquired over time (S202). This tracking of the contact position is continued until no contact is detected between the touch surface and the finger (NO in S203). When contact is not detected in the input unit 11 (YES in S203), the contact information generation unit 21 obtains the contact coordinate information and the contact time information acquired from t = t0 to this time (t = tn). Corresponding contact information is generated (S204).

Here, the gesture determination unit 25 may determine the gesture of the contact operation based on the contact information (S205). In the present embodiment, if the determined gesture is not “enclose” (NO in S206), the gesture determination unit 25 instructs each unit of the control unit 10 to execute a process according to the determined other gesture. Each unit performs processing according to the determined gesture (S207).

On the other hand, when the determined gesture is “enclose” (YES in S206), the gesture determination unit 25 instructs the contact information generation unit 21 to store the contact information in the contact information storage unit 44. . The contact information generation unit 21 stores the contact information generated in S204 in the contact information storage unit 44 (S208).

The object specifying unit 22 includes contact information stored in the contact information storage unit 44 (for example, (b) of FIG. 12) and map information stored in the frame map storage unit 41 (for example, (c) of FIG. )) And the object overlapping the area surrounded by the user is specified as the selected object (S209). In the example shown in FIG. 5C, the object 80 “Photo 1” is specified.

The related item extraction unit 23 refers to the related information (for example, FIG. 6) in the related information storage unit 42 based on the object specified in S209, and extracts the related item of the specified object (S210). Alternatively, identification information of icons assigned to related items may be extracted.

Subsequently, the operation screen processing unit 24 first acquires the contact information generated in S204 from the contact information storage unit 44 (S211). The ring shape determination unit 30 of the operation screen processing unit 24 determines a ring shape for arranging icons from the acquired contact information (S212). For example, based on the heart-shaped locus shown in FIG. 12B, the heart-shaped similarity is determined as a ring shape (for example, the ring shape 81 in FIG. 13).

The operation screen processing unit 24 acquires the icon image of the related item extracted in S210 from the icon storage unit 43 (for example, FIG. 7). Then, the acquired icon image is arranged around the object specified in S209 to generate an operation screen (S213). At this time, the operation screen processing unit 24 places the object in the center and places each icon on the ring shape arranged around the object, that is, on the ring-shaped outline determined in S212 (for example, FIG. 13). The video signal of the operation screen generated as described above is output to the display unit 12.

According to the above-described configuration and method of the present invention, the tablet terminal 100 displays an icon around the object in response to an extremely natural and simple user contact operation of “enclosing” the object. Can be output. That is, the user can obtain, as a result, an operation screen in which icons of related items are arranged so as to surround the object.

Specifically, the user performs an operation of surrounding the object in an arbitrary shape with a free hand, and the locus at this time is held in the tablet terminal 100. Then, on the operation screen created by the tablet terminal 100, each icon is arranged so as to surround the object on the same or similar ring-shaped contour line as the obtained trajectory.

The positional relationship between these icons and the object matches the positional relationship between the object and the locus of the finger by the contact operation previously performed by the user. Further, the trajectory of the finger obtained by enclosing it matches the ring shape where the icon is arranged.

That is, when an object is enclosed, an operation screen in which icons are arranged "as enclosed" around the object can be obtained. It can be said that the transition of this event is a more natural flow that does not contradict the user's intuition. In addition, since the icons are arranged in the shape as enclosed by the user, the playability when the operation screen is displayed and the tablet terminal 100 is operated is improved. In addition, since the user can predict the arrangement of the icons, surround the object as he / she desires, and display the icons of the related items, the operability is further improved.

The icons displayed in the surroundings after selecting an object indicate related items that are deeply related to the object and are likely to be selected next.

In addition, about the structure of the tablet terminal 100 which concerns on this embodiment, ie, the structure which displays an icon along the similar or approximate ring of the locus | trajectory of a pointer's movement, this is shown as input part 11 and display part 12 May be applied to an information processing apparatus provided separately. For example, an information processing apparatus in which the input unit 11 is configured by an input device such as a mouse and the display unit 12 is configured by a display device such as an LCD can be considered. Here, the cursor displayed on the display unit 12 indicates the position of the display unit 12 on the screen. Then, when the user operates the mouse to perform an input operation, the cursor moves. Therefore, in such an embodiment, the mouse is the operating body, the cursor is the pointing body, and the pointing body displayed on the display unit 12 moves as the operating body moves. When the user selects an object by moving the cursor on the screen of the display unit 12 by operating the mouse, the information processing apparatus links the position of the cursor held in advance with the movement of the mouse, The trajectory of the movement is also held, and icons are displayed in a circle around the selected object. At this time, the information processing apparatus can make the shape of the ring a similar shape or an approximate shape of the acquired trajectory. According to the above configuration, an operation screen on which icons are arranged can be obtained in a shape as if the user moved the mouse, so that it is possible to improve playability when operating the information processing apparatus.

As the input unit 11 configured separately from the display unit 12, various input devices such as a keyboard, a joystick, a digitizer, a tablet, and a stylus pen can be employed in addition to the mouse.

<< Embodiment 3 >>
Another embodiment of the information processing apparatus according to the present invention will be described below with reference to FIGS. For convenience of explanation, members having the same functions as those in the drawings described in the first embodiment or the second embodiment are denoted by the same reference numerals, and the description overlapping with those in the first or second embodiment is omitted.

In each of the above-described embodiments, the extracted icons are arranged according to a predetermined rule, such as being arranged clockwise from the top of the object in the order of extraction. However, in this embodiment, it is assumed that a plurality of related items displayed around the object are selected (enclosed) at a time, and the related items that are particularly linked among the extracted related items are mutually connected. The operation screen processing unit 24 has a configuration for determining the arrangement of icons so that they are arranged near (next).

[Tablet function]
FIG. 15 is a functional block diagram illustrating a main configuration of the tablet terminal 100 according to the present embodiment.

Compared with the tablet terminal 100 (FIG. 1) of the first embodiment, the tablet terminal 100 according to the present embodiment further includes the cooperation processing execution unit 26 and the icon arrangement determination unit 33 as functional blocks. It is the structure equipped with.

Here, in the present embodiment, although not indispensable, the storage unit 19 further includes the contact information storage unit 44 and the control unit 10 as function blocks as necessary, as necessary. The structure provided with the shape determination part 30, the icon order | rank determination part 31, and the animation determination part 32 may be sufficient.

The icon arrangement determining unit 33 determines the icon arrangement. The icon arrangement determining unit 33 determines which icon is arranged at which position based on the attribute of the extracted related item.

In the tablet terminal 100 of the present invention, each icon is determined to be arranged on a predetermined or determined ring-shaped outline, but the other icons may be arbitrarily designed.

Therefore, the icon arrangement determination unit 33 determines where, how many, in what arrangement, or at what interval the icons are arranged on the contour line based on the attribute of each related item. Can do.

In particular, in the present embodiment, the icon arrangement determining unit 33 determines the icon arrangement by paying attention to the cooperation between the related items among the attributes of the related items.

FIG. 16 is a diagram illustrating an example of related information stored in the related information storage unit 42 of the present embodiment.

16 is different from the related information shown in FIG. 6 in that “linked” information is associated with each related item. Although not shown in FIG. 16, the related information of the present embodiment may further include “operation attribute” and “condition” information shown in FIG. 6 as an example of the attribute of the related item.

According to the related information example shown in FIG. 16, six related items from A to F are associated with a certain object “tool 1”. Therefore, when the object “tool 1” is selected by the user, the related item extracting unit 23 extracts items A to F shown in FIG. 16 as related items based on the related information.

Attribute “cooperation” is one of the attributes of the related items, and is information indicating the cooperation between the related items. As described above, the related items include those representing “operation”, those representing “operation partner”, and items representing “operation target”. For example, when “operation” is performed, an “operation target” that is a target for performing the operation may be required. In this case, it can be said that “operation” and “operation target” are linked. Further, depending on the content of the “action”, in addition to the “operation target”, an “operation partner” that is the partner to perform the operation may be required. In this case, it can be said that “operation”, “operation target”, and “operation partner” are linked.

In the example illustrated in FIG. 16, there is information indicating that the operation “A: send by mail”, the operation partner “D: best friend information”, and the operation target “E: photograph” are linked. , Stored in the attribute “cooperation”. In the present embodiment, the related items A, D, E, and F are associated with each other by the cooperation number “1”. The icon arrangement determination unit 33 grasps the cooperation of the related items based on the “cooperation” of each related item stored in the related information storage unit 42. Then, the icon arrangement determining unit 33 determines the icon arrangement by arranging icons of related items that cooperate with each other, or by bringing together icons of related items that cooperate with each other.

Note that the related items and the linkage numbers do not necessarily have to be one-to-one. That is, one related item may belong to one cooperation group (one cooperation number), or may belong to a plurality of cooperation groups. One linkage group may consist of a pair of (that is, two) related items, or may consist of three or more related items.

For example, the related item “E: photo” can be an “operation target” of both operations “A: send by e-mail” and “B: upload to blog”. It belongs to both “1” and the cooperation group “2” to which the related item B belongs.

In addition, for example, the related item “C: Call” does not have an “operation target”, and it only needs to be linked to the other party to call, that is, the “operation partner”. The cooperation group “3” is composed of a pair of related items C and D. On the other hand, the “action” of the related item “A: send by e-mail” requires “operation target” and “operation partner”. Furthermore, the related items “E: Photo” and “F: Movie” are “operation targets” that can be transmitted by e-mail. Therefore, the cooperation group “1” to which the related item A belongs includes the related items A, D, E, and F.

Note that, when one related item belongs to a plurality of linkage groups, it is preferable that priorities are set in advance for the linkage groups to which they belong in accordance with the strength of the linkage. For example, in the example illustrated in FIG. 16, the related item “E: photo” is associated with the cooperation number in the order of “1” and “2”. In other words, “E: photo” indicates that the “operation” “A: send by e-mail” is stronger than the “operation” “B: upload to blog”. On the other hand, “F: moving image” is associated with a cooperation number in the order of “2” and “1”. That is, “F: moving image” indicates that the “operation” “B: upload to blog” is stronger than the “operation” “A: send by e-mail”.

(A) and (b) of FIG. 17 is a diagram illustrating a procedure in which the icon arrangement determining unit 33 determines the icon arrangement.

Here, the object “tool 1” has already been specified by the object specifying unit 22, and the six related items “A” to “F” shown in FIG. 16 have been extracted by the related item extracting unit 23. And When a rule that six related items are evenly arranged is determined in advance, the icon arrangement determining unit 33 determines only the arrangement of icons according to “cooperation”.

Specifically, first, the icon arrangement determining unit 33 refers to the related information shown in FIG. 16 and classifies each related item according to the association with the cooperation number. For example, as shown in FIG. 17, each related item is classified into “linkage number 1” to “linkage number 3”. The related item may belong to a plurality of linkages. Further, the icon arrangement determining unit 33 classifies each related item into one of “operation”, “operation target”, and “operation partner”. The result of classifying each related item by the icon arrangement determining unit 33 is shown in FIG.

Note that the icon arrangement determining unit 33 classifies related items according to a predetermined priority order when a plurality of “operation targets” belong to one linkage group. For example, in the example illustrated in FIG. 17A, the related item “E” has priority over the related item “F” in the cooperation group “1”.

Next, as shown in FIG. 17B, the icon arrangement determining unit 33 determines the arrangement of the related items “A” to “F” in accordance with the cooperation of the related items. First, the icon arrangement determining unit 33 recognizes the cooperation of the related items A, D, and E with the cooperation number 1. Then, it is determined that these are arranged adjacent to each other in the order of “operation”, “operation object”, and “operation partner”. Specifically, when the icon arrangement determining unit 33 arranges the related item “A”, next, “D” is arranged next to “A”, and “E” is arranged next to the other of “A”. Deploy. Then, the icon arrangement determining unit 33 determines that “D” that has already been arranged is linked to “C” by the linkage number 3, and places “C” next to an empty “D”. Subsequently, the icon arrangement determination unit 33 determines that “E” that has already been arranged is a sub, but is associated with “B” by the cooperation number 2, and “B” ". Finally, the remaining “F” linked to “B” is arranged next to “B” that is free.

Note that it is conceivable that a plurality of related items are assigned to one arrangement position and confront each other. In such a case, it is only necessary to prioritize related items with stronger cooperation based on the above-described priority order. Further, it is conceivable that one related item is assigned to a plurality of arrangement positions and overlapped. In preparation for such a case, a priority order is determined for each cooperation group separately indicating which cooperation is preferentially held (for example, the cooperation number may indicate the priority order as it is). The arrangement position may be determined so that the related items of the high cooperation group are adjacent to each other as much as possible.

The operation screen processing unit 24 arranges icons corresponding to each related item according to the arrangement determined by the icon arrangement determination unit 33, and generates an operation screen.

FIG. 18 is a diagram showing a specific example of the operation screen obtained as a result of the operation screen processing unit 24 arranging icons according to the determination of the icon arrangement determining unit 33.

The icon 90 corresponds to the related item “A: send by e-mail” and is created with an icon image of “3: e-mail” associated with the related item A. The icon 91 corresponds to the related item “D: best friend information”, and is created with an icon image of “15: avatar” associated with the related item D. This icon image is preferably an avatar of a specific person registered as a best friend. Accordingly, the user can understand at a glance which person the icon indicates. The icon 92 corresponds to the related item “E: photo” and is created with an icon image of “16: thumbnail” associated with the related item E. This icon image is preferably an icon of the thumbnail image of the photo. Thereby, the user can understand what the photograph is at a glance.

As shown in FIG. 18, the

icons

90, 91, and 92 of related items A, D, and E that are linked to each other are arranged next to each other. In this way, the related icons are arranged close together so that the user can show the relationship among the icons even if the icons are equivalent to the object. It becomes possible to grasp easily. In addition, when it is possible to further enclose these three icons to be linked and cause another operation, the three icons are gathered close together so that they can be easily enclosed and the operability is improved. .

Alternatively, the icon arrangement determining unit 33 may determine the icon arrangement interval according to “cooperation”.

FIG. 19 is a diagram showing another specific example of the operation screen obtained as a result of the operation screen processing unit 24 arranging icons according to the determination of the icon arrangement determining unit 33.

In this example, as shown in FIG. 17, the icon arrangement determining unit 33 classifies each related item into “cooperation number 1” to “cooperation number 3”, and then, for each cooperation class, the related item belonging to that class. It is decided to arrange icons with a close interval between icons.

That is, as shown in FIG. 19, the icon arrangement determining unit 33 determines an arrangement in which a cluster of icons is formed for each category of cooperation. When one related item belongs to a plurality of classifications, a plurality of items may be arranged so that one related item belongs to several clusters.

Such an arrangement makes it difficult to see each icon, but it is possible to visualize the cooperation between icons (related items), and the user can easily grasp the cooperation. It becomes. Alternatively, when it is possible to further enclose a plurality of icons and cause another operation, it is easy to remind the user to “enclose a plurality of icons together”.

When there are a plurality of objects (or icons) specified by the object specifying unit 22, the cooperation processing execution unit 26 executes the cooperation processing in consideration of the cooperation of the related items corresponding to these icons. Is. The cooperation processing execution unit 26 may realize the cooperation processing by controlling each execution processing unit so that the plurality of execution processing units operate in conjunction with each other, or the plurality of processes are executed in an order that makes sense. As described above, the cooperative processing may be realized by controlling one or a plurality of execution processing units.

For example, as shown in FIG. 18 or FIG. 19, an icon 90 of the related item “A: Send by email”, an icon 91 of the related item “D: Best Friend Information”, and an icon of the related item “E: Photo” 92 is surrounded at once, the cooperation processing execution unit 26 controls an execution processing unit (not shown) to start a mail application, attaches a photograph indicated by the icon 92 to the mail, A series of processes of sending to the e-mail address is executed.

According to the above configuration, the tablet terminal 100 can easily present icons that operate in cooperation with each other and present them to the user. If a plurality of icons are surrounded, the tablet terminal 100 can be displayed. A plurality of processes can be executed together according to the linkage information.

Thereby, the tablet terminal 100 can display the final result desired by the user in a natural flow that does not contradict the user's intuition while having a simple contact operation and a small number of operations. As a result, the tablet terminal 100 including the touch panel has an effect that it is possible to realize excellent operability.

<< Embodiment 4 >>
Another embodiment of the information processing apparatus according to the present invention will be described below with reference to FIGS. For convenience of explanation, members having the same functions as those in the drawings described in the first, second, or third embodiment are denoted by the same reference numerals, and the same contents as those in the first, second, or third embodiments are described. Omitted.

In the third embodiment described above, the icon arrangement determining unit 33 is configured to determine the arrangement or interval of icons in consideration of the cooperation of related items. In the present embodiment, when the related item corresponds to a device in the vicinity of the tablet terminal 100, the icon arrangement determination unit 33 considers the positional relationship between the tablet terminal 100 and the peripheral device, and displays the icon. It is the structure which determines arrangement | positioning.

[Tablet function]
FIG. 20 is a functional block diagram illustrating a main configuration of the tablet terminal 100 according to the present embodiment.

Compared with the tablet terminal 100 (FIG. 1) of the first embodiment, the tablet terminal 100 according to the present embodiment further includes the device direction specifying unit 27 and the icon arrangement determining unit 33 as functional blocks. It is the structure equipped with. The storage unit 19 further includes a device direction storage unit 45.

Here, in the present embodiment, although not indispensable, if necessary, the storage unit 19 serves as the contact information storage unit 44, and the control unit 10 serves as the function block, the gesture determination unit 25, and the cooperation processing execution. The structure provided with the part 26, the ring shape determination part 30, the icon order | rank determination part 31, and the animation determination part 32 may be sufficient.

Further, in the present embodiment, the tablet terminal 100 is further provided with a position detection unit and a direction detection unit, although not shown in FIG. 2 as a hardware configuration.

The position detection unit is for detecting the current position of the tablet terminal 100. For example, an antenna for measuring a distance from a satellite or a base station, a signal received by the antenna, and a tablet is processed. And a signal processing unit that generates position information indicating the current position of the terminal 100. The position information acquired by the position detection unit is supplied to the device direction specifying unit 27 of the control unit 10. The position detection unit is realized by, for example, GPS (Global Positioning System) or an existing indoor position information acquisition system. For example, when an indoor position information acquisition system is employed, the position detection unit communicates with a plurality of base stations via a WLAN (Wireless Local Area Network) or the like, measures distances, Detect current position. Note that the antenna of the position detection unit may be shared with the antenna of the wireless communication unit 16 or may be a separate body.

The direction detection unit detects which direction the tablet terminal 100 faces in the horizontal direction. For example, the direction detection unit identifies the north direction by detecting the geomagnetism in the front-rear direction and the left-right direction. And a signal processing unit that processes a signal detected by the geomagnetic sensor and generates direction information indicating a direction in which the tablet terminal 100 is facing. The direction information acquired by the direction detection unit is supplied to the device direction identification unit 27 of the control unit 10. There are two-axis type and three-axis type geomagnetic sensors, but any type of geomagnetic sensor may be adopted in the tablet terminal 100 of the present invention. However, when a two-axis type geomagnetic sensor is employed, at least while the tablet terminal 100 is performing the process of specifying the direction of its own device, the user moves the surface of the touch panel (display unit 12) upward. Need to be held horizontally.

In this embodiment, the wireless communication unit 16 performs wireless communication with an external peripheral device and receives position information of each peripheral device. The position information of each peripheral device received by the wireless communication unit 16 is supplied to the device direction specifying unit 27.

The device direction specifying unit 27 specifies the direction of the tablet terminal 100 and specifies in which direction each peripheral device exists as viewed from the tablet terminal 100. Specifically, the device direction specifying unit 27 acquires position information from the position detection unit described above, and specifies the current position of the tablet terminal 100 based on the position information. And the apparatus direction specific | specification part 27 acquires each positional information from each peripheral apparatus via the wireless communication part 16, and specifies the present position of each peripheral apparatus. Moreover, the apparatus direction specific | specification part 27 acquires direction information from the above-mentioned direction detection part, and specifies the direction where the tablet terminal 100 is facing now. Thereby, the device direction specifying unit 27 generates device direction information indicating in which direction each peripheral device exists as viewed from the tablet terminal 100. The device direction information generated by the device direction specifying unit 27 is stored in the device direction storage unit 45 and is read by each unit of the operation screen processing unit 24 as necessary.

Next, the operation of the device direction specifying unit 27 will be described in detail with reference to the specific examples shown in FIGS. FIG. 21 is a diagram illustrating an example of a usage environment when the user U of the tablet terminal 100 is using the tablet terminal 100 in a room indoors. Although the direction in FIG. 21 is not particularly limited, it is assumed that the upper direction in FIG. 21 is north for easy understanding. In the following description, the side of the tablet terminal 100 main body shown in FIG. 3 where the wireless communication unit 16 is provided is the top (or upper side) of the tablet terminal 100 and the side of the tablet terminal 100 is the side where the voice input unit 18 is provided. Called the bottom (or bottom). And when the direction in the horizontal direction of the tablet terminal 100 is represented, the direction of the tablet terminal 100 (direction in which the tablet terminal 100 is facing) refers to the direction in which the upper side of the tablet terminal 100 is facing. That is, in the example shown in FIG. 21, since the upper side of the tablet terminal 100 faces north, the direction in which the tablet terminal 100 currently faces is “north”.

In the use environment shown in FIG. 21, the user U is holding and using the display unit 12 of the tablet terminal 100 so that the surface of the display unit 12 is horizontal. The tablet terminal 100 faces north. In the vicinity of the tablet terminal 100, the digital television 1 is in the direction of approximately north, the digital photo frame 3 is in the direction of approximately east, the printer 2 is in the direction of approximately east, and the personal computer 4 is in the direction of approximately southeast. Is installed. In the present embodiment, each peripheral device of the digital television 1, printer 2, digital photo frame 3, and personal computer 4 can wirelessly communicate with the tablet terminal 100, detects the current position of the own device, and detects the position of the own device. It has a function of transmitting information to the tablet terminal 100.

FIG. 22 is a diagram illustrating a specific example of the device direction information generated by the device direction specifying unit 27. The device direction specifying unit 27 communicates with each peripheral device via the wireless communication unit 16 and acquires position information of each peripheral device. The device direction specifying unit 27 plots each peripheral device in a predetermined coordinate system based on the acquired position information. In the present embodiment, as shown in FIG. 22, the device direction specifying unit 27 uses the coordinate system (Xr, Yr) corresponding to the room shown in FIG. ) And plot the current position of each peripheral device in this coordinate system. In the example shown in FIG. 22, each peripheral device is shown as a block for explanation, but the device direction specifying unit 27 grasps the position of each peripheral device by one point plotted in the coordinate system. Just do it.

The device direction specifying unit 27 further acquires the position information of the tablet terminal 100 from the position detection unit included in the device itself, specifies the current position of the tablet terminal 100, and the direction information of the tablet terminal 100 from the direction detection unit. And the direction of the tablet terminal 100 is specified. The position of the tablet terminal 100 may be grasped by one point plotted on the coordinate system. As illustrated in FIG. 22, the device direction specifying unit 27 plots the tablet terminal 100 at the specified position in the specified direction. Depending on the position and direction of the tablet terminal 100, the coordinate system (X, Y) corresponding to the touch panel of the tablet terminal 100 (the upper left corner of the touch panel screen is the origin) or the ring displayed on the display unit 12 of the touch panel Can be defined in the room coordinate system.

The device direction information generated by the device direction specifying unit 27 in this way is stored in the device direction storage unit 45 and read by the operation screen processing unit 24.

The operation screen processing unit 24 can specify the direction of each peripheral device by reading out the device direction information. In particular, the icon arrangement determining unit 33 of the operation screen processing unit 24 can determine the arrangement positions of icons related to each peripheral device so as to correspond to the direction in which the peripheral device exists.

For example, as shown in FIG. 22, it is assumed that an elliptical ring (thick broken line in FIG. 22) is defined based on the specified position and direction of the tablet terminal 100, with the tablet terminal 100 as a central object. In this case, the icon arrangement determination unit 33 sets the position of the icon related to the peripheral device at the position where the straight line connecting the point representing the position of the tablet terminal 100 and the point representing the position of the peripheral device and the outline of the ring intersect. The position can be determined.

FIG. 23 is a diagram illustrating a specific example of the related information related to the peripheral device stored in the related information storage unit 42. In the present embodiment, as to related items related to the peripheral device, as shown in FIG. 23, related information indicating the correspondence between the device ID of the peripheral device and the icon is further stored.

The operation screen processing unit 24 refers to the related information shown in FIG. 23 when related items related to the peripheral devices “digital television 1”, “printer 2”, “digital photo frame 3”, and “computer 4” are extracted. The icon image associated with each peripheral device is read from the icon storage unit 43. Then, the operation screen processing unit 24 generates an operation screen by arranging the read icon images at the arrangement positions determined by the icon arrangement determining unit 33.

FIG. 24 is a diagram illustrating a specific example of the operation screen obtained as a result of the operation screen generation process executed by the operation screen processing unit 24 in the present embodiment.

As shown in FIG. 24, the operation screen processing unit 24 arranges the icons of the peripheral devices in accordance with the icon arrangement position determined by the icon arrangement determining unit 33 based on the device direction information shown in FIG. Generate. Thereby, the positions of the icons of the peripheral devices displayed on the display unit 12 correspond to the actual directions of the peripheral devices shown in FIG. Specifically, the screen on the touch panel is north in the example shown in FIG. As seen from the center object, the digital TV 1 icon is substantially in the north direction, the digital photo frame 3 icon is in the substantially northeast direction, and the printer 2 icon is in the same direction as the actual existence direction. The icon of the personal computer 4 is displayed in the direction of approximately east and in the direction of approximately southeast.

According to the above configuration, the icon of the related item (peripheral device) can be displayed in a ring shape corresponding to the actual direction of the peripheral device. Therefore, it becomes possible to provide the user with an operation screen that is easier to understand without contradicting the user's intuition, and it is possible to realize a user interface that allows the next operation to be continued with a more natural flow. .

When the subsequent operation is a contact operation as described below, a particularly great effect can be obtained by the above configuration.

FIG. 25 is a diagram showing a specific example of the operation screen obtained following the operation screen shown in FIG. 24 as a result of the operation screen generation process executed by the operation screen processing unit 24.

The icon of each peripheral device shown in FIG. 24 is an icon indicating a related item of transferring a central object (here, an object “Photo 1”) to the peripheral device when the icon is selected. .

In the present embodiment, as a subsequent operation, a contact operation for selecting an icon is an operation of selecting an object 80 with a finger and dragging it to a desired icon (transfer destination peripheral device) and releasing it. Become.

The contact operation as described above is linked to the operation of transmitting data (object 80) to the peripheral device, and is an operation that is easy for the user to understand intuitively. In addition, according to the tablet terminal 100 of the present embodiment, icons are displayed in correspondence with the actual positional relationship between the tablet terminal 100 and each peripheral device (direction in which the peripheral device actually exists). For this reason, the user can intuitively grasp the relevance between his / her contact operation and information processing generated as a result. For example, if the user wants to transfer a photo to the digital television 1, the user may drag the photo object in the direction in which the digital television 1 is actually located.

As a result, it is possible to provide an operation screen that can be operated in a natural flow that does not contradict the user's intuition.

≪Modification≫
A tablet terminal 100 in which the above-described first to fourth embodiments are appropriately combined also falls within the scope of the present invention. In other words, the control unit 10 of the tablet terminal 100 according to each of the first to fourth embodiments includes the gesture determination unit 25, the cooperation processing execution unit 26, and the device direction specifying unit 27 when not essential in each embodiment. Or all may be provided. Moreover, the control part 10 operation screen process part 24 may be provided with the ring shape determination part 30, the icon order | rank determination part 31, the animation determination part 32, and the icon arrangement | positioning determination part 33 partially or all.

(Object and ring-shaped arrangement 1)
In each of the above-described embodiments, the operation screen processing unit 24 places the selected object (for example, the object 80 in FIGS. 8A and 8B) in the center of the touch panel screen when placing an icon. The ring shape determining unit 30 is configured to determine the position of the ring so as to arrange icons around the center object 80. However, the configuration of the present invention is not limited to the above. The operation screen processing unit 24 may be configured to maintain the display position of the selected object 80 as it is. Even in this case, the ring shape determination unit 30 may display the ring shape on the screen as shown in FIG. The position and size of the ring may be determined so as to be displayed large in the center. FIG. 26 shows a modification of the related item icon display method.

(Object and ring arrangement 2)
Alternatively, in the configuration in which the display position of the selected object 80 is maintained as it is, the ring shape determination unit 30 may cause the object 80 at the original position to be in the center of the ring as shown in FIG. The position and size of the ring shape may be determined. FIG. 27 shows a variation of the related item icon display method.

(Animated ring shape)
Or you may provide an animation with respect to the ring shape for arrange | positioning an icon. In this case, the operation screen processing unit 24 includes an animation determination unit 32.

The animation determination unit 32 determines an animation to be given to all objects to be arranged on the operation screen, that is, objects, icons, rings, and the like. Thereby, when displaying an object and an icon, the visual effect (namely, animation) can be given to how to display.

It should be noted that the animation determination unit 32 may give a visual effect such as fade-in (change of transparency) in addition to the movement of the object, icon, or ring.

Further, the animation determination unit 32 may move the icons so that the icons are finally terminated on the contour line from different places instead of causing the icon to be displayed in a circular shape to appear on the contour line of the ring from the beginning. . For example, after the icons are aggregated around the center of the ring, the animation determination unit 32 adds a movement that diffuses each icon, and finally moves each icon so that it is arranged on the outline of the ring. May be attached.

(Object and ring shape arrangement 3-ring shape animation)
In the configuration in which the display position of the selected object 80 is maintained as it is, when the ring shape is largely arranged at the center of the screen regardless of the object 80, it is said that “the icon is displayed surrounding the object 80”. There is a problem in that the relationship between the event and the finally obtained result (eg, FIG. 26) is diminished. On the other hand, in the configuration in which the display position of the object 80 is maintained as it is, when icons are displayed around the object 80 in association with the display position of the object 80, depending on the position where the object 80 is originally displayed, There is a problem in that the shape is not arranged to fit on the screen with a sufficient size, and as a result, the visibility of the icon is lowered.

Therefore, as shown in FIG. 28, the animation determination unit 32 may solve the above problem by giving an animation to a ring for arranging icons. Specifically, the animation determination unit 32 sets the ring shape determined by the ring shape determination unit 30 based on the original display position and size of the object 80 so that the ring shape is largely arranged in the center of the screen over a certain period of time. Add animation to the ring. As a result, the ring of icons once arranged small around the object 80 gradually changes its shape with the passage of time, and is finally arranged large in the center of the screen. 28A shows a state of an icon ring initially arranged around the object 80, and FIG. 28B shows a state in which the icon ring is being expanded. (C) shows a state in which the ring of icons is finally arranged largely at the center of the screen from (a) of FIG.

According to the above configuration, the icon can be displayed in a sufficient size without impairing the relevance between the user's contact operation and the displayed result.

Note that the animation determination unit 32 may give an animation that gradually increases the size of each icon according to the size of the ring, or the icon size is independent of the size of the ring and the size of the icon. The animation may be given such that the size of the icon is fixed and the interval between the icons is gradually increased.

(Icon display timing)
In each of the above-described embodiments, the operation screen processing unit 24 is configured to simultaneously arrange a plurality of icons around the selected object. However, the present invention is not limited to this, and when the operation screen processing unit 24 includes the animation determination unit 32, the animation determination unit 32 may determine the display timing of each icon.

FIG. 29 is a diagram showing a modification of the related item icon display method.

For example, it is assumed that contact information as shown in FIG. 29A is obtained from the user's “enclose” contact operation. The animation determination unit 32 refers to the contact information stored in the contact information storage unit 44, and recognizes that the “enclose” gesture has occurred clockwise from t0 to tn. The animation determination unit 32 determines that the first to eighth icons appear one by one at regular intervals in a clockwise direction so as to match this movement. For example, the animation determining unit 32 causes icons to appear in order at regular intervals such as (b), (c), (d), (e),... In FIG. The display timing of the icons is controlled so that the operation screen shown in FIG.

According to the above configuration, a result with substantially the same movement as the movement (clockwise) when the user is surrounded can be obtained, so that the relevance between the user's contact operation and the displayed result can be further increased. As a result, the operation screen can be provided in a natural flow that does not contradict the user's intuition.

Here, the icon arrangement determining unit 33 roughly matches the display position of the first icon with the start point of the finger trajectory (contact position at time t0), and the display position of the last icon and the end point (tn It is preferable that the contact position at the time is approximately the same.

This makes it possible to provide an operation screen with a more natural flow by further linking the contact operation and the result.

(Icon display timing 2)
Furthermore, it is preferable that the animation determination unit 32 sequentially causes each icon to appear in accordance with not only the finger movement (clockwise or counterclockwise) but also the speed at which the finger moves when the object is surrounded.

Specifically, it is assumed that contact information as shown in FIG. This contact information indicates a trajectory enclosed in a clockwise direction from time t0 to time tn. More specifically, the contact position (tip of the finger) is at the left of the object at the time ta, the contact position is at the upper left of the object at the time tb, and the contact position is at the upper right of the object at the time tc. It can be seen from this contact information.

Therefore, when the eight icons are arranged at equal intervals on the ellipse, the animation determination unit 32 causes the first icon to appear immediately below the object at time t0, and then matches the finger speed. The icon appears up to the left (third) of the object at the time ta, the icon appears up to the upper left (fourth) of the object at the time tb, and the object appears at the time tc. The icons appear up to the upper right (sixth), and finally, it is determined that all objects appear at the time point tn. FIG. 30B shows an operation screen at time t0. FIG. 30C shows the operation screen at the time point ta. FIG. 30D shows the operation screen at time tb. FIG. 30E shows an operation screen at the time tc. FIG. 30 (f) shows the operation screen at time tn.

According to the above configuration, a result is obtained that has substantially the same movement as the movement of the user (clockwise) and the speed of the movement. Therefore, the relationship between the user's contact operation and the displayed result is obtained. The operation screen can be provided in a natural flow that does not contradict the user's intuition.

(Icon arrangement 1)
The icon arrangement determining unit 33 determines which icon is arranged at which position based on the attribute of the extracted related item. Here, the icon arrangement determining unit 33 may determine the icon arrangement position of each related item in consideration of “operation attribute” which is one of the attributes of the related item.

The related item attribute "operation attribute" is information that indicates whether the operation has an attribute of an operation that works on another device or an operation that completes processing in the device itself. is there. As an example of the former operation, for example, an operation of transmitting data to another device can be cited. Examples of the latter operation include an operation for displaying data on the display unit of the own device.

Fig. 6 shows a specific example of "motion attribute". The related information table shown in FIG. 6 has a field of “operation attribute”. In the example shown in FIG. 6, when the operation of the related item is “operation of transmitting data to another device”, the identification information of “other device transmission operation” indicating that is stored in association with the related item. The Further, when the operation of the related item is “operation in which the processing is completed within the own device”, the identification information of “operation in the own device” indicating that is stored in association with the related item.

FIG. 31 shows a variation of the icon arrangement pattern of related items.

The icon arrangement determining unit 33 first refers to each “operation attribute” of the related item extracted by the related item extracting unit 23 from the related information storage unit 42. Then, as shown in FIG. 31, the icon arrangement determining unit 33 displays the icons (icons 70 to 72) of related items whose “operation attribute” is “other device transmission operation” in the upper half (or upper 3 minutes). 1). That is, it is arranged above the object 80. The icon arrangement determination unit 33 determines to arrange the icons (icons 74 to 76) of related items whose “operation attribute” is “in-machine operation” in the lower half (or the lower third) of the ring. . In other words, the icon arrangement determining unit 33 arranged below the object 80, when there is a related item that does not belong to any of “other device transmission operation” and “in-machine operation”, the remaining empty space ( Alternatively, it is determined that the icons (icons 73 and 77) of the related items are arranged in the middle third).

As shown in FIG. 25, when the data indicated by the central object 80 is transmitted to another device, the user next touches the object 80 and drags it to the icon indicating that the data is transmitted to the other device. The contact operation is performed.

Therefore, if the icon related to the operation of transmitting to another device is arranged above the center object, the user causes the tablet terminal 100 to perform the “operation of transmitting to another device (information processing)”. , You will make a gesture to move things away from your place.

The action of “sending to another machine (information processing)” is more intuitive than the gesture of bringing an object in front of you, and the gesture of moving an object away from you. The connection is considered strong.

According to the above configuration, the relevance between the user's contact operation and the obtained result can be further increased, and as a result, the operation screen can be provided in a natural flow that does not contradict the user's intuition.

Note that the “operation attribute” information may be a flag indicating whether or not it is “another device transmission operation”. For example, the icon placement determination unit 33 sets the “operation attribute” to “TRUE”, that is, “others”. It may be determined that the related item “machine transmission operation” is arranged in the upper half of the ring.

(Icon arrangement 2)
In addition, when each extracted related item includes information of “temporal element” as one of the attributes of the related item, the icon arrangement determining unit 33 displays the icon of each related item in time series. May be determined to be arranged.

For example, when the related item is data of a photo and the information of “photographing date / time” is associated with each photo, the icon placement determining unit 33 selects the first The icon of the photo (here, the thumbnail image of the photo is preferred) is placed in the center of the top of the ring, and starting from there, it is decided to arrange the remaining icons evenly in the order of shooting date and time. May be.

Alternatively, the icon arrangement determining unit 33 determines that the icons are arranged in accordance with the time indicated by the “temporal element” information of each related item, with the ring for arranging the icons as a clock face. May be.

FIG. 32 shows a variation of the icon arrangement pattern of related items.

For example, when an album is selected as the object 80, the icon arrangement determining unit 33 arranges each photo included in the album on the screen. At this time, the icon arrangement determining unit 33 arranges, for example, a photograph (group) 78 taken around 1 o'clock on the outline around 1 o'clock when the ring is regarded as a clock face. The photograph (group) 79 photographed around the time is arranged on a contour line around 3 o'clock.

According to the above configuration, the shooting date and time of a photo in the album can be expressed by the arrangement position of the photo, and the final result that is the user's purpose can be displayed with a simple contact operation and a small number of operations. It becomes.

("See" user usage)
As shown in FIGS. 4A and 4B, when the tablet terminal 100 is a small portable terminal and can be operated with one hand or both hands, when operating with one hand, operate with both hands. It is assumed that the area where the user can touch the screen with a finger is different from the time when the user is present. As shown in FIG. 4B, when operating with both hands, any area of the touch panel can be touched. On the other hand, as shown in FIG. 4A, when the operation is performed with one hand, the contact position is the area on the lower left side of the screen (when operated with the left hand) or the area on the lower right side of the screen (operated with the right hand). Tend to be biased. When the user is using the tablet terminal 100 in such a situation, there is a problem that the operation becomes complicated if an object or icon that requires a contact operation is displayed on the upper part of the screen or the lower part of the screen opposite to the hand. . This is because the user cannot touch the target object immediately and must perform an extra operation of rushing to a contactable area or switching to a two-handed operation.

Therefore, the tablet terminal 100 of the present invention solves the above problem by observing the usage status of the user. Specifically, the tablet terminal 100 of the present invention can be configured to detect the bias of the contact position of the finger and place the icon in an area where the user's finger is expected to reach immediately.

In this modification, the contact information generation unit 21 of the tablet terminal 100 does not depend on contact / non-contact switching, and further, regardless of whether the contact operation is a gesture of “enclose” (for example, In this configuration, contact information indicating a user's contact operation that occurred within a few seconds to several minutes) is generated and stored in the contact information storage unit 44.

FIG. 33 is a diagram for explaining the operation of the tablet terminal 100 of the present invention capable of presenting an operation screen in accordance with the usage status of the user. More specifically, FIG. 33A is a diagram illustrating an example of a situation where the user is operating with the left hand. FIG. 33B is a diagram showing a specific example of the contact information generated in accordance with the contact operation of FIG.

As shown in (a) of FIG. 33, for example, it is assumed that a user performs a contact operation by dragging a target object to a contactable region by a flick operation with a thumb and surrounding it.

In this modification, the contact information generation unit 21 generates contact information as shown in FIG. 33B for the above-described series of contact operations for a predetermined period (for example, the past few seconds to several minutes), The information is stored in the contact information storage unit 44. When the memory capacity of the contact information storage unit 44 that stores the locus is limited, the contact information generation unit 21 may be configured to delete the oldest locus every time a new locus is stored. .

Here, it is determined by the gesture determination unit 25 that the “enclose” gesture has occurred. The operation screen processing unit 24 refers back to the past several seconds to several minutes of contact information stored in the contact information storage unit 44, and detects whether or not the finger touch position is biased. In the example shown in FIGS. 33A and 33B, the locus of the finger is biased toward the area 82 on the lower left side of the screen. The operation screen processing unit 24 detects this bias and identifies the user accessible area as the area 82 on the lower left side of the screen. Note that the area 82 on the lower left side of the screen and the area 83 on the lower right side of the screen are defined in advance.

The ring shape determining unit 30 of the operation screen processing unit 24 determines the shape, size, and arrangement position of the ring so that the ring for arranging the icons fits in the area 82 on the lower left side of the screen.

FIG. 34 is a diagram illustrating an example of an operation screen when icons are arranged according to the ring shape determined by the ring shape determination unit 30. As shown in FIG. 34, since the related items of the selected object 80 are displayed so as to fit within the area 82 on the lower left side of the screen, the user does not need to drag the target icon with his thumb, The icon can be selected.

Note that the tablet terminal 100 determines whether or not the user is operating with the thumb based on the thickness of the line of the finger trajectory in order to determine the usage situation that the user is operating with one hand. If it is determined that the operation is performed, it may be determined that the operation is performed with one hand, and an icon may be displayed at the bottom of the screen. Alternatively, a sensor is provided in the casing of the tablet terminal 100, and the tablet terminal 100 determines whether the casing is gripped by four fingers or five fingers, and accordingly One-handed operation or two-handed operation may be determined.

In addition, the operation screen processing unit 24 of the tablet terminal 100 refers to the contact coordinate information of the area surrounded by the locus of the finger, identifies the locus area, and the locus area and the vicinity thereof are user accessible areas. It may be determined that an icon ring is arranged there.

Or the tablet terminal 100 assumes the use condition that a user operates with one hand, and when a user does not arrange | position the ring of an icon so that it may be settled in the area | region 82 or the area | region 83, a user simply sets the target icon, An operation screen that can be selected on a limited accessible area can be provided to the user.

FIG. 35A is a diagram illustrating a state in which the user rotates the ring of the icon by dragging on the contactable area, and FIG. 35B is a diagram illustrating the above-described contact operation “drag”. FIG. 6 is a diagram illustrating an example of an operation screen after the arrangement of icons is changed due to rotation of the ring.

In this modification, assuming that the accessible area is limited because the user operates with one hand, the tablet terminal 100 presents the icons arranged in a ring shape so as to be rotatable along the outline of the ring. It is the structure to do.

For example, it is assumed that after the object 80 is selected, an icon ring is displayed on the touch panel (display unit 12) in the arrangement shown in FIG. Since the user operates the tablet terminal 100 with only the left hand, the contactable area is limited (for example, the area 82 shown in FIG. 33B).

In such a situation, if the user wants to select the TV icon located above the screen of the touch panel, the user must switch to holding with two hands, or fix the tablet terminal 100 to a table or the like and open the left hand. It is inconvenient for the user.

Therefore, in the present modification, when the input unit 11 accepts the user's contact operation “drag”, the gesture determination unit 25 determines the direction in which the finger moves by dragging, and “the icon corresponding to that direction is determined. It is recognized that the instruction “rotate the ring” has been input. The animation determination unit 32 of the operation screen processing unit 24 outputs an operation screen to which an animation for rotating an icon is given to the display unit 12 according to the direction determined by the gesture determination unit 25.

For example, the user continues the contact action “drag (in this case, from the upper right to the lower left)” around the lower right of the ring on the operation screen shown in FIG. With this contact operation, each icon rotates clockwise on the outline of the ring. In this way, the user can drag the icon of the television arranged above the touch panel to the contactable area (for example, the area 82) as shown in FIG. 35 (b). The TV icon can be selected.

(Adjusting the number of icons displayed)
If the icon arrangement determining unit 33 determines that all icons extracted by the related item extracting unit 23 cannot be displayed in consideration of the determined ring shape size and icon size, the icon arrangement determining unit 33 should display the icons. It may be decided to reduce the number of icons.

Further, the icon arrangement determining unit 33 may determine the number of icons to be displayed based on the absolute size of the finger trajectory (or the enclosed area) with reference to the contact information. Thus, the user can intentionally adjust the number of icons to be displayed next by changing whether the object is surrounded by a smaller ring or a larger ring.

Alternatively, even if the related item extracted by the related item extracting unit 23, the icon arrangement determining unit 33 displays a specific related item as an icon based on the information of “condition” which is one of the attributes of the related item. You may decide not to.

Fig. 6 shows a specific example of "condition". The related information table shown in FIG. 6 has a “condition” field. The attribute “condition” of the related item is information indicating a condition for displaying the icon of the related item.

In the example shown in FIG. 6, “condition” “if there is attached information” is associated with each of the related items “display attached information of photo” and “display attached information of album”. This condition prescribes that if there is attached information in the photo, an icon is displayed, and if there is no attached information in the photo, no icon is displayed.

Therefore, the icon placement determination unit 33 must include attached information in the selected photo when the photo object is selected and the eight related items (related item group 60) shown in FIG. 6 are extracted. For example, it is determined that only seven related items except for the related item “display attached information of photograph” are arranged in the ring.

(Icon display priority)
Further, the operation screen processing unit 24 may include an icon rank determining unit 31.

The icon ranking determining unit 31 gives priority to the related items extracted by the related item extracting unit 23 based on the attributes of the related items.

For example, the related information includes a “selection frequency” field as one of the attributes of the related item. “Selection frequency” is information indicating the number of times that the related item has been selected by the user in the past.

The icon order determination unit 31 assigns priorities to each related item in the descending order of the “selection frequency” of the extracted related items. The icon arrangement determining unit 33 can determine the icon arrangement according to the priority order determined by the icon order determining unit 31.

For example, icons can be arranged clockwise from the highest priority, or related item icons with higher priority can be arranged in the upper half of the ring.

Alternatively, when the number of icons to be displayed is reduced, the icon arrangement determining unit 33 can reduce the icons in descending order of priority.

(When selecting multiple objects)
In each of the above-described embodiments and modifications, the case has been described in which the user selects one object from the object list screen (for example, FIG. 5A). However, the tablet terminal 100 of the present invention may accept selection of a plurality of objects on the object list screen. In this case, the tablet terminal 100 can extract related items according to the combination of the selected objects.

FIG. 36A is a diagram illustrating a state in which the user performs a contact operation of “enclosing” an object in order to select a plurality of target objects. FIG. 36B is a diagram illustrating a specific example of the operation screen generated by the operation screen processing unit in accordance with the contact operation illustrated in FIG.

36A, it is assumed that the user selects three photos, that is, an object 80, an object 84, and an object 85, from the object list screen. The object specifying unit 22 specifies that three photographs of the object 80, the object 84, and the object 85 are selected.

Here, even though there are a plurality of selected objects, all these objects are “photographs”. Therefore, the related item extraction unit 23 refers to the related information shown in FIG. 6 and extracts the related item group 60 associated with the object “photograph”.

36 (b), the operation screen processing unit 24 reads icon images corresponding to the related item group 60 from the icon storage unit 43 and arranges them in a ring around the object. However, since there are three selected objects here, the operation screen processing unit 24 displays the three selected photos (

objects

80, 84, 85) in the center of the ring. Thus, the user can obtain an operation screen in which what he / she actually encloses is displayed in the center, and icons related to these are displayed in a ring shape around them. This makes it possible to easily grasp the relationship between what is actually enclosed and what is obtained as a result. In the present modification, three photographs (

objects

80, 84, 85) arranged together in the center can be selected together by one touch operation. For example, when the user drags a picture of three photos displayed in the center once (for example, by dragging to the TV icon), the tablet terminal 100 executes a process of transferring all three pictures to the TV. Can be made. Since there is no need to drag each photo, the number of user actions can be reduced.

(A) of FIG. 37 is a diagram illustrating a state in which the user performs a contact operation of “enclosing” an object in order to select a plurality of different types of objects. FIG. 37B is a diagram illustrating a specific example of the operation screen generated by the operation screen processing unit in accordance with the contact operation illustrated in FIG. FIG. 38 is a diagram illustrating another example of the related information stored in the related information storage unit 42 in the present modification.

As shown in FIG. 37A, the user surrounds an object of a different type from the photograph, that is, the music file 86 in addition to the three photographs (

objects

80, 84, 85) from the object list screen. Suppose you select it. The object specifying unit 22 specifies that a total of four objects of three

photos

80, 84, and 85 and a music file 86 have been selected.

Here, the selected object combination is “photo” and “music file”. Therefore, the related item extraction unit 23 refers to the related information shown in FIG. 38 and selects the related item group 66 associated with the object “photo + music file”.

37 (b), the operation screen processing unit 24 reads out icon images corresponding to the related item group 66 from the icon storage unit 43, and arranges them in a ring around the object. However, here, since the selected object is three photos + one music file, the operation screen processing unit 24 brings together the icons of the three photos and music files selected in the center of the ring. Display in the center. Thus, the user can obtain an operation screen in which what he / she actually encloses is displayed in the center, and icons related to these are displayed in a ring shape around them. This makes it possible to easily grasp the relationship between what is actually enclosed and what is obtained as a result.

Furthermore, when an object consists of a plurality of types of data, relevant items suitable for the combination of the objects are extracted and their icons are displayed. For example, as in this modification, for the combination of “photo” and “music file”, there is a specialization related to “photo” such as “print (photo)” or “edit photo”. Items are not extracted, but related items using both “photos” and “music files” such as “display slide show” are extracted.

In the example shown in FIG. 37B, instead of the icon “2: Printer” representing the related item “Print (photo)”, “9: Slideshow playback” representing the related item “Display slideshow”. Icon 87 is displayed around the center object. In the operation screen shown in FIG. 37B, when the user drags an object in which three photos and one music file are grouped together to the icon 87, the tablet terminal 100 reproduces the music file 86 and displays the icon. The three photos dragged to 87 are displayed in a slide show.

As described above, the tablet terminal 100 according to the present invention can accept a contact operation in which a plurality of objects or different types of objects are selected by enclosing them all at once. Appropriate related items can be extracted according to the combination of objects.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention.

Note that the information processing apparatus and operation screen display method of the present invention can also be expressed as follows.

In an information processing apparatus including a touch panel, at least a part of a contact motion acquisition unit that acquires a trajectory of movement of the indicator that has moved on the touch panel and a region surrounded by the trajectory acquired by the contact motion acquisition unit Is identified by the object identifying means with reference to an object identifying means for identifying an object including the object as a selected object and a related information storage unit for storing the object and an item related to the object in association with each other Related item extraction means for extracting an item associated with an object as a related item, and an operation for displaying on the touch panel the icons of related items extracted by the related item extraction means arranged on the outline of a ring An information processing apparatus comprising screen processing means

In an operation screen display method in an information processing apparatus including a touch panel, the operation is displayed surrounded by a contact motion acquisition step of acquiring a trajectory of movement of the indicator that has moved on the touch panel, and a trajectory acquired in the contact motion acquisition step. Referring to the object specifying step for specifying an object including at least a part in the area as the selected object, and the related information storage unit for storing the object and an item related to the object in association with each other, the object A related item extraction step for extracting an item associated with the object specified in the specific step as a related item, and icons of related items extracted by the related item extraction step are arranged side by side on the outline of the ring. Operation screen processing step displayed on the touch panel Operation screen display method which comprises and.

[Example of software implementation]
Finally, each block of the tablet terminal 100, in particular, the contact information generation unit 21, the object identification unit 22, the related item extraction unit 23, the operation screen processing unit 24, the gesture determination unit 25, the cooperation processing execution unit 26, and the device direction identification unit 27, the ring shape determining unit 30, the icon order determining unit 31, the animation determining unit 32, and the icon arrangement determining unit 33 may be configured by hardware logic, or realized by software using a CPU as follows. May be.

That is, the tablet terminal 100 includes a CPU (central processing unit) that executes instructions of a control program that realizes each function, a ROM (read only memory) that stores the program, a RAM (random access memory) that develops the program, A storage device (recording medium) such as a memory for storing the program and various data is provided. An object of the present invention is to provide a recording medium on which a program code (execution format program, intermediate code program, source program) of a control program of the tablet terminal 100, which is software that realizes the functions described above, is recorded so as to be readable by a computer. This can also be achieved by supplying the tablet terminal 100 and reading and executing the program code recorded on the recording medium by the computer (or CPU or MPU).

Examples of the recording medium include tapes such as magnetic tapes and cassette tapes, magnetic disks such as floppy (registered trademark) disks / hard disks, and disks including optical disks such as CD-ROM / MO / MD / DVD / CD-R. Card system such as IC card, IC card (including memory card) / optical card, or semiconductor memory system such as mask ROM / EPROM / EEPROM / flash ROM.

Further, the tablet terminal 100 may be configured to be connectable to a communication network, and the program code may be supplied via the communication network. The communication network is not particularly limited. For example, the Internet, intranet, extranet, LAN, ISDN, VAN, CATV communication network, virtual private network, telephone line network, mobile communication network, satellite communication. A net or the like is available. Further, the transmission medium constituting the communication network is not particularly limited. For example, even in the case of wired such as IEEE 1394, USB, power line carrier, cable TV line, telephone line, ADSL line, etc., infrared rays such as IrDA and remote control, Bluetooth ( (Registered trademark), 802.11 wireless, HDR, mobile phone network, satellite line, terrestrial digital network, and the like can also be used. The present invention can also be realized in the form of a computer data signal embedded in a carrier wave in which the program code is embodied by electronic transmission.

The present invention can be widely applied to information processing apparatuses including an input unit and a display unit. For example, but not limited to this, a digital TV, personal computer, smartphone, tablet PC, notebook computer, mobile phone, PDA (Personal Digital Assistant), electronic book reader, electronic dictionary, portable, provided with an input unit and a display unit -It can be used suitably for a home game machine, an electronic blackboard, etc. Furthermore, if the present invention is applied to an information processing apparatus including a touch panel, it is possible to realize even better operability.

1 Digital TV 2 Printer 3 Digital Photo Frame 4 Personal Computer 10 Control Unit 11 Input Unit (Touch Panel)
12 Display unit (touch panel)
13 Operation unit 14 External interface 15 Communication unit (communication unit)
16 Wireless communication unit (communication unit)
17 audio output unit 18 audio input unit 19 storage unit 21 contact information generation unit (trajectory acquisition means / contact operation acquisition means)
22 Object identification part (object identification means)
23 Related Item Extraction Unit (Related Item Extraction Unit)
24 Operation screen processing unit (operation screen processing means)
25 Gesture determination unit (gesture determination means)
26 Cooperation process execution part (Cooperation process execution means)
27 Device direction specifying unit (device direction specifying means)
30 Ring shape determining unit (ring shape determining means)
31 Icon ranking determining unit (icon ranking determining means)
32 Animation determination unit (animation determination means)
33 Icon arrangement determining unit (icon arrangement determining means)
41 Frame map storage unit 42 Related information storage unit 43 Icon storage unit 44 Contact information storage unit 45 Device direction storage unit 100 Tablet terminal (information processing apparatus)

Claims

A trajectory acquisition means for acquiring a trajectory by which the indicator pointing to the position of the screen of the display unit has moved;
Object specifying means for specifying, as a selected object, an object at least part of which is included in the area surrounded by the trajectory acquired by the trajectory acquiring means;
A related item extracting unit that extracts an item associated with the object identified by the object identifying unit as a related item with reference to a related information storage unit that associates and stores the object and an item associated with the object; ,
An information processing apparatus, comprising: operation screen processing means for displaying icons on the related items extracted by the related item extracting means side by side on the outline of the ring and displaying them on the display section.
The operation screen processing means includes
The information processing apparatus according to claim 1, wherein the position and size of the ring are determined so that icons are arranged around the selected object.
The operation screen processing means includes
The information processing apparatus according to claim 1, wherein a trajectory acquired by the trajectory acquisition unit, or a similar shape or an approximate shape thereof, is determined as the shape of the ring.
The operation screen processing means includes
The information processing apparatus according to any one of claims 1 to 3, wherein a timing at which each icon is displayed is individually determined.
The trajectory acquisition means includes
The elapsed time from the start of the movement of the indicator is measured, and the movement time information indicating the elapsed time is associated with and held in at least some of the points constituting the trajectory,
The operation screen processing means includes
The information processing apparatus according to claim 4, wherein the icons are sequentially displayed in a clockwise direction or a counterclockwise direction in accordance with a moving direction of the indicator determined from the trajectory and the moving time information.
The operation screen processing means includes
6. The information processing apparatus according to claim 5, wherein a position that is relatively the same as the position of the start point of the locus in the ring is determined as a display position of the first icon.
The operation screen processing means includes
The information processing apparatus according to claim 6, wherein a timing for sequentially displaying the icons is determined so as to correspond to a moving speed of the indicator that forms the locus.
In the related information storage unit, when a plurality of related items can be processed continuously or simultaneously, when the plurality of related items are linked, the link information indicating the presence or absence of the link is the related item. Every one is remembered,
The operation screen processing means includes
The information processing apparatus according to any one of claims 1 to 7, wherein among related items extracted by the related item extracting unit, icons of related items having cooperation are arranged next to each other. .
The information processing apparatus further includes:
A position detector that acquires position information indicating the position of the device itself;
A direction detection unit that acquires direction information indicating the direction of the device itself;
A communication unit that communicates with the peripheral device of its own device to obtain position information of the peripheral device;
Based on the position information of the own device acquired by the position detection unit and the position information of the peripheral device acquired by the communication unit, the positional relationship between the own device and the peripheral device is specified, and the direction detection unit Device direction specifying means for specifying in which direction each peripheral device exists with respect to the own device by specifying the direction of the own device by the direction information acquired by
The operation screen processing means includes
9. An icon arrangement position corresponding to a related item of the peripheral device is determined so as to correspond to a direction in which the peripheral device exists specified by the device direction specifying means. The information processing apparatus according to any one of the above.
The operation screen processing means includes
Among the related items extracted by the related item extracting means, the related item icon associated with an operation attribute indicating that the related information storage unit is an item related to an operation that acts on a device other than the own device. The information processing apparatus according to claim 1, wherein the information processing apparatus is arranged above the selected object.
The trajectory acquisition means includes
Obtaining a locus of the indicator that has occurred in a predetermined period until the indicator moves around the object displayed on the display unit;
The operation screen processing means includes
When it is determined that the trajectory acquired during the predetermined period is biased toward a specific area on the screen of the display unit, the position of the ring is determined so that an icon is arranged in the specific area. The information processing apparatus according to any one of claims 1 to 10, wherein the information processing apparatus is characterized in that:
The operation screen processing means includes
The position of the ring is determined such that an icon is arranged in a region enclosed by the locus of the indicator on the screen of the display unit or a specific region including the enclosed region. The information processing apparatus according to any one of claims 1 to 10.
The input unit and the display unit included in the information processing apparatus constitute a touch panel.
The information processing apparatus according to any one of claims 1 to 12, wherein the trajectory acquisition unit acquires a trajectory of the movement of the indicator that has moved on the touch panel.
The input unit included in the information processing apparatus inputs an instruction to move the cursor displayed on the display unit to the information processing apparatus.
The information processing apparatus according to any one of claims 1 to 12, wherein the trajectory acquisition unit acquires a trajectory of movement of a cursor as the indicator.
An operation screen display method in an information processing apparatus,
A trajectory acquisition step for acquiring a trajectory of movement of an indicator that indicates the position of the screen of the display unit included in the information processing apparatus;
An object specifying step for specifying, as a selected object, an object including at least a part of the region surrounded by the track acquired in the track acquiring step;
A related item extracting step of extracting an item associated with the object identified in the object identifying step as a related item with reference to a related information storage unit that associates and stores the object and an item associated with the object When,
An operation screen display method comprising: an operation screen processing step of arranging and displaying icons of related items extracted in the related item extraction step on a ring outline and displaying them on the display unit.
A control program for causing a computer to function as each unit of the information processing apparatus according to any one of claims 1 to 14.
A computer-readable recording medium on which the control program according to claim 16 is recorded.