WO2014061098A1

WO2014061098A1 - Information display device and display information operation method

Info

Publication number: WO2014061098A1
Application number: PCT/JP2012/076680
Authority: WO
Inventors: 英一有田; 下谷　光生
Original assignee: 三菱電機株式会社
Priority date: 2012-10-16
Filing date: 2012-10-16
Publication date: 2014-04-24
Also published as: JPWO2014061098A1; CN104737221B; US20150234572A1; JP5738495B2; DE112012007167T5; CN104737221A

Abstract

This information display device includes a display unit which has a display surface, an input unit which receives user operations, and a control unit. If a user operation is a gesture operation associated with a pan modification function which controls the display information on the display surface in a control direction set according to the gesture direction, then the control unit displays a composite icon on the display surface. The composite icon is a composite of multiple icons each associated with a pan modification function but having different associated control directions. If the user operation is an execution instructing operation on any of the icons in the composite icon, then the control unit executes the pan modification function with the control direction associated with the icon on which the execution instructing operation was performed. By this means, a high level of convenience can be provided.

Description

Information display device and display information operation method

The present invention relates to an information display device and a display information operation method.

The following Patent Documents 1 and 2 disclose devices using a touch panel.

In the portable information device disclosed in Patent Document 1, when a finger is moved on a screen on which a map image is displayed, the map image is moved and displayed in the moving direction of the finger. According to this, the execution instruction of the scroll operation and the scroll amount are simultaneously input according to the operation history of the finger. Further, when the two fingers are moved away, the map image enlargement instruction and the enlargement amount are input according to the operation history. Further, when two fingers are brought close to each other, a map image reduction instruction and a reduction amount are input according to the operation history. When another finger is rotated around one finger as an axis, a rotation instruction and a rotation amount of the map image are input according to the operation history.

In the navigation device of Patent Document 2, a smooth scroll operation icon for performing continuous smooth scroll processing of a map image is displayed. Specifically, this icon is displayed in the lower right part or the lower left part on the map image in accordance with the position of the driver's seat. When the arrow indicating the predetermined direction of the icon is touched with a finger, the navigation map image moves at a high speed in the direction of the arrow for the duration of the touch.

Further, in the navigation device of Patent Document 2, when an area other than the smooth scroll operation icon is touched, a touch scroll process for moving the point to the center of the screen is executed. Further, when a region other than the smooth scroll operation icon is touched and the finger being touched is moved on the screen, drag scroll processing for sequentially moving the map according to the locus of the finger is executed.

As described above, in the navigation device of Patent Document 2, an area for executing smooth scroll processing (that is, an icon for smooth scroll operation) and an area for executing touch scroll processing and drag scroll processing (that is, smooth scroll). Area other than the operation icon). As a result, compared to the case where the smooth scroll operation and the touch scroll operation are very similar (for example, the difference between the two operations is only the difference in the time of touching the screen), It is said that the scroll process can be instructed accurately.

JP 2000-163031 A JP 2010-32546 A

In the portable information device of Patent Document 1, for example, when scrolling a long distance, the same finger movement must be repeated many times. The same applies to operations other than scrolling.

The navigation device of Patent Document 2 has poor operability when the smooth scroll operation and the touch scroll operation are very similar (for example, when the difference between the two operations is only the difference in the time of touching the screen). It has been proposed to solve this problem. Since the timing for performing the scroll operation depends on the intention of the user, the smooth scroll operation icon must always be displayed so that it can be used at any time.

Also, in the smooth scroll operation icon, each arrow portion indicating the map moving direction needs to be large enough to be touched with a finger. If an arrow portion for eight directions is provided in the icon as described in Patent Document 2, the icon for smooth scroll operation becomes large.

大きな The fact that a large icon is always displayed can imagine a situation that hinders the visibility of the map. In such a case, the adoption of the icon for the smooth scroll operation may decline and the convenience may be reduced.

An object of the present invention is to provide an information display device and a display information operation method having high convenience.

An information display device according to an aspect of the present invention includes a display unit having a display surface, an input unit that receives a user operation, and a control unit. When the user operation is a gesture operation associated with a screen movement deformation type function that controls display information on the display surface in a control direction set according to the gesture direction, the control unit displays the composite icon on the display surface. Display. The composite icon is a composite of a plurality of icons that are all associated with the screen movement deformation type function but have different control direction assignments. When the user operation is an execution instruction operation for any of the icons in the composite icon, the control unit executes the screen movement deformation type function in the control direction assigned to the icon for which the execution instruction operation has been performed.

According to the above aspect, the composite icon is called on the display surface by the gesture operation, and the screen movement deformable function associated with the gesture operation can be executed in various control directions using the composite icon. . For this reason, if a composite icon is used, the number of repeated gesture operations can be reduced to reduce the operation burden. Moreover, if a composite icon is used, the control direction can be selected as appropriate. Thereby, high convenience can be provided.

Also, a composite icon is displayed simply by performing a gesture operation associated with the function to be executed. That is, a composite icon corresponding to the function intended by the user is automatically displayed. For this reason, high convenience can be provided.

Also, for example, in a situation where the user continues to view the display information without performing an operation, the composite icon is not called, so the display information is not hidden by the composite icon.

The objects, features and advantages of the present invention will become more apparent from the following detailed description and the accompanying drawings.

It is a block diagram which illustrates an information display device. It is a perspective view which illustrates an input and display part. It is a conceptual diagram of 1-point touch operation. It is a conceptual diagram of a two-point touch operation. It is a conceptual diagram of drag operation. It is a conceptual diagram of flick operation. It is a conceptual diagram of pinch out operation (two-point movement type). It is a conceptual diagram of pinch out operation (one point movement type). It is a conceptual diagram of pinch-in operation (two-point movement type). It is a conceptual diagram of pinch-in operation (one-point movement type). It is a conceptual diagram of scroll operation. It is a conceptual diagram of display size change operation (enlargement operation and reduction operation). It is a conceptual diagram of rotation operation. It is a figure which illustrates a scroll compound icon. It is a conceptual diagram of a scroll compound icon. It is a figure which illustrates a display size change compound icon. It is a figure which illustrates a rotation compound icon. It is a block diagram which illustrates a control part. It is a flowchart which illustrates the process until a composite icon is displayed. It is a conceptual diagram of end point conditions. It is a conceptual diagram of composite icon call operation. It is a figure explaining the 1st example of the display position of a composite icon. It is a figure explaining the 2nd example of the display position of a composite icon. It is a figure explaining the 3rd example of the display position of a composite icon. It is a figure explaining the 4th example of the display position of a composite icon. It is a figure explaining the 1st example of how to obtain the extension line of a gesture locus. It is a figure explaining the 2nd example of how to obtain the extension line of a gesture locus. It is a figure explaining the 3rd example of how to obtain the extension line of a gesture locus. It is a figure explaining the 5th example of the display position of a composite icon. It is a flowchart which illustrates the process after a composite icon is displayed. It is a conceptual diagram which shows the relationship between a gesture amount or gesture speed, and the control amount or control speed of display information. It is a figure which illustrates the control amount of display information. It is a figure which illustrates the control speed of display information. It is a figure which illustrates size change of a composite icon. It is a flowchart which illustrates the process regarding the deletion of a composite icon. It is a figure which illustrates the compound icon which combined the scroll icon and the display size change icon. It is a conceptual diagram of an element connection display format.

<Overview of overall configuration>
FIG. 1 illustrates a block diagram of an information display device 10 according to an embodiment. According to the example of FIG. 1, the information display device 10 includes a display unit 12, an input unit 14, a control unit 16, and a storage unit 18.

Display unit 12 displays various information. The display unit 12 drives each pixel based on, for example, a display surface configured by arranging a plurality of pixels in a matrix and image data acquired from the control unit 16 (in other words, each pixel's Driving device for controlling the display state). The image displayed on the display unit 12 may be a still image, a moving image, or a combination of a still image and a moving image.

The display unit 12 can be configured by a liquid crystal display device, for example. According to this example, a display area of a display panel (here, a liquid crystal panel) corresponds to the display surface, and a drive circuit externally attached to the display panel corresponds to the drive device. Note that a part of the driver circuit may be incorporated in the display panel. In addition to the liquid crystal display device, the display unit 12 can be configured by an electroluminescence (EL) display device, a plasma display device, or the like.

The input unit 14 receives various information from the user. The input unit 14 includes, for example, a detection unit that detects an indicator used by the user for input, and a detection signal output unit that outputs a result detected by the detection unit to the control unit 16 as a detection signal. .

Here, a case where the input unit 14 is configured by a so-called contact-type touch panel is illustrated, and the input unit 14 may be referred to as a “touch panel 14” below. The touch panel may be referred to as a “touch pad” or the like. Moreover, the case where the said instruction | indication used for an input is a user's finger | toe (specifically fingertip) is illustrated.

The detection unit of the touch panel 14 provides an input surface on which a user places a fingertip, and detects the presence of a finger on the input surface by a sensor group provided for the input surface. In other words, an area where a finger can be detected by the sensor group corresponds to an input area where a user input can be received. In the case of a contact-type touch panel, the input area corresponds to an input surface of a two-dimensional area.

The sensor group may be any of electrical, optical, mechanical, etc., or a combination thereof. Various position detection methods have been developed, and any of them may be adopted for the touch panel 14. In addition to detecting the position of the finger, a configuration capable of detecting the pressing force of the finger on the input surface may be employed.

• The position of the fingertip on the input surface can be specified from the combination of the output signals of each sensor. The identified position is expressed by coordinate data on coordinates set on the input surface, for example. In this case, when the finger is moved on the input surface, the coordinate data indicating the finger position changes, so that the movement of the finger can be detected by a series of coordinate data acquired continuously.

Note that the finger position may be expressed by a method other than coordinates. That is, the coordinate data is an example of finger position data for expressing the position of the finger.

Here, an example is given in which the detection signal output unit of the touch panel 14 generates coordinate data indicating the finger position from the output signals of the sensors, and transmits the coordinate data to the control unit 16 as a detection signal. However, for example, the conversion to coordinate data may be performed by the control unit 16. In such an example, the detection signal output unit converts the output signal of each sensor into a signal in a format that can be acquired by the control unit 16, and transmits the obtained signal to the control unit 16 as a detection signal.

Further, as shown in the perspective view of FIG. 2, the input surface 34 of the touch panel 14 (see FIG. 1) and the display surface 32 of the display unit 12 (see FIG. 1) are overlapped, in other words, the input surface 34. A structure in which the display surface 32 is integrated is illustrated. With such an integrated structure, the input / display unit 20 (see FIG. 1), more specifically, the touch screen 20 is provided.

According to the integrated structure of the input surface 34 and the display surface 32, the input surface 34 and the display surface 32 are identified to the user, giving the user the feeling of performing an input operation on the display surface 32. For this reason, an intuitive operation environment is provided. In view of this point, for example, the expression “the user operates the display surface 32” may be used.

The control unit 16 performs various processes and controls in the information display device 10. For example, the control unit 16 analyzes information input from the touch panel 14, generates image data according to the analysis result, and outputs the image data to the display unit 12.

Here, the control unit 16 includes a central processing unit (for example, configured with one or a plurality of microprocessors) and a main storage unit (for example, one or a plurality of storage devices such as ROM, RAM, flash memory, etc.). ). According to this example, various functions are realized by the central processing unit executing various programs stored in the main storage unit (in other words, by software). Various functions can be realized in parallel.

Various programs may be stored in the main storage unit of the control unit 16 in advance, or may be read from the storage unit 18 during execution and stored in the main storage unit. The main storage unit is used not only for storing programs but also for storing various data. In addition, the main storage unit provides a work area when the central processing unit executes the program. The main storage unit provides an image holding unit for writing an image to be displayed on the display unit 12. The image holding unit may be referred to as “video memory”, “graphic memory”, or the like.

Note that all or part of the processing and control performed by the control unit 16 may be configured as hardware (for example, an arithmetic circuit configured to perform a specific calculation).

The storage unit 18 stores various information. Here, the storage unit 18 is provided as an auxiliary storage unit used by the control unit 16. The storage unit 18 can be configured using one or more storage devices such as a hard disk device, an optical disk, a rewritable and nonvolatile semiconductor memory, and the like.

<User operations and their functions>
Before describing a more specific configuration and processing of the information display device 10, user operations on the touch panel 14 will be described.

User operations are roughly classified into touch operations and gesture operations based on finger movements. Hereinafter, the touch operation and the gesture operation may be referred to as “touch” and “gesture”, respectively. The touch operation is an operation in which at least one fingertip is brought into contact with the input surface of the touch panel and the contacted finger is moved away from the input surface without being moved on the input surface. On the other hand, the gesture operation is an operation in which at least one fingertip is brought into contact with the input surface, and the contacted finger is moved on the input surface (in other words, slid), and then released from the input surface. .

The coordinate data detected by the touch operation (in other words, finger position data) is basically static and static. On the other hand, the coordinate data detected by the gesture operation changes with time and is dynamic. According to such a series of changing coordinate data, the point where the finger starts moving and the point on the input surface, the locus from the moving start point to the moving end point, the moving direction, the moving amount, the moving speed, the moving acceleration, Etc. can be acquired.

FIG. 3 is a conceptual diagram illustrating a one-point touch operation (also simply referred to as “one-point touch”) as a first example of the touch operation. In FIG. 3 and FIGS. 4 to 10 described later, a top view of the input surface 34 is shown in the upper stage, and a side view or a sectional view of the input surface 34 is shown in the lower stage.

As shown in FIG. 3, in the one-point touch, the user makes a point contact with one finger on the input surface 34. In FIG. 3, touch points (in other words, finger detection points) are schematically indicated by black circles. Such an illustration technique is also used in the drawings described later. A black circle may be actually displayed on the display surface.

１ One-point touch can be classified into single tap, multi-tap and long press operations, for example. Single tap is an operation of tapping the input surface 34 once with a fingertip. A single tap is sometimes simply referred to as a “tap”. Multi-tap is an operation of repeating a tap a plurality of times. A double tap is a typical multi-tap. The long press is an operation for maintaining the point contact of the fingertip. These operations can be identified by, for example, the duration and number of times of finger contact (in other words, finger detection).

FIG. 4 is a conceptual diagram illustrating a two-point touch operation (also simply referred to as “two-point touch”) as a second example of the touch operation. The two-point touch is basically the same as the one-point touch except that two fingers are used. For this reason, it is possible to perform each operation of tap, multi-tap, and long press, for example, by two-point touch. In the two-point touch, two fingers of one hand may be used, or one finger of the right hand and one finger of the left hand may be used. The positional relationship between the two fingers is not limited to the example in FIG.

It is also possible to perform touch operations with three or more fingers.

FIG. 5 is a conceptual diagram illustrating a drag operation (also simply referred to as “drag”) as a first example of the gesture operation. Dragging is an operation of shifting the fingertip while it is placed on the input surface 34. The moving direction and moving distance of the finger are not limited to the example in FIG.

In FIG. 5, the movement start point of the finger is schematically shown by a black circle, the movement end point of the finger is schematically shown by a black triangle, the direction of movement of the finger is expressed by the direction of the triangle, and the black circle The trajectory is represented by a line connecting the triangle and the black triangle. Such an illustration technique is also used in the drawings described later. Note that the black circle, the black triangle, and the locus may be actually displayed on the display surface.

FIG. 6 is a conceptual diagram illustrating a flick operation (also simply referred to as “flick”) as a second example of the gesture operation. The flick is an operation for quickly paying the fingertip on the input surface 34. The moving direction and moving distance of the finger are not limited to the example in FIG.

In flicking, unlike dragging, the finger leaves the input surface 34 while moving. Here, since the touch panel 14 is a contact type, the finger movement after leaving the input surface 34 is not detected in principle. However, for example, it is possible to calculate the moving speed of the finger at the final point detected from a series of changes in coordinate data obtained while the finger moves on the input surface 34. A flick can be identified when the moving speed is equal to or higher than a predetermined threshold (referred to as “drag / flick identification threshold”).

Further, for example, from the detected movement direction, movement speed, and movement acceleration of the finger at the final point, the point where the finger finally arrives after it leaves the input surface 34 (more specifically, the point is defined as the input surface 34). Can be estimated. Note that this estimation process can be interpreted as a process of converting a flick into a virtual drag.

Therefore, the information display apparatus 10 treats the estimated point as the end point of finger movement. In this example, the estimation process may be executed by the touch panel 14 or may be executed by the control unit 16.

However, such an estimation is not performed, and the information display device 10 may be modified so that a point away from the input surface 34 is handled as an end point of finger movement.

FIG. 7 is a conceptual diagram illustrating a pinch out operation (also simply referred to as “pinch out”) as a third example of the gesture operation. Pinch out is an operation of moving two fingertips away on the input surface 34. Pinch out is also called “pinch open”.

FIG. 7 illustrates the case where both two fingers are dragged. On the other hand, as shown as a fourth example of the gesture operation in FIG. 8, one fingertip is fixed on the input surface 34 (in other words, one fingertip maintains a touch state), and only the other fingertip is held. It is also possible to pinch out by dragging. 7 and 8 are referred to as “two-point movement type”, and the method in FIG. 8 is referred to as “one-point movement type”.

FIG. 9 is a conceptual diagram illustrating a pinch-in operation (also simply referred to as “pinch-in”) as a fifth example of the gesture operation. Pinch-in is an operation of bringing two fingertips closer on the input surface 34. Pinch-in is also referred to as “pinch close”. FIG. 9 illustrates a two-point movement type pinch-in, but FIG. 10 illustrates a one-point movement type pinch-in as a sixth example of the gesture operation.

Here, pinch-out and pinch-in are collectively referred to as “pinch operation” or “pinch”, and the direction of finger movement is referred to as “pinch direction”. In this case, when the pinch direction is a direction in which the interval between fingers is increased, the pinch operation is particularly referred to as pinch out. Conversely, when the pinch direction is a direction in which the interval between fingers is reduced, the pinch operation is particularly called pinch-in.

In pinch-out and pinch-in, two fingers of one hand may be used, or one finger of the right hand and one finger of the left hand may be used. Further, the positional relationship, the moving direction, and the moving distance of the two fingers are not limited to the examples in FIGS. Further, in the one-point movement type pinch-out and pinch-in, the finger to be dragged is not limited to the examples of FIGS. 8 and 10. It is also possible to pinch out and pinch in using flicks instead of dragging.

Each user operation is associated with a specific function. Specifically, when a user operation is detected, a process associated with the user operation is executed by the control unit 16, thereby realizing a corresponding function. In view of this point, user operations can be classified based on functions to be realized.

For example, a double tap performed on an icon on the display surface 32 is associated with a function for executing a program or command associated with the icon. In this case, the double tap functions as an execution instruction operation.

Also, as illustrated in FIG. 11, dragging performed on display information (a map image is illustrated in FIG. 11) is associated with a scroll function for scrolling the display information. In this case, the drag operation functions as a scroll operation. Note that it is also possible to perform scrolling by flicking instead of dragging.

Further, as illustrated in FIG. 12, the pinch out and the pinch in performed on the display information (a map image is illustrated in FIG. 12) changes the size (in other words, the scale) of the information display. Associated with function. In this case, pinch-out and pinch-in function as a display size change operation (may be referred to as a “display scale change operation”). More specifically, in the example of FIG. 12, pinch out corresponds to an enlargement operation, and pinch in corresponds to a reduction operation.

In addition, as illustrated in FIG. 13, when two fingers are dragged with respect to the display information (a map image is illustrated in FIG. 13) to draw a circle while maintaining a distance, The drag is associated with a function that rotates the information display. In this case, the two-point movement type rotary drag functions as a rotation operation. In addition, you may employ | adopt the rotation drag performed with three or more fingers. Further, the function of associating may be changed according to the number of fingers to be rotated and dragged.

Here, it is also possible to assign multiple types of functions to one user operation. For example, a double tap may be assigned to a folder opening operation for opening a folder associated with an icon in addition to the above execution instruction operation. Further, the drag may be assigned to a scroll function and a drawing function. When a plurality of types of functions are assigned to one user operation, each function is switched according to the operation target, usage status (in other words, usage mode), and the like.

It is also possible to assign multiple types of user operations to one function. For example, the execution instruction function for the icon may be associated with double tap, long press, and flick. In this case, a program or the like associated with the icon can be executed by any of double tap, long press and flick. Further, for example, the scroll function may be associated with both dragging and flicking. Further, for example, the rotation function may be associated with both the two-point movement type rotation drag and the one-point movement type rotation drag.

Here, the functions associated with user operations are roughly classified into screen movement deformation type and non-movement deformation type from the viewpoint of screen movement and deformation. In the following, for example, a gesture operation associated with a screen movement deformation type function may be expressed as “a screen movement deformation type function gesture operation”.

The screen movement deformation type function associated with the gesture operation is a function of controlling (in other words, manipulating) display information on the display surface in a control direction set according to the gesture direction. The screen movement deformation type function includes, for example, a slide function, a display size change function, a rotation function, and a bird's eye view display function (more specifically, an elevation angle and depression angle change function). The slide function can be classified as a screen movement function. Further, if the rotation function is viewed from the viewpoint of angle movement, the rotation function can be classified as a screen movement function. Further, the display size changing function and the bird's eye view display function can be classified into screen deformation functions.

More specifically, in the scroll function, the scroll direction (that is, the control direction) is set according to the gesture direction (for example, the drag direction or the flick direction), and the display information is scrolled in the scroll direction.

In the display size changing function, when the gesture direction (for example, pinch direction) is the enlargement direction, the control direction is set to the enlargement direction, and when the gesture direction is the reduction direction, the control direction is set to the reduction direction. Then, the display information size is changed in the set control direction.

In the rotation function, the control direction is set to the right rotation direction when the gesture direction (for example, the rotation direction in the rotation drag) is the right rotation direction, and the control direction is set to the left rotation direction when the gesture direction is the left rotation direction. The counter is set to the left rotation direction, and the display information is rotated in the set control direction.

Note that the screen movement / deformation function may control the display information by using not only the gesture direction but also the gesture amount (for example, the length of the gesture trajectory). Specifically, the control amount (for example, the scroll amount, the display size change amount, and the rotation amount) of the display information may be set larger as the gesture amount is larger.

Further, the screen movement deformation type function may control the display information using the gesture speed in addition to or instead of the gesture amount. Specifically, the display information control speed (eg, scroll speed, display size change speed, and rotation speed) may be set higher as the gesture speed is higher.

On the other hand, the non-moving deformation type function does not use the gesture direction for realizing the function even if it is associated with the gesture operation. For example, even if a flick to an icon is associated with an execution instruction function of a specific program, the function belongs to the non-moving deformation type. For example, when dragging is used for the drawing function and the handwritten character input function, only the trajectory corresponding to the dragging is displayed, and the display information is not controlled according to the dragging direction.

Note that the user operations and the functions realized thereby are not limited to the above examples.

<Composite icon>
The information display apparatus 10 employs a characteristic operation method called a composite icon. A composite icon is a composite of a plurality of icons. The composite icon is displayed on the display surface when a gesture operation of the screen movement deformation type function is performed. Then, when an execution instruction operation is performed on each icon in the composite icon, the screen associated with the gesture operation related to the appearance of the composite icon (in other words, the opportunity to display the composite icon) A moving deformation type function is executed. In other words, any icon in the composite icon is associated with the screen movement deformation type function associated with the gesture operation related to the appearance of the composite icon. However, different control directions are assigned to the icons in the composite icon. For this reason, when an execution instruction operation is performed on any one of the composite icons, the screen movement deformation type function is executed with the control direction assigned to the icon.

実行 One point touch operation is exemplified as the execution instruction operation for the composite icon. For example, the screen movement deformation type function associated with the composite icon may be continuously executed while the state of touching any one of the composite icons continues. In this case, the control amount (for example, the scroll amount) in the screen movement deformation type function is increased by the long press operation rather than the tap operation. However, it is not limited to this example. For example, the screen movement deformation type function may be continuously executed while tapping is continuously performed.

FIG. 14 shows a scroll composite icon 72 as an example of the composite icon. According to the example of FIG. 14, the scroll composite icon 72 has eight icons 72a to 72h. All of the icons 72a to 72h are associated with the scroll function, but different scroll directions are assigned to the icons 72a to 72h. Specifically, the icon 72a is assigned to scroll in the upward direction, the icon 72b is assigned to scroll in the upper right 45 ° direction, and the icon 72c is assigned to scroll in the right direction. The scroll directions of the

icons

72d, 72e, 72f, 72g, and 72h are respectively assigned to the lower right 45 ° direction, the lower direction, the lower left 45 ° direction, the left direction, and the upper left 45 ° direction. In view of this point, in the example of FIG. 14, the icons 72a to 72h are drawn with a design in which the vertices of the vertically long triangles face the scroll direction. However, the design of the scroll composite icon 72 is not limited to the illustrated example. The icons 72a to 72h may also be referred to as scroll icons 72a to 72h.

FIG. 15 shows a conceptual diagram of the scroll composite icon. Here, a drag 70 that is an example of a gesture operation is associated with a scroll function that is an example of a screen movement deformation type function. When the drag 70 is performed in an arbitrary direction (rightward in the example of FIG. 15), a scroll composite icon 72 capable of executing a scroll function that is a function associated with the drag 70 is displayed. According to the scroll composite icon 72, each of the icons 72a to 72h (see FIG. 14) receives an instruction to execute the scroll function and executes the scroll function in the scroll direction set as described above.

Here, in the example of FIG. 15, by dragging in the right direction, the map image slides in the right direction, and the following map image appears from the left side of the display surface. In this case, the slide direction of the map image is the same right direction as the drag direction. On the other hand, the scroll direction of the map image is generally expressed as the left direction. That is, in the scroll function and the slide function, the scroll direction and the slide direction, which are control directions, differ by 180 °. However, both the scroll function and the slide function are common in that the control direction is set according to the gesture direction (drag direction in the example of FIG. 15) or the direction designated by the scroll icons 72a to 72h.

FIG. 15 illustrates an example in which the left scroll icon 72g (see FIG. 14) is touched. However, by touching icons 72a to 72f and 72h (see FIG. 14) in other scroll directions, It is also possible to scroll in the direction of.

It should be noted that composite icons that receive the display size change function and the rotation function, which are other examples of the screen movement deformation type function, will be referred to as “display size change composite icon” and “rotation composite icon”, respectively. More specifically, as illustrated in FIG. 16, the display size change composite icon 80 includes two display

size change icons

80a and 80b. The two display

size change icons

80a and 80b may be referred to as an enlarged icon 80a and a reduced icon 80b, respectively, depending on the display size change direction (that is, the control direction). Further, as illustrated in FIG. 17, the rotation composite icon 84 includes two

rotation icons

84 a and 84 b. The two

rotation icons

84a and 84b may be referred to as a right rotation icon 84a and a left rotation icon 84b, respectively, depending on the rotation direction (that is, the control direction). The design of the

composite icons

80 and 84 is not limited to the illustrated example.

According to the information display device 10, a composite icon is called on the display surface by a gesture operation, and the screen movement deformable function associated with the gesture operation can be executed in various directions using the composite icon. For this reason, if a composite icon is used, the number of repeated gesture operations can be reduced to reduce the operation burden. If a composite icon is used, the control direction of the display image can be selected as appropriate.

Also, a composite icon is displayed simply by performing a gesture operation associated with the function to be executed. That is, a composite icon corresponding to the function intended by the user is automatically displayed.

<Configuration Example of Control Unit 16>
FIG. 18 illustrates a block diagram of the control unit 16. In FIG. 18, the display unit 12, the input unit 14, and the storage unit 18 are also illustrated for explanation. According to the example of FIG. 18, the control unit 16 includes an input analysis unit 40, an overall control unit 42, a first image forming unit 44, a first image holding unit 46, a second image forming unit 48, A two-image holding unit 50, an image composition unit 52, a composite image holding unit 54, and a composite icon management unit 56 are included.

The input analysis unit 40 analyzes the user operation detected by the input unit 14 and identifies the user operation. Specifically, the input analysis unit 40 acquires coordinate data detected along with a user operation from the input unit 14, and acquires user operation information from the coordinate data. The user operation information is, for example, information such as the type of user operation, the start and end points of finger movement, the trajectory from the start point to the end point, the moving direction, the moving amount, the moving speed, and the moving acceleration.

For identifying the type of user operation, for example, the difference between the start point and the end point can be distinguished from a predetermined threshold value (referred to as a “touch / gesture identification threshold value”) to identify the touch operation and the gesture operation. It is. Further, as described above, the drag and the flick can be identified from the finger moving speed at the end of the trajectory.

Also, for example, when two drags are identified at the same time, pinch out and pinch in can be identified from the moving direction. In addition, when the two drags draw a circle while maintaining a distance, it is possible to identify that the rotation drag has been performed. When a drag and a one-point touch are identified at the same time, it can be identified that the pinch-out, the pinch-in, and the rotational drag are one-point moving types.

The overall control unit 42 performs various processes in the control unit 16. For example, the overall control unit 42 associates a position on the input surface of the input unit 14 with a position on the display surface of the display unit 12. According to this, the touch position in the touch operation, the gesture locus in the gesture operation, and the like are associated on the display surface. With such association, it is possible to identify the position on the display surface where the user operation is intended. Such association can be realized by a so-called graphical user interface (GUI) technique.

Further, the overall control unit 42 identifies a function desired by the user, that is, a user instruction based on, for example, user operation information and function identification information. The function identification information is, for example, information in which an association between a user operation and a function to be executed is defined through the operation status information. The operation status information includes, for example, the usage status of the information display device 10 (in other words, the usage mode), the operation target on which the user operation has been performed, the type of user operation that can be accepted according to the usage status and the operation target, etc. Information.

More specifically, for example, when a drag is performed on the map image as an operation target in a situation where the map browsing software is used, the drag is identified as instructing the execution of the scroll function. Also, for example, when a tap is performed with an enlarged icon on the map image as an operation target, the tap is identified as instructing execution of the display size enlargement function. For example, if no function is associated with the flick for the enlarged icon, it is determined that the flick is an invalid operation.

The overall control unit 42 controls the display information on the display surface by controlling the first image forming unit 44, the second image forming unit 48, and the image composition unit 52. Note that the display information may be changed based on the identification result of the user instruction, or may be based on an instruction on program execution irrespective of the identification result of the user instruction.

The overall control unit 42 performs general control on the other

functional units

40, 44, 46, 48, 50, 52, 54, and 56, for example, adjustment of execution timing.

The first image forming unit 44 reads the first information 60 according to the instruction from the overall control unit 42 from the storage unit 18, forms the first image from the first information 60, and converts the first image into the first image holding unit 46. To store. Similarly, the second image forming unit 48 reads the second information 62 according to the instruction from the overall control unit 42 from the storage unit 18, forms a second image from the second information 62, and converts the second image into the second image. Store in the holding unit 50.

Under the instruction of the overall control unit 42, the image composition unit 52 reads the first image from the first image holding unit 46, reads the second image from the second image holding unit 50, and combines the first image and the second image. The synthesized image is stored in the synthesized image holding unit 54.

The image composition is performed so that the first image and the second image are displayed in an overlapping manner. Here, the case where the first image is the lower image (in other words, the lower layer) and the second image is the upper image (in other words, the upper layer) is illustrated. Here, “up and down” means up and down in the normal direction of the display surface, and the side closer to the user viewing the display surface is expressed as “up”. In practice, image data is superimposed based on such a concept.

In the composite image, that is, the display screen, the lower image is displayed in the transparent portion of the upper image. In other words, the drawing portion of the upper image hides the lower image. However, by setting the transparency in the drawing portion of the upper image, a composite image in which the lower image is transparent can be formed.

The setting of which of the first image and the second image is the upper image may be unchangeable or may be changeable.

Here, an example is given in which two layers of the first image and the second image are combined, but a configuration capable of combining more layers may be adopted. Also, other synthesis methods may be adopted.

The composite image stored in the composite image holding unit 54 is transferred to the display unit 12 and displayed on the display unit 12. The display screen changes when the composite image is updated, that is, when at least one of the first image and the second image is updated.

The composite icon management unit 56 manages display of composite icons under the control of the overall control unit 42. Specifically, the composite icon management unit 56 manages information such as the display position, size, orientation, and display attribute, and controls the second image forming unit 48 and the image composition unit 52 based on the management information. To manage the display of composite icons.

For example, the composite icon management unit 56 reads the composite icon image data from the storage unit 18 with respect to the second image forming unit 48 and forms a composite icon image with a size corresponding to the size of the display surface. And instructing that the formed composite icon image is drawn on the transparent plane according to the display position and orientation and stored in the second image holding unit 50. Further, regarding the deletion of the composite icon, the composite icon management unit 56 causes the second image forming unit 48 to store an image that does not have the composite icon image in the second image holding unit 50, for example. Further, the composite icon management unit 56 instructs the image composition unit 52 to compose the images in the

image holding units

46 and 50.

<Processing Example of Information Display Device 10>
Below, the process (in other words, display information operation method) by the information display apparatus 10 related to a composite icon is illustrated.

<Display of composite icon>
FIG. 19 illustrates a processing flow S10 until a composite icon is displayed. According to the example of FIG. 19, the input unit 14 receives a user operation in step S11, and the control unit 16 identifies the input user operation in step S12. In step S13, the control unit 16 executes a function associated with the user operation based on the identification result in step S12.

Thereafter, in step S14, the control unit 16 refers to the user operation received in step S11 as a condition (“composite icon display start condition” or “display start condition”) set in advance for displaying the composite icon. Judgment) is satisfied. If it is determined that the display start condition is not satisfied, the processing of the information display device 10 returns to step S11. On the other hand, when determining that the display start condition is satisfied, the control unit 16 performs a process of displaying the composite icon in step S15. After the composite icon is displayed, the processing flow S10 in FIG. 19 ends.

<Composite icon display start condition>
Regarding step S14, as a composite icon display start condition, a composite icon is displayed when a gesture operation of the screen movement deformation type function (in other words, a gesture operation that triggers display of the composite icon) is performed once. It is possible to adopt a condition (referred to as “one-time operation condition”). According to the one-time operation condition, the composite icon can be used immediately. Therefore, the operation burden of repeating the same gesture operation many times can be reduced.

Here, when the length of one operation time of the gesture operation of the screen movement deformation type function reaches a predetermined threshold value (hereinafter referred to as “operation time threshold value”), the composite icon is displayed. (Referred to as “operation time condition”) may be added to the operation condition once. The fact that one gesture operation takes a certain amount of time, for example, assumes a situation where the user is performing the gesture operation while watching the display information. For this reason, it is considered that the gesture operation is likely to be repeated further. Therefore, according to the operation time condition, it is possible to display the composite icon while more accurately identifying the user's intention.

In addition, a condition that a composite icon is displayed when a single operation speed of the gesture operation of the screen movement deformation type function is equal to or higher than a predetermined threshold (referred to as “operation speed threshold”) ( (Referred to as “operation speed condition”) may be added to the operation condition once. That the gesture operation is performed quickly means, for example, a situation where the user wants to see the display information after the operation quickly. For this reason, it is considered that the gesture operation is likely to be repeated further. Therefore, according to the operation speed condition, it is possible to display the composite icon while more accurately identifying the user's intention.

Note that the display timing may be specified in the operation speed condition. That is, the operation speed condition is such that when the single operation speed of the gesture operation of the screen movement deformation type function is equal to or higher than the operation speed threshold, the composite icon is displayed at a timing earlier than a predetermined icon display timing. You may deform | transform into the conditions of. According to this, a composite icon can be provided quickly.

In addition, when the gesture amount (for example, drag distance) for one gesture operation of the screen movement deformation type function is equal to or greater than a predetermined threshold (referred to as “gesture amount threshold”), the composite icon is displayed. A condition of displaying (referred to as “gesture amount condition”) may be added to the one-time operation condition. The fact that the gesture operation is performed largely means, for example, a situation where the user desires a large control amount for the display information. For this reason, it is considered that the gesture operation is likely to be repeated further. Therefore, according to the gesture amount condition, it is possible to display the composite icon while more accurately identifying the user's intention.

Further, when the end point of the gesture operation of the screen movement deformation type function corresponds to a point in a predetermined area on the display surface, a condition that a composite icon is displayed (hereinafter referred to as “end point condition”). ) May be added to the operation condition once. The predetermined region on the display surface is, for example, a peripheral region 32b of the display surface 32 as shown in FIG. 20, the peripheral area 32b of the display surface 32 corresponds to the peripheral area 34b of the input surface 34, and the end point 70b of the drag 70 exists in the

peripheral areas

32b and 34b. By forming such a situation, a composite icon (for example, a scroll composite icon) is displayed. For example, it is assumed that the user wants to perform a longer drag but has reached the

peripheral areas

32b and 34b. Further, for example, the user can intentionally use this end point condition in order to display the composite icon. Therefore, according to the end point condition, it is possible to display the composite icon while more accurately identifying the user's intention. In addition, said predetermined area | region is not limited to the peripheral area |

regions

32b and 34b. Further, the drag illustrated in FIG. 20 may be, for example, one drag during a two-point movement type pinch out.

Further, a condition that a composite icon is displayed when a composite icon call operation is performed following a gesture operation of the screen movement deformation type function (hereinafter referred to as a “call operation condition”) is set as a one-time operation condition. May be added. In addition, the condition of “following” includes a condition that the gesture operation and the composite icon call operation are performed within a predetermined operation time interval, and a condition that no other operation is performed during the operation. Including.

The composite icon call operation is, for example, a touch operation.

More specifically, as illustrated in FIG. 21, an operation of touching an arbitrary point on the input surface with another finger without releasing the dragged finger as the gesture operation from the end point is called a composite icon call It can be used as an operation. A tap may be adopted as such a touch operation, or a double tap or a long press may be adopted. Such a touch operation can also be performed when the gesture operation is a pinch-out using a plurality of fingers.

Alternatively, as illustrated in FIG. 21, an operation of touching the end point of the drag performed as the gesture operation or the vicinity thereof can be used as the composite icon calling operation. A tap may be employed as such a touch operation, or a double tap may be employed. Note that such a touch operation can also be performed when the gesture operation is a flick and when the gesture operation is a pinch out using a plurality of fingers or the like. Further, a long press may be employed as the touch operation after dragging. In this case, the finger used for dragging is not released from the input surface, and a long press is performed as it is. Such a composite icon calling operation by long pressing can be performed even when the gesture operation is a pinch out using a plurality of fingers or the like.

Also, as a composite icon call operation, a flick operation may be adopted instead of the touch operation. Specifically, as illustrated in FIG. 21, flicking is performed so as to trace the drag locus.

The composite icon call operation can suppress the accidental display of the composite icon.

In addition, a condition that a composite icon is displayed when a non-operation state continues for a predetermined time (time length) or more after a gesture operation of the screen movement deformation type function (“no-operation progress condition”) May be added to the operation condition once. According to the no operation progress condition, the composite icon is not displayed immediately. For this reason, it contributes to operation mistake prevention.

Note that the above conditions such as the operation time condition may be combined in various ways.

On the other hand, as a composite icon display start condition, a condition that a composite icon is displayed when a gesture operation of the screen movement deformation type function is continuously repeated a predetermined number of times (referred to as “repeated operation condition”). Can be adopted). Here, the condition “continuously” includes a condition that the gesture operation is repeated within a predetermined operation time interval and a condition that no other operation is performed during the repetition.

Note that the above repeated operation conditions do not require that the gesture direction be the same in repeated gesture operations. That is, the gesture directions may be the same or different. For example, when searching for a certain item (such as a certain point on the map) in the display information, the drag may be repeated in the same direction or the drag may be performed in various directions. In view of this point, in any case, when the scroll composite icon appears, high convenience is obtained.

However, it is also possible to add a condition that the gesture operation is repeated in the same gesture direction to the above-described repeated operation condition. Note that the “same gesture direction” is not only the case where the gesture direction of each time is exactly the same, but also the case where the gesture direction of each time is substantially the same (for example, variation in the gesture direction of each time is a predetermined tolerance) In the case)

Note that a condition that a similar gesture operation (for example, dragging and flicking) is handled as the same gesture operation may be added to the repeated operation condition.

The repetition operation condition can detect the repetition of the same gesture operation by monitoring, for example, the type of gesture operation, the gesture direction, the number of repetitions of the loop processing of steps S11 to S14, etc. in step S14 (see FIG. 19). is there.

If the user repeats the gesture operation, it is highly likely that the gesture operation is repeated further. For this reason, according to the repetition condition, it is possible to display the composite icon while more accurately identifying the user's intention.

Here, when the repetition time of the gesture operation reaches a predetermined threshold value (referred to as “repetition total time threshold value”), a condition that a composite icon is displayed (“total repetition time”). May be added to the repeated operation condition. The repetition of the gesture operation takes a certain amount of time, for example, in a situation where the user wants to see the display information that follows continuously. For this reason, it is considered that the gesture operation is likely to be repeated further. Therefore, according to the repeated total time condition, it is possible to display the composite icon while more accurately identifying the user's intention.

In addition, when the repetition speed of the gesture operation of the screen movement deformation type function is equal to or higher than a predetermined threshold value (hereinafter referred to as “repetition speed threshold value”), a condition that the composite icon is displayed (“repetition speed”). May be added to the repeated operation condition. The repetition speed is defined as the number of gesture operations per unit time, for example. That the gesture operation is repeated quickly means, for example, a situation in which the user wants to see the subsequent display information quickly. For this reason, it is considered that the gesture operation is likely to be repeated further. Therefore, according to the repetition speed condition, it is possible to display the composite icon while more accurately identifying the user's intention.

Note that the display timing may be specified in the repetition speed condition. That is, the repetition speed condition is such that the composite icon is displayed at a timing earlier than a predetermined icon display timing when the repetition speed of the gesture operation of the screen movement deformation type function is equal to or higher than the repetition speed threshold. It may be deformed. According to this, a composite icon can be provided quickly.

In addition, the gesture amount (for example, drag distance) is integrated according to the repetition of the gesture operation of the screen movement deformation type function, and the integrated value reaches a predetermined threshold (referred to as “gesture total amount threshold”). In this case, a condition that a composite icon is displayed (referred to as “gesture total amount condition”) may be added to the repeated operation condition. The fact that the integrated value of the gesture amount becomes large means that, for example, the user desires a large control amount for the display information. For this reason, it is considered that the gesture operation is likely to be repeated further. Therefore, according to the gesture total amount condition, it is possible to display the composite icon while more accurately identifying the user's intention.

It should be noted that the above conditions such as the total repetition time condition may be variously combined.

Also, one or more of the above conditions such as the operation time condition exemplified in connection with the one time operation condition may be added to the repeated operation condition. Specifically, one or more of the above conditions such as an operation time condition are applied to each gesture operation. Alternatively, one or more of the above conditions such as an operation time condition may be applied to a predetermined number of gesture operations (for example, the last gesture operation). According to such a condition addition, it is possible to improve the accuracy of identifying the user intention.

<Composite icon display position>
Regarding step S15 (see FIG. 19), the display position of the composite icon is basically arbitrary. On the other hand, as illustrated in FIG. 22, if the scroll composite icon 72 exists in the vicinity of the end point 70 b of the drag 70, the finger used for the drag 70 is moved onto the scroll composite icon 72 with a small amount of movement. Can be made.

Here, in the example of FIG. 22, the composite icon 72 is arranged on the right side of the drag end point 70b. On the other hand, the composite icon 72 may be arranged on another side of the end point 70b or directly above the end point 70b. In view of these, as shown in FIG. 22, if the composite icon 72 exists in a region 70c including the end point 70b (hereinafter referred to as "end point region"), the above effect can be obtained.

The size and shape of the end point area 70c may be a variable value according to the operation situation (the detected finger size, finger movement speed, etc.), or a fixed value that does not depend on the operation situation. It may be. Further, the center of the end point region 70c does not necessarily coincide with the end point 70b.

The end point area 70c can be obtained in the coordinate system of the display surface after associating the end point 70b of the drag 70 on the display surface. Alternatively, before associating the end point 70b of the drag 70 on the display surface, the end point region 70c may be obtained in the coordinate system of the input surface, and the obtained end point region 70c may be associated with the coordinate system of the display surface.

In addition, when the repeated operation condition is applied, for example, the average end point position is obtained for all gesture operations or a part of the gesture operations that are the determination target of the repeated operation condition, and the obtained The end point area 70c may be set from the end point position. Alternatively, the end point region 70c for a predetermined number of gesture operations (for example, the last gesture operation) may be set.

Alternatively, as illustrated in FIG. 23, the composite icon 72 may be arranged on the extension line 70d of the locus of the drag 70. According to this, since the composite icon 72 can be reached by moving the finger used for the drag 70 in the same direction as it is, the movement of the finger is smooth.

Alternatively, as illustrated in FIG. 24, a composite icon 72 may be displayed at a position where the extension line 70d passes in the end point region 70c.

Alternatively, as illustrated in FIG. 25, a composite icon 72 may be displayed at a position where the extension line 70d passes in the peripheral area 32b of the display surface 32. According to this, it is possible to avoid the display information at the center of the display surface that is considered to have a high degree of user attention from being hidden by the composite icon 72. In addition, although the case where the setting range of the peripheral area 32b is the same as the above-described FIG. 20 (related to the end point condition among the composite icon display start conditions) is illustrated here, it is not limited to this example.

Here, an example of how to obtain the extension line 70d will be described with reference to FIGS. Although FIG. 26 to FIG. 28 illustrate curved drag trajectories, the following description also applies to linear drag trajectories.

26, the extension line 70d is determined as a straight line connecting two points in the drag trajectory. FIG. 26 illustrates the case where the two points in the locus are the start point 70a and the end point 70b of the drag 70, but the present invention is not limited to this example. For example, as shown in FIG. 27, the end point 70b of the drag 70 and points other than the end point 70b may be used.

On the other hand, according to the example of FIG. 28, the extension line 70d is determined as a straight line that contacts one point in the drag locus. FIG. 28 illustrates a case where one point in the locus is the end point 70b of the drag 70, but is not limited to this example.

According to these methods, the extension line 70d can be easily obtained.

Here, it is preferable to set the extension line 70d using the end point side portion 70f of the drag trajectory, in other words, excluding the start point side portion 70e of the drag trajectory, as in the examples of FIGS. In the example of FIGS. 27 and 28, the drag trajectory is divided into two parts: a start point side portion 70e including a trajectory start point 70a and an end point side portion 70f including a trajectory end point 70f.

The user's intention is considered to be clearer in the end point side portion 70f than in the start point side portion 70e. For example, it can be imagined that the trajectories in FIGS. 27 and 28 have changed direction during dragging. For this reason, the composite icon 72 can be displayed at a position reflecting the user's intention by using the end point side portion 70f.

It is also possible to use a portion other than the end point 70b in the end point side portion 70f. However, in view of clarity of the user's intention, a straight line (see FIG. 27) passing through the end point 70b and another point in the end side portion (see FIG. 27) or a tangent line at the end point 70b (see FIG. 28) is an extension line 70d. It is more preferable to set to.

In addition, it is considered that the user's intention is reflected more strongly as the end point side portion 70f is smaller than the start point side portion 70e.

The extension line 70d can be obtained in the coordinate system of the display surface after associating the locus of the drag 70 on the display surface. Alternatively, before associating the locus of the drag 70 on the display surface, the extension line 70d may be obtained in the coordinate system of the input surface, and the obtained extension line 70d may be associated with the coordinate system of the display surface.

In addition, when the repeated operation condition is applied, for example, an average extension line is obtained for all the gesture operations that are the target of the repeated operation condition determination, or a part of the gesture operations. The extended line may be used as the extended line 70d. Alternatively, an extension line 70d for a predetermined number of gesture operations (for example, the last gesture operation) may be used.

The above-mentioned various items related to the display position of the composite icon can also be applied to flicks. In addition, since the pinch out and the like include a drag, the various items described above can be applied.

Note that, as illustrated in FIG. 29, in the two-point movement type gesture operation (pinch out is illustrated in FIG. 29), a composite icon (display size change composite icon 80 in FIG. 29) is displayed for each drag. May be provided). In this example, it is assumed that the user only needs to selectively operate one of the two composite icons 80.

<Display attribute of composite icon>
The composite icon may be displayed with a display attribute different from other icons (in other words, a display method). For example, the composite icon is displayed by display attributes such as blinking, stereoscopic display, animation display, and translucency, or by a combination of a plurality of display attributes. According to this, the visibility of the composite icon is improved, which contributes to prevention of an operation error.

<Use of composite icons>
FIG. 30 illustrates a processing flow S30 during display of the composite icon. In the example of FIG. 30, steps S31 and S32 are the same as steps S11 and S12 of FIG. That is, in step S31, the input unit 14 receives a user operation, and in step S32, the control unit 16 identifies the input user operation.

In step S33, the control unit 16 determines whether or not the user operation received in step S31 is an execution instruction for any icon in the composite icon. Specifically, it is determined whether or not the input position of the user operation corresponds to the display position of any icon in the composite icon, and the user operation is a predetermined operation as an execution instruction operation for the composite icon ( Here, it is determined whether or not the one-point touch is exemplified as described above.

If it is determined in step S33 that the user operation is an execution instruction for any icon in the composite icon, the control unit 16 determines in step S34 that the screen movement deformation type function associated with the composite icon, Execute the screen movement deformation function associated with the gesture operation involved in the appearance of the composite icon. At this time, the screen movement deformation type function is executed in the control direction assigned to the icon given the execution instruction. Thereafter, the processing of the information display device 10 returns to step S31.

On the other hand, if it is determined in step S33 that the user operation is not an execution instruction for any icon in the composite icon, the control unit 16 performs a function associated with the user operation received in step S31 in step S35. Execute. Thereafter, the processing of the information display device 10 returns to step S31.

Note that, for example, even when the scroll composite icon is being displayed, the drag associated with the scroll function is accepted in step S31, and the scroll is executed in step S33. According to this, even when the scroll composite icon is displayed, it is possible to finely adjust the display information by dragging and the like. The same applies to composite icons other than the scroll composite icon.

<Control amount and control speed>
Regarding step S34 (see FIG. 30), for example, when a composite icon is tapped (more specifically, when any icon in the composite icon is tapped), a predetermined control amount and a predetermined control speed are set. Thus, the screen movement deformation type function associated with the composite icon is executed. Further, for example, while the composite icon is pressed and held, the screen movement deformation type function associated with the composite icon is continuously executed. In this case, the control amount of the display information is determined according to the long press time. Further, the control speed of the display information may be a predetermined constant value or may be gradually increased.

On the other hand, the gesture amount or gesture speed of the gesture operation related to the appearance of the composite icon may be reflected in the control amount of the display information when the execution instruction operation is performed on the composite icon. Similarly, the gesture amount or the gesture speed may be reflected on the control speed of the display information when the execution instruction operation is performed on the composite icon.

In the example of FIG. 31, the control amount or the control speed of the display information is set to be larger as the gesture amount or the gesture speed is larger. More specifically, the scroll amount is set larger as the drag distance is longer. Alternatively, the scroll speed is set larger as the drag distance is longer. Alternatively, the scroll amount is set larger as the drag speed is higher. Alternatively, the scroll speed is set higher as the drag speed is higher. As the drag speed, for example, an average speed or a maximum speed can be used. However, it is not limited to the linear relationship shown in FIG.

Alternatively, when the amount of gesture operation related to the appearance of the composite icon is set to one unit of the control amount of the display information and the execution instruction operation is performed on the composite icon, the unit is intermittently displayed for each unit. Display information may be controlled. For example, as shown in FIG. 32, when the scroll composite icon is tapped once, the display information is scrolled by the one unit, and when the scroll composite icon is pressed for a long time, the scroll for the one unit is interrupted. Do it. According to this, it becomes easy to confirm the change of display information.

Alternatively, a change in gesture speed of the gesture operation (in other words, acceleration of the gesture operation) may be reflected in the control speed of the display information when the execution instruction operation is performed on the composite icon. For example, as shown in FIG. 33, when the scroll composite icon is tapped once, the speed history of the gesture operation is reproduced once, and when the scroll composite icon is pressed for a long time, the speed history of the gesture operation is Repeat during long press. Since the gesture speed generally decreases at the beginning and end of the gesture operation, a situation similar to the above-described intermittent scroll is provided. For this reason, it becomes easy to confirm the change of display information.

Regarding the control amount and control speed, any of the above examples can be applied to gesture operations other than dragging and screen movement deformation type functions other than scrolling.

Further, when the touch panel 14 is configured to detect the pressing force of the finger against the input surface, the control information amount and the control speed may be set larger as the pressing force with respect to the composite icon increases.

<Resize composite icon>
Regarding step S35 (see FIG. 30), when it is determined in step S33 that the user operation is a display size change operation performed on the composite icon, the control unit 16 sets the display size of the composite icon itself in step S35. change. The display size changing operation is, for example, pinch out and pinch in as shown in FIG. The pinch operation may be a two-point movement type (see FIGS. 7 and 9) or a one-point movement type (see FIGS. 8 and 10). When the user changes the composite icon to a desired size, the operability is improved.

<Erase composite icon>
FIG. 35 illustrates a processing flow S50 related to erasure of composite icons (that is, display end). According to the example of FIG. 35, in step S51, the control unit 16 satisfies a predetermined condition (hereinafter referred to as “composite icon deletion condition” or “deletion condition”) for deleting the composite icon. Judge whether or not.

If it is determined that the erasure condition is satisfied, the control unit 16 performs a process of erasing the composite icon from the display surface in step S52. Thereafter, the processing of the information display device 10 returns to the processing flow S10 (see FIG. 19) until the composite icon is displayed. On the other hand, if it is determined that the erasure condition is not satisfied, the processing of the information display device 10 returns to step S51.

The process flow S50 is executed in parallel with the process flow S30 displaying the composite icon. Specifically, step S51 is repeated until the composite icon deletion condition is satisfied, and step S52 is executed as an interrupt process when the composite icon deletion condition is satisfied.

<Composite icon deletion conditions>
As the composite icon erasure condition, it is possible to adopt a condition (hereinafter referred to as “operation waiting condition”) that the composite icon is erased from the display surface when the execution instruction operation for the composite icon is not input. If the composite icon is not used for a certain period of time, the user is likely not to use the composite icon for a while. Therefore, according to the deletion waiting time condition, it is possible to provide high convenience in that the composite icon is deleted after the user's intention is more accurately identified.

As the length of the waiting time until the composite icon is deleted, for example, a predetermined constant value can be adopted. Alternatively, the length of the waiting time may be set based on the gesture speed of the gesture operation involved in the appearance of the composite icon. For example, if the gesture operation is performed quickly, it is considered that there is a high possibility that the gesture operation is repeated as described above. That is, it is considered that there is a high possibility that a composite icon is used. For this reason, when the gesture speed is high, it is preferable to set the erasure waiting time longer.

Alternatively, as a composite icon erasing condition, when the user operation is a predetermined composite icon erasing operation, a condition that the composite icon is erased from the display surface (hereinafter referred to as “erase instruction condition”) is adopted. May be. An operation different from the execution instruction operation for the composite icon (for example, flick for the composite icon) is assigned to the composite icon deletion operation. According to the delete instruction condition, the user can delete the composite icon at any time.

Alternatively, both the operation waiting condition and the erasure instruction condition may be adopted, which further improves convenience.

<Number of composite icons>
It is also possible to display a plurality of composite icons at the same time. For example, a scroll composite icon, a display size change composite icon, and a rotation composite icon may be displayed. In this case, the processing flows S10, S30, and S50 are managed in parallel for each composite icon. In addition, a limit may be set on the number of composite icons displayed simultaneously.

<Combination of icons>
A composite icon may be configured by combining icons associated with different types of screen movement deformation type functions. For example, the composite icon 88 shown in FIG. 36 includes scroll icons 72a to 72h and display

size change icons

80a and 80b. According to the composite icon 88, the scrolling, enlargement, and reduction functions can be operated at one place, so that a favorable operation environment can be provided. Note that scrolling, enlargement, and reduction can be performed independently or in combination. For example, if the two-point touch targeting the upward scroll icon 72a and the enlargement icon 84a is identified, the control unit 16 executes the upward scroll and enlargement simultaneously. Note that the combination of icons is not limited to the example of FIG.

<Effect>
According to the information display device 10, the various effects described above can be obtained, and as a result, high convenience can be provided. In the above, the case where the gesture operation is a drag and the screen movement deformation type function associated with the drag is mainly exemplified as a scroll, but other gesture operations and other screen movement deformation type functions are also exemplified. Similar effects can be obtained.

<Modification>
In the above, the case where the display information displayed on the display unit 12 is a map image is mainly exemplified. However, the use of composite icons is not limited to map images. For example, the composite icon can be used for a list of titles such as books and music and a slide for a list of web search results. Further, for example, the composite icon can be used for turning pages of an electronic book or the like and selecting content such as an electronic album.

Further, the display information to be controlled by the gesture operation and the composite icon may be displayed on the entire display surface or may be displayed on a part of the display surface. The display information displayed on a part of the display surface is, for example, display information in a window provided on the part. Further, a part of the display surface may be one-dimensional as illustrated in FIG. That is, in the example of FIG. 37, the elements A, B, C, D, E, F, G, H, and I forming the display information form a row on the zigzag path (in other words, connected to each other). ) Move and the movement is controlled by drag or flick.

In the above, a contact type touch panel is exemplified as the input unit 14. On the other hand, a non-contact type (also referred to as a three-dimensional (3D) type) touch panel can be used for the input unit 14.

According to the non-contact type, a detectable region of the sensor group (in other words, an input region that can accept user input) is provided as a three-dimensional space on the input surface, and a finger in the three-dimensional space is placed on the input surface. The position projected on is detected. Some non-contact types can detect the distance from the input surface to the finger. According to this method, the finger position can be detected as a three-dimensional position, and further, the approach and retreat of the finger can also be detected. Various systems have been developed as non-contact type touch panels. For example, a projection capacity system which is one of electrostatic capacity systems is known.

In the above description, the finger is exemplified as the indicator used by the user for input. However, for example, a part other than the finger can be used as the indicator. For example, a tool such as a touch pen (also referred to as a stylus pen) may be used as an indicator.

Also, a so-called motion sensing technology may be used for the input unit 14. Various methods have been developed as motion sensing technology. For example, a method is known in which a user's movement is detected by a user holding or wearing a controller equipped with an acceleration sensor or the like. In addition, for example, a method of extracting a feature point such as a finger from a captured image of a camera and detecting a user's movement from the extraction result is known. An intuitive operation environment is also provided by the input unit 14 using the motion sensing technology.

In addition, although the input / display unit 20 is illustrated above, the display unit 12 and the input unit 14 may be arranged separately. Even in this case, an intuitive operation environment is provided by configuring the input unit 14 with a touch panel or the like.

Further, the information display device 10 may further include elements other than the

above elements

12, 14, 16, and 18. For example, an audio output unit that outputs auditory information, a communication unit that performs wired or wireless communication with various devices, and the current position of the information display device 10 conform to, for example, a GPS (Global Positioning System) system. One or more of the current position detection units to be detected may be added.

The voice output unit can output, for example, operation sounds, sound effects, guidance sounds, and the like. For example, a notification sound can be output at each timing of appearance, use, and deletion of a composite icon. The communication unit can be used for, for example, new acquisition and update of information stored in the storage unit 18. Further, the current position detection unit can be used for a navigation function, for example.

The use of the information display device 10 is not particularly limited. For example, the information display device 10 may be a portable or desktop information device. Alternatively, the information display device 10 may be applied to a navigation device or an audio / visual device mounted on a moving body such as an automobile.

Further, in the present invention, the embodiments can be appropriately modified and omitted within the scope of the invention.

10 information display device, 12 display unit, 14 input unit, 16 control unit, 18 storage unit, 20 input / display unit, 32 display surface, 32b peripheral region, 34 input surface (input region), 34b peripheral region, 70 drag, 70a start point, 70b end point, 70c end point area, 70d extension line, 70e start point side part, 70f end point side part, 72 scroll composite icon, 72a to 72h scroll icon, 80 display size change composite icon, 80a enlarge icon (display size change icon ), 80b reduction icon (display size change icon), 84 rotation composite icon, 84a right rotation icon (rotation icon), 84b left rotation icon (rotation icon), 88 composite icon S10, S30, S50 processing flow.

Claims

A display unit having a display surface;
An input unit that accepts user operations;
When the user operation is a gesture operation associated with a screen movement deformation type function for controlling display information on the display surface in a control direction set in accordance with the gesture direction, both of the screen movement deformation type functions A composite icon that is a composite of a plurality of icons that are associated with each other but have different control direction assignments is displayed on the display surface, and the user operation performs an execution instruction operation on any of the icons in the composite icon In the case, the information display device includes a control unit that executes the screen movement deformation function in the control direction assigned to the icon for which the execution instruction operation has been performed.
The information display device according to claim 1, wherein the control unit displays the composite icon when the gesture operation is continuously repeated a predetermined number of times.
The information display device according to claim 2, wherein the control unit displays the composite icon when a length of a repetition time of the gesture operation reaches a predetermined threshold value.
The information display device according to claim 1, wherein the control unit displays the composite icon when a length of one operation time of the gesture operation reaches a predetermined threshold value.
The information display apparatus according to claim 2, wherein the control unit displays the composite icon when a repetition speed of the gesture operation is equal to or higher than a predetermined threshold.
The said control part displays the said composite icon at a timing earlier than a predetermined icon display timing, when the said repetition speed of the said gesture operation is more than the said predetermined threshold value. Information display device.
The information display device according to claim 1, wherein the control unit displays the composite icon when a single operation speed of the gesture operation is equal to or higher than a predetermined threshold.
The said control part displays the said composite icon at timing earlier than a predetermined icon display timing, when the said operation speed of the said gesture operation is more than the said predetermined threshold value. The information display device described in 1.
The information display device according to claim 2, wherein the control unit accumulates the gesture amount according to repetition of the gesture operation, and displays the composite icon when the accumulated value reaches a predetermined threshold value.
The information display device according to claim 1, wherein the control unit displays the composite icon when the amount of one gesture of the gesture operation reaches a predetermined threshold value.
The information display device according to claim 1, wherein the control unit displays the composite icon when an end point of the gesture operation corresponds to a point in a predetermined area on the display surface.
The information display device according to claim 1, wherein the control unit displays the composite icon when a composite icon call operation is performed subsequent to the gesture operation.
The information display device according to claim 1, wherein the control unit displays the composite icon when a no-operation state continues for a predetermined time after the gesture operation.
The control unit reflects the gesture amount or gesture speed of the gesture operation on the control amount or control speed of the display information when the execution instruction operation is performed on the composite icon. The information display device described.
The control unit sets a gesture amount of the gesture operation to one unit of a control amount of the display information, and when the execution instruction operation is performed on the composite icon, the control unit intermittently performs the unit operation. 15. The information display device according to claim 14, which controls display information.
15. The control unit according to claim 14, wherein the control unit reflects a change in gesture speed of the gesture operation on a change in control speed of the display information when the execution instruction operation is performed on the composite icon. Information display device.
The control unit associates an end point of the gesture operation or an end point region defined including the end point on the display surface, and displays the composite icon in the end point region on the display surface. The information display device described in 1.
The information according to claim 1, wherein the control unit associates a gesture trajectory of the gesture operation or an extension line of the gesture trajectory on the display surface, and displays the composite icon on the extension line on the display surface. Display device.
The information display device according to claim 18, wherein the control unit displays the composite icon at a position where the extension line passes in a peripheral area of the display surface.
The information display device according to claim 18, wherein the control unit sets the extension line by using an end point side portion that is a part of the gesture trajectory and includes an end point of the gesture trajectory.
21. The information display device according to claim 20, wherein the extension line is a tangent line at the end point of the gesture trajectory, or a straight line passing through the end point of the gesture trajectory and another point in the end point side portion.
The information display device according to claim 1, wherein the composite icon is displayed with a display attribute different from that of the other icons.
The information display device according to claim 1, wherein the control unit erases the composite icon from the display surface when the execution instruction operation on the composite icon is not input.
The information display device according to claim 1, wherein the control unit erases the composite icon from the display surface when the user operation is a composite icon erase operation.
The screen movement deformation type function is a scroll function for scrolling the display information in the control direction,
The control direction is a scroll direction of the display information,
The composite icon is a scroll composite icon in which any of the plurality of icons is associated with the scroll function, but the scroll direction of each icon is different.
The information display device according to claim 1.
The scroll compound icon is
An enlarge icon associated with the function of enlarging the display size of the display information;
The information display device according to claim 25, further comprising a reduction icon associated with a function of reducing a display size of the display information.
The screen movement deformation type function is a display size changing function for changing a display size of the display information,
The control direction is a display size change direction of the display information,
The composite icon is a display size change composite icon in which any of the plurality of icons is associated with the display size change function, but the display size change direction by each icon is different.
The information display device according to claim 1.
The screen movement deformation type function is a rotation function for rotating the display information,
The control direction is a rotation direction of the display information,
The composite icon is a rotation composite icon in which any of the plurality of icons is associated with the rotation function, but the rotation direction of each icon is different.
The information display device according to claim 1.
The information display device according to claim 1, wherein the control unit changes a display size of the composite icon when the user operation is a display size change operation performed on the composite icon.
(A) receiving a user operation;
(B) identifying the user operation;
(C) When the user operation is a gesture operation associated with a screen movement deformation type function for controlling display information on the display surface in a control direction set in accordance with the gesture direction, any of the screen movement deformations Displaying on the display surface a composite icon that is a composite of a plurality of icons that are associated with a type function but have different control direction assignments;
(D) When the user operation is an execution instruction operation with respect to any of the icons in the composite icon, the screen movement deformable function with the control direction assigned to the icon for which the execution instruction operation has been performed A display information operation method comprising the steps of: