WO2011128959A1

WO2011128959A1 - Tactile sensation control information generation device

Info

Publication number: WO2011128959A1
Application number: PCT/JP2010/007604
Authority: WO
Inventors: 光洋阿曽; 秦　秀彦; 雄基篠本; 紹二大坪
Original assignee: パナソニック株式会社
Priority date: 2010-04-13
Filing date: 2010-12-28
Publication date: 2011-10-20

Abstract

Provided is a tactile sensation control information generation device which achieves GUI operation that is intuitive and easy to use on a touch panel. An input processing unit acquires input information from a touch panel display, notifies the acquired input information to a drawing processing unit, and also instructs the drawing processing unit to draw a GUI element. An operation target extraction unit determines the GUI element that is the operation target of the user, said determination being made from the input information acquired by the input processing unit and GUI element attribute values which are managed by a GUI attribute management unit. A control information generation unit acquires from the GUI attribute management unit the attribute values of the GUI element determined as being the operation target by the operation target extraction unit. The control information generation unit then generates, from the acquired GUI element attribute values and the input information acquired by an input acquisition unit, control information which instructs a tactile sensation reproduction device to reproduce a tactile sensation in the part of the user's body which operates the touch panel display, when said part of the user's body touches the GUI element displayed on the touch panel display.

Description

Tactile control information generator

The present invention relates to an information terminal having a touch panel display that operates a graphical user interface (GUI) displayed on a display screen by a touch operation, and simulates the type and operation of the GUI to the fingertip of the user who operates the GUI. The present invention relates to a technique for making GUI operation on a touch panel without unevenness more intuitive and easy to use by reproducing as typical tactile information.

In an information terminal equipped with a touch panel display such as a PDA (Personal Digital Assistant), a bank ATM (Automatic Teller Machine), or a mobile phone, the user can directly operate the GUI on the screen with a fingertip. Therefore, it is possible to operate the GUI more intuitively than a mouse or keyboard that indirectly operates the GUI.

On the other hand, in the operation using only the touch panel without unevenness, since the user must operate while looking at the screen in order to determine the display position and display area of the GUI to be operated, the information terminal is in an invisible state. There was a problem that it was difficult to operate.

For example, in a music player, the information terminal is provided with buttons for volume adjustment with unevenness and buttons for music selection operation, so that the user can adjust the volume and perform music selection operation by groping while the music player is in a pocket or bag. Was possible. However, in the touch panel type display, unless the GUI is visually observed, it is difficult to determine the display position and display area of the GUI for adjusting the sound volume and the GUI for music selection operation. Met.

To solve this problem, as a conventional technique, when the user touches the GUI on the touch panel, the user is informed by voice whether the GUI exists on the contact point between the user's fingertip and the touch panel. There are some which improve the operability in (for example, see Patent Document 1).

As another conventional technique, there is a technique for making a fine texture on the surface, such as an object image displayed on a display, or a feeling of roughness as a pseudo tactile sensation on the fingertip of the user who touched the dill play (for example, Non-Patent Document 1). By realizing a display that reproduces such a tactile sensation, a user can feel the texture of an object displayed on the display with a fingertip.

Japanese Patent No. 2760289

However, in the conventional technique described in Patent Document 1, it is possible to notify the user of the presence or absence of the GUI by voice, but the voice information alone is like an uneven physical button or keyboard. It was difficult to intuitively convey the shape and operation to the user.

In addition, in the conventional technique described in Non-Patent Document 1, by realizing a display that reproduces a tactile sense, a user can feel the texture of an object displayed on the display with a fingertip. It was not disclosed to apply to the operation of.

Therefore, the present invention solves the above-described conventional problems, and reproduces a pseudo tactile sensation on the user's fingertip in accordance with the characteristics of the user's operation target GUI in an information terminal provided with a touch panel. It is an object of the present invention to provide a tactile control information generation device that can inform a user of the presence and characteristics of a user and realize an intuitive and easy-to-use GUI operation.

An object of the present invention is directed to a haptic control information generation device that controls a haptic reproduction device that reproduces a haptic sensation on a part of a user's body that operates a touch panel display. In order to achieve the above object, the haptic control information generation apparatus of the present invention includes a GUI attribute management table that holds attribute values of GUI elements displayed on the touch panel display, and a GUI attribute that manages the GUI attribute management table. A drawing processing unit that acquires an attribute value of a GUI element to be drawn from the management unit and the GUI attribute management unit, draws the GUI element based on the acquired attribute value, and outputs the drawn GUI element to a touch panel display The input information from the touch panel display, notifies the drawing processing unit of the acquired input information, and instructs the drawing processing unit to draw a GUI element, and the input information acquired by the input processing unit. And the GUI element to be operated by the user from the attribute value of the GUI element managed by the GUI attribute management unit The attribute value of the GUI element determined as the operation target by the operation target extraction unit and the operation target extraction unit is acquired from the GUI attribute management unit, and the acquired attribute value of the GUI element and the input information acquired by the input acquisition unit From the above, when a part of the body of the user who operates the touch panel display touches the GUI element displayed on the touch panel display, the tactile reproduction device is configured to reproduce the tactile sensation on the part of the user's body. A control information generation unit that generates control information to be instructed.

According to the haptic control information generation device of the present invention, it is possible to reproduce a haptic according to the characteristics of the GUI on the fingertip of the user who operates the touch panel. Thereby, it is possible to provide an intuitive and easy-to-use GUI operation for a user who traces the GUI on the touch panel.

FIG. 1 is a diagram illustrating an example of a configuration of an information terminal 100 including a haptic control information generation device according to the first embodiment of the present invention. FIG. 2A is a diagram showing an overlapping relationship between the touch panel 1011 and the display screen 200 according to the first embodiment of the present invention. FIG. 2B is a diagram showing an example of a GUI element displayed on the display screen 200 according to the first embodiment of the present invention. FIG. 3A is a diagram showing an example of the GUI attribute management table 108 according to the first embodiment of the present invention. FIG. 3B is a diagram showing an example of the GUI attribute management table 108b according to the first embodiment of the present invention. FIG. 4 is a flowchart showing an example of processing of the information terminal 100 according to the first embodiment of the present invention. FIG. 5A is a diagram illustrating a correspondence relationship between GUI elements and frictional forces according to the first embodiment of the present invention. FIG. 5B is a diagram illustrating a correspondence relationship between GUI elements and frictional forces according to the first embodiment of the present invention. FIG. 5C is a diagram illustrating a correspondence relationship between GUI elements and frictional forces according to the first embodiment of the present invention. FIG. 6 is a diagram illustrating an example of the relationship between the GUI element and the frictional force according to the second embodiment of the present invention. FIG. 7 is a diagram illustrating an example of the relationship between the GUI element and the frictional force according to the second embodiment of the present invention. FIG. 8 is a diagram illustrating an example of the relationship between the GUI element and the frictional force according to the second embodiment of the present invention. FIG. 9 is a diagram illustrating an example of table information indicating a correspondence relationship between the attribute value of the GUI element and the control information calculation method according to the second embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram illustrating an example of a configuration of an information terminal 100 including a haptic control information generation device according to the first embodiment of the present invention. In FIG. 1, the information terminal 100 includes an input device 101, a display device 102, a haptic control information generation device, and a haptic reproduction device 109. The display device 102 includes a display screen 200. FIG. 2A is a diagram illustrating an overlapping relationship between the touch panel 1011 and the display screen 200 included in the display device 102. FIG. 2B is a diagram illustrating an example of a GUI element displayed on the display screen 200 of the display device 102. A structure in which the touch panel 1011 and the display screen 200 as illustrated in FIG. 2A are overlapped may be called a touch panel type display.

The input device 101 includes a touch panel 1011 in its configuration. In the input device 101, the touch panel 1011 is arranged so as to overlap the display screen 200 of the display device 102 as shown in FIG. 2A. When the user performs an input operation on the touch panel 1011, the input apparatus 101 uses the touch position of the user's fingertip on the touch panel 1011 as the coordinate value Pin (Xin, Yin) of the XY coordinate system to the input processing unit 103. Notice. Here, the XY coordinate system defines the upper left end of the touch panel 1011 as the origin, the right direction as the positive direction of the X axis, and the downward direction as the positive direction of the Y axis. Further, it is assumed that the XY axes on the touch panel 1011 coincide with the xy axes on the display screen 200 shown in FIG. 2B. In addition, it is assumed that a plurality of GUI elements 201 to 208 are displayed on the display screen 200 shown in FIG. 2B. Note that the touch panel 1011 is touched by the user's fingertip, but is not limited to the fingertip and may be a part of the user's body.

The tactile control information generation device includes an input processing unit 103, a drawing processing unit 104, a control information generation unit 105, an operation target extraction unit 106, a GUI attribute management unit 107, and a GUI attribute management table 108. Note that the haptic control information generation device may further include a haptic reproduction device 109 described later.

In the haptic control information generation device, when the input processing unit 103 receives the touch position Pin (Xin, Yin) of the user's finger from the input device 101 as input information, the input processing unit 103 receives the drawing processing unit 104 and the control information generation unit 105. Then, the contact position Pin (Xin, Yin) is notified and the start of each process is instructed.

When the touch position Pin (Xin, Yin) is notified from the input processing unit 103, the drawing processing unit 104 displays the display from the GUI attribute management table 108 managed by the GUI attribute management unit 107 via the GUI attribute management unit 107. The attribute value of the GUI element to be displayed on the device 102 is acquired. Then, the drawing processing unit 104 draws a GUI on the display screen 200 of the display device 102 according to the acquired attribute value of the GUI element. Details of the GUI attribute management table 108 will be described later.

When the contact position Pin (Xin, Yin) is notified, the control information generation unit 105 notifies the operation target extraction unit 106 of an acquisition request for a GUI element to be operated. In addition, the control information generation unit 105 generates control information for the haptic reproduction device 109 to reproduce a haptic from the attribute value of the GUI element extracted by the operation target extraction unit 106 and the input information acquired by the input processing unit 103. Generate. In addition, the control information generation unit 105 notifies the tactile reproduction device 109 of the generated control information, thereby instructing the tactile reproduction device 109 to reproduce the tactile sensation.

The tactile reproduction device 109 reproduces a pseudo tactile sensation on the fingertip of the user who touched the touch panel 1011 according to the control information notified from the control information generation unit 105. In the embodiment of the present invention, the frictional force generated on the contact surface between the user's fingertip and the touch panel or the strength of the propulsive force can be designated as control information to the haptic reproduction device 109 for reproducing the haptics. Shall.

Note that the frictional force is a force generated in a direction opposite to the tracing direction when the user performs a tracing operation on the touch panel. Further, the propulsive force is a force that moves the user's fingertip in contact with the touch panel in an arbitrary direction. Therefore, the control information generation unit 105 rubs the direction indicating the direction opposite to the user's drag direction (for example, the direction of the unit vector with respect to the XY axes on the touch panel 1101) as the control information, along with the strength of the frictional force. The tactile playback device 109 is notified of the force generation direction.

In addition, the direction in which the frictional force is generated is changed from the current contact position Pin to the previous contact position by the control information generation unit 105 storing the contact position notified from the input processing unit 103 during the previous process. It can be calculated as a heading unit vector. Moreover, what is necessary is just to invert the sign of said unit vector, when generating a driving force. In the following description, a tactile reproduction device 109 that generates a frictional force will be described as an example.

Further, when the operation target extraction unit 106 receives the operation target acquisition request, the operation target extraction unit 106 acquires all the GUI elements existing on the contact position Pin (Xin, Yin) from the GUI attribute management unit 107, and the overlapping order is the highest. A GUI element is extracted as an operation target. Here, the highest-level GUI element is a GUI element that is displayed on the forefront (user side) on the display screen when the GUI elements are displayed on the display screen in an overlapping manner. The GUI element is a component of the GUI such as a button, an image, text, or a scroll bar.

The GUI attribute management unit 107 manages the GUI attribute management table 108. The GUI attribute management unit 107 receives a GUI element acquisition request, and extracts a GUI element that matches the condition of the acquisition request from the GUI attribute management table 108.

Here, a specific example of the GUI attribute management table 108 will be described with reference to FIG. 3A. In FIG. 3A, the attribute value of the GUI element managed by the GUI attribute management table 108 includes the element ID, the overlapping order value indicating the overlapping relationship between the GUI elements, the element type indicating the type of the GUI element, the display position of the GUI element, and the display Defines size, background type, and supplementary information. Next, details of each attribute value will be described in order.

Element ID is an ID uniquely determined for each GUI element. In FIG. 3A, GUI elements corresponding to element ID 3201 to element ID 3208 are defined. The overlapping order value is the order in which the GUI is displayed, and is defined so that the larger the overlapping order value, the closer to the front (user side) is displayed on the display screen. Note that GUI elements with the same overlap order value are shown in a non-overlapping state.

Next, the element type of the GUI element indicates a display or behavior characteristic of the GUI element such as a button, a scroll bar, an image, or a container. The display position (x, y) of the GUI element is represented by the coordinate value of the xy coordinate system on the display screen 200 of the display device 102, and is the coordinate value of the upper left point of the GUI element. The display size represents the width and height of the GUI element.

The background type of the GUI element indicates how the background of the GUI element is drawn, such as whether it is painted with a single color, painted as a gradation, or whether a specified image is displayed. Information is defined. Further, the supplemental information of the GUI element is information defined individually for each GUI element depending on the element type or background type of the GUI element.

Subsequently, the features of the GUI element defined in FIG. 3A will be briefly described. In the element ID 3201 in FIG. 3A, “left / right scroll bar” is defined as the element type. In the element ID 3207, “up / down scroll bar” is defined as the element type. The element ID 3201 and the element ID 3207 define “scroll position, scroll direction” as supplementary information.

In addition, in the

element IDs

3202, 3203, 3205, and 3206, “button” is defined as the element type. Among these, since the element ID 3205 and the element ID 3206 define “gradation” as the background type, the start color and end color of the gradation are defined as supplementary information.

In addition, in a GUI element (element ID 3204) for displaying an image or a GUI element (element ID 3202 or 3203) in which the background type is designated as an image, “image” for acquiring an image to be displayed on the background is used as supplementary information. Data name "is defined.

Using such a GUI attribute management table 108, the operation target extraction unit 106 has the largest overlap order value among all the GUI elements that include the contact position Pin (Xin, Yin) in the display area of the GUI element. Extract an element as an operation target.

Here, GUI elements 201 to 208 in FIG. 2B represent the result of the rendering processing unit 104 displaying the element IDs 3201 to 3208 in the GUI attribute management table 108 in FIG. 3A on the display device 102. For example, the GUI element with the element ID 3201 represents the left / right scroll bar 201 in FIG. 2B. The same applies to the element IDs 3202 to 3208, and the GUI elements 202 to 208 in FIG. 2B are display examples.

Note that the operation target extraction unit 106 may determine a GUI element to be extracted as an operation target by an operation target availability flag for determining whether the attribute of the GUI element can be an operation target. In this case, in the GUI attribute management table 108b, as shown in FIG. 3B, an operation target propriety flag indicating whether “operation is possible” or “operation is not possible” is set. Then, the operation target extraction unit 106 refers to the GUI attribute management table 108b, can determine that the operation target availability flag is “operation possible”, and extracts the GUI element with the highest overlapping order as the GUI element to be operated. To do.

Next, a processing flow of the information terminal 100 according to the first embodiment of the present invention will be described. FIG. 4 is a flowchart showing an example of processing of the information terminal 100 according to the first embodiment of the present invention. Referring to FIG. 4, in step S <b> 401, when input device 101 detects that the user touches touch panel 1011, input processing unit 103 causes touch position Pin (Xin, Yin) of the user's fingertip on touch panel 1011 from input device 101. ) And notifies the drawing processing unit 104 and the control information generation unit 105 of the received contact position Pin (YES in step S401). Hereinafter, the contact position Pin (530, 130) will be described.

Subsequently, in step S <b> 402, the drawing processing unit 104 acquires the attribute value of the GUI element to be displayed on the display device 102 from the GUI attribute management table 108 managed by the GUI attribute management unit 107 via the GUI attribute management unit 107. Then, the GUI is drawn on the display screen 200 of the display device 102. Here, the GUI element drawn on the display screen 200 is assumed to be FIG. 2B as described above.

Subsequently, in step S403, the control information generation unit 105 notifies the contact position Pin (530, 130) notified from the input processing unit 103, and extracts the operation target GUI element so as to extract the operation target GUI element. Is notified of the operation target extraction instruction.

Subsequently, in step S404, the operation target extraction unit 106 extracts a GUI element including the contact position Pin (530, 130) in the display area from the GUI attribute management table managed by the GUI attribute management unit 107. Here, it is assumed that the operation target extraction unit 106 extracts the element ID 3205 and the element ID 3208. Subsequently, as described above, the operation target extraction unit 106 extracts, from the element ID 3205 and the element ID 3208, the GUI element of the element ID 3205 having the higher overlapping order value as the operation target.

Subsequently, in step S <b> 405, the control information generation unit 105 generates control information from the attribute value of the GUI element extracted by the operation target extraction unit 106 and the input information acquired by the input acquisition unit 103. Notify the playback device 109.

Finally, in step S406, the tactile reproduction device 109 generates a frictional force as a tactile sensation on the touch panel 1011 in contact with the user's finger according to the control information notified from the control information generating unit 105.

Here, the control information generated by the control information generation unit 105 in step S405 will be described with reference to FIGS. 5A to 5C. 5A to 5C show a correspondence relationship between the scroll position X of the scroll bar element (for example, the GUI element 201 in FIG. 2B) and the frictional force F. FIG. As shown in FIGS. 5A to 5C, the control information generation unit 105 dynamically changes the frictional force F according to the scroll position X of the scroll bar element.

The graph of FIG. 5A shows a state in which the center of the scroll bar element is the minimum value of the friction force F, and the friction force F gradually increases toward the left end and the right end. For example, in the graph of FIG. 5A, the frictional force F is given by a relational expression such as F = a · X ² (a is a positive coefficient). The control information generation unit 105 can calculate the control information from the contact position Pin by holding such a relational expression. Thereby, the user can sense the scroll position with the fingertip by the change of the frictional force.

The frictional force F is given by a relational expression such as F = a · X ² , but is not limited to this relational expression. For example, the frictional force F may be given by a function such as F = a · X + b (b is an arbitrary value) or an arbitrary function. Further, the control information generation unit 105 calculates the friction force F by holding the correspondence relationship between the friction force F and the scroll position X as table information instead of holding the relational expression as described above. May be.

Further, as shown in the graph of FIG. 5B, the control information generation unit 105 may set the center of the scroll bar element as the maximum value of the frictional force and gradually decrease the frictional force F toward the left end and the right end. In the graph of FIG. 5B, the frictional force F is given by a relational expression such as F = c · X ² (where c is a negative coefficient). Similarly, as shown in the graph of FIG. 5C, the control information generation unit 105 sets the left end or the right end as the minimum value of the frictional force and gradually increases the frictional force F toward the right end or the left end. Good.

Also, in the slider element (a GUI element in which a value can be input with a GUI by sliding the knob with a finger), the frictional force can be changed according to the slider position in the same manner as the scroll bar element. The slider element showing the frictional force as described above can be applied not only to a screen scroll operation as shown in FIG. 2B but also to an operation such as volume adjustment. Further, the control information generation unit 105 may change the frictional force in accordance with the volume level reproduced by the information terminal 100.

Next, the basic operation of the haptic reproduction device 109 will be described. For the basic principle of presenting a tactile sensation to the user in the tactile reproduction device 109, for example, the technique disclosed in Non-Patent Document 1 can be applied.

Here, the basic principle of tactile presentation according to Non-Patent Document 1 will be briefly described. In Non-Patent Document 1, the deformation on the skin surface of the fingertip that occurs when the user traces the object surface with a fine roughness at a velocity v using the fingertip is expressed as a superposition formula of sinusoidal traveling waves. The sinusoidal traveling wave is characterized by a spatial frequency k and a time frequency ω (ω = kv). In Non-Patent Document 1, this sine traveling wave superposition formula and a similar sine traveling wave (in Non-Patent Document 1, an elastic wave called a Lamb wave is used) are intentionally generated and sent to the user's fingertip. Tell. At this time, deformation of the skin surface similar to when a rough object is traced occurs at the fingertip of the user who has detected the elastic wave, and the user feels a tactile sensation as if he / she traced the rough object.

Using the basic principle of the tactile sensation presented in Non-Patent Document 1, the haptic reproduction device 109 converts the strength of the frictional force notified from the control information generation unit 105 into a rough feeling, and copes with the rough feeling. By exciting elastic waves on the touch panel 1011, a tactile sensation is given to the user's fingertip that has touched the touch panel 1011. The traveling direction of the elastic wave is a direction defined by the unit vector notified from the control information generation unit 105.

In addition, the tactile reproduction device 109 holds, as an index for defining a rough object surface, a correspondence relationship among the friction coefficient μ of the object surface, the spatial frequency k of the elastic wave, and the time frequency ω as table information in advance. It may be left. Here, it is defined that a large friction coefficient is associated with an object having a large roughness, and a small friction coefficient is associated with an object having a small roughness. Thus, the tactile reproduction device 109 uses the frictional force F notified by the control information generating unit 105 using the relational expression F = μN (N is a force acting in the vertical direction when the user traces the object surface). It is possible to calculate the friction coefficient μ from the force F and generate an elastic wave corresponding to the calculated friction coefficient μ. An appropriate value may be set in advance for the force N acting in the vertical direction when the user traces the touch panel 1011.

As described above, according to the haptic control information generation device according to the first embodiment of the present invention, it is possible to reproduce a haptic according to the feature of the GUI on the fingertip of the user who operates the touch panel. Thereby, an intuitive GUI operation can be provided to a user who performs a tracing operation on the GUI on the touch panel.

(Second Embodiment)
In the second embodiment, a variation to which the present invention is applied will be described. Since the configuration of the information terminal 100 including the haptic control information generation device according to the second embodiment and the configuration of the display device 102 are the same as those in the first embodiment, FIGS. Moreover, FIG. 2B is used also about the GUI element displayed on the display screen 200. FIG.

(First variation)
FIG. 6 is a diagram for explaining a correspondence relationship between a tactile sensation reproduced on a user's fingertip and a GUI element in the second embodiment of the present invention. FIG. 6 illustrates the relationship between the element ID 3205 (the enlargement button 205 in FIG. 2B) extracted by the operation target extraction unit 106, the touch position Pin of the touch panel by the user, and the strength of the frictional force generated at the touch position Pin. ing.

In the graph of FIG. 6, the frictional force F that gradually increases from the display center portion (point O) of the GUI element toward the display boundary direction, the frictional force F at the contact position Pin, and the contact position Pin from the display center portion. It is illustrated as a relationship with the distance D to. For example, in the graph of FIG. 6, the frictional force F is given by a relational expression such as F = a × D ² (a is an arbitrary coefficient). The control information generation unit 105 can calculate the control information from the contact position Pin by holding such a relational expression. In addition, the control information generation unit 105 may calculate the frictional force F from any relational expression other than those described above, and the correspondence information between the frictional force F and the distance D from the display center to the contact position is table information. As a result, the frictional force F may be calculated.

The control information generation unit 105 generates a frictional force F that gradually increases in the GUI element display boundary direction from the attribute value of the GUI element extracted by the operation target extraction unit 106 and the input information acquired by the input acquisition unit 103. Is generated and notified to the tactile playback device 109 as control information. Based on the notified frictional force F, the tactile reproduction device 109 generates a frictional force as a tactile sensation on the touch panel 1011 in contact with the user's finger. Thereby, the user who operates the touch panel can feel the display boundary of the GUI element with the fingertip even if the touch panel is a flat touch panel.

Note that the direction in which the frictional force is generated may be reversed. This is because the frictional force F gradually changes toward the display boundary direction of the GUI element, so that the user can feel the display boundary of the GUI element with the fingertip.

Further, although the control information generation unit 105 generates the frictional force F, when the tactile reproduction device 109 can generate the propulsive force, the control information generation unit 105 may generate the propulsive force. For example, the control information generation unit 105 can control to guide the user's fingertip to a specific GUI element by generating a driving force toward the center of the area of the specific GUI element. Further, the direction in which the propulsive force is generated may be reversed.

(Second variation)
In addition to the frictional force and the operation direction of the touch panel of the user, the control information generation unit 105 is based on the relationship between the movement distance length from the previous user contact position to the current contact position Pin and the elapsed time from the previous input. The user's tracing speed v (v = movement distance / elapsed time) can be calculated. For this reason, the control information generation part 105 can also change a frictional force according to a user's tracing speed. Specifically, the control information generation unit 105 generates a new friction force Fnew (= v · F) obtained by multiplying the friction force F shown in the graphs of FIGS. 5A, 5B, 5C, 6 and the like by the tracing speed v. ) Can be notified to the tactile reproduction device 109. The new frictional force Fnew can also be calculated by any relational expression other than Fnew = v · F.

As described above, by increasing the frictional force according to the tracing speed, the user can perform the tracing operation of the GUI element too early, thereby suppressing the early operation on the GUI element. For example, in a music player, if the slider element controls the volume, it is possible to suppress a sudden increase in volume due to a quick slider operation by the user.

Further, when calculating the control information, the control information generation unit 105 may determine whether to multiply the friction force by the tracing speed according to the element type of the GUI element. As this determination method, for example, “multiply by the tracing speed” is set to the supplementary information of the table information shown in FIG. 9 described later. The control information generation unit 105 can determine that the friction force is multiplied by the tracing speed by referring to the supplementary information of the table information.

(Third variation)
The control information generation unit 105 may generate a luminance distribution from the background color of the GUI element, and may generate control information that increases the frictional force as the luminance of the acquired background color increases. For example, FIG. 7 shows an example in which a luminance distribution is generated from a GUI element whose background color is gradation. If the gradation is such that the background color is brighter from the center of the display area of the GUI element toward the periphery of the area, the luminance distribution is as shown in FIG. In this case, the control information generation unit 105 generates control information such that the frictional force of the GUI element increases from the center of the GUI element with low luminance toward the region boundary with high luminance. Further, the correspondence relationship between the luminance and the frictional force may be reversed, or a driving force may be generated instead of the frictional force. By changing the generated frictional force or propulsive force in accordance with the change in luminance of the background color of the GUI element, the user can feel the luminance of the GUI element with the fingertip.

(Fourth variation)
A case is assumed in which a plurality of GUI elements are arranged in pairs such as

GUI elements

202 and 203 defined as buttons shown in FIG. 2B. In such a case, the control information generation unit 105 may arrange a GUI element 209 between the GUI element 202 and the GUI element 203 as shown in FIG. The control information generation unit 105 changes the frictional force F generated on the GUI element 209 according to the contact position between the user's fingertip and the GUI element 209. FIG. 8 shows a state in which the center of the GUI element 209 is set to the minimum value of the frictional force, and the frictional force F gradually increases toward the GUI element 202 and the GUI element 203. As a result, the user who operates the touch panel can estimate the positions of the buttons (GUI elements 202 and 203) arranged in pairs by the change in the frictional force of the GUI element 209 even in the case of a flat touch panel. it can. Note that the frictional force F generated on the GUI element 209 is not limited to the above example, and may change as shown in FIGS. 5A to 5C, for example.

(5th variation)
The control information generation unit 105 may switch the control information calculation method according to the GUI element extracted by the operation target extraction unit 106. FIG. 9 is a diagram illustrating an example of table information indicating a correspondence relationship between an attribute value of a GUI element and a control information calculation method. FIG. 9 shows a control information calculation method and supplementary information corresponding to the element type and the background type. In the example shown in FIG. 9, it is described that the relational expressions described in FIGS. 5A, 5B, 6, and 7 described above are used as the control information calculation method.

For example, when calculating the control information, the control information generation unit 105 refers to the table information in FIG. 9 and, if the element type of the GUI element is “scroll bar”, as shown in the graph of FIG. 5A. Control information is calculated using a relational expression. Further, if the element type of the GUI element is “slide bar”, the control information generation unit 105 says that the supplementary information is “multiply the tracing speed”, so the relational expression as shown in the graph of FIG. Control information is calculated by a relational expression multiplied by the tracing speed. In addition, if the element type of the GUI element is “button” and the background type is “gradation”, the control information generation unit 105, as described with reference to FIG. Generate control information that increases the frictional force of the elements.

Each processing procedure performed by the haptic control information generation device according to the embodiment of the present invention described above is predetermined program data that can execute the processing procedure described above stored in a storage device (ROM, RAM, hard disk, etc.). May be realized by being interpreted and executed by the CPU. In this case, the program data may be introduced into the storage device via the storage medium, or may be directly executed from the storage medium. Note that the storage medium refers to a semiconductor memory such as a ROM, a RAM, and a flash memory, a magnetic disk memory such as a flexible disk and a hard disk, an optical disk memory such as a CD-ROM, a DVD, and a BD, and a memory card. The storage medium is a concept including a communication medium such as a telephone line or a conveyance path.

Further, each component of the haptic control information generation apparatus (FIG. 1) according to the embodiment of the present invention described above can be realized as an LSI which is an integrated circuit. These may be individually made into one chip, or may be made into one chip so as to include all or part of each. Although referred to as LSI here, it may be referred to as IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

Also, the method of circuit integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used. Further, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Possible applications of biotechnology are possible.

The tactile control information generation device according to the present invention is useful for improving the operability of the GUI in an information terminal equipped with a touch panel display.

DESCRIPTION OF SYMBOLS 100 Information terminal 101 Input device 1011 Touch panel 102 Display device 103 Input processing part 104 Drawing processing part 105 Control information generation part 106 Operation target extraction part 107 GUI attribute management part 108 GUI attribute management table 109 Tactile reproduction apparatus 200 Display screen 201 Scroll bar 202 Button 203 Button 204 Image 205 Button 206 Button 207 Scroll Bar 208 Container 209 GUI Element

Claims

A tactile control information generation device that controls a tactile reproduction device that reproduces a tactile sensation on a part of a user's body that operates a touch panel display,
A GUI attribute management table that holds attribute values of GUI elements displayed on the touch panel display;
A GUI attribute management unit for managing the GUI attribute management table;
A drawing processing unit that acquires an attribute value of a GUI element to be drawn from the GUI attribute management unit, draws the GUI element based on the acquired attribute value, and outputs the drawn GUI element to the touch panel display;
An input processing unit that acquires input information from the touch panel display, notifies the acquired input information to the drawing processing unit, and instructs the drawing processing unit to draw a GUI element;
An operation target extraction unit that determines a GUI element to be operated by the user from the input information acquired by the input processing unit and the attribute value of the GUI element managed by the GUI attribute management unit;
The attribute value of the GUI element determined as the operation target by the operation target extraction unit is acquired from the GUI attribute management unit, and the attribute value of the acquired GUI element and the input information acquired by the input acquisition unit are When a part of a user's body that operates a touch panel display touches a GUI element displayed on the touch panel display, the haptic reproduction device is configured to reproduce a haptic sensation on the part of the user's body. A tactile control information generation device comprising a control information generation unit that generates control information to be instructed.
The control information generation unit reproduces a tactile sensation to be reproduced on a part of the user's body as the strength of frictional force or propulsive force, and sets the attribute value of the GUI element determined as the operation target by the operation target extraction unit. When the acquired GUI element is a scroll bar element or a slider element, the scroll position of the scroll bar element or the slider position of the slider element is obtained from the GUI attribute management unit. 2. The haptic control information generation according to claim 1, wherein the control information is generated so as to change a frictional force reproduced by the haptic generation device according to the acquired scroll position or slider position. apparatus.
The control information generation unit reproduces the tactile sensation to be reproduced on a part of the user's body as the strength of frictional force or propulsive force, and the attribute value of the GUI element determined as the operation target by the operation target extraction unit. Among them, the attribute value that defines the display area of the GUI element is acquired from the GUI attribute management unit, and the frictional force that the tactile sensation generating device reproduces from the center of the acquired display area of the GUI element toward the display boundary direction. The haptic control information generation device according to claim 1, wherein the control information is generated such that a propulsive force changes.
The control information generation unit reproduces the tactile sensation to be reproduced on a part of the user's body as the strength of frictional force or propulsive force, and the attribute value of the GUI element determined as the operation target by the operation target extraction unit. Among them, the background color information of the GUI element is acquired from the GUI attribute management unit, the brightness of the background color of the GUI element is generated from the acquired background color information, and the tactile sensation is generated according to the brightness of the background color The haptic control information generation device according to claim 1, wherein the control information is generated such that a frictional force reproduced by the device changes.
The drawing processing unit draws a new GUI element between a plurality of GUI elements arranged in pairs,
The control information generation unit reproduces the tactile sensation to be reproduced on a part of the user's body as the strength of frictional force or propulsive force, and the attribute value of the GUI element determined as the operation target by the operation target extraction unit. Among them, the frictional force that the tactile sensation generating device reproduces according to the contact position of the user with respect to the acquired GUI element display area is acquired from the GUI attribute management unit an attribute value that defines the display area of the GUI element The haptic control information generation device according to claim 1, wherein the control information is generated such that a propulsive force changes.
The control information generation unit reproduces a tactile sensation to be reproduced on a part of the user's body as a frictional force or a propulsive force, and the tactile sensation is generated according to a tracing speed of the user tracing the touch panel display. The haptic control information generation device according to claim 1, wherein the control information is generated such that a frictional force or a propulsive force reproduced by the device changes.
The haptic control information generation according to claim 6, wherein the control information generation unit generates the control information such that a frictional force or a propulsive force reproduced by the haptic generation device increases as the tracing speed increases. apparatus.
The control information generation unit reproduces the tactile sensation reproduced on a part of the user's body as the strength of the frictional force or the propulsive force, and the frictional force reproduced by the tactile sensation generating device according to the magnitude of the generated sound volume. The haptic control information generation device according to claim 1, wherein the control information is generated such that a propulsive force changes.
A method executed by a haptic control information generation device that controls a haptic reproduction device that reproduces a haptic sensation on a part of a user's body that operates a touch panel display,
The tactile control information generation device includes a GUI attribute management table that holds attribute values of GUI elements displayed on the touch panel display.
A drawing process step of acquiring an attribute value of a GUI element to be drawn from the GUI attribute management table, drawing the GUI element based on the acquired attribute value, and outputting the drawn GUI element to the touch panel display;
An input processing step of acquiring input information from the touch panel display, notifying the acquired input information to the drawing processing step, and instructing the drawing processing step to draw a GUI element;
An operation target extraction step for determining a GUI element to be operated by the user from the input information acquired by the input processing step and the attribute value of the GUI element held by the GUI attribute management table;
The attribute value of the GUI element determined to be the operation object by the operation object extraction step is acquired from the GUI attribute management table, and from the acquired attribute value of the GUI element and the input information acquired by the input acquisition step, the When a part of a user's body that operates a touch panel display touches a GUI element displayed on the touch panel display, the haptic reproduction device is configured to reproduce a haptic sensation on the part of the user's body. A control information generating step of generating control information to be instructed.
A program to be executed by a haptic control information generation device that controls a haptic reproduction device that reproduces a haptic sensation on a part of a user's body that operates a touch panel display,
The tactile control information generation device includes a GUI attribute management table that holds attribute values of GUI elements displayed on the touch panel display.
A drawing process step of acquiring an attribute value of a GUI element to be drawn from the GUI attribute management table, drawing the GUI element based on the acquired attribute value, and outputting the drawn GUI element to the touch panel display;
An input processing step of acquiring input information from the touch panel display, notifying the acquired input information to the drawing processing step, and instructing the drawing processing step to draw a GUI element;
An operation target extraction step for determining a GUI element to be operated by the user from the input information acquired by the input processing step and the attribute value of the GUI element held by the GUI attribute management table;
The attribute value of the GUI element determined to be the operation object by the operation object extraction step is acquired from the GUI attribute management table, and from the acquired attribute value of the GUI element and the input information acquired by the input acquisition step, the When a part of a user's body that operates a touch panel display touches a GUI element displayed on the touch panel display, the haptic reproduction device is configured to reproduce a haptic sensation on the part of the user's body. A program for executing a control information generation step for generating control information to be instructed.
An integrated circuit that controls a haptic reproduction device that reproduces a haptic sensation on a part of a user's body that operates a touch panel display,
A GUI attribute management table that holds attribute values of GUI elements displayed on the touch panel display;
A GUI attribute management unit for managing the GUI attribute management table;
A drawing processing unit that acquires an attribute value of a GUI element to be drawn from the GUI attribute management unit, draws the GUI element based on the acquired attribute value, and outputs the drawn GUI element to the touch panel display;
An input processing unit that acquires input information from the touch panel display, notifies the acquired input information to the drawing processing unit, and instructs the drawing processing unit to draw a GUI element;
An operation target extraction unit that determines a GUI element to be operated by the user from the input information acquired by the input processing unit and the attribute value of the GUI element managed by the GUI attribute management unit;
The attribute value of the GUI element determined as the operation target by the operation target extraction unit is acquired from the GUI attribute management unit, and the attribute value of the acquired GUI element and the input information acquired by the input acquisition unit are When a part of a user's body that operates a touch panel display touches a GUI element displayed on the touch panel display, the haptic reproduction device is configured to reproduce a haptic sensation on the part of the user's body. An integrated circuit comprising: a control information generation unit that generates control information to be instructed.