JP5496032B2 - Tactile sensation presentation apparatus and control method for tactile sensation presentation apparatus - Google Patents

Description

  The present invention relates to a tactile sensation providing apparatus and a control method for the tactile sensation providing apparatus.

  In recent years, in order to display an object three-dimensionally, in the field of computer graphics, a method of displaying unevenness on the surface of an object by bump mapping or the like and displaying a three-dimensional image is generally used. Furthermore, in addition to 3D video display, a method of presenting a tactile sensation when touching an object using a tactile display has been proposed. For example, the shape of an object is read by a three-dimensional scanner, information on the shape of the object is sent to the tactile display, and the tactile display drives the control pins arranged in a grid based on the received information, There has been proposed a method of expressing the shape (see, for example, Patent Document 1).

JP 2003-3116299 A

  However, an object displayed as a three-dimensional image by bump mapping is apparently recognized as a three-dimensional object due to unevenness, but is only a plane on the data. For this reason, the unevenness cannot be expressed by a tactile display as disclosed in Patent Document 1.

  Further, in Patent Document 1, for example, in order to read fine irregularities such as the surface of a cloth with a three-dimensional scanner and present a tactile sensation with a tactile display, a very high resolution tactile display is required, and the configuration is complicated. It becomes. Moreover, the tactile display disclosed in Patent Document 1 can present tactile sensation only for data that can be measured by a three-dimensional scanner. Therefore, for example, it is impossible to present a realistic tactile sensation for a water ripple, a flag swaying in the wind, and the like included in a two-dimensional image (moving image).

  Accordingly, an object of the present invention made in view of such points is to provide a tactile sensation providing apparatus and a control method for the tactile sensation providing apparatus that can present a realistic tactile sensation from two-dimensional image data with a simple configuration.

The invention of the tactile sensation providing apparatus according to the first aspect to achieve the above object is as follows:
A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A control unit,
The controller is
A coordinate calculation unit that calculates coordinates on the touch surface of an object that touches the touch surface based on information of the touch sensor;
An image three-dimensionalization unit that three-dimensionalizes the image of the predetermined area including the coordinates calculated by the coordinate calculation unit;
A normal vector calculation unit that calculates a normal vector of a surface of the image based on the three-dimensional image;
A touch surface movement vector calculation unit for calculating a movement vector of the object on the touch surface;
A friction vector calculation unit that calculates a friction vector that the image in the predetermined region should give to the object as friction based on the movement vector and the normal vector;
A tactile sensation control unit that controls driving of the tactile sensation providing unit based on the friction vector;
It is characterized by providing.

The invention of the tactile sensation providing apparatus according to the second aspect of achieving the above object is as follows:
A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A control unit,
The controller is
A coordinate calculation unit that calculates coordinates on the touch surface of an object that touches the touch surface based on information of the touch sensor;
An image three-dimensionalization unit that three-dimensionalizes the image of the predetermined area including the coordinates calculated by the coordinate calculation unit;
A normal vector calculation unit that calculates a normal vector of a surface of the image based on the three-dimensional image;
An image movement vector calculation unit for calculating a movement vector of the image;
A friction vector calculation unit that calculates a friction vector that the image in the predetermined area should give to the object as friction based on the normal vector and the movement vector of the image;
A tactile sensation control unit that controls driving of the tactile sensation providing unit based on the friction vector;
It is characterized by providing.

The invention of the tactile sensation providing apparatus according to the third aspect of achieving the above object is as follows:
A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A control unit,
The controller is
A coordinate calculation unit that calculates coordinates on the touch surface of an object that touches the touch surface based on information of the touch sensor;
An image three-dimensionalization unit that three-dimensionalizes the image of the predetermined area including the coordinates calculated by the coordinate calculation unit;
A normal vector calculation unit that calculates a normal vector of a surface of the image based on the three-dimensional image;
A touch surface movement vector calculation unit for calculating a movement vector of the object on the touch surface;
An image movement vector calculation unit for calculating a movement vector of the image;
A friction vector calculation unit that calculates a friction vector that the image in the predetermined region should give to the object as friction based on the movement vector, the normal vector, and the movement vector of the image;
A tactile sensation control unit that controls driving of the tactile sensation providing unit based on the friction vector;
It is characterized by providing.

The invention of the control method of the tactile sensation providing apparatus according to the fourth aspect of achieving the above object,
A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A tactile sensation presentation apparatus comprising a control unit,
Calculate coordinates on the touch surface of an object touching the touch surface based on the information of the touch sensor,
Three-dimensionalize the image of the predetermined area including the coordinates calculated by the coordinate calculation unit,
Based on the three-dimensional image, calculate a normal vector of the surface of the image,
Calculating a movement vector of the object on the touch surface;
Based on the calculation result of the movement vector and the normal vector, calculate a friction vector that the image in the predetermined region should give to the object as friction,
The drive of the tactile sensation providing unit is controlled based on the friction vector.

The invention of the control method of the tactile sensation providing apparatus according to the fifth aspect of achieving the above object,
A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A tactile sensation presentation device comprising a control unit,
Calculate coordinates on the touch surface of an object touching the touch surface based on the information of the touch sensor,
Three-dimensionalize the image of the predetermined area including the coordinates calculated by the coordinate calculation unit,
Based on the three-dimensional image, calculate a normal vector of the surface of the image,
Calculating a movement vector of the image;
Based on the normal vector and the movement vector of the image, calculate a friction vector that the image in the predetermined region should give to the object as friction,
The drive of the tactile sensation providing unit is controlled based on the friction vector.

The invention of the control method of the tactile sensation providing apparatus according to the sixth aspect of achieving the above object is as follows:
A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A tactile sensation presentation device comprising a control unit,
Calculate coordinates on the touch surface of an object touching the touch surface based on the information of the touch sensor,
Three-dimensionalize the image of the predetermined area including the coordinates calculated by the coordinate calculation unit,
Based on the three-dimensional image, calculate a normal vector of the surface of the image,
Calculating a movement vector of the object on the touch surface;
Calculating a movement vector of the image;
Based on the movement vector, the normal vector, and the movement vector of the image, calculate a friction vector that the image in the predetermined region should give to the object as friction,
The drive of the tactile sensation providing unit is controlled based on the friction vector.

  According to the present invention, it is possible to provide a tactile sensation providing apparatus capable of presenting a realistic tactile sensation from two-dimensional image data and a control method for the tactile sensation providing apparatus with a simple configuration.

1 is a block diagram illustrating a schematic configuration of a tactile sensation providing apparatus according to a first embodiment of the present invention. It is a flowchart which shows operation | movement of the tactile sense presentation apparatus shown in FIG. It is a flowchart which shows the image three-dimensionalization process in step S104 shown in FIG. It is a figure for demonstrating the image three-dimensionalization process in step S104 shown in FIG. It is a figure for demonstrating the image three-dimensionalization process in step S104 shown in FIG. It is a figure for demonstrating the image three-dimensionalization process in step S104 shown in FIG. It is a figure for demonstrating the image three-dimensionalization process in step S104 shown in FIG. It is a flowchart which shows the normal vector calculation process in step S105 shown in FIG. It is a figure for demonstrating the normal vector calculation process in step S105 shown in FIG. It is a figure for demonstrating the normal vector calculation process in step S105 shown in FIG. It is a flowchart which shows the touch surface movement vector calculation process in step S106 shown in FIG. It is a figure for demonstrating the touch surface movement vector calculation process in step S106 shown in FIG. It is a flowchart which shows the friction vector process in step S107 shown in FIG. It is a figure for demonstrating the friction vector process in step S107 shown in FIG. It is a figure for demonstrating the friction vector process in step S107 shown in FIG. It is a figure for demonstrating an example of the specific operation | movement of the tactile sense presentation apparatus shown in FIG. It is a figure for demonstrating an example of the specific operation | movement of the tactile sense presentation apparatus shown in FIG. It is a figure for demonstrating an example of the specific operation | movement of the tactile sense presentation apparatus shown in FIG. It is a figure for demonstrating other examples of the specific operation | movement of the tactile sense presentation apparatus shown in FIG. It is a figure for demonstrating other examples of the specific operation | movement of the tactile sense presentation apparatus shown in FIG. It is a figure for demonstrating other examples of the specific operation | movement of the tactile sense presentation apparatus shown in FIG. It is a functional block diagram which shows roughly the principal part structure of the tactile sense presentation apparatus which concerns on 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the tactile sense presentation apparatus shown in FIG. It is a flowchart which shows the image movement vector calculation process in step S602 shown in FIG. It is a figure for demonstrating the image movement vector calculation process in step S602 shown in FIG. It is a figure for demonstrating the image movement vector calculation process in step S602 shown in FIG. It is a figure for demonstrating the image movement vector calculation process in step S602 shown in FIG. It is a figure for demonstrating an example of operation | movement of the tactile sense presentation apparatus shown in FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
FIG. 1 is a functional block diagram schematically showing a main configuration of the tactile sensation providing apparatus according to the first embodiment of the present invention. As shown in FIG. 1, the tactile sensation providing apparatus 100 includes a touch sensor 1, a display unit 2, a tactile sensation providing unit 3, and a control unit 4.

  When the touch sensor 1 detects that an object such as an operator's finger or a pen-type input device such as a stylus pen touches (contacts) the touch surface, the touch sensor 1 outputs a signal including position information. Hereinafter, in the present embodiment, the object is described as an operator's finger. The touch sensor 1 can be configured by a known sensor that outputs position information of a touch position, such as a resistive film method, a capacitance method, and an optical method.

  The display unit 2 displays a still image or a moving image, and is configured using, for example, a liquid crystal display panel or an organic EL display panel. The touch sensor 1 is disposed on the display unit 2. The operator visually observes an image displayed on the display unit 2 through the touch sensor 1.

  The tactile sensation providing unit 3 vibrates the touch surface of the touch sensor 1 and is configured using, for example, a piezoelectric vibrator or an eccentric motor. The tactile sensation providing unit 3 varies the vibration frequency and driving pattern in various ways to stimulate the operator's tactile sensation by giving a tactile sensation, and gives information through vision through an image displayed on the display unit 2. A realistic tactile sensation is generated as if the operator is touching an object displayed on the display unit 2.

  The control unit 4 includes a coordinate calculation unit 5, an image stereoscopic unit 6, a normal vector calculation unit 7, a friction vector calculation unit 9, a touch surface movement vector calculation unit 8, and a tactile sensation control unit 10. Then, the entire tactile sensation providing apparatus 100 is controlled. The coordinate calculation unit 5 calculates the coordinates on the touch surface of the operator's finger that touches the touch surface based on the position information output from the touch sensor 1. Specifically, the coordinates on the touch surface correspond to the coordinates of the display unit 2. For example, when the touch surface and the display unit 2 are rectangular, a predetermined corner is the origin (0, 0). It is an xy coordinate system. The image three-dimensionalization unit 6 three-dimensionalizes an image of a predetermined area including the coordinates calculated by the coordinate calculation unit 5.

  The normal vector calculation unit 7 calculates a normal vector of the surface of the image based on the three-dimensional image. The touch surface movement vector calculation unit 8 calculates a movement vector on the touch surface of the operator's finger. The friction vector calculation unit 9 calculates, based on the movement vector and the normal vector, a friction vector to be given to the operator's finger as a friction by the image in the predetermined region that is three-dimensionalized by the image three-dimensionalization unit 6. . The tactile sensation control unit 10 controls driving of the tactile sensation providing unit 3 based on the friction vector.

  Each functional block included in the control unit 4 may be configured by a single CPU or may be configured by an individual DSP (Digital Signal Processor).

  FIG. 2 is a flowchart showing the operation of the tactile sensation providing apparatus shown in FIG. When the tactile sensation providing apparatus 100 is activated, the control unit 4 performs a loop until it is finished (S101, S102). The coordinate calculation unit 5 calculates coordinates based on a signal including position information output from the touch sensor 1 (S103). When the coordinate calculation unit 5 does not calculate the coordinates, the control unit 4 performs a loop until the system ends. On the other hand, when the coordinate calculation unit 5 calculates the coordinates, the image three-dimensionalization unit 6 performs image three-dimensionalization by processing to be described later (S104).

  Then, the normal vector calculation unit 7 calculates a normal vector by a process described later (S105). And the touch surface movement vector calculation part 8 calculates the movement vector on the touch surface of an operator's finger | toe by the process mentioned later (S106). Then, the friction vector calculation unit 9 gives the image in the region three-dimensionalized in step S104 to the operator's finger that touches the touch surface based on the normal vector and the movement vector. The direction and magnitude of the friction (friction vector) are calculated (S107). The tactile sensation control unit 10 controls the driving of the tactile sensation providing unit 3 based on the friction vector calculated by the friction vector calculation unit 9 (S108).

  Next, the operation of each unit described above will be described. FIG. 3 is a flowchart showing the image stereoscopic processing in step S104 shown in FIG. Based on the coordinates calculated by the coordinate calculation unit 5, the image three-dimensionalization unit 6 extracts a plurality of pixels P in the display unit 2 corresponding to the peripheral area of the coordinates (S201). Specifically, as shown in FIG. 4, assuming a circle B having a radius of 6 pixels centering on the position A of the coordinates corresponding to the center of the area where the operator's finger touches the touch surface. A plurality of pixels P passing through the circle and the circumference line are extracted. Then, the image stereoscopic unit 6 obtains the luminance of the plurality of pixels P extracted in step S201 (S202). In this embodiment, the case of six pixels has been described. However, the number of pixels may be set as appropriate, and the extraction may not be based on a circle.

For example, when the input image is an RGB color image, ITU-R (International Telecommunication Union) BT. According to the 709 standard, the luminance is calculated according to the following equation.
Y = 0.2126 × R + 0.7152 × G + 0.0722 × B
According to this equation, the luminance when the RGB components are combined is calculated. The calculated luminance distribution in the plurality of pixels P is, for example, as shown in FIG. Note that the XY plane illustrated in FIG. 5 corresponds to the XY plane of the display unit 2.

  Then, the image three-dimensionalization unit 6 applies the luminance calculated in step S202 to each pixel P as the height component (z component) of the plurality of pixels P (S203). This process is a process for determining a low-luminance (dark) part included in an image as a boundary part in an image that presents a tactile sensation. Specifically, as shown in FIG. 6, each pixel P is three-dimensionalized based on the luminance by the process of step S <b> 203. Then, as shown in FIG. 7, the image solid lower part 6 generates a surface by connecting the center points of the adjacent pixels (S204). By this processing, a surface having three vertices (that is, a triangular surface) is formed in a region where there is a difference in luminance between adjacent pixels, which has not been formed at the time of step S203.

  FIG. 8 is a flowchart showing normal vector calculation processing in step S105 shown in FIG. The normal vector calculation unit 7 shares one vertex of the three sides constituting the surface for all the inclined surfaces (the surfaces having different values of at least one of the three vertices). When these two sides are regarded as vectors, their outer product is calculated (S301, see FIG. 9). In FIG. 9, the arrows indicate the outer product for each surface. Thereby, a vector (that is, a normal vector) perpendicular to the surface can be calculated. Further, the normal vector calculation unit 7 does not calculate a normal vector for a surface having no inclination. This is because a non-inclined surface can ignore the influence of friction on the direction of movement of an operator's finger moving on the touch surface, so that calculation of friction is unnecessary. Then, as shown in FIG. 10, the normal vector calculation unit 7 projects the normal calculated in step S301 on the XY plane, and removes the z component of the normal vector (S302). This facilitates calculation of the inner product of the normal vector and the movement vector (corresponding to step S107) in step S501 described later.

  FIG. 11 is a flowchart showing the touch surface movement vector calculation process in step S106 shown in FIG. The touch surface movement vector calculation unit 8 acquires information of the immediately preceding coordinate A ′ (see FIG. 12) calculated by the coordinate calculation unit 5 (S401). Subsequently, the touch surface movement vector calculation unit 8 acquires information on the current coordinate A (see FIG. 12) calculated by the coordinate calculation unit 5 (S402). Then, the touch surface movement vector calculation unit 8 calculates a movement vector (A′−A) of the finger of the operator who moves on the touch surface based on the coordinate information acquired in Step S401 and Step S402 ( S403, see FIG.

  FIG. 13 is a flowchart showing the friction vector processing in step S107 shown in FIG. The friction vector calculation unit 9 calculates the inner product of all the normal vectors calculated by the normal vector calculation unit 7 and the movement vector calculated by the touch surface movement vector calculation unit 8 (S501). Here, the value of the inner product calculated by the friction vector calculation unit 9 in step S501 is a positive value for a surface where the angle formed by the normal vector and the touch surface movement vector is less than 90 degrees, as shown in FIG. As for the surface where the angle formed by the two vectors to be calculated is greater than 90 degrees, it is calculated as a negative value.

  That is, if the value of the inner product is positive, the direction of the normal vector is a direction that includes a component in the same direction as the movement direction of the operator's finger on the touch surface (the angle difference is within 90 degrees), On the other hand, if the value of the inner product is negative, the direction of the normal vector is a direction including a component opposite to the moving direction of the operator's finger on the touch surface. In the latter case, it is necessary to generate vibration by the tactile sensation providing unit 3 on the operator's finger.

  Then, the friction vector calculation unit 9 calculates the sum of the negative inner products among the inner products calculated in step S501 (S502). The friction vector calculation unit 9 is opposite to the movement vector and has a magnitude calculated in step S502 in order to calculate a friction vector to be given as friction to the operator's finger moving on the touch surface. A vector that is the sum is obtained as a friction vector (S503, see FIG. 15).

  16-18 is a figure for demonstrating an example of the specific operation | movement of the tactile sense presentation apparatus shown in FIG. In this example, an image of a fish having a scale is displayed on the display unit 2. The tactile sensation control unit 10 controls the driving of the tactile sensation providing unit 3 based on the direction and magnitude of the friction vector obtained in step S503. Specifically, when the finger is slid in the right direction as the scale, the vibration (friction) applied to the operator's finger is reduced to recognize that the three-dimensional friction (friction vector) is small. The tactile sensation is presented, and when the finger is slid in the opposite left direction, the vibration is increased to present a tactile sensation that recognizes that the three-dimensional friction (friction vector) is large. In FIG. 16, the width of the arrow corresponds to the magnitude of vibration (friction). FIG. 17 is an enlarged view of an image near the touch position on the touch surface of the operator's finger.

  When the operator slides his / her finger from the left in the figure to the right in the figure on the X axis in FIG. 17 (that is, when the movement vector is “X axis positive direction”), the friction vector calculation unit 9 The value of the calculated friction vector is small, and the friction given to the operator's finger is small. On the other hand, when the operator slides his / her finger in a direction opposite to the X-axis positive direction (hereinafter referred to as “X-axis negative direction”), the value of the friction vector is large and is given to the operator's finger. Friction is great. Here, “the friction is small (large)” is the magnitude of the friction (friction vector) in three dimensions, and when the friction is small (large), it is actually touched by making the vibration small (large). Such a realistic tactile sensation can be presented.

  This will be described in more detail with reference to FIG. FIG. 18 shows the image shown in FIG. 17 formed on the xz plane by forming a surface based on the luminance by the processing described with reference to FIGS. The arrows indicated by the solid line and the broken line indicate the normal vectors of the respective surfaces, and the broken line arrow is parallel to the X axis. When the finger is slid in the X-axis positive direction, when calculating the inner product of the vector in the X-axis positive direction and the solid arrow (normal vector), the angle θ formed by both vectors is 90 ° <θ <180 °. Since cos θ is 1 degree, −1 <cos θ <0. On the other hand, when the inner product is calculated when the finger is slid in the negative direction of the X-axis, cos θ = −1 because the angle θ between both vectors is θ = 180 °. Therefore, the absolute value of the inner product value when the finger is slid in the X-axis negative direction is larger than the absolute value of the inner product value when the finger is slid in the X-axis positive direction. For this reason, when the finger is slid in the negative direction of the X axis, the value of the friction vector is large, and the friction given to the operator's finger is large.

  19-21 is a figure for demonstrating the other example of the specific operation | movement of the tactile sense presentation apparatus shown in FIG. In this example, an image of a dog with fur is displayed on the display unit 2. The dog shown in FIG. 19 has fur that flows from top to bottom. In this case, the tactile sensation providing apparatus 100 increases the friction when the operator slides the finger in the X-axis direction (left-right direction in FIG. 19) and slides the finger in the Y-axis direction (up-down direction in FIG. 19). Reduces friction. That is, the width of the arrow in FIG. 19 corresponds to the magnitude of friction. FIG. 20 is an enlarged view of an image near the touch position on the touch surface of the operator's finger. In the figure, the vertical line is a line schematically showing the fur.

  When the operator slides his / her finger in the X-axis direction in FIG. 20 (that is, when the movement vector is in the X-axis direction), the friction vector calculated by the friction vector calculation unit 9 is large, and the operator's The friction given to the finger is great. On the other hand, when the operator slides his / her finger in the Y-axis direction from the top to the bottom, the value of the friction vector is small and the friction given to the operator's finger is small (see FIG. 21). 21 is a view of the animal's fur as shown in FIG. 20 as viewed from the X-axis direction. The vertical axis (Z-axis) is the luminance direction, and in the case of a fur that has a high luminance portion and a low luminance portion alternately appearing, a surface having a normal vector parallel to the X-axis direction at the boundary is generated. The normal vector is indicated by an arrow. On the other hand, it is the figure which looked at the fur of the animal shown in FIG. 20 from the Y-axis direction under FIG. Since there is no change in luminance in the Y-axis direction, the luminance is constant and a surface having a normal vector is not generated.

  In FIG. 21, as in FIG. 18, arrows indicated by solid lines and broken lines indicate normal vectors of each surface. Both solid and dashed arrows are assumed to be parallel to the X axis. In this case, regardless of whether the finger is slid in the X-axis positive direction or the negative direction, the angle θ formed by the movement vector and the normal vector is 180 °, and the absolute values of the inner products are equal. Therefore, the magnitude of the friction vector is the same whether the finger is slid in either the positive direction or the negative direction of the X axis. On the other hand, as described above, since a surface having a normal vector is not generated in the Y-axis direction, a friction vector is not calculated in the Y-axis direction (see the lower part of FIG. 21).

  As described above, the tactile sensation providing apparatus according to the present embodiment includes the touch sensor 1, the display unit 2 that displays an image, the tactile sensation providing unit 3 that vibrates the touch surface, and the control unit 4. Includes a coordinate calculation unit 5 that calculates the coordinates of an object touching the touch surface based on the information of the touch sensor 1, an image three-dimensionalization unit 6 that three-dimensionalizes an image of a predetermined area including the coordinates, and the image A normal vector calculation unit 7 for calculating a normal vector of the surface of the object, a touch surface movement vector calculation unit 8 for calculating a movement vector of the object on the touch surface, the movement vector, the normal vector, A friction vector calculation unit 9 that calculates a friction vector that the image in the predetermined region should give to the object as friction, a tactile sensation control unit 10 that controls the driving of the tactile sensation providing unit 3 based on the friction vector, With , With a simple configuration, it can exhibit a realistic tactile from the two-dimensional image data. In addition, the image stereoscopic unit 6 uses, for example, the luminance of each pixel in a single image without using three-dimensional information measured from a three-dimensional scanner or a stereo image as information used for image stereoscopicization. Therefore, a realistic tactile sensation can be presented for an image using bump mapping.

[Second Embodiment]
FIG. 22 is a functional block diagram schematically showing the main configuration of the tactile sensation providing apparatus according to the second embodiment of the present invention. Functional blocks similar to those of the tactile sensation providing apparatus according to the first embodiment are denoted by the same reference numerals.

  The control unit 4 ′ of the tactile sensation providing apparatus 101 according to the present embodiment includes an image movement vector calculation unit 11 in addition to the functional blocks of the control unit 4 of the tactile sensation providing apparatus 100 according to the first embodiment. When the tactile sensation providing apparatus 101 according to the present embodiment displays a moving image on the display unit 2, the movement vector of the image itself (hereinafter referred to as “image movement vector”) is also used as a parameter for calculating the friction vector. It is configured to be able to.

  The image movement vector calculation unit 11 calculates an image movement vector. Specifically, the moving direction of the image is calculated. The friction vector calculation unit 9 ′ calculates a friction vector that the image in the predetermined area should give to the operator's finger as vibration (friction) based on the movement vector, the normal vector, and the image movement vector.

  FIG. 23 is a flowchart showing the operation of the tactile sensation providing apparatus shown in FIG. Similar to the flowchart shown in FIG. 2, after each process shown in steps S <b> 101 to S <b> 105 is performed in each functional block of the control unit 4 ′, the image movement vector calculation unit 11 calculates the movement vector of the image. (S601).

  Then, the friction vector calculation unit 9 ′ calculates a friction vector that the image in the predetermined region should give to the operator's finger as friction based on the movement vector, the normal vector, and the movement vector of the image (S602). ). Specifically, as described in steps S502 and S503 with reference to FIG. 13, the friction vector calculation unit 9 ′ first calculates the sum of negative values of the inner product of the normal and the movement vector as a vector. And a vector having a direction opposite to the movement vector is obtained. Then, the friction vector calculation unit 9 ′ calculates a combined vector of the calculated vector and the image movement vector, and calculates a friction vector that the moving image in the predetermined area should give to the operator's finger as friction. Then, the tactile sensation control unit 10 controls the driving of the tactile sensation providing unit 3 based on the friction vector calculated by the friction vector calculation unit 9 (S603).

FIG. 24 is a flowchart showing image movement vector calculation processing in step S602 shown in FIG. The image movement vector calculation unit 11 obtains the pixel position of the pixel at the coordinates calculated by the coordinate calculation unit 5 within the frame one frame before the moving image data (S701). Specifically, as shown in FIG. 25, the coordinates A _k− in the frame one frame before corresponding to the center coordinates A _k of the region (solid line) where the operator's finger is currently touching the touch surface are shown. seek _1.

Then, the image three-dimensionalization unit 6 extracts pixels in a predetermined area (broken line) centered on the coordinates A _k-1 obtained in step S701, and as shown in FIG. 26, the peripheral area of the coordinates A _k-1 As described above, the image three-dimensionalization process is executed on each pixel included in the image based on the luminance of each pixel (S702). Next, the image movement vector calculation unit 11 is generated by the image three-dimensional process in step S702 using the result of the image three-dimensional process in step S103 and the image three-dimensional process result in step S702 in FIG. A vector connecting the surface and the surface generated by the image three-dimensional processing in step S103 is obtained (S703).

Specifically, for example, the normal vector calculated in step S104 in FIG. 23 is performed on all surfaces having components opposite to (in the reverse direction from) the movement vector calculated in step S105. Further, when determining the vector, as shown in FIG. 27, the plane P _k-1 generated by the image three-dimensional processing in step S702, on each side P _k generated by the image three-dimensional processing in S103 For example, one corresponding point such as the outer center is selected, and the vector P _k-1 -P _k is obtained by connecting these two points. Then, the image movement vector calculation unit 11 calculates an average vector for all the vectors calculated in step S703 and sets it as an image movement vector (S704).

  FIG. 28 is a diagram for explaining an example of a specific operation of the tactile sensation providing apparatus according to the second embodiment. In this example, a moving image of waves taken from above is displayed on the display unit 2 of the tactile sensation providing apparatus 101. FIG. 28A shows the current frame, and FIG. 28B shows the immediately preceding frame. If the position touched by the operator's finger in the image of the previous frame is the wavefront, and the wave moves upward in the display unit 2 in the image of the current frame, the image in the upward direction is displayed in step S704. Since the movement vector is detected and the friction vector in the upward direction as a whole is synthesized based on the normal vector and the movement vector, vibration (friction) of a predetermined magnitude is applied to the finger of the operator by the tactile sensation providing unit 3. Is generated. Thereby, when the finger is moved upward, a feeling of being pushed back by the wave flow on the moving image can be obtained. Furthermore, since the operator receives the stimulus through not only the tactile sense but also the visual sense by visually observing the display unit 2 on which the moving image is displayed, a realistic tactile sensation as if the finger was pushed upward by the wave. Get.

  In this embodiment, the case where the finger is moved and the movement vector is calculated (the case where the movement vector is generated) has been described. However, the movement vector is not generated, that is, the coordinate calculation is based on the position where the finger is left. A friction vector may be synthesized based on normal vector calculation and image movement vector calculation, and vibration may be applied by the tactile sensation providing unit 3 based on this friction vector. In this manner, even when the finger is simply placed (even when the movement vector is not calculated), a tactile sensation in accordance with the motion of the moving image can be presented to the finger. Specifically, in FIG. Since the operator is stimulated by visual and tactile sensations, a realistic tactile sensation can be obtained as if the finger was pushed upward by waves.

  As described above, the tactile sensation providing apparatus according to the present embodiment includes the touch sensor 1, the display unit 2 that displays an image, the tactile sensation providing unit 3 that vibrates the touch surface, and the control unit 4. Includes a coordinate calculation unit 5 that calculates the coordinates of an object touching the touch surface based on the information of the touch sensor 1, an image three-dimensionalization unit 6 that three-dimensionalizes an image of a predetermined area including the coordinates, and the image A normal vector calculation unit 7 for calculating a normal vector of the surface of the object, a touch surface movement vector calculation unit 8 for calculating a movement vector of the object on the touch surface, and an image movement vector calculation for calculating a movement vector of the image. A friction vector calculation unit 9 ′ that calculates a friction vector to be given to the object by the image in the predetermined region as a friction based on the unit 11, the movement vector, the normal vector, and the image movement vector; A tactile control unit 10 for controlling the drive of the tactile sensation providing unit 3 based on torque, so provided with, it is possible to reproduce the tactile sensation when touching the objects can not be obtained depth information included in the moving image. Moreover, in the tactile sensation presentation apparatus according to the present embodiment, the image movement vector calculation unit 11 calculates an image movement vector and uses the calculation result for calculation of the friction vector in the friction vector calculation unit 9 ′. A realistic tactile sensation can also be presented for an image such as an image that changes with time.

  In addition, this invention is not limited only to embodiment mentioned above, Many change or deformation | transformation is possible. For example, in the second embodiment, after the image movement vector calculation unit 11 calculates the image movement vector in step S601, the friction vector calculation unit 9 ′ calculates the normal vector, the movement vector, and the image movement vector in step S602. The friction vector is calculated based on the friction vector, and the tactile sensation control unit 10 controls the driving of the tactile sensation providing unit 3 based on the friction vector in step S603. However, the present invention is not limited to this, and the tactile sensation is such that the friction vector calculation unit 9 ′ calculates the friction vector based on the normal vector and the movement vector before the calculation of the image movement vector in step S601. The presentation apparatus 101 can also be configured. Then, after calculating the friction vector by the friction vector calculation unit 9 ′, the image movement vector calculation unit 11 calculates the image movement vector, and the tactile sensation control unit 10 obtains a combined vector of the friction vector and the image movement vector, The driving of the tactile sensation providing unit 3 may be controlled based on the combined vector. Since the tactile sensation providing apparatus 101 having such a configuration does not require an image movement vector for calculating the friction vector, data including a mixture of a still image and a moving image can be used as an input image.

  In the present invention, “calculation of vector” stores information necessary for storage (not shown) in advance (calculated in advance, such as magnitude and direction of force), and the vector is based on information from the storage. The determination also falls under “vector calculation”.

DESCRIPTION OF SYMBOLS 1 Touch sensor 2 Display part 3 Tactile sense presentation part 4, 4 'Control part 5 Coordinate calculation part 6 Image three-dimensional part 7 Normal vector calculation part 8 Touch surface movement vector calculation part 9, 9' Friction vector calculation part 10 Tactile sense control part 11 Image movement vector calculation unit 100, 101 Touch display device

Claims (6)

A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A control unit,
The controller is
A coordinate calculation unit that calculates coordinates on the touch surface of an object that touches the touch surface based on information of the touch sensor;
An image three-dimensionalization unit that three-dimensionalizes the image of the predetermined area including the coordinates calculated by the coordinate calculation unit;
A normal vector calculation unit that calculates a normal vector of a surface of the image based on the three-dimensional image;
A touch surface movement vector calculation unit for calculating a movement vector of the object on the touch surface;
A friction vector calculation unit that calculates a friction vector that the image in the predetermined region should give to the object as friction based on the movement vector and the normal vector;
A tactile sensation control unit that controls driving of the tactile sensation providing unit based on the friction vector;
A tactile sensation presentation apparatus comprising: A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A control unit,
The controller is
A coordinate calculation unit that calculates coordinates on the touch surface of an object that touches the touch surface based on information of the touch sensor;
An image three-dimensionalization unit that three-dimensionalizes the image of the predetermined area including the coordinates calculated by the coordinate calculation unit;
A normal vector calculation unit that calculates a normal vector of a surface of the image based on the three-dimensional image;
An image movement vector calculation unit for calculating a movement vector of the image;
A friction vector calculation unit that calculates a friction vector that the image in the predetermined area should give to the object as friction based on the normal vector and the movement vector of the image;
A tactile sensation control unit that controls driving of the tactile sensation providing unit based on the friction vector;
A tactile sensation presentation apparatus comprising: A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A control unit,
The controller is
A coordinate calculation unit that calculates coordinates on the touch surface of an object that touches the touch surface based on information of the touch sensor;
An image three-dimensionalization unit that three-dimensionalizes the image of the predetermined area including the coordinates calculated by the coordinate calculation unit;
A normal vector calculation unit that calculates a normal vector of a surface of the image based on the three-dimensional image;
A touch surface movement vector calculation unit for calculating a movement vector of the object on the touch surface;
An image movement vector calculation unit for calculating a movement vector of the image;
A friction vector calculation unit that calculates a friction vector that the image in the predetermined region should give to the object as friction based on the movement vector, the normal vector, and the movement vector of the image;
A tactile sensation control unit that controls driving of the tactile sensation providing unit based on the friction vector;
A tactile sensation presentation apparatus comprising: A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A tactile sensation presentation apparatus comprising a control unit,
Calculate coordinates on the touch surface of an object touching the touch surface based on the information of the touch sensor,
Three-dimensionalize the image of the predetermined area including the coordinates calculated by the coordinate calculation unit,
Based on the three-dimensional image, calculate a normal vector of the surface of the image,
Calculating a movement vector of the object on the touch surface;
Based on the calculation result of the movement vector and the normal vector, calculate a friction vector that the image in the predetermined region should give to the object as friction,
A control method comprising controlling driving of the tactile sensation providing unit based on the friction vector. A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A tactile sensation presentation device comprising a control unit,
Calculate coordinates on the touch surface of an object touching the touch surface based on the information of the touch sensor,
Three-dimensionalize the image of the predetermined area including the coordinates calculated by the coordinate calculation unit,
Based on the three-dimensional image, calculate a normal vector of the surface of the image,
Calculating a movement vector of the image;
Based on the normal vector and the movement vector of the image, calculate a friction vector that the image in the predetermined region should give to the object as friction,
A control method comprising controlling driving of the tactile sensation providing unit based on the friction vector. A touch sensor;
A display for displaying an image;
A tactile sensation providing unit that vibrates the touch surface of the touch sensor;
A tactile sensation presentation device comprising a control unit,
Calculate coordinates on the touch surface of an object touching the touch surface based on the information of the touch sensor,
Three-dimensionalize the image of the predetermined area including the coordinates calculated by the coordinate calculation unit,
Based on the three-dimensional image, calculate a normal vector of the surface of the image,
Calculating a movement vector of the object on the touch surface;
Calculating a movement vector of the image;
Based on the movement vector, the normal vector, and the movement vector of the image, calculate a friction vector that the image in the predetermined region should give to the object as friction,
A control method comprising controlling driving of the tactile sensation providing unit based on the friction vector.

Images