JP2021135656A - Tactile feedback method, system, and program - Google Patents

Abstract

To provide a tactile feedback method, system and program for visually presenting tactile sense of an arbitrary object displayed on a display and requested by a user, to the user.SOLUTION: A tactile feedback system 10 which visually presents tactile sense of an object displayed on a touch panel display 102 that can detect touch operation includes: a tactile information management unit 200 which holds tactile parameters for each kind of objects; a tactile parameter acquisition unit which acquires, from a tactile sense storage DB, a tactile parameter of an object corresponding to a tactile sense request position P on the touch panel display where touch operation has been performed; and a tactile indicator display unit which displays a tactile indicator visually representing the tactile sense of the object on the basis of the acquired tactile parameter.SELECTED DRAWING: Figure 1

Description

The present invention relates to a tactile presentation method, a system and a program, and in particular, a tactile presentation method, a system and a system capable of visually and intuitively presenting the tactile sensation of an object displayed at a position on a display touched by the user to the user. Regarding the program.

Non-Patent Document 1 discloses a system in which a position input by a pointing device such as a mouse is displayed on a display as an arrow cursor, and a tactile sensation is presented to a user by controlling the movement and size of the cursor. For example, if you try to move the arrow cursor over the rattling surface shown on the display, the mouse cursor will appear to sway up and down. Also, if you try to move the arrow cursor at the part where the adhesive surface of the gum tape is displayed, the cursor will move while being stuck. In Non-Patent Document 1, the user can obtain a simulated tactile sensation of an object displayed on the display by such a change in the movement of the cursor.

Patent Document 1 describes a server device that generates a web page that provides various product information in the form of an HTML document, and drives an actuator based on touch input from a touch panel, mainly assuming use for online shopping. This discloses a tactile sensation presenting device that presents the tactile sensation of an object.

In Patent Document 1, an icon described as "tactile sensation" is displayed on a web page generated by the server device, and the user can present the tactile sensation from the server device by clicking the icon from a consumer terminal such as a smartphone. You can download the control parameters of the device. The consumer terminal and the tactile sensation presenting device are connected by wire or wirelessly, the downloaded control parameters are applied to the tactile sensation presenting device, and the user can experience the tactile sensation of the desired product by touching the touch panel of the tactile sensation presenting device with a fingertip.

Japanese Unexamined Patent Publication No. 2019-128738

Keita Watanabe Michiaki Yasumura, VisualHaptics: Generating Haptic Sensation Using Only Visual Cues, ACE '08: Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology

As in Non-Patent Document 1, a pointing device such as a mouse is required in the method of expressing the tactile sensation of the cursor position by the display form of the cursor. Therefore, it has been difficult to apply it to smartphones and tablets. Further, in Non-Patent Document 1, the cursor expressing the tactile sensation is limited to the shape of an arrow, and it is not suitable for expressing a wide variety of tactile sensations.

The device of Patent Document 1 is optimized for use in providing a tactile sensation to a user in online shopping. Non-Patent Document 1 is also a system that presents the tactile sensation of an object prepared in advance. Therefore, the following points are technical issues when configuring a device that visually and intuitively provides the user with the tactile sensation of an arbitrary object in an arbitrary scene.

First, since it is necessary to prepare the tactile parameter associated with the object in advance, the number of times the tactile parameter is set increases as the number of points of the target object for which the tactile sensation is experienced increases, resulting in a time cost.

Secondly, since one tactile parameter is set for one article, the tactile sensation of a plurality of objects cannot be reproduced at the same time in the same scene. For example, in the kitchen scene, there are various objects such as cups and frying pans. In such a scene, the tactile sensation of a cup and the tactile sensation of a frying pan cannot be visually and intuitively expressed at the same time.

An object of the present invention is to solve the above technical problems and visually and intuitively present various tactile sensations of an object displayed on a display, without using a pointing device such as a mouse. The purpose is to provide a tactile presentation method, system and program that can be performed.

In order to achieve the above object, the present invention is characterized in that it has the following configuration in a tactile sensation presentation system that visually presents the tactile sensation of an object displayed on a touch-operated display.

(1) Based on the tactile memory means for holding the tactile parameters for each type of object, the means for acquiring the tactile parameters of the article corresponding to the tactile required position of the touch-operated display from the tactile memory means, and the acquired tactile parameters. A means for displaying a tactile indicator that visually expresses the tactile sensation of the article is provided.

(2) Multiple tactile indicators are displayed over the entire background area of an object.

(3) Tactile information that associates each position on the display with a tactile parameter based on the means for identifying the type and position of the object displayed on the display, the identification result of the type and position of the object, and the tactile memory means. The means for generating the tactile sensation and the means for acquiring the tactile sensation parameter corresponding to the tactile sensation request position based on the tactile sensation information are further provided.

(4) When the tactile sensation required position slides, the tactile sensation indicator changes according to the tactile sensation of each slide position.

(5) The display position of the tactile indicator moves according to the slide of the tactile required position.

(6) A means for correcting the tactile sensation information generated based on the current frame image based on each tactile sensation information generated based on each frame image before the previous time is provided.

(7) A means for correcting the type of all pixels or tactile sensation request positions of this frame by the distance between each pixel or tactile sensation request position and each pixel of each frame image before the previous time and a weighted linear sum based on the type. Equipped with.

(8) A type histogram was created for each frame, and the above-mentioned correction means was not applied between frames in which the change of the type histogram exceeded a predetermined threshold value.

(1) When an object displayed on the display is touched, the tactile sensation parameter of the article displayed at the touch position is acquired and the tactile sensation is expressed by an indicator, so that the user does not need to use a pointing device such as a mouse. You will be able to visually and intuitively obtain the tactile sensation of any object displayed on the display.

(2) Since the tactile indicator is displayed over the entire background area of the object, it is possible to fully express various tactile sensations of the article without interfering with the display of the article for which the tactile sensation is desired.

(3) The type and position of the object displayed on the display are identified, and when the user touches the display position of the object for which he / she wants to experience the tactile sensation, the tactile parameter is changed based on the identification result of the article displayed at the touch position. Since the tactile sensation is acquired and expressed by an indicator, the tactile sensation of any object displayed on the display can be visually and intuitively presented.

As a result, it becomes possible to realize movies with tactile sensations and online shopping where products can be selected while visually experiencing the tactile sensations of each product contained in an image. In addition, since processing can be performed in real time, a telecommunications system with a tactile sensation can be realized. Furthermore, by identifying a fictitious character and giving it a tactile sensation, it becomes possible to create a more realistic application such as a manga book or animation that allows the character to be touched.

(4) When the tactile sensation required position slides, the tactile sensation indicator changes according to the tactile sensation of each slide position, so that the tactile sensation of the article whose tactile sensation changes according to the position can be visually expressed without omission.

(5) Since the display position of the tactile indicator moves according to the slide of the tactile required position, the difference in texture between different articles can be visually and dynamically expressed.

(6) Since the tactile information generated based on the current frame image can be corrected based on each tactile information generated based on each frame image before the previous time, even if the type of the object cannot be correctly identified. The correct tactile sensation of the object can be presented to the user.

(7) Since the type of each pixel or tactile request position of this frame is corrected by the distance between each pixel or tactile request position and each pixel of each frame image before the previous time and the weighted linear sum based on the type. The estimation result of the type can be abstracted temporally and spatially. Therefore, even if a temporary or partial misrecognition occurs in the type estimation, the influence on the tactile expression can be reduced.

(8) A type histogram was created for each frame, and the above correction was not applied between frames in which the change in the type histogram exceeded a predetermined threshold value, so that the tactile information before the image changed significantly changed significantly. This makes it possible to prevent erroneous corrections that would be made based on later tactile information.

It is a functional block diagram of the tactile sensation presentation system to which this invention is applied. It is a figure which showed the example of the tactile sensation information. It is a figure explaining the request method of the tactile sensation. It is a figure which showed the display method of the tactile indicator which concerns on 1st Embodiment of this invention. It is a figure which showed the display method of the tactile indicator which concerns on 2nd Embodiment of this invention. It is a figure which showed the example which expresses the tactile sensation by the shape of the tactile sensation indicator. It is a figure which showed the example which expresses the tactile sensation of a hedgehog by the shape of the tactile indicator. It is a figure which showed the example which expresses the tactile sensation by the size of the tactile sensation indicator. It is the figure which expressed the tactile sensation of the cheek part of a human face by the size of the tactile sensation indicator. It is a figure which showed the example which expresses the temperature by the color of a tactile indicator. It is a figure which showed the example which expresses the smoothness by the movement of a tactile indicator. It is a figure which showed the example which expresses the tactile sensation of the slide position of a touch operation by the movement of an indicator. It is a figure which showed the example which expresses the tactile sensation when a touch operation is linearly slid by the indicator is displaced to the left and right in the slide direction. It is a figure which showed the example which moves the tactile indicator of a background area in synchronization with the slide of a touch operation, and further changes the shape and the like according to the tactile sensation of a slide position. It is a figure which showed the example which partially switches the display of the tactile sensation indicator in the process of changing the tactile sensation when the position of a touch operation is slid. It is a figure which showed typically the display switching method of the tactile sensation indicator when the touch position is slid between articles having different tactile sensations. It is a flowchart of the tactile sensation presentation method to which this invention is applied. It is a functional block diagram of an embodiment in which a tactile sensation presentation system is composed of a local terminal and a server device. It is a sequence flow of the tactile sensation presentation system shown in FIG. It is a functional block diagram of an embodiment in which a tactile sensation presentation system is composed of a local terminal and a server device. It is a sequence flow of the tactile sensation presentation system shown in FIG. It is a functional block diagram of the embodiment which configured the tactile sensation presentation system by a local terminal. It is a functional block diagram of a tactile presentation system provided with a map correction part. It is a figure explaining the function of the map correction part. It is a functional block diagram of a tactile presentation system equipped with a map correction part in a local terminal. It is the figure which showed typically the method of correcting the type of the tactile sensation request position of a frame this time. It is the figure which showed typically the method of detecting the change of a camera and the change of an object. It is a functional block diagram of the tactile presentation system provided with the map correction part in the server device.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram of a tactile presentation system 10 to which the present invention is applied. The tactile presentation system 10 of the present invention realizes tactile presentation to the user by visually expressing the tactile information of the object specified by the user on the display displaying a plurality of objects. The following embodiments are examples that embody the present invention, and do not limit the technical scope of the present invention.

The image acquisition unit 100 acquires a still image or a moving image (hereinafter, represented by an image or a frame image) including an object as a subject via a network or locally, and displays it on a display 102 provided with a touch panel 102a capable of detecting a touch operation. do. The tactile information management unit 200 holds the tactile parameters of the object displayed on the display 102.

In the present embodiment, the tactile information management unit 200 includes an object identification unit 101, a tactile database (DB) 103, and a tactile information generation unit 104, but the configuration of the tactile information management unit 200 is not limited to this.

The object identification unit 101 includes a type identification unit 101a and a position identification unit 101b, and identifies the type and position coordinates of an object displayed on the touch panel display 102, respectively. The position coordinates may be two-dimensional 2D position coordinates or three-dimensional 3D position coordinates to which depth information (z-axis direction) is added.

As the type identification unit 101a, an algorithm such as YOLO that estimates two-dimensional position coordinates as a rectangular area or DeepLab that estimates two-dimensional position coordinates as a free-form area can be used. PlaneNet or the like can be used as an algorithm for estimating depth information. Further, when the depth information of the object is acquired by the TOF (Time of Flight) sensor or the like, the value may be used.

The tactile sensation database (DB) 103 holds tactile sensation parameters described in detail later for each type of object. Tactile parameters include parameters corresponding to material properties such as "Young's modulus", "surface roughness" or "thermal conductivity" that are unique to each object, and / or "hardness", "roughness" or "roughness". Parameters corresponding to the perceived intensity of tactile sensation such as "cool feeling" are included.

The tactile information generation unit 104 acquires tactile parameters corresponding to the types of each object acquired from the object identification unit 101 from the tactile database 103, and as shown in FIG. 2, the recognized object map d1 and the tactile data subset. Generates tactile information D composed of d2. The recognized object map d1 defines the position coordinates of each object on the touch panel display 102. The tactile data subset d2 associates the type of each object with the tactile parameters.

The tactile request detection unit 105 includes a position detection unit 105a and a tactile sensation detection unit 105b. The position detection unit 105a detects the position of the object for which the user is requesting a tactile sensation as the tactile sensation request position P based on the user's touch operation on the touch panel 102a of the touch panel display 102 displaying the object. The touch method detection unit 105b detects a time-series change in touch speed and touch pressure as "touch method Q" based on the touch operation.

The tactile parameter acquisition unit 106 acquires the tactile parameter of the object for which the user has requested the tactile sensation from the tactile sensation information management unit 200 based on the tactile sensation request position P. The tactile indicator generation unit 107 applies the tactile sensation request position P, the tactile sensation Q, and the tactile sensation parameter to a predetermined model formula learned in advance to generate the tactile sensation indicator. The tactile indicator display unit 108 superimposes and displays the tactile indicator on the frame image.

FIGS. 3, 4 and 5 are views showing a method of displaying the tactile indicator In according to the present invention, and as shown in FIG. When a tactile sensation request is made by touching a portion, the tactile parameter of the tactile sensation request position P is acquired.

FIG. 4 is a diagram showing a display method of the tactile indicator In according to the first embodiment of the present invention. In the present embodiment, the tactile indicator In having a predetermined shape that visually expresses the tactile sensation of the tactile request position P is superimposed and displayed around the tactile request position P.

FIG. 5 is a diagram showing a display method of a tactile indicator according to a second embodiment of the present invention. In the present embodiment, a large number of tactile indicators In that visually represent the tactile sensation of the tactile request position P are discrete and regular in the background area (including the vehicle area and the dog area) other than the article at the tactile request position P. It is superimposed and displayed as an array. At this time, in order to make the tactile indicator In easier to see, the display color of the article for which the tactile sensation is not required may be lightened or lightened.

The indicator In can visually express the tactile sensation of the tactile sensation request position P by its shape, color, size, movement, and the like. FIG. 6 is a diagram showing an example in which the tactile sensation is expressed by the shape of the tactile sensation indicator In. If the tactile sensation is sufficiently smooth, the tactile sensation indicator In is a circular shape that intuitively expresses the smoothness. As the roughness becomes rougher, protrusions appear on the circumference of the protrusions, and the shape of the protrusions becomes sharper to intuitively express the roughness of the surface.

FIG. 7 is a diagram showing an example of visualizing the tactile sensation when a tactile sensation is requested for the body surface portion of the hedgehog in the second embodiment, and is sharpened to the outer peripheral portion of the tactile sensation indicator In displayed in the background area. A large number of protrusions have appeared.

FIG. 8 shows an example in which the hardness of the tactile sensation request position P is expressed by the size of the tactile sensation indicator In. When the touch panel sensor 102a is configured to be able to detect the contact pressure, in the present embodiment, the larger the pressing pressure at the time of touch operation and the softer the tactile sensation, the larger the size of the tactile sensation indicator In. FIG. 9 is a diagram showing the hardness of the cheek portion of the human face by the size of the tactile indicator In, and the tactile indicator In is displayed larger than that of FIG. 5, indicating that the tactile sensation is softer.

FIG. 10 is a diagram showing an example in which the temperature of the tactile sensation request position P is expressed by the tactile sensation indicator In, and when the tactile sensation information has the temperature information, the difference in temperature is visualized by the color of the tactile sensation indicator In. In this embodiment, the coldest tactile sensation is expressed in purple, and the color changes to indigo, blue, green, yellow, orange, and red as the temperature rises.

FIG. 11 is a diagram showing an example in which the smoothness of the tactile request position P is expressed by the movement of the tactile indicator In, and the smoothness of the tactile request position P is expressed by the amplitude and / or frequency of the tactile indicator In. .. In the present embodiment, the coarser the tactile sensation at the tactile sensation request position P, the larger the amplitude / lower frequency of the tactile indicator In. The amplitude and / or frequency of the tactile indicator In may be changed according to the distance from the tactile sensation request position P. For example, the amplitude and / or frequency of the tactile indicator In, which is farther from the tactile sensation request position P, can be suppressed. In addition, it is possible to express elasticity and softness like rubber by giving a time delay to vibration according to the distance to the tactile sensation required position P.

In the above embodiment, the tactile indicator In has been described as being represented by any one of shape, color, size and movement according to the tactile sensation, but the present invention is not limited to this. For example, the smoothness, temperature and softness of the tactile required position P may be expressed by a combination of the shape, color and size of the tactile indicator In.

Further, in the above embodiment, the visual expression method of the tactile sensation has been described by taking as an example the case where one part of the article is fixedly touch-operated, but the position of the touch operation is slid while maintaining the touch state. Then, the shape of the indicator In may be continuously changed according to the tactile sensation of each slide position.

FIG. 12 is a diagram showing an example in which when the position of the touch operation is slid, the movement of the indicator In is not synchronized with the slide operation according to the tactile sensation of the slide position to express the tactile sensation. In the present embodiment, when the tactile sensation of the slide position is sufficiently smooth, the slide operation of the touch position and the movement of the indicator In are synchronized as shown in FIG. On the other hand, if the tactile sensation of the slide position is not smooth, as shown in FIG. ..

FIG. 13 is a diagram showing an example in which when the position of the touch operation is slid linearly, the indicator In moves while being displaced to the left and right in the slide direction according to the tactile sensation of the slide position. In the present embodiment, the coarser the tactile sensation of the slide position, the larger the amount of displacement to the left and right.

FIG. 14 shows the shape of the tactile indicator In, which moves the tactile indicator In in the background area in the same direction as or in the opposite direction to the slide direction in synchronization with the movement of the tactile indicator In when the position of the touch operation is slid. It is a figure which showed the example which changes the size, color, etc. according to the tactile sensation of a slide position.

FIG. 15 is a diagram showing an example in which the display of the tactile indicator In is partially switched in the process of changing the tactile sensation when the position of the touch operation is slid. The case of sliding to is described as an example.

When a touch operation on a person's hair is first detected, a round tactile indicator In that visualizes a smooth tactile sensation is displayed on the entire surface of the background area, as shown in FIG. Then, when the touch position slides to the front right and reaches the position that straddles the hair and the dog's body surface as shown in Fig. (B), the background on the left side of the screen corresponding to the hair part is "smooth". While the tactile indicator In that visualizes the "roughness" is displayed in the background on the right side of the screen corresponding to the body surface of the dog, the tactile indicator In that visualizes "roughness" is displayed.

When the touch position advances further to the right and reaches the dog's body surface, as shown in Fig. (C), all the shapes and colors of the tactile indicator In displayed in the background area other than the dog are the dog's. It changes to a shape and color that corresponds to the tactile sensation of the body surface.

FIG. 16 is a diagram schematically showing a display switching method of the tactile sensation indicator In when the touch position is slid between articles having different tactile sensations.

In the present embodiment, the tactile information D near the tactile sensation request position is divided into minute rectangular regions corresponding to the positions of the tactile indicator In, and the shape and color of each tactile indicator In are determined according to the tactile sensation of each rectangular region. Will be done. In the present embodiment, the tactile sensation of each rectangular area in the left three columns is "hair", and the tactile sensation of each rectangular area in the right three columns is "dog". Is displayed.

On the other hand, "hair" and "dog" are mixed in the center row, but since the area of "dog" is larger than the area of "hair", the tactile indicator In corresponding to "dog" is displayed. Will be done.

FIG. 17 is a flowchart showing the operation of the tactile sensation presenting system. In step S1, an image of an object as a subject is acquired by the image acquisition unit 100 via a network or from a built-in memory in frame units.

In step S2, the image is displayed on the display 102, and in parallel with or before and after this, in step S3, the type identification unit 101a identifies the type of each object extracted from the frame image by the identification unit 101, and in step S4. , The position identification unit 101b identifies the position coordinates of each object on the display 102, and in step S5, the tactile information generation unit 104 generates the tactile information D based on the identification result of the type and position of the object and the tactile database 103. Generate.

In step S6, the tactile request detection unit 105 determines whether or not the user has touched the display 102. When the touch operation is detected, the tactile sensation request detection unit 105 recognizes the position as the tactile sensation request position P, further detects the touch method Q, and proceeds to step S7. In step S7, the tactile sensation information D is referred to, and the tactile sensation parameter associated with the tactile sensation request position P on the recognized object map d1 is acquired from the tactile sensation data subset d2.

In step S8, the tactile sensation request position P, the tactile sensation Q, and the tactile sensation parameter are applied to the model formula to generate the tactile sensation indicator In. In step S9, the tactile indicator In is superimposed and displayed on the frame image and presented to the user. Therefore, the user can visually and intuitively experience the tactile sensation of any object displayed on the display.

According to the present embodiment, the user can visually experience the tactile sensation of any object displayed on the touch panel display, so that the product can be experienced while experiencing the tactile sensation of a movie with the tactile sensation or the tactile sensation of the product contained in the image. It will be possible to realize online shopping that can be selected. In addition, since processing can be performed in real time, a telecommunications system with a tactile sensation can be realized. Furthermore, by identifying a fictitious character and giving it a tactile sensation, it becomes possible to create a more realistic application such as a manga book or animation that allows the character to be touched.

FIG. 18 is a functional block diagram showing a first dispersion example of each function constituting the tactile sensation presentation system, and the same reference numerals as those above represent the same or equivalent parts, and thus the description thereof will be omitted. In the present embodiment, each function of the tactile sensation presentation system 10 is distributed and arranged in a local terminal 1 represented by a smartphone and a server device 2 on a network, and the tactile sensation presentation is realized by the cooperative operation of each function. There is a feature in.

In the local terminal 1, the tactile information acquisition unit 104a transmits the image information acquired by the image acquisition unit 100 to the server device 2, and transmits the tactile information D composed of the object identification map d1 and the tactile data set d2 to the server. Obtained from device 2.

FIG. 19 is a sequence flow showing the operation of the first dispersion example, and at time t1, the video acquisition unit 100 of the local terminal 1 acquires a frame image of an object. At time t2, the tactile information acquisition unit 104a transmits a tactile information request including frame image information to the server device 2 via the network. At time t3, the frame image is displayed on the touch panel display 102.

When the server device 2 receives the tactile information request, the identification unit 101 extracts an object from the frame image at time t4, and further identifies the type and position coordinates of each object. At time t5, a parameter request describing the type of the object is transmitted from the server device 2 to the tactile DB 103. The tactile DB 103 responds to the server device 2 with tactile parameters corresponding to the object type at time t6.

At time t7, the server device 2 generates the tactile information D described with reference to FIG. 2 based on the acquired tactile parameters and position coordinates of each object, and at time t8, the tactile information D is generated by the local terminal 1. Respond to.

In parallel with this, at time t9, the tactile request detection unit 105 of the local terminal 1 sets the position of the object for which the user is requesting tactile sensation to the tactile sensation request position P based on the user's touch operation on the touch panel unit 102b. And further, the time-series changes in touch speed and touch pressure are detected as touch Q.

At the local terminal 1, at time t10, the tactile parameter acquisition unit 106 acquires the tactile parameter of the object requested by the user from the tactile information D based on the tactile request position P. At time t11, the tactile indicator generation unit 107 generates the tactile indicator In by applying the tactile request position P, the tactile Q, and the tactile parameters to the model formula. At time t12, the tactile indicator display unit 108 superimposes and displays the tactile indicator In on the current frame image.

According to the present embodiment, since a plurality of functions required for tactile presentation can be distributed to the local terminal 1 and the server device 2, it is possible to provide a visual tactile presentation function even to a local terminal having a low processing capacity.

FIG. 20 is a functional block diagram showing a second dispersion example of each function constituting the tactile sensation presentation system, and the same reference numerals as those described above represent the same or equivalent parts, and thus the description thereof will be omitted. This embodiment is characterized in that the identification unit 101 is mounted on the local terminal 1 as compared with the first dispersion example.

FIG. 21 is a sequence flow showing the operation of the second dispersion example. At time t21, the video acquisition unit 100 of the local terminal 1 acquires a frame image and displays it on the touch panel display 102 at time t22. At time t23, the identification unit 101 extracts an object from the frame image and identifies the type of each object and the position coordinates on the display 102.

At time t24, a tactile information request describing the type of the object is transmitted from the local terminal 1 to the server device 2. At time t25, the parameter request including the object type described in the tactile information request is transmitted from the server device 2 to the tactile DB 103. At time t26, the tactile parameter corresponding to the object type is returned from the tactile DB3 to the server device 2 in response to the parameter request. At time t27, the tactile information generation unit 104 of the server device 2 generates tactile information D composed of the object identification map d1 and the tactile data set d2 based on the acquired tactile parameters and the position coordinates of the object.

In parallel with this, at time t29, the tactile request detection unit 105 of the local terminal 1 detects the tactile request position P and the touch method Q based on the user's touch operation on the touch panel unit 102b. At time t30, the tactile parameter acquisition unit 106 acquires tactile parameters from the tactile information D based on the tactile request position P.

At time t31, the tactile indicator generation unit 107 applies the tactile sensation request position P, the tactile sensation Q, and the tactile sensation parameter to the model formula to generate the tactile sensation presentation signal. At time t32, the tactile indicator In is superimposed and displayed on the frame image.

According to the present embodiment, the processing load of the local terminal 1 related to the tactile sensation presentation increases as compared with the first distributed example, but the traffic amount between the local terminal 1 and the server device 2 can be reduced. ..

FIG. 22 is a functional block diagram showing a third dispersion example of each function constituting the tactile sensation presentation system, and the same reference numerals as those described above represent the same or equivalent parts, and thus the description thereof will be omitted. The present embodiment is characterized in that all the functions of the tactile presentation system 10 are implemented in the local terminal 1.

FIG. 23 is a functional block diagram showing another configuration of the tactile sensation presentation system 10, and is characterized in that it includes a map correction unit 109 that modifies the perceptible object map d1 generated by the tactile sensation information generation unit 104.

If the video is a moving image, the tactile information generation unit 104 generates the perceptible object map d1 on a frame-by-frame basis based on the identification result by the identification unit 101. At this time, as shown in FIG. 24, the type of the object is temporarily and partially erroneously recognized between the frames due to the influence of the identification accuracy of the identification unit 101a, the shade in the image, the image noise, and the like, and the recognition object map d1 An error may occur in a part of. Such misrecognition occurs remarkably especially in the identification algorithm that estimates the type of the object based on the free shape.

When the misrecognized area is designated as the tactile request position P, the wrong tactile sensation is presented based on the erroneous tactile parameter, and when the correct tactile sensation is presented based on the correct tactile parameter in the next frame, a short time The change in the tactile sensation in the above causes noise, and the quality of the presented tactile sensation is significantly deteriorated.

In order to prevent the quality of the presented tactile sensation from deteriorating due to an error in a part of the recognized object map d1 due to an erroneous recognition of the object type, the map correction unit 109 uses the recognized object map d1 of the frame in the past. Modify based on the perceptible object map d1 for multiple frames. As a result, even if the type of the object cannot be correctly identified, the correct tactile sensation of the object can be presented to the user.

FIG. 25 is a functional block diagram showing a distribution example of each function in the tactile presentation system having the function of the map correction unit 109, and is characterized in that the map correction unit 109a is mounted on the local terminal 102.

In addition to the perceptible object map d1 of the frame this time, the map correction unit 109a holds the perceptible object map d1 for the past n frames (in the present embodiment, the past 4 frames), and as shown in FIG. 26, the tactile sensation. When the requested position P is detected, the identification results for, for example, 5 pixels square, 125 pixels in total, centered on the coordinates of the tactile demanded position P, are extracted from each perceptible object map d1. Then, the weighted linear sum of the tag IDs based on the time or the inter-pixel distance is calculated for each pixel of the past frame starting from the coordinates of the tactile sensation request position P. Here, the tag ID is an identifier unique to the identification result of each object.

In this embodiment, the position coordinates of the tactile sensation request position P in the perceptible object map d1 of the frame this time are set to (xt, yt), and the position coordinates (x, y) of each pixel in the perceptible object map d1 before the f frame are supported. Let the tag ID be vxyf, go back to the past k frames, extract the tag ID for the rectangular area of (2N + 1) pixels × (2M + 1) pixels, and obtain the linear sum. The linear sum E (T) for the tag ID (T) can be expressed by the following equation (1). However, wxyf is the linear sum weight from the tactile sensation request position P (xt, yt) in the frame to each pixel coordinate (x, y) in the perceptible object map j1 before the f frame, and f = 0 is the current frame. show.

Then, among the linear sums obtained for each tag ID, the linear sum having the largest value and the corresponding tag ID are selected, and if the linear sum is larger than a certain threshold value, the position coordinates of the tactile sensation request position P ( xt, yt) is judged to be the area of the selected tag ID. On the other hand, if the linear sum is smaller than the threshold value, the position coordinates (xt, yt) are judged to be the background area.

According to the present embodiment, since the estimation result is abstracted temporally and spatially, even if a temporary or partial erroneous recognition occurs in the tactile information generation unit 104, it has an influence on the visual tactile sensation presentation. Can be reduced.

However, if the camera viewpoint is switched or the objects in the image are replaced during the image, the map correction unit 109 may not be able to obtain the correct estimation result if the weighted linear sum is calculated by going back to the past frame. There is.

Here, as shown in FIG. 27, when a histogram is created using the object identification result (tag ID) as pixel data in the recognized object map j1, which object occupies what area in the frame image. Indicates whether it is. Therefore, when the frame image in which "people", "dogs", and "cars" are shown is switched to the frame image in which only "people" are shown, the histogram changes significantly as shown in the figure.

Therefore, in the present embodiment, the histogram value for the tag ID is held for the past several frames, and the histogram of the current frame is compared with the histogram of each past frame image to detect the change of the camera or the change of the object. do.

For example, the volatility between the past frame and the current frame is calculated for each tag ID, and the average value of the volatility between the tag IDs is calculated. If the calculated average value exceeds the threshold value, it can be determined that the camera has been switched or the object has been replaced.

However, if the target area of the histogram is the entire frame, it is not possible to deal with partial fluctuations in the frame. For this reason, it is actually desirable to divide one frame image into subsections and calculate a histogram for each section to detect camera changes and object changes in each section.

When the user touches a part of the area when the camera change or the object change is detected in a certain area by the above method, the weight of the linear sum is reset again in the frame before the detection. Determine the type of object at the touch position again. As a result, it is possible to present a visual tactile sensation that reduces the influence of erroneous recognition of an object even in an image in which cameras are switched or objects are replaced.

According to the present embodiment, it is possible to prevent erroneous correction such that the tactile information before the image is significantly changed is corrected based on the tactile information after the large change. Further, according to the present embodiment, only the tag ID of the pixel corresponding to the tactile sensation request position P is corrected based on the tactile sensation information map M of the past frame, so that the amount of calculation can be reduced. Therefore, it is possible to implement the implementation on the local terminal 1 which generally has low computing power.

FIG. 28 is a functional block diagram showing another distribution example of each function in the tactile presentation system having the function of the map correction unit 109, and is a map correction that realizes the function of the map correction unit 109 in the server device 2. The feature is that the part 109b is mounted.

In the present embodiment, when the tactile sensation request position P is detected, only the tag ID of the pixel corresponding to the tactile sensation request position P is modified based on the tactile sensation information map M of the past frame. P is unknown. Therefore, the map correction unit 109b corrects the recognized object map d1 by repeating the same procedure as above for all the pixels of the recognized object map generated for the frame this time.

According to the present embodiment, since the server device can bear the correction processing of the perceptible object map d1 which generally has a high processing load, the processing load of the local terminal 1 can be reduced.

1 ... local terminal, 2 ... server device, 10 ... tactile presentation system, 100 ... image acquisition unit, 101 ... object identification unit, 101a ... type identification unit, 101b ... position identification unit, 102 ... touch panel display, 103 ... tactile database ( DB), 104 ... Tactile information generation unit, 105 ... Tactile request detection unit, 106 ... Tactile parameter acquisition unit, 107 ... Tactile indicator generation unit, 108 ... Tactile indicator display unit, 109, 109a, 109b ... Map correction unit, 200 ... Tactile information management department

Claims (18)

In a tactile presentation system that visually presents the tactile sensation of an object displayed on a display that can detect touch operations.
Tactile memory means that holds tactile parameters for each type of object,
A means for acquiring the tactile sensation parameter of an article corresponding to the tactile sensation required position of the touch-operated display from the tactile sensation storage means, and a means for acquiring the tactile sensation parameter.
A tactile presentation system comprising a means for displaying a tactile indicator that visually expresses the tactile sensation of the article based on the acquired tactile sensation parameters. The tactile sensation presentation system according to claim 1, wherein the tactile sensation indicator is displayed at the tactile sensation request position. The tactile presentation system according to claim 1, wherein a plurality of tactile indicators are displayed over the entire background area of the object. A means of identifying the type and position of an object displayed on a display,
A means for generating tactile information that associates each position on the display with a tactile parameter based on the identification result of the type and position of the object and the tactile memory means.
The tactile sensation presentation system according to any one of claims 1 to 3, further comprising a means for acquiring a tactile sensation parameter corresponding to the tactile sensation required position based on the tactile sensation information. The tactile sensation presentation system according to any one of claims 1 to 4, wherein when the tactile sensation required position slides, the tactile sensation indicator changes according to the tactile sensation of each slide position. The tactile sensation presentation system according to claim 5, wherein the display position of the tactile sensation indicator moves according to the slide of the tactile sensation request position. The tactile sensation presentation system according to claim 6, wherein the display position of the tactile sensation indicator moves asynchronously with the slide at the tactile sensation request position according to the tactile sensation of each slide position. The tactile sensation presentation system according to any one of claims 1 to 7, wherein the shape of the tactile sensation indicator changes according to the tactile sensation. The tactile sensation presentation system according to any one of claims 1 to 8, wherein the display color of the tactile sensation indicator changes according to the tactile sensation. The tactile presentation system according to any one of claims 1 to 9, wherein the tactile indicator vibrates with an amplitude and / or a frequency corresponding to the tactile sensation. The tactile presentation system according to claim 10, wherein the amplitude and / or frequency changes according to a distance from a tactile required position. The tactile presentation system according to any one of claims 1 to 11, wherein the size of the tactile indicator changes according to the tactile sensation and the pressing pressure of the touch operation. The video is each frame image of the moving image.
According to any one of claims 1 to 12, the tactile sensation information generated based on the current frame image is further provided with a means for correcting the tactile sensation information generated based on the tactile sensation information generated based on each frame image before the previous time. Described tactile presentation system. The correction means is characterized in that the type of the tactile sensation request position of the frame this time is corrected by the distance between the tactile sensation request position and each position of each frame image before the previous time and the weighted linear sum based on the type. The tactile presentation system according to claim 13. The tactile presentation system according to claim 14, wherein the correction means creates a type histogram for each frame, and does not apply the correction between frames in which the change of the type histogram exceeds a predetermined threshold value. Further provided with means for detecting how to touch the display based on the touch operation.
The tactile presentation system according to any one of claims 1 to 15, wherein the means for displaying the tactile indicator displays the tactile indicator based on the acquired tactile parameters and the tactile sensation. In a tactile presentation method in which a computer visually presents the tactile sensation of an object displayed on a display capable of detecting a touch operation.
The tactile parameter of the article corresponding to the tactile required position of the touch-operated display is acquired from the tactile memory means that holds the tactile parameter for each type of object.
A tactile sensation presentation method comprising displaying a tactile sensation indicator that visually expresses the tactile sensation of the article based on the acquired tactile sensation parameters. In a tactile presentation program that visually presents the tactile sensation of an object displayed on a display that can detect touch operations.
A procedure for acquiring the tactile parameter of an article corresponding to the tactile required position of a touch-operated display from a tactile memory means that holds the tactile parameter for each type of object, and
A procedure for displaying a tactile indicator that visually expresses the tactile sensation of the article based on the acquired tactile sensation parameters, and
A tactile presentation program that causes a computer to execute.

Images