KR102132693B1 - Haptic interaction system and method for probing real objects in remote places - Google Patents

Abstract

A haptic interaction system for remote sensing according to an embodiment of the present invention includes a remote client; And a server communicating in real time with the client, wherein the client includes a camera, a first haptic device, and a first control unit, and the server includes a second haptic device, a second control unit, and a display unit. , The length of the base vector is the length of the base vector minus the position of the haptic pointer of the first haptic device minus the position of the haptic pointer of the second haptic device received from the server when the first collision between the haptic pointer of the first haptic device and the object occurs At the same time, the user can visually detect a real object at a remote location while using a haptic device to tactilely explore the object at a remote location, and the user can experience a smooth haptic feedback similar to the real life.

Description

HAPTIC INTERACTION SYSTEM AND METHOD FOR PROBING REAL OBJECTS IN REMOTE PLACES}

The present invention relates to a haptic interaction system and method for remote sensing.

Recently, the demand for virtual reality technology in the IT (information technology) industry has increased exponentially. The IT industry mainly requires a Head Mount Display (HMD) device that plays a role in the visual aspect of virtual reality. Most virtual reality applications can experience virtual reality only with visual and auditory senses. However, in certain applications, tactile feedback can create a more realistic virtual environment. Haptic technology is a technology that allows a user to sense the tactile sensation of force and movement. Haptic technology plays an important role in simulation in virtual environments such as virtual surgical training or haptic gaming systems. In the implementation of haptic technology, haptic devices are an essential part. Typical haptic devices include Touch (formerly PHAMTOM Omni) and Touch X (formerly PHANTOM Desktop) of 3D system, Falcon of Novint Technologies, as shown in FIG. 1.

When a haptic device collides with a virtual object, the haptic device calculates the reaction force at the moment to perform a haptic rendering process. Then, it provides haptic feedback to the user so that the user can feel the touch. In order to provide smooth and sophisticated tactile feedback to the user, an update rate of 1000 kHz or higher must be maintained. However, in the case of a real-time haptic application through a network, the responsiveness of the haptic feedback may be deteriorated due to a delay or loss on the network.

In order to solve this tactile feedback degradation problem, various studies have been conducted to enable smooth feedback in a network-based haptic application. One of the earliest studies is the "Network-Based Force Feedback Algorithm" by MIT researchers. In this study, "passive transmission line modeling" and "haptic dead- reckoning" were proposed to reduce haptic feedback errors due to network delay. In addition, researchers at Ottawa University have proposed a smooth synchronous collaboration transport protocol (SCTP) for tactile data transmission. The experimental results of this study showed that the proposed protocol can increase the efficiency of tactile-based collaboration compared to existing transmission protocols such as SCTP or Light Transmission Control Protocol (TCP). The University of Waterloo and Handshake VR Inc. have commercialized the Handshake proSENSE™ 1.3, a tele-haptic toolbox. This study provides various network delay compensation technologies and enables users to easily create network-based tactile collaboration programs by dragging and dropping.

One of the inventors' studies of haptic collaboration in a network environment is "integrated haptic data transmission in a haptic collaborative virtual environment" [12]. This study analyzes changes in delay, jitter and loss in networked haptic collaboration applications. In addition, it proposes an algorithm that can be applied to actual network traffic. Packets that are lost or affected by jitter are compensated by a simple linear prediction method to reduce packet loss and errors caused by jitter. The concurrency problem between clients due to the delay of network collaboration is solved by setting the buffering time.

Another study by the inventor, "MuseSpace: a touchable 3D museum that maximizes the use of haptics," provides a variety of haptic interactions in a network environment. This study embodies a virtual exhibition space in 3D and enables an exhibition represented by 3D objects that can be experienced with visual, auditory, and tactile sensations. In addition, users can interact with other users in real time over the network.

[1] LapSim®: The Proven Training System. https://surgicalscience.com/systems/lapsim/ (Retrieved June 1, 2017). [2] S. M. Kim, M. Y. Sung, “A Haptic Gaming System for Tactile Textures and 3D Shapes Discrimination,” International Journal of Multimedia & Ubiquitous Engineering, Vol. 9 Issue 9, pp. 319-334, September 2014. [3] The Touch™ Haptic Device. https://www.3dsystems.com/haptics-devices/geomagic-touch/ (Retrieved June 1, 2017). [4] The Touch X Haptic Device. https://www.3dsystems.com/haptics-devices/geomagic-touch-x (Retrieved June 1, 2017). [5] Novint Falcon, http://www.novint.com/index.php /products/novintfalcon/ (Retrieved June 1, 2017). [6] K. Salisbury and F. Barbagli, “Haptic rendering: introductory concepts,” Computer Graphics and Applications, IEEE vol. 24, no. 2, pp. 24-32, 2004. [7] G. Vineet, N. Jayakrishnan and C. Subhasis, “Opportunistic Adaptive Haptic Sampling on Forward Channel in Telehaptic Communication,” 2016 IEEE Haptics Symposium, Philadelphia, Pennsylvania, USA, April 8-11, 2016. [8] D. Sabine, WA William and FA Aldo, “Gaze-based teleprosthetic enables intuitive continuous control of complex robot arm use: writing & drawing,” 2016 6th IEEE International Conference on Biomedical Robotics and Biomechatronics, UTown, Singapore, June 26 -29, 2016. [9] P. W. John, J. K-S. Robert, A. K. Marc and H. Neville, “Algorithms for Network-Based Force Feedback,” Proceedings of the fourth PHANTOM User Group Workshop, MIT, (1999) October 9-12. [10] B. Azzedine and M. Haifa, “An Efficient Hybrid Multicast Transport Protocol for Collaborative Virtual Environment with Networked Haptic,” International Workshop on Haptic Audio Visual Environments and their Applications, Ottawa, Canada, (2006) November 4-5. [11] Mehran A. and Kevin T., Haptic enabled robotics training system and method, U.S. Patent 20,090,253,109, October 8, 2009. [12] Y. H. You, M. Y. Sung, K. K. Jun, “An Integrated Haptic Data Transmission in Haptic Collaborative Virtual Environments,” Computer and Information Science, 2007. ICIS 2007. 6th IEEE/ACIS International Conference, July 11-13, 2007. [13] Y. H. You, M. Y. Sung, K. K. Jun, and S. R. Lee, “MuseSpace: a touchable 3D museum with maximum usage of haptics,” Proceeding SIGGRAPH '07 ACM SIGGRAPH 2007, San Diego, California, posters, Article No. 154, ACM, August 05-09, 2007. [14] CHAI 3D. http://www.chai3d.org/ (Retrieved June 1, 2017). [15] J. Postel, “User Datagram Protocol,” RFC 768, 1980. [16] G. C. Vinton and E. K. Robert, “A Protocol for Packet Network Intercommunication,” Transactions on Communications, IEEE vol. Com-22, no. 5, (1974), pp. 637-648. [17] FSR ® 400 Series Data Sheet. https://www.interlinkelectronics.com/datasheets/Datasheet_FSR.pdf (Retrieved June 1, 2017). [18] Arduino UNO. https://www.arduino.cc/en/Main/ ArduinoBoardUno/ (Retrieved June 1, 2017). [19] OpenCV. http://opencv.org/ (Retrieved June 1, 2017).

The problem to be solved by the present invention is that a user can visually detect a real object at a remote location while using a haptic device to tactilely detect the object at a remote location, and the user can experience remote haptic feedback similar to the real experience. It is to provide a haptic interaction system for.

Another problem to be solved by the present invention is that a user can visually detect an object at a remote location using a haptic device while the user visually identifies a real object at a remote location, and the user can experience a smooth haptic feedback similar to the real experience. It provides a haptic interaction method for exploration.

Haptic interaction system for remote sensing according to an embodiment of the present invention for solving the above problems,

Remote clients; And

And a server communicating in real time with the client,

The client,

A camera for acquiring an image of a real object, a first haptic device that senses a force at a point in contact with the object and adjusts the position of a haptic pointer according to a position control signal received from the server, and the first Control the operation of a haptic device, transmit the haptic data including the position of the haptic pointer of the first haptic device, the sensed force value and the length of the base vector, and an image of the real object to the server, from the server It includes a first control unit for receiving a position control signal for controlling the position of the haptic pointer of the first haptic device,

The server,

The position of the haptic pointer is adjusted by the user, and the intensity of the actual object corresponding to a force value sensed by the first haptic device received from the client when a collision occurs between the haptic pointer of the first haptic device and the object A second haptic device that provides the user with a tactile sense, and provides the position of the haptic pointer of the second haptic device to the client as a position control signal for controlling the position of the haptic pointer of the first haptic device of the client, and the The haptic rendering is performed based on the haptic data received from the client, and when the collision between the haptic pointer of the first haptic device and the object occurs, the force corresponding to the force value detected by the first haptic device received from the client A second control unit for converting the intensity of a real object into a reaction force and providing it to the second haptic device, and a display unit for displaying an image of the real object received from the client,

The length of the base vector is the base obtained by subtracting the position of the haptic pointer of the first haptic device from the position of the haptic pointer of the second haptic device received from the server when the first collision between the haptic pointer of the first haptic device and the object occurs. The length of the vector.

In the haptic interaction system for remote sensing according to an embodiment of the present invention, when the second control unit performs haptic rendering, the position of the haptic pointer of the first haptic device of the client received from the client, the base vector By calculating the reaction force based on the length of, the sensed force value and the position of the haptic pointer of the second haptic device, discrete force values received from the client may be interpolated into successive values.

Further, in the haptic interaction system for remote sensing according to an embodiment of the present invention, the reaction force is calculated by multiplying the force value received from the client by the force scale and the length of the normalized direction vector, and the force scale is The length of the current direction vector is divided by the length of the base vector, which is the length of the direction vector when the first collision occurs, and the direction vector is the first haptic device at the position of the current haptic pointer of the second haptic device. It may be subtracted from the position of the haptic pointer.

Haptic interaction method for remote sensing according to an embodiment of the present invention for solving the above other problems,

(a) a client at a remote site, using a camera to obtain an image of a real object and transmitting it to a server that communicates with the client in real time;

(b) the server displaying an image of an actual object received from the client on a display unit;

(c) a position for controlling the position of the haptic pointer of the second haptic device by the user according to the position adjustment of the haptic pointer of the second haptic device by the user, to control the position of the haptic pointer of the first haptic device of the client Providing the client as a control signal;

(d) the client adjusting a position of a haptic pointer of the first haptic device according to a position control signal received from the server to contact the object and sensing the force at the contact point;

(e) the client transmitting haptic data including the position of the haptic pointer of the first haptic device, the sensed force value, and the length of the base vector to the server; And

(f) the server, by performing haptic rendering based on haptic data received from the client, by the first haptic device received from the client when a collision occurs between the haptic pointer of the first haptic device and the object And converting the intensity of the real object corresponding to the sensed force value into a reaction force and providing it to the second haptic device,

The length of the base vector determines the position of the haptic pointer of the first haptic device at the position of the haptic pointer of the second haptic device received from the server when the first collision between the first haptic pointer of the first haptic device and the object occurs. The length of the subtracted base vector.

In the haptic interaction method for remote sensing according to an embodiment of the present invention, when the server performs haptic rendering, the position of the haptic pointer of the first haptic device of the client received from the client, the length of the base vector , By calculating the reaction force based on the sensed force value and the position of the haptic pointer of the second haptic device, discrete force values received from the client may be interpolated into successive values.

In addition, in the haptic interaction method for remote sensing according to an embodiment of the present invention, the reaction force is calculated by multiplying the force value received from the client by the force scale and the length of the normalized direction vector, and the force scale is The length of the current direction vector is divided by the length of the base vector, which is the length of the direction vector when the first collision occurs, and the direction vector is the first haptic device at the position of the current haptic pointer of the second haptic device. It may be subtracted from the position of the haptic pointer.

According to a haptic interaction system and method for remote sensing according to an embodiment of the present invention, a user can visually detect an object at a remote location using a haptic device while simultaneously observing a real object at a remote location by a user Can experience realistic haptic feedback.

1 shows haptic devices, (a) Touch, (b) Touch X, (c) Novint Falcon.
2 is a view showing the configuration of a haptic interaction system for remote sensing according to an embodiment of the present invention.
3 is a diagram illustrating the conceptual structure of a haptic interaction system for remote sensing according to an embodiment of the present invention.
4 is a diagram illustrating devices of a haptic interaction system for remote sensing according to an embodiment of the present invention, (a) a haptic device of a client, (b) a screenshot of a server monitor, (c) a haptic of a server Device.
Fig. 5 is a view showing the definition of haptic data packets (left: packet structure from server to client, right: packet structure from client to server).
6 is a diagram illustrating a process flow of a client for haptic data transmission.
7 is a view showing a process flow of a server for haptic data transmission.
8 is a diagram showing a haptic rendering algorithm of the server.
9 is a view for explaining the calculation of the force scale.
10 is a flowchart of a haptic interaction method for remote sensing according to an embodiment of the present invention.
11 is a view showing a hard book, a rubber ball, and a sponge.
Fig. 12 is a graph showing a distance error graph.

The objects, specific advantages and novel features of the present invention will become more apparent from the following detailed description and preferred embodiments that are associated with the accompanying drawings.

Prior to this, the terms or words used in the specification and claims should not be interpreted in a conventional and lexical sense, and the inventor can appropriately define the concept of terms in order to best describe his or her invention. Based on the principles, it should be interpreted as meaning and concept consistent with the technical idea of the present invention.

In addition, it should be noted that, in addition to reference numerals to the components of each drawing in the present specification, the same components have the same numbers as possible, even if they are displayed on different drawings.

Further, the terms "first", "second", "one side", "other side", and the like are used to distinguish one component from another component, and the component is limited by the terms. It is not.

Hereinafter, in describing the present invention, detailed descriptions of related well-known technologies that may unnecessarily obscure the subject matter of the present invention are omitted.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In general, haptic devices are used to convey the touch sense of an object to a user in a virtual environment. However, in the present invention, unlike the general usage, it is proposed to transmit a tactile feeling to a user of a real object existing in a remote place, not a virtual environment.

Past efforts on haptic interactions have only been relevant to haptic feedback on simulated virtual objects. The present invention proposes a haptic interaction with a real object existing in a remote location. In the present invention, a tele-haptic system is proposed that includes remote sensing of a server and a client. The system structure and haptic rendering are described based on the data obtained from the remote site. In order to evaluate the quality of haptic transmission and tactile recognition of the method proposed in the present invention, many user studies on rigid and deformable objects have been performed. As a result, it was confirmed that the method according to the present invention is suitable for recognizing different stiffness of an object in the distance and tactile feedback of a rigid body.

An object of the present invention is to allow a user to sense the feeling of a real object at a remote location using a haptic device. The system proposed in the present invention is divided into a server and a client. The server is responsible for user interaction and the client is responsible for the user's control over objects at remote locations. The server is composed of a computer and a haptic device, and the client is additionally equipped with a camera such as a force sensor and a webcam.

Figure 2 shows the configuration of a haptic interaction system for remote sensing according to an embodiment of the present invention.

The haptic interaction system for remote exploration according to an embodiment of the present invention shown in FIG. 2 includes a remote client 200 and a server 202 in real-time communication with the client 200.

The client 200, the camera 204 for acquiring the image of the real object 200, detects the force at the point of contact with the object 220, and controls the position received from the server 202 The position of the haptic pointer of the first haptic device 206 is controlled by controlling the operation of the first haptic device 206 and the first haptic device 206, where the position of the haptic pointer is adjusted according to a signal. Haptic data including the force value and the length of the base vector and the image of the real object 220 are transmitted to the server 202, and the location of the haptic pointer of the first haptic device 206 from the server 202 It includes a first control unit 208 for receiving a position control signal for controlling the.

The first haptic device 206 includes a force sensor 212 in contact with the real object 220 and a haptic pointer 210 for exploring the real object 220. In one embodiment of the present invention, the force sensor 212 is attached to the haptic pointer 210 and detects a force at a point in contact with the object 220 when in contact with the object 220.

The server 202, the position of the haptic pointer is adjusted by the user, the first haptic received from the client 200 when a collision between the haptic pointer of the first haptic device 206 and the object 220 occurs The position of the haptic pointer of the second haptic device 216, the second haptic device 216, which provides the user with the tactile intensity of the real object 220 corresponding to the force value sensed by the device 206 Haptic rendering based on haptic data received from the client 200 and provided to the client 200 as a position control signal for controlling the position of the haptic pointer of the first haptic device 206 of the client 200 By performing the, the haptic pointer of the first haptic device 206 when the collision of the object 220 occurs, the force corresponding to the force value detected by the first haptic device 206 received from the client 200 A second control unit 218 for converting the intensity of the real object 220 into a reaction force and providing it to the second haptic device 216, and displaying an image of the real object 220 received from the client 200 It includes a display unit 214 for.

The length of the base vector is determined by the position of the haptic pointer of the second haptic device 216 received from the server 202 when the first collision between the haptic pointer of the first haptic device 206 and the object 220 occurs. 1 is the length of the base vector minus the position of the haptic pointer of the haptic device 206.

When performing the haptic rendering, the second control unit 218 receives the position of the haptic pointer of the first haptic device 206 of the client 200 received from the client 200, the length of the base vector, and the sensed force. By calculating the reaction force based on a value and the position of the haptic pointer of the second haptic device 216, discrete force values received from the client 200 are interpolated into successive values. By interpolating discrete force values into successive values, the user can experience smooth haptic feedback that is realistic.

The reaction force is calculated by multiplying the force value received from the client 200 by the force scale and the length of the normalized direction vector, and the force scale is the length of the current direction vector when the first collision occurs. It is a value divided by the length of the in-base vector, and the direction vector is obtained by subtracting the position of the haptic pointer of the first haptic device 206 from the position of the current haptic pointer of the second haptic device 216.

3 shows a conceptual structure of a haptic interaction system for remote sensing according to an embodiment of the present invention.

Referring to FIG. 3, in step S314, the client 200 receives the image data of the real object 220 from the camera 204, and in step 316, the client 200 transmits the image data to the server 202.

In step S318, the server 202 receives the image data and displays it on the display unit 214.

In step S300, the server 202 transmits the haptic data of the server 202 to the client 200.

In step S302, the client 200 receives the haptic data of the server 202, and in step S304, the client 200 applies the haptic data received from the server 202 to the haptic device of the client 200.

In step S306, the client 200 receives the force value from the force sensor 212, and in step S308, the client 200 transmits the haptic data and force value of the client 200 to the server 202.

In step S310, the server 202 receives the haptic data and force values of the client 200, and in step S312, the server 202 applies the haptic data and force values to the haptic device 216 of the server 202.

The live image transmission process is performed separately from the haptic data transmission. It is a process of receiving image information captured by the camera 204 of the client 202 and displaying live image information on a monitor that is the display unit 214 of the server 202. The user can recognize the remote situation through live video and perform appropriate control on the remote situation.

The camera 204 of the client 200 may act as the eye of a remote mobile detection robot and the haptic device 206 may be the arm of the robot. Since the haptic device 206 of the client 200 is only needed to indicate the location of the haptic device 216 of the server 202, it can be replaced with another mechanical device such as a robotic arm rather than a haptic device. Therefore, the present invention can be applied to an autonomous mobile robot or a robot vehicle for searching for a dangerous place.

avatar

Using the cross-platform C++ simulation framework CHAI3D[14], we construct a software environment for the system proposed in the present invention. Communication between the server 202 and the client 200 is implemented using a Windows socket library. Because the protocol requires real-time high-speed processing, UDP (User Datagram Protocol) is chosen over TCP (Transmission Control Protocol). The haptic pointer of the client 200 is a force that is a force sensor 212 that is controlled by the Arduino Uno microcontroller, which is the first control unit 208, in order to obtain a reaction force against the object 220 present at a remote location. It is connected to a sense resistor (Interlink Electronics FSR 400). The process of capturing, encoding and decoding video information from the camera 204 including the webcam is implemented in OpenCV. The computing environment for the experiment is shown in Table 1.

4(a) is a first haptic device 206 of the client 200, FIG. 4(b) is a screenshot of the server 202 display 214, and FIG. 4(c) is a server 202 The second haptic device 216 is illustrated.

Information transmitted from the server 202 to the client 200 through the network 222 is defined as shown in the left data structure of FIG. 5. The right data structure of FIG. 5 shows information transmitted from the client 200 to the server 202.

The process of transmitting, receiving and processing the above haptic data is shown in FIGS. 6 and 7.

Referring to FIG. 6, in step S600, the client 200 repeatedly checks whether data input from the server 202 exists. When the corresponding data is detected, the location data is updated to the latest data of the server 202 in step S608, and the haptic pointer is moved to the most recent location in step S602. In step S604, the force value is read from the force sensor 212 at the position where the haptic pointer is moved, and in step S606, the force value and the current position value are transmitted to the server 202. This process is repeated until the application ends.

Referring to FIG. 7, in step S700, the server 202 continuously checks whether data is received from the client 200. When data is received from the client 200, the position data is updated to the latest data in step S708. In step S702, the server 202 checks whether a collision occurs on the client 200 side. If a collision occurs on the client 200 side, in step S704, the server 202 performs haptic rendering according to the exploration (haptic) data of the client 202. If no collision occurs, in step S706, the current location of the haptic pointer of the server 202 is transmitted to the client 202. This process is repeated until the application ends. The haptic rendering of the server 202 is a process of calculating the force feedback so that the user can feel the tactile feel of the real object 220 existing in the remote location.

The haptic rendering algorithm performed on the server 202 is summarized in FIG. 8.

The haptic rendering algorithm of the server 202 starts by checking whether a collision has been detected by checking the value of the force sensor 212 received from the client 200. If this value is 0, no collision has occurred. When a collision occurs, the direction vector is obtained by subtracting the current server 202 location and the current client 200 location. The force value is calculated by multiplying the normalized value of the direction vector by the force vector. The force vector is calculated by multiplying the force sensor value and the force scale value. The force scale value is the length of the current direction vector divided by the length of the base vector corresponding to the direction vector at the moment of the first collision.

9 shows the calculation of the force value. When the haptic pointer of the server 202 moves from sp0 (server position 0) to sp1 following the current cp (client position) and then to sp2, the force scale values divide the current force vector by the length of the base vector. Is calculated.

The purpose of multiplying the force scale is to interpolate discrete force values received from the client 200 into successive haptic values according to a fast haptic rendering frequency (1000 Hz). By interpolating these discrete force values into successive haptic values, the user can experience smooth haptic feedback that is realistic.

The reason that discrete values occur is that the haptic rendering speed is faster than the update rate of the most recent location data. The final value of the force calculation is applied to the haptic device of the server 202.

10 is a flowchart illustrating a haptic interaction method for remote sensing according to an embodiment of the present invention.

2 and 10, in step S1000, the remote client 200 acquires an image of the real object 220 using the camera 204 and communicates with the client 200 in real time (202). ).

In step S1002, the server 202 displays an image of the actual object 220 received from the client 200 on the display unit 214.

In step S1004, the server 202 adjusts the position of the haptic pointer of the second haptic device 216 by the user, so that the position of the haptic pointer of the second haptic device 216 is the first of the client 200. The position control signal for controlling the position of the haptic pointer of the haptic device 206 is provided to the client 200.

In step S1006, the client 200 adjusts the position of the haptic pointer of the first haptic device 206 according to the position control signal received from the server 202 to contact and contact the object 220. Senses its power.

In step S1008, the client 200 transmits haptic data including the position of the haptic pointer of the first haptic device 206, the sensed force value, and the length of the base vector to the server 202.

In step S1010, the server 202 performs haptic rendering based on haptic data received from the client 200, and when a collision occurs between the haptic pointer of the first haptic device 206 and the object 220, The intensity of the real object 220 corresponding to the force value sensed by the first haptic device 206 received from the client 200 is converted into a reaction force and provided to the second haptic device 216.

The length of the base vector is determined by the position of the haptic pointer of the second haptic device 216 received from the server 202 when the first collision between the haptic pointer of the first haptic device 206 and the object 220 occurs. 1 is the length of the base vector minus the position of the haptic pointer of the haptic device 206.

The server 202, when performing haptic rendering, the position of the haptic pointer of the first haptic device 206 of the client 200 received from the client 200, the length of the base vector, the sensed force value and By calculating the reaction force based on the position of the haptic pointer of the second haptic device 216, discrete force values received from the client 200 are interpolated into successive values.

The reaction force is calculated by multiplying the force value received from the client 200 by the force scale and the length of the normalized direction vector, and the force scale is the length of the current direction vector when the first collision occurs. It is a value divided by the length of the in-base vector, and the direction vector is obtained by subtracting the position of the haptic pointer of the first haptic device 206 from the position of the current haptic pointer of the second haptic device 216.

Experiment

In Fig. 11, hard books, rubber balls and sponges with different stiffness are shown. User tests were conducted according to the following survey questions.

-Location Synchronization: Is the server's haptic pointer synchronized with the client's haptic pointer?

-Haptic rendering smoothness: Is the change in force feedback smooth?

-Sensitivity of stiffness: Does the degree of stiffness feel realistic?

-Haptic realism for deformable objects: For deformable objects, is the feeling of deformation realistic?

Through user studies, various feedbacks were collected from test participants. In general, test participants felt adequate tactile sensation for three different objects (hard books, rubber balls, and sponges). They also said that the position synchronization of the two haptic pointers is good enough. In addition, it was evaluated that the haptic feedback is smoothly transmitted to the rigid body, but is not satisfactory for the deformable object. He also pointed out that some unstable vibrations sometimes occur and the system responsiveness at the moment of the first crash is poor.

Table 2 shows the experimental results for calculating the Euclidean distance between the two haptic pointers of the server and the client. Data were collected for 15 seconds at 0.1 second (10 ms) intervals. In the experimental workspace, the haptic pointer is -0.09 to 0.0945 for the x coordinate (range of 0.1845), -0.22 to 0.2145 for the y coordinate (range of 0.4345), and -0.1 to 0.21 for the z coordinate ( 0.31). The results of this experiment confirmed that the system proposed in the present invention can be relatively well synchronized with an average error of 0.00679891 (Euclidean distance) between two haptic pointers. The maximum Euclidean distance in this experimental space is 0.56473935 and a simple calculation of the possible error rate is 1.2×e ^-10 .

12 shows the distance error between the haptic pointer of the server and the haptic pointer of the client.

conclusion

The present invention proposes a system in which a user can directly sense the tactile sense of a real object located at a remote location using a networked haptic device. The system consists of a server and a client, and performs a haptic rendering process based on data obtained from a remote client. Several experiments were performed to evaluate the system proposed in the present invention. As a result, it has been found that while unstable vibrations may occur for deformable objects, the system according to the invention can provide relatively good haptic feedback to the rigid body. The experimental results indicate the waiting time for responsiveness at the moment of the first collision.

The present invention can contribute to the development of autonomous mobile robots or robotic vehicles for exploring dangerous places. The client camera of the system according to the invention is similar to the robot's eye, but the client's haptic device is similar to the functional robotic arm.

The present invention has been described in detail through specific examples, but this is for specifically describing the present invention, and the present invention is not limited to this, and to those of ordinary skill in the art within the technical spirit of the present invention. It will be apparent that the modification and improvement are possible.

All simple modifications and variations of the present invention belong to the scope of the present invention, and the specific protection scope of the present invention will become apparent by the appended claims.

200: client 202: server
204 camera 206 first haptic device
208: first control unit 210: haptic pointer
212: force sensor 214: display
216: second haptic device 218: second control unit
220: real object 222: network

Claims (6)

Remote clients; And
And a server communicating in real time with the client,
The client,
A camera for acquiring an image of a real object, a first haptic device that senses a force at a point in contact with the object and adjusts the position of a haptic pointer according to a position control signal received from the server, and the first Control the operation of a haptic device, transmit the haptic data including the position of the haptic pointer of the first haptic device, the sensed force value and the length of the base vector, and an image of the real object to the server, from the server It includes a first control unit for receiving a position control signal for controlling the position of the haptic pointer of the first haptic device,
The server,
The position of the haptic pointer is adjusted by the user, and the intensity of the actual object corresponding to a force value sensed by the first haptic device received from the client when a collision occurs between the haptic pointer of the first haptic device and the object A second haptic device that provides the user with a tactile sense, and provides the position of the haptic pointer of the second haptic device to the client as a position control signal for controlling the position of the haptic pointer of the first haptic device of the client, and the The haptic rendering is performed based on the haptic data received from the client, and when the collision between the haptic pointer of the first haptic device and the object occurs, the force corresponding to the force value sensed by the first haptic device received from the client A second control unit for converting the intensity of a real object into a reaction force and providing it to the second haptic device, and a display unit for displaying an image of the real object received from the client,
The length of the base vector is a base obtained by subtracting the position of the haptic pointer of the first haptic device from the position of the haptic pointer of the second haptic device received from the server when the first collision between the haptic pointer of the first haptic device and the object occurs. Haptic interaction system for remote sensing, which is the length of the vector. The method according to claim 1,
When performing the haptic rendering, the second control unit receives the position of the haptic pointer of the first haptic device of the client, the length of the base vector, the sensed force value, and the position of the haptic pointer of the second haptic device received from the client. A haptic interaction system for remote sensing, by interpolating discrete force values received from the client into successive values by calculating the reaction force based on the. The method according to claim 2,
The reaction force is calculated by multiplying the force value received from the client by the force scale and the length of the normalized direction vector, and the force scale is the base vector that is the length of the current direction vector when the first collision occurs. The value divided by the length of the direction vector, wherein the direction vector is characterized in that the position of the haptic pointer of the first haptic device is subtracted from the position of the current haptic pointer of the second haptic device. (a) a client at a remote site, using a camera to obtain an image of a real object and transmitting it to a server that communicates with the client in real time;
(b) the server displaying an image of an actual object received from the client on a display unit;
(c) a position for controlling the position of the haptic pointer of the second haptic device by the user according to the position adjustment of the haptic pointer of the second haptic device by the user, to control the position of the haptic pointer of the first haptic device of the client Providing the client as a control signal;
(d) the client adjusting a position of a haptic pointer of the first haptic device according to a position control signal received from the server to contact the object and sensing the force at the contact point;
(e) the client transmitting haptic data including the position of the haptic pointer of the first haptic device, the sensed force value, and the length of the base vector to the server; And
(f) the server, by performing haptic rendering based on haptic data received from the client, by the first haptic device received from the client when a collision occurs between the haptic pointer of the first haptic device and the object And converting the intensity of the real object corresponding to the sensed force value into a reaction force and providing it to the second haptic device,
The length of the base vector determines the position of the haptic pointer of the first haptic device at the position of the haptic pointer of the second haptic device received from the server when the first collision between the first haptic pointer of the first haptic device and the object occurs. Haptic interaction method for remote sensing, which is the length of the subtracted base vector. The method according to claim 4,
When performing haptic rendering, the server is based on the position of the haptic pointer of the client's first haptic device, the length of the base vector, the sensed force value, and the position of the haptic pointer of the second haptic device received from the client. By calculating the reaction force, interpolating the discrete force values received from the client into successive values, a haptic interaction method for remote sensing. The method according to claim 5,
The reaction force is calculated by multiplying the force value received from the client by the force scale and the length of the normalized direction vector, and the force scale is the base vector that is the length of the current direction vector when the first collision occurs. It is a value divided by the length of, and the direction vector is characterized by subtracting the position of the haptic pointer of the first haptic device from the position of the current haptic pointer of the second haptic device, the haptic interaction method for remote sensing.

Images