KR20100067005A - Apparatus and method for providing haptic augmented reality - Google Patents

Abstract

PURPOSE: A device and a method for providing haptic augmented reality are provided to add and subtract the reaction force which is augmented by a haptic device about a real repulsive power, thereby experiencing augmented haptic information. CONSTITUTION: A haptic augmented reality providing device(100) comprises a control unit(110) and a drive unit. The control unit calculates repulsive power of an object. The control unit determines a driving power considering the calculated repulsive power and preset solidity. The driving unit drives a haptic device by the determined driving force. The driving unit transfers a repulsive power strengthened by the control unit to a user not a repulsive power according to the collision with a real object.

Description

Apparatus and method for providing haptic augmented reality

The present invention relates to augmented reality, and more particularly, to an augmented reality providing method for providing a user with a mixed result of a real environment (virtual environment) and a virtual environment (virtual environment).

Suppose you hold a pen-shaped tool and write something on a real desk. Can you feel like writing on a piece of paper with a pencil or ballpoint pen, or with a marker pen on a piece of rubber? You could use this tool to create a system for teaching calligraphy on a sheet of paper. It is haptic augmented reality that makes these possible.

Augmented reality, or more commonly a mixed reality environment, is an environment in which a computer-generated virtual and real sense is mixed. As suggested by Milgram et al., Mixed reality can be located on a reality-virtuality continuum (Milgram, P., Colquhoun Jr., H .: A taxonomy of real and virtual world display integration. Tamura, Y. (ed.) Mixed Reality.Merging Real and Virtual Worlds, pp. 1.16.Springer, Berlin (1999)). In this continuum, whether an environment is real or virtual is determined by how much information is stored on the computer to manage the environment. For example, the head-up display of an airplane cockpit is a kind of augmented reality, and adding a real picture of a celebrity's face to a virtual body in a game is a kind of augmented virtuality. Can be. Although this continuum is primarily designed for visual augmented reality, it can be applied to the sense of touch. And when there is a mixed reality environment that provides both visual and tactile sensitivities, the real or virtual degree of this environment can be defined in a composite visuo-haptic continuum.

Existing studies can be classified using the continuum proposed by Milgram et al. Although this taxonomy is not strictly applied to existing visual mixed reality studies, it will be a tool that makes it easier to understand the meaning of haptic augmented reality. First of all, haptic reality is the environment in which the user directly touches the real object and feels only the tactile sense coming from the real object. A typical example is the tangible AR, which means that a user holds a prop made of real objects and interacts in visual mixed reality. Second, 'haptic virtuality' can be located at the other end of the continuum, which is the most researched. In this case, a pure virtual object is added to the visual mixed reality environment and the haptic rendering method of this virtual object is rendered by the haptic rendering method in the existing virtual environment. Therefore, the exact issue of the exact location registration of the coordinate system and visual coordinate system of haptic equipment has become a major issue. The third field is haptic augmented virtuality. In this environment, a real prop is attached to the haptic device, which provides the surface texture of the real object, which is responsible only for the force feedback of the virtual object.

The present invention relates to haptic augmented reality, which aims to provide a user with augmented haptic information by adding virtual tactile information to tactile information generated in a collision between a real object and a haptic device.

Accordingly, at least one embodiment of the present invention is to provide a haptic augmented reality providing apparatus that allows a user integrated into the haptic equipment colliding with an object to experience a different intensity than the actual stiffness of the object. have.

Another technical problem to be achieved by at least one embodiment of the present invention is to provide a haptic augmented reality providing method for a user integrated in the haptic device collided with the object to experience a different intensity than the actual stiffness of the object.

Another technical problem to be achieved by at least one embodiment of the present invention is a computer-readable computer program having a computer program stored therein that allows a user integrated into a haptic device to collide with an object to experience an actual stiffness that is different from the actual stiffness of the object. To provide a recording medium.

In order to achieve the above object, the present invention, the control unit for calculating the repulsive force from the object, in consideration of the calculated repulsive force and the predetermined strength; And it provides a haptic augmented reality providing apparatus comprising a drive unit for driving the haptic equipment with the determined driving force.

Preferably, the controller measures the displacement of the object, and determines the driving force in consideration of the measured displacement, the calculated repulsive force, and the predetermined strength.

Preferably, the controller detects a collision between the object and the haptic device, generates a control signal while the object is deformed due to a collision with the haptic device, and the driving unit is operated in response to the control signal It is characterized by.

The controller may detect whether the haptic device collides with the object by checking whether the repulsive force from the object exceeds a threshold.

On the other hand, according to another embodiment of the present invention, the present invention, the step of calculating the repulsive force from the object; Determining a driving force in consideration of the calculated repulsive force and a predetermined strength; And it provides a haptic augmented reality providing method comprising the step of driving the haptic equipment with the determined driving force.

Preferably, the haptic augmented reality providing method further comprises the step of measuring the displacement of the object, the step of determining the driving force in consideration of the calculated repulsive force, the measured displacement, and the predetermined strength Decide

Preferably, the haptic augmented reality providing method further comprises the step of detecting a collision between the object and the haptic device, the calculating, the determining and the driving step is the object and the haptic It is characterized in that it is performed while a collision between equipment is detected.

In the sensing step, it is desirable to detect whether a collision between the haptic device and the object is detected by checking whether the repulsive force from the object exceeds a threshold.

On the other hand, according to another embodiment of the present invention, the present invention, the step of calculating the repulsive force from the object; Determining a driving force in consideration of the calculated repulsive force and a predetermined strength; And a computer readable recording medium storing a computer program for causing the computer to execute the step of driving the haptic device with the determined driving force.

According to the present invention, the user interacts with the real object and feels the reaction force, instead of directly feeling the repulsive force generated by the collision with the real object, but by adding the reaction force augmented by the haptic device to the actual repulsive force. You can feel the haptic information.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, in adding reference numerals to the components of each drawing, it should be noted that the same reference numerals have the same reference numerals as much as possible even if displayed on different drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In addition, the following will describe a preferred embodiment of the present invention, but the technical idea of the present invention is not limited thereto and may be variously modified and modified by those skilled in the art.

Hereinafter, an apparatus and method for providing haptic augmented reality according to at least one embodiment of the present invention will be described with reference to the accompanying drawings.

1 is a block diagram of a haptic augmented reality providing apparatus 100 according to at least one embodiment of the present invention, including a control unit 110 and a driving unit 120, at least one of the control unit 110 and the driving unit 120 May be integrated into the haptic equipment.

In the present specification, the haptic device includes a plurality of joints connected to each other in a link structure, and when one joint moves, another device connected to the one joint may be interlocked, and at least some of the plurality of joints (for example, , The joint located at the end of the plurality of joints) refers to a device that is integrated with at least a portion of the user's body (for example, a finger) to move at the user's will. For example, the user can grab the most extreme joint and move the most extreme joint at will.

The user may move the haptic device to cause the haptic device to collide with the object. At this time, the part of the part of the haptic equipment that collides with the object is the most end of any one of the plurality of joints constituting the haptic device, in this specification, the collision end means such an end. In the present specification, the 'object' may be a virtual object, but it is assumed that the object is a real object for convenience of description.

The controller 110 detects whether the collision stage of the haptic device collides with the object. To this end, the controller 110 checks whether the repulsive force from the object exceeds the 'preset threshold', and detects a collision between the collision stage of the haptic device and the object. That is, if the repulsive force from the object exceeds the threshold, the controller 110 recognizes that a collision between the collision stage and the object has occurred.

The controller 110 detects a collision between the collision stage of the object and the haptic device, generates a control signal while the object is deformed due to the collision, and calculates a repulsive force from the object while the collision between the object and the collision stage is detected.

In calculating the repulsive force, the controller 110 calculates the repulsive force from the object by using the force sensed by the sensor provided between the collision stage and the last link of the haptic device and the inertial force of the collision stage.

When the collision stage of the haptic device collides with the object to calculate the repulsive force from the object, the controller 110 determines the driving force to drive the driving unit 120 and controls the driving unit 120. Accordingly, the driving unit 120 may transmit the repulsive force enhanced by the control unit 110 to the user, not the repulsive force due to the collision with the real object.

2 is a diagram illustrating an example of a haptic device and a driving force calculation method according to an embodiment of the present invention.

As described above, the haptic augmented reality providing apparatus 100 according to FIG. 1 may be integrated with the haptic device 200 illustrated in FIG. 2.

As shown in FIG. 2, the haptic device 200 includes a plurality of links 212, 214, and 216, and the first link 212 is pivotally connected to the base portion 210. The first link 212 and the second link 214 are connected by the first joint 213, and the second link 214 and the third link 216 are connected by the second joint 215. The driving unit 120 illustrated in FIG. 1 may be provided at the base, the first joint 213, and the second joint 215 of the first link 212 connected to the base 210. A force sensor 217 is provided at the end of the third link 216, and a collision end 218 is provided below the force sensor 217.

The user 220 may grab and move the third link 216 and cause the collision stage 218 of the haptic device 210 to collide with the object 230, which situation is expressed by Equations 1 and 2 below. Is expressed.

In Equations 1 and 2, t is time, f _r (t) is the repulsive force due to the collision with the object 230, f _s (t) is the force detected by the force sensor 217, m _t is The mass of the impact stage 218, a (t), is the acceleration in the direction of pushing the (object) of the collision stage 218, and f _i (t) means the inertia force of the impact stage.

The controller 110 may determine that a collision occurs when f _r (t) is greater than a decision threshold for collision determination.

However, in an actual implementation, it may be difficult to obtain the acceleration a (t) of the collision stage 218. In this case, it may be considered to obtain a (t) using a specialized acceleration sensor. However, when using an acceleration sensor, a system may become complicated and expensive. In an easy way, acceleration may be obtained from position information of the collision stage 218 provided by the haptic device. However, since the derivative of the discrete position values gives a noise-containing result, it is also preferable to use a low pass filter for each derivative to find the acceleration.

The controller 110 changes the force actually felt by the user by controlling the driving force of the haptic device 200 with respect to the repulsive force f _r (t) obtained according to Equation (1). That is, the user may not feel the actual force due to the repulsion but may feel the force augmented by the haptic augmented reality providing apparatus 100.

로 바꾸 는데 있다. 햅틱 장비(200)가 내는 힘을 f_d(t), 사용자의 손이 내는 힘을 f_h(t)라고 하면, 이 두 가지의 힘 때문에 실제 물체의 표면이 변형되어 x_r(t)만큼의 위치변위를 만들어 낸다. 그리고 이때 실제 물체로부터 오는 힘은 위의 두 힘의 합이 되며, 이는 수학식 3과 같이 표현된다. The relationship between force and positional displacement is usually called stiffness, and let stiffness of the real object at a certain time t be k (t). If there is no control of the AR controller 100, this value will be the intensity felt by the user. Our goal is to use this haptic device 200 appropriately so that k (t) is the value we want.

To change it. If the force produced by the haptic device 200 is f _d (t) and the force produced by the user's hand is f _h (t), these two forces deform the surface of the real object, which is equal to x _r (t). Create positional displacement. At this time, the force coming from the real object is the sum of the two forces, which is expressed by Equation 3.

In Equation 3, a force f _h (t) felt by the user may be represented by Equation 4, and the force produced by the haptic device 200 may be represented by Equation 5.

The controller 110 controls f _d * (t) to be generated by the driver 120 according to Equation 5. Accordingly, the user feels the force obtained by subtracting the driving force f _d * (t) generated by the haptic device 200 from the repulsive force f _r (t) at which the collision stage 218 collides with the object. You can feel the power increased by).

The driving unit 120 generates a force command f _c (t) for driving the haptic device 200 with the driving force f _d * (t). The haptic device 200 generates a driving force f _d * (t) according to the generated force command.

Here, f _c (t) may be equal to f _d * (t), but in this case, the noise caused by the force sensor 217 provided at the collision stage 218 in the process of sensing the force is applied to the force. Directly affecting the command, the driving unit 120 may not provide a stable driving force.

To address this concern, f _c (t) can be applied to the traditional closed-loop stiffness method using PD and Proportional and Differential control. The closed loop stiffness control can be expressed as Equation 6.

로서 이는 힘 에러를 의미하고, K_p와 K_d는 각각 proportional, derivative 게인을 의미한다. 다만, 수학식 6에서의 문제점은 f_d(t)를 정확하게 측정하는 것이 어렵다는 것이다. 물론 f_d(t)는 도 2를 참조할 때 제 3 링크(216)와 제 2 링크(214)의 사이에 추가로 힘 센서를 부착하면 구할 수 있 으나 이는 시스템의 복잡성을 증가시킨다. here,

This means force error, and K _p and K _d are proportional and derivative gains, respectively. However, a problem in Equation 6 is that it is difficult to accurately measure f _d (t). Of course, f _d (t) can be obtained by additionally attaching a force sensor between the third link 216 and the second link 214 when referring to FIG. 2 but this increases the complexity of the system.

Accordingly, the present invention proposes to set f _d (t) = f _c (t-1). This assumes that the force command prior to the sampling time in a heuristic manner is the same as the force currently being produced by the haptic device 200. That is, the driving force determined at the present time by the driver 120 may be the driving force according to the force command generated by the driver 120 at the previous time. This assumption is valid when the user stops the hand movement to feel the intensity, but in general this is not exactly the same. If the user is forced into the object to feel the strength and the tool is moving inside the object, or the tool is moved out of the object after the strength is felt, the two forces are slightly different. However, in general, only a few milliseconds of phase delay occur even in this case, because the speed range that a person can move is limited. For example, for a phantom haptic device with a 2.5 Hz period, 2 N intensity sine wave as the force command, the values of f _d (t) and f _c (t-1) only generate a delay of 4 ms. The force error caused by this delay is much lower than the human notice of just noticeable difference (JND).

Thus, the haptic augmented reality providing apparatus according to at least one embodiment of the present invention, the user integrated into the haptic equipment colliding with the object to experience the intensity different from the actual stiffness of the object. For example, according to at least one embodiment of the present invention, even if the impact stage of the haptic equipment collides with the sponge (moves one joint of the haptic equipment to cause the collision stage to collide with the sponge), the user does not feel the impact with the sponge. It may have a feeling of colliding with the same hard object.

3 is a flowchart illustrating a haptic augmented reality providing method according to at least one embodiment of the present invention, which will be described below with reference to FIG. 1.

First, the haptic augmented reality providing apparatus 100 checks whether the object and the collision stage collide with each other (step 310).

If it is checked in step 310 that the collision, the haptic augmented reality providing apparatus 100 calculates the repulsive force from the object (step 320).

After operation 320, the haptic augmented reality providing apparatus 100 determines the driving force in consideration of the repulsive force calculated in operation 320, the displacement of the object due to the collision, and the predetermined strength (operation 330).

After operation 330, the haptic augmented reality providing apparatus 100 drives the haptic device with the driving force determined in operation 330 (operation 340).

The program for executing the haptic augmented reality providing method according to at least one embodiment of the present invention on a computer may be stored in a computer-readable recording medium. Here, the computer-readable recording medium may be a magnetic storage medium (for example, a ROM, a floppy disk, a hard disk, etc.), and an optical reading medium (for example, a CD-ROM, a DVD). : Digital Versatile Disc).

On the other hand, the inventors performed the experiment to configure the system as shown in Figure 4 and to apply the present invention directly to the four real objects for the performance verification according to the present invention.

4 is a diagram illustrating a practical implementation for the application of the method and apparatus for providing haptic augmented reality according to the present invention.

Using the haptic augmented reality providing apparatus according to Figure 4 to obtain the acceleration a (t) by differentiating the position information provided by the haptic device twice in obtaining f _r (t) according to equations (1) and (2) The result of the calculation is shown in FIG. When the fr (t) and f _i (t) were compared in the absence of collision to measure the delay caused by the low pass filter, the difference did not exceed 0.015 N. This value does not exceed the noise level of the force sensor. This value also confirms that the error due to delay caused by the use of low pass filters is not higher than the noise level due to the force sensor.

Four real objects tested using the haptic augmented reality providing apparatus according to FIG. 4 are sponge blocks, foam balls, rubber balls, and rubber erasers, respectively. The relationship between the force-position displacement of each real object is shown in FIG. 6. 6 was measured in a well controlled tapping experiment with phantom haptic equipment and a force sensor. The force exerted by the haptic device at the tap changed from 0 N to 4 N at a rate of 0.5 N / s. As a result, the rubber eraser showed the most linear result, and the sponge block was the most nonlinear and had many hysteresis.

Then, the actual tapping was performed and the displacement of the tool and the repulsive force from the real object were measured. As mentioned earlier, the force produced by the haptic device was replaced by the force command value passed to the haptic device. Using these values, we calculated the force acting on the real user to find out the actual stiffness and compare it with our desired stiffness. As shown in FIG. 7, the stiffness of the real object was all deformed effectively. In Fig. 7, the upper left is for the sponge block, the upper right is for the foam ball, the lower left is for the rubber ball, and the lower right is for the rubber eraser.

The above description is merely illustrative of the technical idea of the present invention, and various modifications, changes, and substitutions may be made by those skilled in the art without departing from the essential characteristics of the present invention. will be. Accordingly, the embodiments disclosed in the present invention and the accompanying drawings are not intended to limit the technical spirit of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by the embodiments and the accompanying drawings. . The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

1 is a block diagram of a haptic augmented reality providing apparatus according to at least one embodiment of the present invention.

2 is a diagram illustrating an example of a haptic device and a driving force calculation method according to an embodiment of the present invention.

3 is a block diagram of a haptic augmented reality providing apparatus according to at least one embodiment of the present invention.

4 is a diagram illustrating a practical implementation for the application of the method and apparatus for providing haptic augmented reality according to the present invention.

FIG. 5 is a diagram illustrating an experiment result of performing a collision test using another haptic augmented reality providing device of FIG. 4.

6 is a diagram showing the relationship between the force and the positional displacement of the four objects used in the experiment.

FIG. 7 is a view showing stiffness enhanced by the experimental result. FIG.

Claims (19)

A controller configured to calculate a repulsive force from an object and determine a driving force in consideration of the calculated repulsive force and a predetermined strength; And Haptic augmented reality providing apparatus comprising a drive unit for driving the haptic equipment with the determined driving force. According to claim 1, And the controller measures the displacement of the object and determines the driving force in consideration of the measured displacement, the calculated repulsive force, and the predetermined strength. The apparatus of claim 1, wherein the controller detects a collision between the object and the haptic device, generates a control signal while the object is deformed due to a collision with the haptic device, and the driver is configured to respond to the control signal. Haptic augmented reality providing device in operation. The method of claim 3, The controller checks whether the repulsive force from the object exceeds a threshold, and detects a collision between the haptic device and the object. According to claim 1, And the preset intensity is a preset intensity corresponding to the object. According to claim 1, The preset intensity is a haptic augmented reality providing device which is a predetermined strength by the user regardless of the object. The haptic augmented reality providing apparatus of claim 1, wherein the driving force determined at the present time is a driving force according to a force command generated by the driving unit at a previous time. The haptic augmented reality providing apparatus of claim 1, wherein the controller calculates a repulsive force from the object by using a force sensed by a sensor provided at a collision stage of the haptic device and an inertial force of the collision stage. The apparatus of claim 1, wherein the object is a real object. Calculating a repulsive force from the object; Determining a driving force in consideration of the calculated repulsive force and a predetermined strength; And Haptic augmented reality providing method comprising the step of driving the haptic equipment with the determined driving force. 11. The method of claim 10, further comprising measuring displacement of the object, The determining may include determining the driving force in consideration of the calculated repulsive force, the measured displacement, and the predetermined strength. The method of claim 10, further comprising: detecting a collision between the object and the haptic device, wherein the calculating, determining, and driving the collision between the object and the haptic device are detected. Haptic augmented reality is provided during the method. The haptic augmented reality providing method of claim 12, wherein the detecting of the collision detects a collision between the haptic device and the object by checking whether the repulsive force from the object exceeds a threshold. The method of claim 10, The preset intensity is a haptic augmented reality providing method corresponding to the predetermined intensity. The method of claim 10, The preset intensity is a haptic augmented reality providing method is a predetermined strength by the user irrespective of the object. The method of claim 10, The haptic augmented reality providing method of the driving force determined at the current time is the driving force according to the force command generated at the previous time of the driving step. The haptic augmented reality providing method of claim 10, wherein the calculating comprises calculating a repulsive force from the object by using a force sensed by a sensor provided at a collision stage of the haptic device and an inertial force of the collision stage. 11. The method of claim 10, wherein the object is a real object. A computer-readable recording medium storing a computer program for executing the method of any one of claims 10 to 18.

Images