KR102582312B1 - System for Interaction Collaboration of Virtual Environment and Method thereof - Google Patents

Abstract

The present invention relates to a virtual environment interaction collaboration system and method that allows multiple people to collaborate through intuitive interaction to jointly work on virtual objects in virtual maintenance training equipment.
The disclosed virtual environment interaction collaboration system includes components to be manipulated in the real world; An image capture device that acquires an image by photographing the component to be manipulated; A motion controller mounted on the hand and detecting the movement of the hand relative to the manipulation target component according to six degrees of freedom (x, y, z, pitch, roll, yaw); Receives the image acquired from the video camera, creates a virtual object to be manipulated according to the object representation (transformation) of the virtual world, obtains location information and direction information about the generated virtual object to be manipulated, and follows the movement of the hand. a control device that applies control points detected by the motion controller to the virtual object to be manipulated; and a head mounted display (HMD) that displays the virtual object to which the control points sensed by the motion controller are applied.

Description

Virtual environment interaction collaboration system and method {System for Interaction Collaboration of Virtual Environment and Method there}

The present invention relates to a virtual environment interaction collaboration system and method. More specifically, in virtual maintenance training, trainees collaborate on the same components in a virtual space to create a system similar to the real world through intuitive interaction. It relates to a virtual environment interaction collaboration system and method that enables collaboration in manipulating virtual objects.

In accordance with the 4th Industrial Revolution, new education/training methods using AR/VR technology are becoming established, and interest in the defense field is also increasing, especially in the maintenance field.

The value of virtual maintenance education/training can be further improved in fields that deal with high-risk and expensive equipment, such as complex and precise weapon systems.

In the real world, maintenance activities are mainly carried out in teams of two or three people, and the process of cooperating with each other is essential for heavy and dangerous weapon system equipment.

KR 10-2020-0070058 A(2020. 06. 17.)

The purpose of the present invention is to enable virtual maintenance training in which trainees can collaborate on the same components in a virtual space to manipulate virtual objects similar to the real world through intuitive interaction. To provide an environmental interaction collaboration system and method.

A virtual environment interaction collaboration system according to an embodiment of the present invention for achieving the above-described object includes: components to be manipulated in the real world; An image capture device that acquires an image by photographing the component to be manipulated; A motion controller mounted on the hand and detecting the movement of the hand relative to the manipulation target component according to six degrees of freedom (x, y, z, pitch, roll, yaw); Receives the image acquired from the video camera, creates a virtual object to be manipulated according to the object representation (transformation) of the virtual world, obtains location information and direction information about the generated virtual object to be manipulated, and follows the movement of the hand. a control device that applies control points detected by the motion controller to the virtual object to be manipulated; and a head mounted display (HMD) that displays the virtual object to which the control points sensed by the motion controller are applied.

When a trigger button is pressed, the motion controller recognizes the detected control points as an action of picking the virtual object to be manipulated.

The control device uses one vector (x, y, z) information for the position information, and two vector information orthogonal to each other among the x, y, and z axes for the direction information.

When there are a plurality of control points detected by the motion controller, the control device calculates the center of the control points as the average value of the coordinates of each control point according to the following equation based on the position reference selection method.

When there are a plurality of control points detected by the motion controller, the control device determines that for the plurality of control points, the distance from the first control point group (First Control Group) that is closest based on the camera viewpoint of the image capture device is It is divided into a second control group, and for the center point of each group, the average value of the coordinates of the control points forming the first control point group is calculated to determine the first control point group center point (ⓐ) and the second control point group. The second control point group center point (ⓑ) is obtained by calculating the average value of the coordinates of the forming control points, and connecting the first control point group center point (ⓐ) divided by the first direction and the second control point group center point (ⓑ). Align the line segment along the first direction axis ( ), and regenerate the first control point group center point (ⓒ') and the second control point group center point (ⓓ') divided by the first direction, and the control point connection line connecting these two center points ( ), and the first direction axis ( ) and the control point connecting line segment ( ), using the cross product according to the following equation, the vertical direction axis of the two line segments ( ) is calculated to calculate the second direction axis ( ), and according to changes in control points detected by the motion controller, the position and direction displacement values for each frame are applied to the virtual object to be manipulated to determine the position and direction, using the center point reference and direction reference. The movement of the virtual object to be manipulated is output by applying a displacement value offset of the movement of the component according to the displacement value that changes every frame.

Meanwhile, a virtual environment interaction collaboration method according to an embodiment of the present invention for achieving the above-described object includes the steps of: (a) acquiring an image by photographing a component to be manipulated with an image capture device; (b) a control device receiving the acquired image and creating a virtual object to be manipulated according to the object representation (transformation) of the virtual world; (c) obtaining, by the control device, position information and direction information about the generated virtual object to be manipulated; (d) a motion controller detecting hand movement relative to the manipulation target component according to six degrees of freedom (x, y, z, pitch, roll, yaw); (e) displaying, by a head worn display (HMD), the virtual object to which control points detected by the motion controller are applied; and (f) the control device applying control points detected by the motion controller to the virtual object to be manipulated according to hand movement.

In step (d), when the trigger button is pressed, the motion controller moves the detected control points and recognizes it as an act of picking the virtual object to be manipulated.

In step (c), the control device uses one vector (x, y, z) information for the position information and two vector information orthogonal to each other among the x, y, and z axes for the direction information. .

In step (f), if there are multiple control points detected by the motion controller, the control device calculates the center of the control points as the average value of the coordinates of each control point according to the following equation based on the position reference selection method. do.

In step (f), when there are a plurality of control points detected by the motion controller, the control device selects a first control point group (First Control Group) close to the camera viewpoint of the image capture device for the plurality of control points. ) is divided into a second control point group (Second Control Group) that is far from The second control point group center point (ⓑ) is obtained by calculating the average value of the coordinates of the control points forming the second control point group, and the first control point group center point (ⓐ) and the second control point group center point (ⓐ) are divided by the first direction. ⓑ), the line segment connecting the first direction axis ( ), and regenerate the first control point group center point (ⓒ') and the second control point group center point (ⓓ') divided by the first direction, and the control point connection line connecting these two center points ( ), and the first direction axis ( ) and the control point connecting line segment ( ), using the cross product according to the following equation, the vertical direction axis of the two line segments ( ) is calculated to calculate the second direction axis ( ), and according to changes in control points detected by the motion controller, the position and direction displacement values for each frame are applied to the virtual object to be manipulated to determine the position and direction, using the center point reference and direction reference. The movement of the virtual object to be manipulated is output by applying a displacement value offset of the movement of the component according to the displacement value that changes every frame.

According to the present invention, in virtual maintenance training, trainees can collaborate on the same components in a virtual space to manipulate virtual objects similar to the real world through intuitive interaction.

In addition, according to the present invention, through interaction with virtual objects within the virtual maintenance training system, it has a significant impact on visual immersion, training experience and intuition, and provides an educational effect by providing training experience to trainees. It can be maximized.

Figure 1 is a configuration diagram schematically showing the overall configuration of a virtual environment interaction collaboration system according to an embodiment of the present invention.
Figure 2 is a diagram showing an example of a component to be manipulated in the real world according to an embodiment of the present invention.
Figure 3 is a diagram showing an example of mounting and using a motion controller according to an embodiment of the present invention.
Figure 4 is a diagram showing an example of dividing control points into two groups according to the position of the hand detected by the motion controller according to an embodiment of the present invention.
Figure 5 is a diagram showing an example of finding a control center point for each group in a control device according to an embodiment of the present invention.
Figure 6 is a diagram showing an example of creating a second direction axis using the first direction axis to find the control center point for each group in the control device according to an embodiment of the present invention.
Figure 7 is an operation flowchart showing a virtual environment interaction collaboration method according to an embodiment of the present invention.
Figure 8 is a flowchart showing detailed collaboration operations in a virtual environment interaction collaboration system according to an embodiment of the present invention.
Figure 9 is a diagram showing an example of moving a virtual component in a virtual world according to an embodiment of the present invention.

Hereinafter, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein.

The concept of the present invention relates to an intuitive interaction implementation system and method for multiple people to collaborate on virtual objects in virtual maintenance training equipment. For example, the act of lifting heavy objects together in a virtual environment is an example. You can.

Interaction with virtual objects within an immersive virtual maintenance training system has a significant impact on the training experience and intuitiveness as well as visual immersion, and can maximize educational effectiveness by providing training experience to trainees.

Figure 1 is a configuration diagram schematically showing the overall configuration of a virtual environment interaction collaboration system according to an embodiment of the present invention.

Referring to Figure 1, the virtual environment interaction collaboration system 100 according to an embodiment of the present invention includes a manipulation target component 10, an image capture device 20, a motion controller 30, a control device 40, and a head. It may include a worn indicator (50: Head mounted display, hereinafter HMD), etc.

The manipulation target component 10 is a component that workers want to manipulate in the real world, and is a rigid and non-deformable object.

As shown in FIG. 2, the operating target component 10 may be heavy and dangerous weapon system equipment that must be operated in a two-person or three-person team in the real world. Figure 2 is a diagram showing an example of a component to be manipulated in the real world according to an embodiment of the present invention. As shown in FIG. 2, examples of the component 110 to be manipulated include ballistic missiles that require manpower to move. A ballistic missile is a missile that flies along a ballistic curve. After being launched, it is accelerated by the rocket's propulsion and flies into the atmosphere. When the rocket's combustion is completed and acceleration is completed, only the warhead is separated and continues to fly, and when it approaches the target, it flies into the Earth's atmosphere. It is a missile that falls in free fall due to manpower, accelerates again at a very high speed, and finally reaches the target.

Accordingly, since ballistic missiles are large in volume and heavy in weight, they must be moved collaboratively in a group of three, so it takes a long time to prepare for launch.

Additionally, ballistic missiles are largely composed of rocket engines and jet engines. Rocket engines are equipped with solid or liquid fuel and an oxidizer that acts as oxygen, and obtain thrust by burning the fuel in the combustion chamber to generate high-pressure gas. Jet engines, like jet airplanes, take in air and use the oxygen in the air to burn fuel to generate thrust.

Therefore, since the components 110 to be manipulated, such as missiles, as described above, are dangerous explosives, it is important for workers to set the exact location of which point they should each hold when collaborating.

The image capture device 20 captures the image of the component 10 to be manipulated through a camera and transmits the image to the control device 40 . For this purpose, the video camera 20 may include a DSLR camera, a mirrorless camera, a digital camcorder, etc.

The motion controller 30 is mounted on a person's hand and, as shown in FIG. 3, moves the hand toward the manipulation target component 10 according to six degrees of freedom (x, y, z, pitch, roll, yaw). sense Figure 3 is a diagram showing an example of mounting and using a motion controller according to an embodiment of the present invention. As shown in FIG. 3, the motion controller 30 can be mounted on a person's hand, such as an Oculus Touch, and can be used with an HMD 50 mounted on a person's head, such as an HTC Vive Controller.

For example, the Oculus Touch consists of a pair of handheld devices each with an analog stick, three buttons, and two triggers (one typically used for holding and the other for shooting or firing), which the user can use with their fingers. Detect gestures. Each controller's ring contains a set of infrared LEDs, allowing the controller to be fully tracked in 3D space by Oculus Rift's Constellation system and displayed in a virtual environment. Each controller has a rumble motor for haptic feedback and is powered by a single AA alkaline cell.

For example, the HTC Vive Controller is designed to utilize room-scale technology to transform a room into a three-dimensional space through sensors, allowing users to naturally explore and walk through virtual reality, and motion-tracking handheld devices. Using the controller, you can vividly manipulate objects, interact and communicate with precision, and experience an immersive environment. In other words, the Vive Controller is a device that allows free wireless play in virtual reality and consists of 24 sensors, a multi-function trackpad, dual stage trigger, HD haptic feedback, and a rechargeable battery. Also includes one controller, wrist strap, micro USB cable, and power adapter.

In addition, the motion controller 30 is 3D tracked so that workers, etc. can use both hands in a virtual environment, and is provided with 6 degrees of freedom (x, y, z, pitch, roll, yaw) to manipulate the component 10. The movement position of the grasping hand is detected and the detected movement positions are provided as control point information.

The control device 40 is a device that acts as a server that performs overall control operations. It receives the image obtained from the image capture device 20 and generates a virtual object to be manipulated according to the object representation (transformation) in the virtual world. Position information and direction information for the virtual object to be manipulated are acquired, and control points detected by the motion controller 30 according to the movement of the hand are applied to the virtual object to be manipulated.

The control device 40 can use one vector (x, y, z) information for position information and two vector information orthogonal to each other among the x, y, and z axes for direction information.

The HMD 50 displays a virtual object to be manipulated to which control points detected by the motion controller 30 are applied. Therefore, when worn on a person's head, the person can perform work operations while viewing the virtual object to be manipulated through the HMD 50.

When the trigger button is pressed, the motion controller 30 may recognize it as an act of picking (picking) a virtual object to be manipulated by moving control points detected through a sensor.

In FIG. 1, the image capture device 20, motion controller 30, control device 40, and HMD 50 are connected to each other through a wireless network or a wired network to transmit and receive data. Therefore, through this network sharing, two or more workers can participate in the same space in a virtual environment and perform collaboration.

1. Set location criteria

When there are a plurality of control points detected by the motion controller 30, the control device 40 calculates the center of the control points as the average value of the coordinates of each control point according to the following equation 1 by using a position reference selection method. You can.

For example, the control device 40 controls the x coordinate ( ) The average value of the control points for ( ), y coordinate ( ) The average value of the control points for ( ), and z coordinate ( ) The average value of the control points for ( ) is calculated respectively.

2. Separate control points into two groups

When there are a plurality of control points detected by the motion controller 30, the control device 40, as shown in FIG. 4, moves the control points closer to the camera viewpoint of the image capture device 20 for the plurality of control points. 1 Set the control point group (First Control Group) by dividing it into a second control group (Second Control Group) that is far away from the control point group (First Control Group). Here, FIG. 4 is a diagram illustrating an example of dividing control points into two groups according to the position of the hand detected by the motion controller according to an embodiment of the present invention.

As shown in FIG. 4, in order to establish a direction standard, a number of control points are first divided into two groups. The standard is to divide the control point into a close control point group and a far control point group based on the current camera viewpoint.

For example, if five hands are holding the component 10 to be manipulated (five control points), there is a first control point group at a close distance based on the camera viewpoint and a second control point group at a distance ( Second Control Group).

3. Find control center point for each group and select first direction axis

The control device 40 finds the center position for each group as shown in FIG. 5. The method of finding the center point for each group is the same as 1. Location standard selection method. In other words, the central point for each group can be found through the process below. Here, FIG. 5 is a diagram illustrating an example of finding a control center point for each group in a control device according to an embodiment of the present invention.

1) The control device 40 calculates the average value of the coordinates of the control points forming the first control point group with respect to the center point of each group, and calculates the average value of the coordinates of the center point ⓐ of the first control point group and the control points forming the second control point group. The second control point group center point (ⓑ) is obtained by calculating .

2) The control device 40 divides the line segment connecting the first control point group center point (ⓐ) and the second control point group center point (ⓑ) divided by the first direction into the first direction axis ( ).

Ultimately, two control points are determined, and the object's position standard can be defined as the midpoint of these two control points. Additionally, the line segment connecting the two control points is divided into the first direction axis ( ) is used.

4. Find the second direction axis reference

The control device 40 regenerates the two groups divided by the first direction reference as shown in FIG. 6. Here, Figure 6 is a diagram showing an example of creating a second direction axis using the first direction axis to find the control center point for each group in the control device according to an embodiment of the present invention.

1) The control device 40 regenerates the first control point group center point (ⓒ') and the second control point group center point (ⓓ') divided by the first direction, and a control point connection line segment connecting these two center points ( ) is set.

2) The control device 40 controls the first direction axis ( ) and the control point connecting line segment ( ), using the cross product according to Equation 2 below ( ) is calculated.

3) The control device 40 divides the vertical axis of the two line segments calculated as above into the second direction axis ( ).

4. Virtual object conversion using two axes: position reference and direction reference

1) The control device 40 determines the position and direction by applying the position displacement and direction displacement values for each frame to the virtual object to be manipulated according to changes in control points detected by the motion controller 30.

2) The control device 40 controls the movement output of the virtual object to be manipulated by applying a displacement value offset of the movement of the component according to the displacement value that changes in each frame using the center point reference and the direction reference.

Figure 7 is an operation flowchart showing a virtual environment interaction collaboration method according to an embodiment of the present invention.

Referring to FIG. 7, the virtual environment interaction collaboration system 100 according to an embodiment of the present invention acquires an image by photographing the manipulation target component 10 with the image capture device 20 (S710).

Next, the control device 40 receives the acquired image and creates a virtual object to be manipulated according to the object representation (transformation) of the virtual world (S720).

Next, the control device 40 obtains location information and direction information about the generated virtual object to be manipulated (S730).

At this time, the control device 40 can use one vector (x, y, z) information for position information and two vector information orthogonal to each other among the x, y, and z axes for direction information.

Next, the motion controller 30 detects the movement of the hand with respect to the manipulation target component 10 according to six degrees of freedom (x, y, z, pitch, roll, yaw) (S740).

At this time, when the trigger button is pressed, the motion controller 30 may recognize it as an action of moving the detected control points and grabbing the virtual object to be manipulated (Pick).

Next, the HMD 50 displays a virtual object to be manipulated to which the control points detected by the motion controller 30 are applied (S750).

Next, the control device 40 applies the control points detected by the motion controller 30 according to the movement of the hand to the virtual object to be manipulated (S760).

At this time, when there are a plurality of control points detected by the motion controller 30, the control device 40 calculates the center of the control points as the average value of the coordinates of each control point according to Equation 1 by using a position reference selection method. You can.

In addition, when there are a plurality of control points detected by the motion controller 30, the control device 40 provides a first control point group ( It is separated into a second control group (Second Control Group) that is far from the First Control Group.

In addition, the control device 40 calculates the average value of the coordinates of the control points forming the first control point group with respect to the center point of each group, and calculates the average value of the coordinates of the center point ⓐ of the first control point group and the control points forming the second control point group. The second control point group center point (ⓑ) is obtained by calculating the line segment connecting the first control point group center point (ⓐ) and the second control point group center point (ⓑ), which are divided by the first direction, along the first direction axis ( ).

Meanwhile, in order to determine the second direction axis, the control device 40 regenerates the first control point group center point (ⓒ') and the second control point group center point (ⓓ') divided by the first direction, and these two center points The control point connecting line segment ( ) is set.

Additionally, the control device 40 controls the first direction axis ( ) and the control point connecting line segment ( ) using the cross product according to Equation 2 to determine the vertical direction axis of the two line segments ( ) is calculated to calculate the second direction axis ( ).

Additionally, the control device 40 determines the position and direction by applying the position displacement and direction displacement values for each frame to the virtual object to be manipulated according to changes in control points detected by the motion controller 30.

In addition, the control device 40 controls to output the movement of the virtual object to be manipulated by applying a displacement value offset of the movement of the component according to the displacement value that changes in each frame using the center point reference and the direction reference. .

Figure 8 is a flowchart showing detailed collaboration operations in a virtual environment interaction collaboration system according to an embodiment of the present invention.

Referring to FIG. 8, in the virtual environment interaction collaboration system 100 according to the embodiment, the control device 40 first determines whether a plurality of control points for the manipulation target component 10 are captured (S802).

That is, the control device 40 controls the control points detected through the motion controller 30 according to the grasping action or grasping action of the hands of the collaborator with respect to the component 10 to be manipulated, such as a missile in the real world. This is to determine whether the message is received from the controller 30.

At this time, when the virtual environment interaction collaboration operation is performed for the first time (S804-Yes), the control device 40 executes a position reference selection operation to find the center point of the control points (S806).

For example, when there are a plurality of control points detected by the motion controller 30, the control device 40 determines the coordinates of each control point according to the above-described equation 1 by selecting the center of the control points based on the position. Calculated as the average value.

Next, the control device 40 determines the position offset of the virtual component and the center point (S808).

Here, the virtual component is obtained by the control device 40 capturing an image of the component 10 to be manipulated in the real world through the image capture device 20 and generating the image as a virtual object to be manipulated.

In the real world, collaboration occurs due to the presence of physical forces such as gravity and centrifugal force, but in the virtual environment, there is no such force element, so for manipulation similar to the real world, the manipulation object component 10 in the real world is connected to the manipulation object virtual object in the virtual world. It is created with .

Object representation (transformation) in the virtual world can be defined through location information and direction information. Location information is one Vector (x, y, z) information, and direction information is two Vectors (x, y, z axes). Two of them, orthogonal to each other, can use information.

The worker's hand (actually the controller held in the hand) in the real world can be expressed as a control point with 6 degrees of freedom within the virtual environment, and pressing the trigger button (trigger) of the motion controller (30) is the act of picking an object. It can be defined as:

When there are a plurality of control points, the control device 40 sets a second control point group (Second Control Group) that is distant from the first control point group (First Control Group) that is close to the plurality of control points based on the camera viewpoint. Set separately.

Then, the control device 40 calculates the average value of the coordinates of the control points forming the first control point group to create the first control point group center point ⓐ, and calculates the average value of the coordinates of the control points forming the second control point group to create the second control point. Acquire the group center point (ⓑ).

Next, the control device 40 selects the direction reference axis (S810).

That is, the control device 40 sets the first direction axis and the second direction axis.

For example, the control device 40 divides the line segment connecting the first control point group center point (ⓐ) and the second control point group center point (ⓑ) divided by the first direction into the first direction axis ( ).

In addition, the control device 40 regenerates the two groups divided by the first direction into a first control point group center point (ⓒ') and a second control point group center point (ⓓ'), and a control point connection line connecting these two center points. ( ), and set the first direction axis ( ) and the control point connecting line segment ( ) using the cross product according to Equation 2 to determine the vertical direction axis of the two line segments ( ) is calculated, and the direction axis perpendicular to the two calculated line segments is set to the second direction axis ( ).

Next, the control device 40 sets the direction of the virtual component and the offset with the direction reference axis (S812).

That is, the control device 40 determines the position and direction by applying the position displacement and direction displacement values for each frame to the virtual object to be manipulated according to changes in control points detected by the motion controller 30.

In addition, the control device 40 controls the movement output of the virtual object to be manipulated by applying a displacement value offset of the movement of the component according to the displacement value that changes in each frame using the center point reference and the direction reference.

Through the same process as described above, a scenario was performed where two people (four hands) participated in the virtual experience space as shown in FIG. 9 through the network and the two people moved the 'target detection unit' components together. Figure 9 is a diagram showing an example of moving a virtual component in a virtual world according to an embodiment of the present invention. Here, there are a total of 4 control points, and the position reference is found by finding the center point of the 4 control points, and the direction reference is found by using 2 axes using groups. Using the center point standard and direction standard, the movement of the virtual component was expressed by applying the displacement value offset of the component's movement according to the displacement value that changes in each frame.

As described above, when the virtual environment interaction collaboration task is completed (S814-Yes), the control device 40 terminates all virtual environment interaction collaboration operations.

However, if the collaborative operation is not terminated (S814-No), it is determined whether a plurality of control points for the manipulation target component 10 are captured, and then the operation returns to determining whether the collaborative operation is the first.

Next, if the virtual environment interaction collaboration operation is not the first (S804-No), the control device 40 performs an operation to find the center point of the control points (S822).

That is, the control device 40 calculates the average value of the coordinates of the control points forming the first control point group to create the first control point group center point ⓐ, and calculates the average value of the coordinates of the control points forming the second control point group to create the second control point. The goal is to obtain the group center point (ⓑ).

Next, the control device 40 obtains a displacement value using the position offset and applies it to the virtual component (S824).

That is, the control device 40 applies a plurality of control points obtained through the motion controller 30 for the component 10 to be manipulated to the virtual object to be manipulated and displays location information of each control point.

Next, the control device 40 finds the direction reference axis (S826).

That is, the control device 40 divides the line segment connecting the first control point group center point (ⓐ) and the second control point group center point (ⓑ) divided by the first direction into the first direction axis ( ).

In addition, the control device 40 regenerates the two groups divided by the first direction into a first control point group center point (ⓒ') and a second control point group center point (ⓓ'), and connects control points connecting these two center points. Line segment ( ), and set the first direction axis ( ) and the control point connecting line segment ( ) using the cross product according to Equation 2 to determine the vertical direction axis of the two line segments ( ) is calculated, and the direction axis perpendicular to the two calculated line segments is set to the second direction axis ( ).

Next, the control device 40 obtains a displacement value using the direction offset and applies it to the virtual component (S828).

That is, the control device 40 determines the position and direction by applying the position displacement and direction displacement values for each frame to the virtual object to be manipulated according to changes in the control points, and uses the center point reference and direction reference to change the position and direction for each frame. The movement of the virtual object to be manipulated is output by applying the displacement value offset of the movement of the component according to the displacement value.

As described above, according to the present invention, in virtual maintenance training, trainees can collaborate on the same components in a virtual space to manipulate virtual objects similar to the real world through intuitive interaction. A virtual environment interaction collaboration system and method that enable virtual environment interaction can be realized.

Those skilled in the art to which the present invention pertains should understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features, and that the embodiments described above are illustrative in all respects and not restrictive. Just do it. The scope of the present invention is indicated by the claims described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. .

10: Component to be operated 20: Video camera
30: motion controller 40: control device
50: HMD 100: Virtual environment interaction collaboration system

Claims (8)

Components subject to manipulation in the real world;
An image capture device that acquires an image by photographing the component to be manipulated;
A motion controller mounted on the hand and detecting the movement of the hand relative to the manipulation target component according to six degrees of freedom (x, y, z, pitch, roll, yaw);
Receives the image acquired from the video camera, creates a virtual object to be manipulated according to the object representation (transformation) of the virtual world, obtains location information and direction information about the generated virtual object to be manipulated, and follows the movement of the hand. a control device that applies control points detected by the motion controller to the virtual object to be manipulated; and
A head mounted display (HMD) that displays the virtual object to be manipulated to which control points sensed by the motion controller are applied;
Includes,
The motion controller is,
When the trigger button is pressed, the detected control points are moved and recognized as an act of picking the virtual object to be manipulated,
The control device is,
One vector (x, y, z) information is used for the position information, and two vector information orthogonal to each other among the x, y, and z axes are used for the direction information,
The control device is,
When there are multiple control points detected by the motion controller, the multiple control points are divided into a first control group and a second control group based on the camera viewpoint of the video capture device. And, with respect to the center point of each group, the average value of the coordinates of the control points forming the first control point group is calculated to obtain the first control point group center point (ⓐ), and the first control point group center point ⓐ is calculated as the average value of the coordinates of the control points forming the second control point group. 2 A virtual environment interaction collaboration system characterized by acquiring the control point group central point (ⓑ).
delete delete In the virtual environment interaction collaboration method using the virtual environment interaction collaboration system of claim 1,
(a) acquiring an image by photographing a component to be manipulated using an image capture device;
(b) a control device receiving the acquired image and creating a virtual object to be manipulated according to the object representation (transformation) of the virtual world;
(c) obtaining, by the control device, position information and direction information about the generated virtual object to be manipulated;
(d) a motion controller detecting hand movement relative to the manipulation target component according to six degrees of freedom (x, y, z, pitch, roll, yaw);
(e) displaying, by a head worn display (HMD), the virtual object to which control points detected by the motion controller are applied; and
(f) the control device applying control points detected by the motion controller according to hand movement to the virtual object to be manipulated;
Includes,
In step (d), the motion controller,
When the trigger button is pressed, the detected control points are moved and recognized as an act of picking the virtual object to be manipulated,
The control device is,
One vector (x, y, z) information is used for the position information, and two vector information orthogonal to each other among the x, y, and z axes are used for the direction information,
The control device is,
When there are multiple control points detected by the motion controller, the multiple control points are divided into a first control group and a second control group based on the camera viewpoint of the video capture device. And, with respect to the center point of each group, the average value of the coordinates of the control points forming the first control point group is calculated to obtain the first control point group center point (ⓐ), and the first control point group center point ⓐ is calculated as the average value of the coordinates of the control points forming the second control point group. 2 A virtual environment interaction collaboration method characterized by obtaining the control point group center point (ⓑ).
delete According to paragraph 1,
The control device is,
A line segment connecting the first control point group center point (ⓐ) and the second control point group center point (ⓑ) is formed along the first direction axis ( ), and regenerate the first control point group center point (ⓒ') and the second control point group center point (ⓓ') divided by the first direction, and the control point connection line connecting these two center points ( ) A virtual environment interaction collaboration system characterized by setting up.
According to clause 6,
The control device is,
The first direction axis ( ) and the control point connecting line segment ( ), using the cross product according to the following equation, the vertical direction axis of the two line segments ( ) is calculated to calculate the second direction axis ( ) A virtual environment interaction collaboration system characterized by setting.

In clause 7,
The control device is,
According to changes in control points detected by the motion controller, the position and direction are determined by applying the position displacement and direction displacement values for each frame to the virtual object to be manipulated,
A virtual environment interaction collaboration system characterized by outputting the movement of the virtual object to be manipulated by applying a displacement value offset of the movement of the component according to the displacement value that changes in each frame using a center point reference and a direction reference.