KR20220058122A - Collaborative systems and methods for field and remote site using HMD - Google Patents

Abstract

According to the present invention, a collaborative system between a field and a remote plate using an HMD, comprises: a worker HMD worn by a worker located in a field; one or more expert HMDs worn by a remotely located expert, and communicating with the worker HMD; a main camera provided in the worker HMD to photograph the field; one or more auxiliary cameras provided in the worker HMD to photograph the field, and remotely controlled by the expert HMD; and a gyro sensor for detecting a head motion of the worker in the worker HMD. According to the present invention, a worker located in a field and at least one expert located in a remote place can check different points on the field. Accordingly, a worker can check a location of a point currently viewed by an expert, such that collaborative efficiency between a field and a remote place can be improved.

Description

Field and remote collaboration systems and methods using HMD {Collaborative systems and methods for field and remote site using HMD}

According to the present invention, a worker located in the field and an expert located at a remote location wear an HMD (Head Mounted Device) and communicate with each other, so that a remote expert can easily understand the situation on the site, and the expert remotely adjusts the operator's camera to the field It relates to a field and remote collaboration system and method using HMD that can increase the efficiency of collaboration between and remote locations.

Various wearable devices are being developed according to the trend of lighter and smaller digital devices, and one of these wearable devices, a Head Mounted Device, is worn on the user's head to receive multimedia contents such as VR and AR. It is a device that has Such a head-mounted device (hereinafter, HMD) may be worn on the user's head to provide content to the user in various environments as the user moves. In addition, HMDs are divided into see-through and see-closed types. Through-beam HMDs are mainly used for Augmented Reality (AR), and closed HMDs are mainly used for Virtual Reality (VR). ) is used for

These HMDs are put into construction, disaster, rescue, and medical situations while being worn by the user to be used to remotely share the situation of the site. For example, in hazardous areas such as fire, construction, nuclear power plant, chemical plant, tunnel construction, police operation, military training, and overseas dispatch among work sites, mutual communication between field personnel or between field personnel and managers due to the nature of the site This will be very important.

Collaboration between the field and the remote location using the conventional HMD depends only on the point where the remote expert or manager is photographed by the HMD for the worker located in the field. The problem with this HMD collaboration method is that the efficiency of collaboration is lowered because the HMD screen for a remote expert moves according to the movement of the head of the worker located in the field.

Korean Patent Registration No. 10-2103404 (2020.04.23. Announcement)

doesn't exist

An object of the present invention is to provide a worker HMD worn by a worker located in the field, a professional HMD worn by an expert located at a remote location while communicating with the worker HMD, and the remote location expert using the expert HMD To provide a field and remote collaboration system and method using the HMD that can increase the efficiency of collaboration by remotely controlling the gaze of the auxiliary camera of the HMD for workers to simultaneously see the point desired by the expert in addition to the point illuminated by the operator. there is.

Another object of the present invention is to utilize the HMD that can increase the efficiency of collaboration by displaying the location of the point the expert is looking at on the screen of the HMD for the worker when the worker located in the field and the expert located in a remote place are looking at different points It is to provide a system and method for collaboration between one site and a remote site.

Another object of the present invention is to recognize a specific object on the expert's screen in order to minimize the influence of the screen viewed by the expert by the head movement of the worker in the field, and the auxiliary camera of the expert HMD to track it, so that the worker's head It is to provide a system and method for collaboration between on-site and remote locations using HMD that can minimize the effect of movement.

Another object of the present invention is to minimize the influence of the operator's head movement by correcting the screen by adding or subtracting the operator's head movement speed to the movement speed of the auxiliary camera when a field worker and a remote expert adjust the screen at the same time. It is to provide a field and remote collaboration system and method using HMD that can be used.

In order to achieve the above object, there is provided a system for collaboration between a field and a remote location using an HMD according to the present invention, comprising: an HMD for a worker worn by a worker located in the field; one or more professional HMDs worn by remote-located professionals and in communication with the operator HMDs; a main camera provided in the HMD for the worker to photograph the site; one or more auxiliary cameras provided in the HMD for the worker to photograph the field, and the HMD for the expert remotely controllable; And the HMD for the operator is characterized in that it comprises a gyro sensor for detecting the movement of the operator's head.

In addition, the HMD for the worker communicates with at least one HMD for the expert, and the HMD for the expert remotely controls the one auxiliary camera, respectively.

In addition, the expert HMD includes a gyro sensor for detecting the head movement of the expert, wherein the expert HMD is characterized in that the remote control of the gaze of the auxiliary camera by the head movement of the expert.

In addition, when the HMD for the worker faces the main camera and the auxiliary camera at different points, the HMD for the worker displays the position the auxiliary camera is looking at in any one of virtual reality, augmented reality, and mixed reality. characterized.

In addition, the HMD for the worker and the HMD for the expert further include an acceleration sensor, and when the main camera and the auxiliary camera move at the same time, the screen of the HMD for the expert adds or decreases the movement speed of the HMD for the operator to the movement speed of the auxiliary camera It is characterized in that it is corrected.

In addition, the method for collaboration between a field and a remote location using an HMD according to the present invention for achieving the above object is a method in which a worker located on the site and an expert located at a remote location collaborate through the HMD, (a) the HMD for the field worker and Interworking with a remote expert HMD; (b) transmitting the video signals of the main camera and the auxiliary camera of the HMD for the worker to the HMD for the expert; and (c) the HMD for the expert controlling the auxiliary camera of the HMD for the operator.

In addition, the step (c) includes transmitting, by the HMD for the expert, movement and direction information of the HMD for the operator to the HMD for the operator, and the HMD for the operator is a worker based on the movement and direction information of the HMD for the expert. It characterized in that it further comprises the step of displaying the gaze position of the auxiliary camera on the screen.

In addition, when the screen of the HMD for the expert is affected by the movement of the HMD for the worker, the method further comprising the step of tracking the singularity by the HMD for the expert, wherein the step of tracking the singularity is present in the screen of the HMD for the expert Recognizing at least one or more fixed objects, designating an object closest to the center of the HMD screen for experts among the recognized objects as a singularity, setting a reference line on the HMD screen for experts, and the expert and controlling the auxiliary camera so that the HMD screen for use maintains the coordinates of the singular point.

In addition, when the singularity deviates from the reference line on the expert HMD screen, designating a fixed object closest to the center of the expert HMD screen as a new singular point is included.

In addition, when the main camera and the auxiliary camera move at the same time, the HMD for experts corrects the movement of the auxiliary camera by adding or subtracting the movement speed of the main camera to the movement speed of the auxiliary camera.

According to the present invention as described above, since a worker located in the field and at least one expert located in a remote location can check different points of the field, the effect of increasing the efficiency of collaboration between the field and the remote site is obtained.

In addition, the present invention has the effect of increasing the efficiency of collaboration between the field and the remote site because the operator located in the field can check the location of the point currently being viewed by the expert located at a remote site.

In addition, the present invention has the effect of minimizing the effect of the head movement of the worker by recognizing a specific object on the screen of the expert and tracking it by the auxiliary camera of the HMD for the worker even if the worker located in the field moves the head.

In addition, the present invention corrects the screen by adding or subtracting the operator's head movement speed to the movement speed of the auxiliary camera even when the on-site operator and the remote operator adjust the screen at the same time, so the effect of the operator's head movement can be minimized. effect is obtained.

1 is a block diagram of a field and remote collaboration system using an HMD according to an embodiment of the present invention;
2 is a block diagram of an HMD for a worker according to an embodiment of the present invention;
3 and 4 are flowcharts showing a method for an operator and an expert to interlock each other's HMD;
5 and 6 are flowcharts showing a screen display method of the HMD;
7 and 8 are diagrams for explaining the location display using the movement direction information of the HMD;
9 is a flowchart of a singularity algorithm method of an expert HMD 20 according to the present invention;
10 is a flowchart for explaining the control of the screen according to the singularity algorithm according to the present invention;
11 is a diagram illustrating an example of a singularity algorithm according to the present invention, and
12 is a flowchart for explaining a case in which the HMD 10 for workers and the HMD 20 for professionals according to the present invention simultaneously control a camera.

Hereinafter, the present invention will be described by describing embodiments of the present invention with reference to the accompanying drawings.

Like reference numerals in each figure indicate like elements. Also, in describing the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Also, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

1 is a block diagram of a system for collaboration between a field and a remote location using an HMD according to an embodiment of the present invention, and FIG. 2 is a block diagram of an HMD for a worker according to an embodiment of the present invention.

As shown in Figure 1, the field and remote collaboration system using the HMD according to the present invention is a worker HMD 10 worn by a worker located in the field, and a professional HMD worn by an expert located in a remote location. It is composed of 20 , and the HMD 10 for an operator may include a control module 100 , a camera module 110 , a sensor module 120 , an input/output device 130 , and a communication module 140 . .

The HMD 10 for workers is an HMD (Head Mounted Device) worn by workers located in the field of construction, disaster, rescue and medical situations, etc. It is a device worn on the head. The frame of the HMD 10 for the operator may have an optical system disposed in front of both eyes of the operator so that the real world through the operator's field of view can be transmitted and viewed.

In addition, the camera module 110 of the HMD 10 for a worker may include a main camera 111 and one or more auxiliary cameras 112 so as to provide a photograph of a field situation. The main camera 111 may be disposed between the optical systems of the frame so as to take an image that matches the field of view of the operator who is the user of the HMD 10 for the operator, for example. In addition, the HMD 10 for the operator may include a display 150 for outputting image light into the HMD 10 for the operator so as to provide the image captured by the main camera 111 as an optimized screen to the operator. can

The auxiliary camera 112 is a camera provided on one side of the HMD 10 for a worker, and may be implemented as a PTZ (Pan-Tilt-Zoom) camera capable of remotely controlling the direction and enlargement/reduction. In the present invention, the auxiliary camera 112 is basically the same as the main camera 111, but can photograph the position of the operator's gaze, but to the expert HMD 20 for an expert located in a remote location, which will be described later, or a separate PTZ controller. can be remotely controlled by

On the other hand, the auxiliary camera 112 of the HMD 10 for the operator is provided with the auxiliary camera 112 according to the number of the HMD 20 for the professional to be connected. Therefore, the HMD 10 for workers may provide different images to the HMD 20 for professionals.

The control module 100 processes the images obtained from the main camera 111 and the auxiliary camera 112 together with information obtained by the sensor module 120 to be described later to provide the images. The control module 100 may include a memory and an image processing unit as an image processing device.

The memory may store image data transmitted by the main camera 111 and one or more auxiliary cameras 112 . The memory may store a model for generating a display image provided to an operator or an expert, and may store a certain machine learning model.

The image processing unit is an image device capable of processing image data. The image processing unit may edit the image to provide additional information to the output image by inputting the image to the machine learning model stored in the memory.

The sensor module 120 is a sensor built into the HMD 10 for a worker, and a sensor for providing additional information to an operator who is a user and an image provided to the outside is provided. The sensor module 120 may include, for example, a gyro sensor 121 , an acceleration sensor 122 , and a visual sensor 123 .

The gyro sensor 121 senses the movement of the HMD 10 for the operator in the three-axis direction, that is, the movement direction, and provides the sensed signal to the control module 100 . The acceleration sensor 122 detects the movement speed of the HMD 10 for a worker and provides a sensed signal to the control module 100 , and the visual sensor 123 is provided from the main camera 111 and the auxiliary camera 112 . A graphic signal input to the collected image data is generated. The generated graphic signal may be edited together with the image data in the image processing unit of the control module 100 so that additional information displayed on the image may be implemented in any one of virtual reality, augmented reality, or mixed reality.

The input/output device 130 is a device that can additionally connect an external device to the HMD 10 for a worker, and a wearable device for assisting a worker located in the field can be connected, for example, a headphone, a microphone, etc. are connected to the professional It may communicate with the HMD 20 . In addition, the input/output device 130 may further include an input/output control device to control the input/output device 130 by interworking with the HMD. For example, a sound controller or a voice controller may be provided.

The communication module 140 is a communication device for receiving or transmitting data and commands through communication of the HMD 10 for workers. The communication module 140 may include a communication device that performs wireless communication (eg, Wi-Fi).

The HMD 10 for a worker configured as described above is worn by a worker located in the field, and it is possible to communicate with the outside by collecting images about the field situation.

On the other hand, the HMD for professionals 20 is a Head Mounted Device (HMD) worn by at least one expert located in a remote location other than the field of construction, disaster, rescue, and medical situations, and the HMD 10 for workers at a remote location. ), it is a device worn on the head of an expert to receive an image of the situation on the spot and to deliver the situation to the expert vividly through the display. The expert HMD 20 includes a display 150 for outputting an image of the field received from the worker HMD 10, a control module 100, a sensor module 120, an input/output device 130, and a communication module ( 140) may be included.

The display 150 of the HMD 20 for professionals receives and displays the images taken from the main camera 111 and the auxiliary camera 112 of the HMD 10 for workers in the field. At this time, the display 150 of the professional HMD 20 may provide an image in a split-screen manner to display both the image photographed by the main camera 111 and the image photographed by the auxiliary camera 112 on the display 150 . there is.

The control module 100 of the professional HMD 20 is an image processing device that processes the main camera 111 image and the auxiliary camera 112 image received from the worker HMD 10, and includes a memory and an image processing unit. and may further include a control unit capable of remotely operating the auxiliary camera 112 of the HMD 10 for the operator. For example, the control unit collects movement direction information of the gyro sensor 121 of the expert HMD 20 to be described later, and generates a control signal of the auxiliary camera 112 based on the collected information through the communication module 140 . It can be transmitted to the HMD 10 for the operator.

The sensor module 120 of the professional HMD 20 may include a gyro sensor 121 , an acceleration sensor 122 , and a visual sensor 123 similarly to the sensor module 120 of the HMD 10 for an operator. . However, the information collected by the sensor module 120 of the HMD 20 for professionals collects information necessary to remotely control the HMD 10 for the operator.

The gyro sensor 121 senses the movement in the three-axis direction of the expert HMD 20 , that is, the movement direction, and provides a sensed signal to the control module 100 , and the acceleration sensor 122 is the expert HMD 20 . ) and provides the sensed signal to the control module 100 by detecting the movement speed. The control module 100 may generate a control signal of the auxiliary camera 112 based on the received movement direction signal and movement speed signal and transmit it to the HMD 10 for the operator through the communication module 140 . That is, an expert located at a remote location can directly control the auxiliary camera 112 of the HMD 10 for a worker worn by a worker located in the field by wearing the HMD 20 for the expert and using a head movement.

On the other hand, the expert HMD 20 can additionally connect an external device through the input/output device 130, and when the expert's head cannot move or fine screen control is required, the auxiliary camera 112 is interlocked with a PTZ controller. can be made to control.

The collaboration system between the field and the remote location using the HMD according to the present invention configured as described above enables a worker located in the field and an expert located in a remote location to wear each HMD and communicate with each other through each HMD, The HMD 20 remotely controls the auxiliary camera 112 of the HMD 10 for the operator separately from the operator's control, so that the remote expert can check points other than the point directly illuminated by the operator through the main camera 111 This can increase the efficiency of collaboration between workers and experts.

Hereinafter, a method of collaboration between a field and a remote location using the HMD 10 for workers and the HMD 20 for experts will be described. 3 and 4 are flowcharts showing a method for an operator and an expert to link HMDs with each other, FIGS. 5 and 6 are flowcharts showing a screen display method of the HMD, and FIGS. 7 and 8 are HMD movement direction information It is a drawing for explaining the utilized position display.

3 to 8, the method of collaboration between the field and the remote location using the HMD includes the steps of linking the HMD 10 for a field worker and the HMD 20 for a remote location, and a main camera of the HMD for the worker. (111) and the step of transmitting the video signal of the auxiliary camera to the expert HMD 20, and the expert HMD 20 controlling the auxiliary camera 112 of the worker HMD 10 can do.

First, the worker HMD 10 determines whether the worker wears the worker HMD 10 in the field and interlocks with the expert HMD 20 in order to communicate with an expert located in a remote location. When the operator wants to work alone in the field without interlocking with the HMD 20 for the operator, only the image signal of the main camera 111 of the HMD 10 for the operator is output to the display 150 of the HMD 10 for the operator. do. When the operator wants to interwork with the HMD 20 for the professional, the HMD for the operator 10 and the HMD 20 for the professional are interlocked through communication setting, and the main camera 111 of the HMD 10 for the operator is interlocked. The video signal of the auxiliary camera 112 is transmitted to the professional HMD 20 . In this case, when the HMD 10 for the worker interlocks with the plurality of HMDs for experts 20, the image signals are transmitted to the HMDs 20 for experts determined respectively of the plurality of auxiliary cameras 112 provided in the HMD 10 for the worker. to do it That is, the remote expert HMD 20 receives the image signal of the main camera 111 and the image signal of the predetermined auxiliary camera 112 .

The expert HMD 20 receives the interlocking request from the worker HMD 10 of the worker, sets the communication, and receives the image signals of the main camera 111 and the auxiliary camera 112 from the worker HMD 10. . In this case, the expert can control whether the professional HMD 20 simultaneously views the images of the main camera 111 and the auxiliary camera 112 through the split screen.

The expert HMD 20 detects the expert's head movement through the built-in gyro sensor 121 and generates a control signal for the auxiliary camera 112 based on the movement direction information of the expert HMD 20, and then It can be transmitted to the HMD (10) to control the auxiliary camera (112). In addition, the expert HMD 20 may connect a separate external device such as a PTZ controller to control the auxiliary camera 112 by manipulation other than the expert's head movement.

On the other hand, the worker located in the field can see the image taken by the main camera 111 of the HMD 10 for the worker, which the worker directly controls through the movement of the head, but the remote expert directly through the HMD 20 for the expert The image of the controlled auxiliary camera 112 (ie, the image the expert is watching) cannot be confirmed, so a problem may occur in mutual communication between the operator and the expert.

Therefore, the HMD 10 for the operator is the operator of the HMD 10 in order to inform that the expert located at a remote location is looking at a point different from the point the main camera 111 is looking at (that is, the point the operator is looking at). The position of the point (expert's screen) being photographed by the auxiliary camera 112 may be displayed on the display 150 .

The position display of the auxiliary camera 112 of the display 150 of the HMD 10 for a worker according to an embodiment may be displayed in an arrow manner, and the viewpoint direction of the auxiliary camera 112 based on 8 directions on the operator's screen can be displayed. Determination of the position of the auxiliary camera 112 may be implemented by using the gyro sensor 121 of each HMD to display the relative position according to the head movement of the expert and the head movement of the operator.

9 is a flowchart of a singularity algorithm method of the expert HMD 20 according to the present invention, FIG. 10 is a flowchart for explaining screen control according to the singularity algorithm according to the present invention, and FIG. It is a diagram explaining an example of an algorithm.

On the other hand, in the process of mutual communication between the operator located in the field and the expert located in a remote location, even if the expert remotely adjusts the auxiliary camera 112 to view a specific screen, the point where the auxiliary camera 112 is illuminated by the operator's head movement is Moving, the expert's screen 200 is affected. In this case, the expert HMD 20 designates a specific object existing in the screen as a singularity, and moves the auxiliary camera 112 in the opposite direction as much as the direction in which the singular point moves, so that the expert's screen 200 is fixed.

The step of performing the singularity algorithm according to an embodiment of the present invention includes recognizing at least one or more fixed objects existing in the screen of the expert HMD 20, and selecting the screen of the expert HMD 20 among the recognized objects. The step of designating the object closest to the center as the singularity, the step of setting the reference line 210 on the screen of the expert HMD 20, and the auxiliary camera 112 so that the screen of the expert HMD 20 maintains the coordinates of the singular point control may be included.

First, when an expert located at a remote location wants to fix a specific point to be observed on the expert's screen 200, the expert manipulates the expert HMD 20 to fix the auxiliary camera 112. In this case, even if the expert moves his head, the auxiliary camera 112 may not move.

Next, the expert HMD 20 recognizes at least one or more fixed objects currently appearing on the expert's screen 200, and designates coordinates of the recognized objects. In this case, as the coordinates, X, Y coordinates may be applied based on the center of the screen 200 of the expert. Then, among the recognized objects, the center of the expert's screen 200, that is, the object closest to the central coordinates, is designated as a singularity, and the remaining recognized objects are designated as alternative singularities.

Next, the expert HMD 20 sets the reference line 210 in a certain range based on the edge of the current expert's screen 200 to end the creation of the singularity algorithm.

In this way, when the professional HMD 20 sets a singularity for a specific screen and the auxiliary camera 112 is affected by the operator's head movement thereafter, the expert HMD 20 moves the auxiliary camera 112 . The direction and distance in which the singular point moves are collected, and a control signal is transmitted to the operator HMD 10 so that the auxiliary camera 112 moves in the opposite direction to the direction in which the singular point moves.

On the other hand, if the operator's head moves excessively, a singularity point appearing on the expert's screen 200 may go out of the screen and the auxiliary camera 112 may not be able to track the location of the singular point.

11 is a view for explaining an example of a singularity algorithm according to the present invention, (a) of FIG. 11 shows recognition of a number of fixed objects on the screen 200 of the expert, (b) is the screen of the expert (200) shows that the singular point has moved, and (c) is a diagram for explaining a case where the singular point deviates from the reference line 210 .

Referring to FIG. 11 , when the expert wants to fix the expert screen 200 , the expert HMD 20 detects fixed objects on the expert's screen 200 . As shown in Fig. 11 (a), if the fixed objects P1, P2, and P3 are detected, the expert HMD 20 sets P1 as a singularity point because the P1 object is closest to the center of the expert screen, and the remaining P2 and P3 is set as an alternative singularity.

In a state in which the singularity is set, when the auxiliary camera 112 is moved by the movement of the operator's head, the screen 200 of the expert moves and the coordinates of the recognized objects are moved, as shown in Fig. 11 (b). The P1 coordinates designated by will also move. The professional HMD 20 recognizes the direction and size of the movement of the auxiliary camera 112 by moving the coordinates of the moved P1, and the auxiliary camera 112 so that the auxiliary camera 112 moves in the opposite direction to the movement direction. transmits the control signal of the operator to the HMD (20). Accordingly, the auxiliary camera 112 may track the singular point P1.

On the other hand, as shown in FIG. 11 ( c ), if the operator's head movement is excessive enough that the singular point P1 deviates from the preset reference line 210 on the expert screen 200 , the expert HMD 20 is currently an expert An alternative singularity closest to the center coordinate of the screen 200 is set as a new singularity, and the tracking algorithm as described above is executed. In this case, it is preferable that the expert HMD 20 detects a fixed object whenever the expert screen 200 moves to continuously detect an alternative singularity.

12 is a flowchart for explaining a case in which the HMD 10 for a worker and the HMD 20 for an expert according to the present invention simultaneously control a camera.

On the other hand, if a worker located in the field and an expert located at a remote location move their heads at the same time while wearing the worker HMD 10 and the professional HMD 20 respectively worn during the collaboration process, the remote expert located at the remote location moves the worker's head There is a problem in that it is difficult to adjust the auxiliary camera 112 as desired under the influence of .

For example, if the operator moves the head while the expert controls the auxiliary camera 112 using the expert HMD 20, the main camera 111 and the auxiliary camera 112 move at the same time, at this time the main camera 111 ) and the auxiliary camera 112 move in the same direction, there is a problem in that the auxiliary camera 112 moves larger than the control signal of the auxiliary camera 112 controlled by the expert, and conversely, the main camera 111 and When the moving direction of the auxiliary camera 112 is different, the auxiliary camera 112 moves smaller than the control signal of the auxiliary camera 112 controlled by the expert. That is, when a field worker moves his head, it becomes difficult for a remote expert to control the auxiliary camera 112 as desired.

Therefore, in order to minimize the influence on the head movement of the operator when the expert remotely controls the auxiliary camera 112, the HMD 20 for the expert adjusts the movement direction and speed of the auxiliary camera 112 currently controlled by the expert. By controlling the auxiliary camera 112 by adding or subtracting the direction and speed of the movement of the HMD 20 for use, the auxiliary camera 112 can minimize the influence of the head movement of the operator in the field.

According to an embodiment, the expert HMD 20 measures the expert's head movement direction and speed through the built-in gyro sensor 121 to know the movement direction and speed of the auxiliary camera 112, and the operator's head The movement direction and speed are also measured through the gyro sensor 121 . Then, when the moving direction of the main camera 111 and the auxiliary camera 112 using the collected information, that is, the direction of the operator's head movement and the expert's head movement direction are the same, the expert HMD 20 is the auxiliary camera 112 ) to control the auxiliary camera 112 by generating a control signal for the auxiliary camera 112 by subtracting the operator's head movement speed from the movement speed.

Conversely, when the moving directions of the main camera 111 and the auxiliary camera 112 are different, that is, when the moving direction of the operator's head and the expert's head moving direction are different, the expert HMD 20 moves the auxiliary camera 112 . To control the auxiliary camera 112 by generating a control signal for the auxiliary camera 112 as much as the speed plus the speed of the operator's head movement.

On the other hand, when the movements of the main camera 111 and the auxiliary camera 112 according to the expert's intention are different from each other, the expert HMD 20 transmits the control signal of the auxiliary camera 112 according to the expert's head movement to the operator's HMD. The HMD 10 for the worker, which is transmitted to (10) and receives the control signal of the auxiliary camera 112, extracts the movement direction information of the operator corresponding to the opposite direction to the control signal of the received auxiliary camera 112, It is reflected in the control signal of the auxiliary camera (112).

For example, if the expert moves the HMD 20 for the expert in the southeast direction to control the auxiliary camera 112 in the southeast direction, the HMD 20 for the expert transmits the control signal of the auxiliary camera 112 to the HMD 10 for the operator ), and the worker HMD 10 determines that the movement direction of the control signal of the received auxiliary camera 112 is the southeast direction, and extracts and receives the movement information in the northwest direction of the current operator HMD 10 reflected in the control signal.

Therefore, since the expert HMD 20 can reflect the direction and speed of the operator's head movement in the control of the auxiliary camera 112, even if the operator in the field moves the head, the expert is It is possible to control the auxiliary camera 112 as well.

The above description of the present invention is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be.

10: HMD for workers 20: HMD for professionals
100: control module 110: camera module
120: sensor module 130: input/output device
140: communication module

Claims (10)

Worker HMDs worn by field-located workers;
one or more professional HMDs worn by remotely located professionals and in communication with the operator HMDs;
a main camera provided in the HMD for the worker to photograph the site;
one or more auxiliary cameras provided in the HMD for the worker to photograph the field, and the HMD for the expert remotely controllable; and
The HMD for the worker is a field and remote collaboration system using the HMD, characterized in that it comprises a gyro sensor for detecting the head movement of the worker. According to claim 1,
The HMD for the worker communicates with at least one HMD for the expert, and the HMD for the expert remotely controls the one auxiliary camera. 3. The method of claim 2,
The expert HMD includes a gyro sensor for detecting the expert's head movement,
The HMD for experts is a collaboration system between a field and a remote location using an HMD, characterized in that the gaze of the auxiliary camera is remotely controlled by the head movement of the expert. 4. The method of claim 3,
When the HMD for the worker faces a different point between the main camera and the auxiliary camera, the HMD for the worker displays the position viewed by the auxiliary camera in any one of virtual reality, augmented reality, and mixed reality. On-site and remote collaboration system using HMD. 4. The method of claim 3,
The HMD for workers and the HMD for professionals further include an acceleration sensor,
When the main camera and the auxiliary camera move simultaneously, the screen of the professional HMD is corrected by adding or subtracting the movement speed of the operator HMD to the movement speed of the auxiliary camera. A method in which a worker located in the field and an expert located in a remote location collaborate through an HMD,
(a) interlocking the HMD for workers in the field and the HMD for experts at a remote location;
(b) transmitting the video signals of the main camera and the auxiliary camera of the HMD for the worker to the HMD for the expert; and
(c) the on-site and remote location collaboration method using the HMD, characterized in that it comprises the step of controlling the auxiliary camera of the HMD for the worker by the HMD for the expert. 7. The method of claim 6,
In the step (c), the HMD for the expert transmits movement and direction information of the HMD for the expert to the HMD for the operator;
The method for collaboration between the field and the remote location using the HMD, characterized in that the HMD for the worker further comprises the step of displaying the gaze position of the auxiliary camera on the screen for the worker based on the movement and direction information of the HMD for the expert. 7. The method of claim 6,
When the screen of the professional HMD is affected by the movement of the worker HMD, the method further comprising the step of tracking the singularity of the professional HMD,
The step of tracking the singular point may include recognizing at least one or more fixed objects existing in the screen of the expert HMD;
Designating an object closest to the center of the HMD screen for experts among the recognized objects as a singularity;
setting a baseline on the professional HMD screen, and
and controlling the auxiliary camera so that the HMD screen for the expert maintains the coordinates of the singular point. 9. The method of claim 8,
When the singularity deviates from the reference line on the HMD screen for experts, designating a fixed object closest to the center of the HMD screen for experts as a new singularity. How to collaborate. 8. The method of claim 7,
When the main camera and the auxiliary camera move at the same time, the HMD for experts corrects the movement of the auxiliary camera by adding or subtracting the movement speed of the main camera to the movement speed of the auxiliary camera. Way.

Images