KR20200083359A - Method and apparatus for controlling a robot using head mounted display - Google Patents

Abstract

Provided is a method for controlling, by a computing device, a robot in accordance with a user input received by head-mounted display (HMD) equipment. According to an embodiment of the present invention, the method for controlling a robot using an HMD comprises the steps of: sensing a head motion of a user wearing an HMD; sensing a position of a pupil of the user by using a sensor embedded in the HMD; determining whether a gaze of the user is toward a first position for a first reference time or more by using a result that the position of the pupil is sensed; determining that a first user input determined in accordance with an object positioned in the first position and a first head motion occurs when it is determined that the gaze of the user is toward the first position for the first reference time or more, and the first head motion of the user wearing the HMD is sensed within a second reference time with respect to a time point at which the gaze of the user is toward the first position for the first reference time or more; and displaying a feedback in accordance with the first user input on the HMD.

Description

Robot control method using HMD equipment {METHOD AND APPARATUS FOR CONTROLLING A ROBOT USING HEAD MOUNTED DISPLAY}

The present invention relates to a robot control method and apparatus using HMD equipment. More specifically, the present invention relates to a method of controlling a camera's mode and robot movement provided to a user using HMD equipment and an apparatus to which the method is applied.

The initial introduction of the remote robot control was applied to replace humans in dangerous work fields, but recently, the field of application has been expanded to reduce human fatigue and increase work ability. Many robots with excellent mobility and workability have been developed, but there are limitations in performing tasks on behalf of people in an unstructured environment due to lack of robot intelligence. Therefore, if a new robot control method capable of supplementing the intelligence of the insufficient robot at a long distance is developed, more efficient work will be possible.

The conventional remote control robot control method was performed by a user sitting in a cockpit and controlling a robot at a remote location directly with a hand of a pilot after a user mounts a HMD (Head-Mounted Display). However, control of the user's work interface as well as control of the robot itself is required, so the control method depending on the hand of the manipulator is limited to perform precise and special tasks. Accordingly, there is a need for a method of controlling a robot using feedback according to user input information without moving a user's hand using HMD.

Moreover, there is no solution to this, although the processing of multiple overlapping objects existing in the image information provided by the robot corresponds to a very important matter in remote control of the robot.

Republic of Korea Patent Registration Publication No. 10-1759444

The technical problem to be solved by the present invention is to provide a method for controlling a robot according to a user input obtained from a sensor embedded in the HMD equipment using the HMD equipment and a device to which the method is applied.

Another technical problem to be solved by the present invention is to provide a method for controlling selection of a plurality of camera modes provided by a robot and an apparatus to which the method is applied.

Another technical problem to be solved by the present invention is to provide a method for controlling selection of a plurality of overlapping objects existing in image information provided by a robot and an apparatus to which the method is applied.

The technical problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the following description.

In order to solve the above technical problem, the robot control method using the HMD equipment according to an embodiment of the present invention is whether or not the user is watching the overlapping position of a plurality of objects displayed on the HMD for a first reference time or more. And if it is determined that the overlapping position is more than the first reference time, the focus object may provide the changed feedback to any one of the plurality of objects.

In one embodiment, the providing of the feedback is based on a reference period from the object located at the front to the object located at the back of the plurality of overlapping objects displayed on the HMD in turn, and the focus object provides the changed feedback It may include steps.

In one embodiment, the reference period may be shortened when the number of the plurality of objects displayed on the HMD increases.

In an embodiment, when the target object to be changed to the focus object corresponds to the first condition, the method may further include providing feedback to which the focus object is changed, as an object behind the target object.

In one embodiment, the step of providing the feedback is an object located behind a focus object displayed based on a first user input time point among the plurality of overlapping objects displayed on the HMD, and the feedback of the focus object is changed. And providing steps.

In one embodiment, the step of providing the feedback is from a camera mode displayed based on a first user input time point among a plurality of camera modes provided by a camera embedded in the robot and displayed on the HMD, to another camera mode, And providing the feedback in which the focus object is changed.

In one embodiment, acquiring head motion information of a user wearing an HMD, determining whether the user is watching the first location for a first reference time or more, and the user determining the first location It is determined that the user is watching at least one reference time, and the first head motion of the user wearing the HMD within the second reference time is detected based on the time when the user watches the first position for at least the first reference time. In the case, it may include determining that an object located in the first position and a first user input determined according to the first head motion are generated, and providing feedback according to the first user input to the HMD. have.

In one embodiment, providing the feedback according to the first user input to the HMD may include providing an enlarged feedback of the object located at the first location displayed on the HMD.

In one embodiment, providing feedback according to the first user input to the HMD may include providing feedback in which a focus object is changed to any one of a plurality of objects.

In one embodiment, while feedback according to the first user input is displayed on the HMD, it is determined that the user is watching the second location for at least the first reference time, and the user is viewing the second location. When a second head motion of the user wearing the HMD is detected within the second reference time based on a point of time over which the first reference time is watched, it is determined according to the object located in the second position and the second head motion The method may further include determining that a second user input has occurred, and providing feedback according to the second user input to the HMD.

In one embodiment, the step of providing feedback according to the first user input to the HMD provides a feedback in which an action option for controlling the movement of the robot with respect to the object located in the first position is displayed on the HMD. It may include the steps.

In one embodiment, obtaining the motion information of the muscles around the eye of the user, determining that a first user input corresponding to a motion pattern based on the motion of the muscles around the eye of the user has occurred and the first user input Accordingly, the method may include providing feedback in which the focus object is changed to any one of the plurality of objects.

In one embodiment, determining whether the user is viewing the location of the first object displayed on the HMD for a first reference time or more, and determining whether the user is viewing the location of the first object for the first reference time or more. If it is determined to be watching, an action option for controlling the movement of the robot with respect to the first object may include providing feedback displayed on the HMD.

A robot control apparatus using HMD equipment according to an embodiment of the present invention includes a processor, a network interface, a memory, and a computer program loaded into the memory and executed by the processor, wherein the computer program includes:

Instruction for obtaining head movement information of a user wearing an HMD, an instruction for determining whether the user is watching the first location for a first reference time or longer, using the sensor built in the HMD, the user It is determined that the first position is looking at the first reference time or more, and the user wears the HMD within a second reference time based on a time when the user watches the first position at least the first reference time. When the first head motion is detected, an instruction for determining that an object located at the first position and a first user input determined according to the first head motion is generated and feedback according to the first user input are generated by the HMD It may include the steps to provide.

1 is a configuration diagram of a robot control system using HMD equipment according to an embodiment of the present invention.
2 is an exemplary diagram for a problem to be solved according to an embodiment of the present invention.
3 is a flowchart illustrating a process for performing a method of controlling a robot using HMD equipment according to an embodiment of the present invention.
FIG. 4 is a detailed flow chart for describing in detail some operations of the method described with reference to FIG. 3.
FIG. 5 is a detailed flowchart for describing in detail some operations of the method described with reference to FIG. 4.
6 is an exemplary view for explaining a method of controlling a robot using HMD equipment according to the method described with reference to FIG. 5.
7 is a detailed flowchart for describing in detail some operations of the method described with reference to FIG. 4.
8 is an exemplary diagram for explaining a method of controlling a robot using HMD equipment according to the method described with reference to FIG. 7.
FIG. 9 is a detailed flowchart for explaining in detail a method in which some operations are further added to the method described with reference to FIG. 3.
10 to 11 are flowcharts illustrating a process of performing a method of controlling a robot using HMD equipment according to another embodiment of the present invention.
12 is a diagram illustrating an exemplary computing device capable of implementing various embodiments of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. Advantages and features of the present invention, and methods for achieving them will be clarified with reference to embodiments described below in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains. It is provided to completely inform the person having the scope of the invention, and the present invention is only defined by the scope of the claims.

It should be noted that in adding reference numerals to the components of each drawing, the same components have the same reference numerals as possible even though they are displayed on different drawings. In addition, in describing the present invention, when it is determined that detailed descriptions of related well-known structures or functions may obscure the subject matter of the present invention, detailed descriptions thereof will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used as meanings commonly understood by those skilled in the art to which the present invention pertains. In addition, terms defined in the commonly used dictionary are not ideally or excessively interpreted unless specifically defined. The terminology used herein is for describing the embodiments and is not intended to limit the present invention. In the present specification, the singular form also includes the plural form unless otherwise specified in the phrase.

In addition, in describing the components of the present invention, terms such as first, second, A, B, (a), and (b) may be used. These terms are only for distinguishing the component from other components, and the nature, order, or order of the component is not limited by the term. When a component is described as being "connected", "coupled" or "connected" to another component, that component may be directly connected to or connected to the other component, but another component between each component It should be understood that elements may be "connected", "coupled" or "connected".

As used herein, "comprises" and/or "comprising" refers to the elements, steps, operations and/or elements mentioned above, the presence of one or more other components, steps, operations and/or elements. Or do not exclude additions.

Hereinafter, some embodiments of the present invention will be described in detail according to the accompanying drawings.

1 is a configuration diagram of a robot control system using HMD equipment according to an embodiment of the present invention.

Referring to FIG. 1, a robot control system using HMD equipment may include a target server 100, an HMD equipment 200, and a robot 300. 1 illustrates an embodiment in which the target server 100 receives image information provided by the robot 300 and controls sensor by receiving sensor detection information by the HMD equipment 200.

According to an embodiment, the target server 100, the HMD equipment 200, and the robot 300 may be configured to include a computing device that can be connected to each other through a network. For example, the target server 100 may include a server device configured with one or more processors, a device such as a desktop PC, a notebook, and a smartphone. The HMD equipment 200 may include, for example, a device having a display capable of outputting a screen. Further, it may be configured to include a sensor for detecting the position of the pupil of the user, the movement of the muscles around the eyes, and the head movement. In other embodiments, HMD equipment 200 may be configured to include devices configured to include one or more processors. The robot 300 may be configured to include a camera capable of providing various information of the work progress area to the user. For example, it may be configured to include a device such as a visible light camera, a thermal imaging camera, an ultrasonic camera. However, the present invention is not limited to the above example.

The target server 100 according to an embodiment of the present invention may obtain various information on user input from the HMD equipment 200. Here, the information about the user input includes the location information of the user pupil, the location information of the user's gaze using the location information of the pupil, the gaze time information using the location information of the gaze, the motion information of the muscles around the user's eyes, and the user's It may include head movement information.

In another embodiment of the present invention, the target server 100 may obtain information about the user input from the HMD equipment 200 and process the information to determine the user input. Information about the user input may be location information of the user pupil. In this embodiment, the target server 100 may obtain the location information of the user's gaze by obtaining the location information of the user pupil from the HMD equipment 200. Furthermore, the target server 100 may acquire gaze time information by utilizing the location information of the user's gaze.

The target server 100 according to an embodiment of the present invention may obtain various information about the work progress area from the robot 300. Here, the information on the work progress area may include image information of a visible light region photographed by the robot 300 at a remote location, thermal image information, and ultrasound image information.

The HMD equipment 200 according to an embodiment of the present invention may include a sensor that detects information on the user input. In one embodiment, a sensor that detects an IR signal and an eyeball potential (EOG) may be used to obtain location information of the user's gaze utilizing the location of the pupil of the user and location information of the pupil. A specific method for this can be found in Korean Patent Registration No. 10-1817436. Using the location information of the gaze, the user's gaze time information about the point where the gaze is located may be obtained. In another embodiment, a sensor that detects an IR signal may be used to obtain motion information of muscles around the user's eyes. A specific method for this can be found in Korean Patent Registration No. 10-1759444. In another embodiment, an acceleration sensor and a gyro sensor may be used to obtain motion information of the user's head.

The HMD equipment 200 according to an embodiment of the present invention may include a display providing image information of a work progress area to a user. The image information may be provided directly from the robot 300. In another embodiment, it may be processed and provided by the target server 100.

The robot 300 according to an embodiment of the present invention may acquire various information about a work progress area. Here, information on the work area can be obtained by a visible light camera. In other embodiments, the information can be obtained by a thermal imaging camera. In another embodiment, the information can be obtained by an ultrasound camera.

The robot 300 according to an embodiment of the present invention may be controlled by the target server 100 to perform work. In another embodiment, the robot 300 may be controlled directly from the HMD equipment to perform work.

Hereinafter, a specific example of a problem to be solved according to an embodiment of the present invention will be described with reference to FIG. 2. In recent years, the field of remote control robot control has been expanded to reduce human fatigue and increase work capability. The trend is expanding not only in industrial sites but also in shops and homes.

2, the plurality of shirts illustrated corresponds to an example of image information provided by a store robot. For example, the user may perform shopping based on the image information provided by the robot at a remote location, and may request product information on several shirts among the plurality of shirts in the image information provided by the robot prior to purchase. Since the image information provided by the robot is provided in a form of overlapping with each other among a plurality of shirts, a problem arises in that it becomes difficult to select objects for some shirts to check information before purchasing among the plurality of shirts. In some embodiments to be described later, by selecting a target object by changing a focus object corresponding to a user's input wearing a HMD at a remote location, it is possible to facilitate selection of a specific object in a situation where a plurality of objects overlap. That is, by providing an interface for selecting a specific object among a plurality of objects superimposed on the display, a method for pre-controlling an operation for a robot to perform a special and precise operation is provided. By providing the correct selection for the target object, the robot can perform elaborate and intuitive commands.

In FIG. 3, a robot control method using HMD equipment according to an embodiment of the present invention will be described in detail. The robot control method may be performed by the target server 100.

3 is a flowchart illustrating a process for performing a method of controlling a robot using HMD equipment according to an embodiment of the present invention.

In step S101, the position of the pupil is detected in the HMD equipment 200. The location information of the pupil detected by the HMD equipment 200 may be provided to the target server 100. The position of the pupil can be detected by a sensor that detects an IR signal or an EOG signal. However, the present invention is not limited thereto, and various methods for detecting the position of the pupil may be included. Furthermore, since the pupil position corresponds to one of several means for specifying the first location located on the HMD, the present invention is not limited to this.

Next, the first position gazing time is determined in step S103. The first position located on the HMD is specified using the position of the pupil detected in step S101, and the time at which the user's gaze stays within a reference distance from the first position may be calculated. However, since the position of the pupil corresponds to one of several means for specifying the first position, the present invention is not limited thereto. The gaze time may be calculated by the HMD equipment 200 and provided to the target server 100. In another embodiment, the target server 100 obtains only the real-time pupil location information from the HMD equipment 200 to calculate the first location located on the HMD, and the gaze time for the location can be determined.

Next, in step S105, when the first position is being watched for a first reference time or longer, the head motion is detected in step S107. The head motion detected by the HMD device 200 may be provided to the target server 100. The head motion can be detected by an acceleration sensor and a gyro sensor. By utilizing the acceleration sensor and the gyro sensor, the user's vertical and horizontal head motion can be detected.

Next, in step S109, when the first head motion is detected within the second reference time based on the point in time at which the first position is watched for at least the first reference time, in step S111, the object and the user located at the first position on the HMD The user's input is determined based on the combination of the first head motions. The determination of the user input may be performed by the target server 100 based on the detection information provided by the HMD equipment 200.

Next, in step S113, feedback according to the determined user input is displayed on the HMD. Feedback corresponding to the user input determined by the target server 100 may be transmitted to the HMD equipment 200 and displayed.

In a specific embodiment in which step S113 is embodied, the feedback in which the object located at the first position determined in step S111 is enlarged is displayed. For example, when a fixed head motion is detected within a second reference time based on a point in time at which a specific object located on the HMD is watched for a first reference time or longer, the position of the specific object and the fixed head motion The feedback in which a specific object is enlarged by the combination can be displayed. However, the present invention is not limited to the first position and the first head motion set by the present example. By displaying the feedback that the specific object is enlarged, it is possible to more easily perform a subsequent operation using the robot for the enlarged target object.

In another embodiment, a feedback displaying an action option for controlling the movement of the robot with respect to the object located at the first position determined in step S111 is displayed. For example, when a motion of rotating the head 360 degrees within a second reference time is detected based on a point in time at which the first position with a specific object located on the HMD is monitored for a first reference time or longer, the position of the specific object and The combination of the head motion rotating 360 degrees may display a feedback indicating an action option controlling the movement of the robot with respect to a specific object. However, the present invention is not limited to the first position and the first head motion set by the present example.

In this embodiment, the movement of the robot relative to a specific object is provided in various action options. For example, when it corresponds to the store robot illustrated in FIG. 2 above, a specific object may correspond to a product sold in a store. The options provided by the movement of the robot may include information about the product, move the product to be viewed from a different angle, and order and package the product. For another example, in the case of a security robot, a specific object may be an unauthorized person in a space requiring security. The options provided by the movement of the robot may include continuing tracking of the person, reporting to the manager, and requesting tracking of other methods of the robot.

As illustrated in FIG. 3, feedback according to a user input may be displayed on the HMD. By using such a control method as a pre-control method of the robot for controlling the future operation of the robot, an interface for creating a user's precise working environment is provided. Furthermore, by not requiring the use of a user's hand, it provides a means to control the interface even without a complicated operation key.

In FIG. 4, the robot control method according to some embodiments of the present invention embodying the operation of step S113 will be described in detail. The robot control method may be performed by the target server 100.

In step S130, the feedback in which the focus object is changed to any one of the plurality of objects is displayed. Feedback corresponding to the user input determined by the target server 100 may be transmitted to the HMD equipment 200 and displayed. The focus object means a specific object selected and displayed among various objects displayed on the HMD. However, the present invention is not limited to a specific object that is selected and displayed, and may mean a specific camera mode selected and displayed among a plurality of camera modes provided by the robot 300. That is, the focus object means a specific object selected and displayed among a plurality of selectable objects. The focus object may be displayed with a specific color on the object border or shaded with a specific color on the HMD. However, the present invention is not limited to this.

In this embodiment, in response to a user input, a feedback in which the focus object is changed to any one of a plurality of objects may be displayed. In this embodiment, the user input may be the overlapping position viewing time when the user is watching the overlapping position where a plurality of objects overlap on the HMD for a first reference time or longer.

In another embodiment, when the user is watching the first position on the HMD for a first reference time or more, the user's first head motion is within a second reference time from a point in time that the user is looking at the first reference time or more. If present, it may be a combination of the object located at the first position and the user's first head motion. For example, when a user is watching a location where a plurality of objects overlap on the HMD for a first reference time or more, there is a head motion toward the user within a second reference time from a point in time that the first reference time is being watched. In this case, a combination of a position where a plurality of objects overlap and a user's head motion upward may correspond. However, the present invention is not limited to the first position and the first head motion set by the present example.

In another embodiment, the user input may be a pattern based on muscle movement around the user's eyes. For example, a pattern based on the user's right eye wink may be applied. However, the present invention is not limited to the pattern set by the present example.

As illustrated in FIG. 4, as a pre-control method prior to performing the actual operation of the robot, an interface that enables a specific object to be selected among a plurality of selectable objects helps a user to perform a precise operation. Furthermore, it is possible to transmit an intuitive command to the robot by focusing the operation of the robot on a specific object.

Hereinafter, in FIGS. 5 to 8, a robot control method according to some embodiments of the present invention embodying the operation of step S130 will be described in detail. The robot control method may be performed by the target server 100.

5 is a flow chart for some embodiments of the present invention embodying the operation of step S130.

In step S131, a feedback object with a changed focus object is displayed as an object located immediately behind the displayed focus object based on a user input viewpoint among a plurality of overlapping objects. Feedback corresponding to the user input determined by the target server 100 may be transmitted to the HMD equipment 200 and displayed.

In this embodiment, in response to a user input, a feedback is displayed in which a focus object displayed at a user input time among a plurality of objects superimposed on the HMD is changed to a focus object. In this embodiment, one focus object movement is displayed.

In step S133, a feedback cycle in which a focus object is changed is displayed in order from a plurality of overlapping objects to a reference cycle from the object located at the front to the object at the back. Feedback corresponding to the user input determined by the target server 100 may be transmitted to the HMD equipment 200 and displayed.

In this embodiment, in response to a user input, a feedback in which the focus object is continuously changed from a frontmost object to a rearmost object among a plurality of objects superimposed on the HMD at a reference period is displayed.

In some embodiments, the focus object is displayed with feedback to select a specific object once or continuously among a plurality of overlapping objects on the HMD. Hereinafter, a specific example according to the flowchart shown in FIG. 5 will be described with reference to FIG. 6.

As described above with reference to FIG. 2, a problem of selecting a specific object from image information in which a plurality of objects overlap each other corresponds to a preliminary step required for precise robot motion control. Referring to FIG. 6, among the feedbacks in which the focus object described in FIG. 5 is changed, feedback in which the focus object is continuously changed is displayed. The focus object is changed at a reference cycle from the shirt 400 positioned at the front of the plurality of shirts superimposed, and after one cycle, the shirt 410 immediately after the shirt 400 positioned at the front is changed to the focus object, and a series of After passing through a process in which all the shirts overlapping through the cycle are changed to the focus object, the shirt 420 positioned at the rear is changed to the focus object.

In one embodiment, the reference period changed to the focus object may be shortened by the number of overlapping objects. When the reference period is constant irrespective of the number of overlapping objects, there is a limit in providing a convenient user interface. For example, when the target object desired by the user is close to the object located at the rearmost, a series of processes in which the focus object is changed from the frontmost object needs to be omitted quickly. By adjusting the reference period by placing a subtraction value on the number of overlapping objects, it is possible to reduce user inconvenience in the illustrated situation.

In another embodiment, when a target object to be changed into a focus object corresponds to a specific condition, feedback in which the focus object is changed to an object behind the target object may be displayed. For example, a condition that a red shirt among a plurality of overlapping shirts is not considered as a focus object may be set. When the target object to be changed corresponds to the red shirt in the process of changing the focus object in turn, the focus object is changed to the shirt behind the red shirt. In this way, a user-friendly interface can be provided by changing a focus object by giving a user a desired condition. For another example, in a robot for an industrial facility, a condition may be set that a fixed object that cannot be moved among a plurality of overlapping objects is not considered as a focus object. The robot can perform an intuitive and precise command by changing an object that can be operated using a robot for an industrial facility among a plurality of overlapping objects into a focus object. For another example, in a security robot, a condition may be set that a fixed and non-movable object among a plurality of overlapping objects is not considered as a focus object. By changing only the fluid of the plurality of overlapping objects to the focus object, it is possible to clarify the object of attention of the security robot. However, the present invention is not limited by the examples. In changing the focus object, all conditions for excluding the object to be changed may be included.

In some embodiments, by providing an interface for selecting a specific object among a plurality of objects superimposed on a display, a target object may be selected as a preliminary step of controlling the operation of the robot to cause the robot to perform sophisticated and complex commands. have. Furthermore, by providing conditions to the selection, a more user-friendly interface can be provided.

7 is a flowchart for an embodiment of the present invention embodying the operation of step S130.

In step S135, a feedback in which a focus object is changed is displayed from a camera mode displayed based on a user input point of view among a plurality of camera modes provided by a camera embedded in the robot and displayed on the HMD. Feedback corresponding to the user input determined by the target server 100 may be transmitted to the HMD equipment 200 and displayed.

In this embodiment, the plurality of camera modes provided may be a thermal imaging camera, a visible light camera, and an ultrasonic camera. However, it is not limited thereto. In determining the displayed priority of the camera image information provided by the robot, the camera mode may be changed based on information sensed by a sensor built into the HMD without movement of a user's hand. For example, in a security robot, information on an object behind an obstacle that cannot be confirmed by a camera in a visible light region may be converted into a thermal imaging camera by changing the camera mode to obtain information. By using such a control method as a pre-control method of the robot for controlling the future operation of the robot, an interface for creating a sophisticated working environment of the user is provided.

Hereinafter, a specific example according to the flowchart shown in FIG. 7 will be described with reference to FIG. 8.

The problem of selecting a specific camera mode from image information that can provide multiple camera modes corresponds to a preliminary step required for precise robot motion control. Referring to FIG. 8, among the feedbacks in which the focus object described in FIG. 7 is changed, feedbacks in which the camera mode is changed are displayed. The display 500 provided by the visible light camera, the display 510 provided by the thermal imaging camera, and the display 520 provided by the ultrasonic camera are changed according to the user input.

The display 500 provided by the visible light camera delivers image information such as an environment viewed by a real user. A remote robot can perform tasks such as visually observing the area.

The display 510 provided by the thermal imaging camera tracks heat and visually transmits information about it. The thermal imaging camera tracks and displays only the heat, so that the object can be identified not only in the absence of light, but also when an obstacle is placed in front of the object. You can visually check the sensed heat information by expressing the temperature in color.

The display 520 provided by the ultrasound camera tracks the location where the ultrasound is generated and visually transmits the information on it. The amount of ultrasonic waves generated is expressed in color according to decibels to provide information on the region where vibration occurs.

By using a plurality of camera modules embedded in the robot 300 at a remote location, a camera mode suitable for a user's work environment can be set. In selecting a camera mode, hand movement is not required, so that the robot can perform an intuitive command without having a complicated operation key.

Hereinafter, in FIG. 9, a flowchart in which a robot control method using HMD equipment according to an embodiment of the present invention is more specifically described will be described. The robot control method may be performed by the target server 100.

In step S201, when the second user input is generated after the first user input is generated, in step S203, feedback according to the second user input is displayed. In the present embodiment, while feedback according to the first user input is displayed on the HMD, it is determined that the user's gaze is watching the second position for a first reference time or longer, and the user's gaze is the first reference for the second position. When the user's second head motion is detected within the second reference time based on the time point over which the user watches the time, it may be determined that the second user input determined according to the object located in the second position and the second head motion is generated. have. Feedback corresponding to the second user input may be displayed on the HMD.

For example, the first user input may correspond to a position where a plurality of objects overlap on the HMD and a head motion upward. As feedback for this, the feedback in which the focus object is changed with a reference period from the frontmost object to the rearmost object among the overlapping multiple objects may be displayed. While the feedback is displayed, the second user input may correspond to a position deviating by a reference distance or more from a position where a plurality of objects overlap and a fixed head motion. As a feedback to this, the change of the focus object according to the first user input may be stopped. In the present embodiment, by stopping the continuous change of the focus object, the user can easily select a desired target object and perform convenient work. However, the present invention is not limited to the first position, the second position, the first head motion, and the second head motion set by the present example.

10 and 11, a flowchart for a robot control method using HMD equipment according to an embodiment of the present invention will be described in detail. In particular, FIGS. 10 and 11 correspond to an embodiment when the user input is changed.

In FIG. 10, the user input is an overlap position viewing time when the user is watching the overlap position where a plurality of objects overlap on the HMD for a first reference time or longer.

As described above, in step S301, the position of the pupil is detected as in S101, and then, similarly to step S303 in step S303, the gaze time for the overlapping position where a plurality of objects overlap is calculated from the position of the pupil. Similar to S105 in step S305, when the gaze time is equal to or greater than the first reference time, similar to S130 in step S307, feedback in which the focus object is changed to any one of the plurality of objects is displayed.

In another embodiment, the user input is a location viewing time of the first object when the user is watching the location of the first object on the HMD for a first reference time or longer. As feedback corresponding to this, a feedback in which an action option for controlling the movement of the robot with respect to the first object is displayed on the HMD is displayed. This can be confirmed in detail through an embodiment of S113 described above with reference to FIG. 3.

In FIG. 11, the user input is a pattern based on muscle movement around the user's eyes.

In step S401, movement of muscles around the eyes is detected. The motion information of the muscles around the eyes sensed by the HMD device 200 may be provided to the target server 100. The motion information of the muscles around the eye may be detected by a sensor that detects an IR signal. However, the present invention is not limited thereto, and various methods for detecting muscles around the eyes may be included.

In step S403, a pattern is generated by using the IR signal reflection intensity of the muscle corresponding to the feature point among the muscles around the eye according to the movement of the muscles around the eye of the user. When the same pattern as the pattern corresponding to the muscle movement information around the eye of the user set in advance in step S405 is input, it is determined in step S407 that there is user input. Similar to S130 in step S409, the feedback in which the focus object is changed to any one of the plurality of objects is displayed according to the determined user input.

In some embodiments, a method of controlling a robot from information that can be detected by a sensor embedded in the HMD device without using a user's hand has been identified. By controlling the user input, it is possible to obtain outputs related to a plurality of robot control methods. The user input may be a gaze time for a specific object, a gaze object and a user's head motion, or a movement pattern of muscles around the eyes. In one embodiment, the detection information regarding the user input may go through a pre-processing process to be used for deep learning. As an output related to the robot control method, the selected target object may be enlarged, the focus object may be changed, and the robot's action options for the selected target object may be displayed.

According to an embodiment of the present invention, it is possible to easily select a target object corresponding to a preliminary step for controlling a robot without hand intervention, and it is easy to select an appropriate camera mode of a task performed by the robot at a remote location. You can do it. It can provide an interface that creates a user's precise working environment, and can transmit more intuitive commands to the robot. Furthermore, by not requiring the use of a user's hand, it provides a means to control the interface even without a complicated operation key.

Hereinafter, an exemplary target server 100 capable of implementing the methods described in various embodiments of the present invention will be described with reference to FIG. 12.

12 is an exemplary hardware configuration diagram showing the target server 100.

As shown in FIG. 12, the target server 100 loads one or more processors 110, buses 150, communication interfaces 170, and computer programs 191 performed by the processors 110. The memory 130 may include a storage 190 storing the computer program 191. However, only components related to the embodiment of the present invention are illustrated in FIG. 12. Therefore, a person skilled in the art to which the present invention pertains may recognize that other general-purpose components may be further included in addition to the components illustrated in FIG. 12. The target server 100 illustrated in FIG. 12 may point to any one of physical servers belonging to a server farm that provides an infrastructure-as-a-service (IaaS) cloud service.

The processor 110 controls the overall operation of each component of the target server 100. The processor 110 may include at least one of a central processing unit (CPU), a micro processor unit (MPU), a micro controller unit (MCU), a graphics processing unit (GPU), or any type of processor well known in the art. It can be configured to include. Further, the processor 110 may perform operations on at least one application or program for executing a method/operation according to various embodiments of the present invention. The target server 100 may include one or more processors.

The memory 130 stores various data, commands and/or information. The memory 130 may load one or more programs 191 from the storage 190 to execute methods/operations according to various embodiments of the present invention. For example, when the computer program 191 is loaded into the memory 130, logic (or module) as illustrated in FIG. 3 may be implemented on the memory 130. An example of the memory 130 may be RAM, but is not limited thereto.

The bus 150 provides a communication function between components of the target server 100. The bus 150 may be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

The communication interface 170 supports wired and wireless Internet communication of the target server 100. The communication interface 170 may support various communication methods other than Internet communication. To this end, the communication interface 170 may include a communication module well known in the technical field of the present invention.

The storage 190 may store one or more computer programs 191 non-temporarily. The storage 190 may include a non-volatile memory such as a flash memory, a hard disk, a removable disk, or any type of computer-readable recording medium well known in the art.

The computer program 191 may include one or more instructions implemented with methods/operations according to various embodiments of the present invention. When the computer program 191 is loaded into the memory 130, the processor 110 may perform methods/operations according to various embodiments of the present invention by executing the one or more instructions.

So far, some embodiments of the present invention and effects are described with reference to FIGS. 1 to 12. The effects of the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

The concept of the present invention described so far with reference to FIGS. 1 to 12 may be embodied as computer readable code on a computer readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray Disc, USB storage device, removable hard disk), or a fixed recording medium (ROM, RAM, hard disk equipped with a computer). Can. The computer program recorded on the computer-readable recording medium may be transmitted to another computing device through a network such as the Internet and installed on the other computing device, and thus used on the other computing device.

In the above, even if all the components constituting the embodiment of the present invention are described as being combined or operated as one, the present invention is not necessarily limited to these embodiments. That is, within the object scope of the present invention, all of the components may be selectively combined and operated.

Although the operations are shown in a specific order in the drawings, it should not be understood that the operations must be performed in a specific order or in a sequential order, or all illustrated operations must be executed to obtain a desired result. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of various configurations in the above-described embodiments should not be understood as such a separation is not necessarily necessary, and the described program components and systems may generally be integrated together into a single software product or packaged into multiple software products. It should be understood that there is.

Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains may be implemented in other specific forms without changing the technical spirit or essential features of the present invention. Can understand. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. The scope of protection of the present invention should be interpreted by the claims below, and all technical spirits within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

Claims (14)

In a method performed on a computing device,
Determining whether the user is watching the overlapping position of the plurality of objects displayed on the HMD for a first reference time or more; And
When it is determined that the overlapping position is being watched for at least the first reference time, the focusing object includes providing changed feedback to any one of the plurality of objects.
Robot control method using HMD equipment. According to claim 1,
The step of providing the feedback,
The step of providing a feedback in which the focus object is changed, in turn, with a reference period from the object located at the front to the object at the back of the plurality of overlapped objects displayed on the HMD.
Robot control method using HMD equipment. According to claim 2,
The reference period is shortened when the number of the plurality of objects displayed on the HMD increases.
Robot control method using HMD equipment. According to claim 2,
When the target object to be changed to the focus object corresponds to the first condition, further comprising the step of providing feedback that the focus object is changed to an object behind the target object,
Robot control method using HMD equipment. According to claim 1,
The step of providing the feedback,
And providing a feedback, in which the focus object is changed, to an object located behind a focus object displayed based on a first user input time point among the plurality of overlapped objects displayed on the HMD.
Robot control method using HMD equipment. According to claim 1,
The step of providing the feedback,
Providing feedback from which the focus object is changed, from a camera mode displayed based on a first user input point of view among a plurality of camera modes provided by a camera embedded in a robot and displayed on the HMD, to another camera mode. doing,
Robot control method using HMD equipment. In a method performed on a computing device,
Obtaining head motion information of the user wearing the HMD;
Determining whether the user is watching the first location for a first reference time or longer;
It is determined that the user is watching the first position for more than the first reference time, and the HMD is worn within a second reference time based on a time when the user watches the first position for more than the first reference time. If a first head motion of a user is detected, determining that an object located at the first location and a first user input determined according to the first head motion are generated; And
And providing feedback according to the first user input to the HMD.
Robot control method using HMD equipment. The method of claim 7,
Providing feedback according to the first user input to the HMD,
The object located at the first position displayed on the HMD comprises providing enlarged feedback,
Robot control method using HMD equipment. The method of claim 7,
Providing feedback according to the first user input to the HMD,
Providing a feedback in which the focus object is changed to any one of the plurality of objects,
Robot control method using HMD equipment. The method of claim 7,
While feedback according to the first user input is displayed on the HMD, it is determined that the user is watching the second location for at least the first reference time, and the user is viewing the second location for the first reference time or more. When the second head motion of the user wearing the HMD is detected within the second reference time based on the point of view, the second user input determined according to the object located in the second position and the second head motion Determining that it has occurred; And
And providing feedback to the HMD according to the second user input.
Robot control method using HMD equipment. The method of claim 7,
Providing feedback according to the first user input to the HMD,
Providing a feedback displayed on the HMD is an action option for controlling the movement of the robot with respect to the object located in the first position,
Robot control method using HMD equipment. In a method performed on a computing device,
Obtaining motion information of muscles around the user's eyes;
Determining that a first user input corresponding to a movement pattern based on the movement of the muscle around the eye of the user is generated; And
And providing feedback in which the focus object is changed to any one of a plurality of objects according to the first user input.
Robot control method using HMD equipment. In a method performed on a computing device,
Determining whether the user is watching the location of the first object displayed on the HMD for a first reference time or longer; And
When it is determined that the user is watching the location of the first object for more than the first reference time, providing a feedback in which an action option for controlling the movement of the robot with respect to the first object is displayed on the HMD. Containing,
Robot control method using HMD equipment. Processor;
Network interface;
Memory; And
A computer program loaded into the memory and executed by the processor,
The computer program,
An instruction for obtaining head movement information of the user wearing the HMD;
An instruction for determining whether the user is watching the first position for a first reference time or longer using a sensor embedded in the HMD;
It is determined that the user is watching the first position for more than the first reference time, and the HMD is worn within a second reference time based on a time when the user watches the first position for more than the first reference time. An instruction for determining that an object located at the first location and a first user input determined according to the first head motion are generated when a first head motion of a user is detected; And
And providing feedback according to the first user input to the HMD.
Robot control device using HMD equipment.

Images