KR102549631B1 - Method for real-time control of working device and camera by using pose of body part and tilt of device and apparatus thereof - Google Patents

Abstract

The present invention is for controlling a work device and a photographing angle in real time using a posture of a body part and a tilt of a camera that captures the body part, and an operation method of a second device for remotely controlling a first device includes: Receiving video data of a first device captured using a first camera located at , displaying a real-time video of the first device based on the video data, based on a movement of a designated body part of a user generating at least one device control command for controlling the first device, generating at least one camera control command for controlling the first camera based on a tilt change of a second device, and the and transmitting at least one device control command and the at least one camera control command to control the first device and the first camera.

Description

Real-time working device and camera control method and device using posture of body part and inclination of device

The present invention relates to control of a working device and a camera that captures the working device, and more particularly, to a method and apparatus for controlling a working device and a camera in real time using a posture of a body part and an inclination of the device.

It was a traditional method in the past to perform various physical tasks required in industrial sites by human labor. However, depending on the various conditions of the field and the contents or characteristics of the work, there may be an environment in which manpower is difficult to directly input. In this case, more effective work may be performed by using a mechanical device such as a robot. Even if a mechanical device is used, its effectiveness will be somewhat low if a human has to directly control it while being in close proximity to the device. Therefore, even if there is a difference in the degree of distance, remotely controlling the mechanical device can provide greater usability.

The present invention provides a method and apparatus for more effectively controlling a device.

The present invention provides a method and apparatus for conveniently controlling a device remotely.

The present invention provides a method and apparatus for controlling a device in real time using a camera.

The present invention provides a method and apparatus for controlling a device based on a recognition result of a body part photographed using a camera.

The present invention provides a method and apparatus for controlling a posture of a camera for photographing a body part based on a posture of a camera for photographing a body part.

The present invention provides a method and apparatus for synchronizing movement of a camera photographing a device with a camera photographing a body part.

According to an embodiment of the present invention, a method for operating a second device for remotely controlling a first device includes receiving video data of a first device captured using a first camera located at a work site, the video Displaying a real-time video of the first device based on data, generating at least one device control command for controlling the first device based on a movement of a user's designated body part, Generating at least one camera control command for controlling the first camera based on a tilt change, and transmitting the at least one device control command and the at least one camera control command to the first device and the first camera. It may include transmitting to control.

According to an embodiment of the present invention, the generating of the at least one camera control command may include converting the tilt change into an angular change amount in at least one rotation axis supported by the first camera; It may include generating at least one instruction that instructs.

According to an embodiment of the present invention, the method includes receiving information about the first device and the first camera from a server, and displaying a list of controlled devices including the first device. Information on one camera may include information on the number and direction of rotational axes supported by the first camera, and a movable range.

According to an embodiment of the present invention, in addition to the real-time video, a graphic representing the posture of a part of the user's body, a first joystick for controlling the first device, and a first control for controlling the shooting direction of the first camera The method may further include displaying at least one of the two joysticks.

According to an embodiment of the present invention, an application program stored in a medium may execute the above method when operated by a processor.

The present invention makes it possible to more effectively remotely control a device while observing the state of the device located at a distance.

1 shows a system according to an embodiment of the present invention.
2 shows the structure of a control device according to an embodiment of the present invention.
3 shows the structure of a work device according to an embodiment of the present invention.
4 shows the structure of a photographing device according to an embodiment of the present invention.
5 shows the structure of a server according to an embodiment of the present invention.
6 illustrates a concept of a remote control service according to an embodiment of the present invention.
7 illustrates an operating procedure for controlling a work device and a camera according to an embodiment of the present invention.
8 illustrates an example of a structure of a camera for photographing a work device according to an embodiment of the present invention.
9 illustrates an operating procedure for generating a signal for controlling a camera that photographs a work device according to an embodiment of the present invention.
10 illustrates an example of controlling a remote camera according to an inclination of a user device according to an embodiment of the present invention.
11 illustrates an operating procedure for generating a signal for controlling a work device according to an embodiment of the present invention.
12 shows a driving model of a work device according to an embodiment of the present invention.
13 illustrates a signal exchange procedure for real-time control of a work device according to an embodiment of the present invention.
14 illustrates an example of an interface for controlling a work device according to an embodiment of the present invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. And, in describing the present invention, if it is determined that a detailed description of a related known function or configuration may unnecessarily obscure the gist of the present invention, the detailed description will be omitted.

Hereinafter, a technology for remotely controlling a working device and a camera that photographs the working device in real time will be described. Specifically, the present invention generates a command for controlling a work device using a part of a user's body photographed by a camera included in or interlocked with a user device, and provides the user with real-time movement of the work device controlled according to the command. Various embodiments for sharing will be described. In addition, the present invention describes various embodiments for controlling a camera that takes pictures of a work device using a pose of the user device.

1 shows a system according to an embodiment of the present invention.

Referring to FIG. 1 , the system includes a user device 110, a controlled device module 120, and a server 130. The user device 110, a controlled device module 120, and a server 130 ) can transmit and receive mutual signals through a communication network.

The user device 110 is a device used at a site where a user is located, and generates commands for controlling a working device 120a and a photographing device 120b. In addition, the user device 110 receives an image of the movement or state of the work device 120a and outputs the image so that the user can view it. To this end, the user device 110 may access the server 130 and obtain information about accessible controlled devices from the server 130 . The user device 110 includes a display for displaying an image of the work device 120a. For example, the user device 110 may be a mobile device such as a smart phone or tablet. As another example, the user device 110 may be a virtual reality (VR) device that can be worn on a user's face. In some cases, the user device 110 may include a camera. For convenience of description below, a camera included in the user device 110 may be referred to as a 'user camera'.

The controlled device module 120 is equipment used in the field where the work device 120a is located, and the work device 120a controlled by commands generated by the user device 110 and the work device 120a are photographed. A photographing device 120b is included. The work device 120a and the photographing device 120b operate according to commands received from the user device 110 . The work device 120a is a device capable of causing physical movement, such as a robot, heavy equipment, or other machine. The photographing device 120b photographs the work device 120a. According to one embodiment, the photographing device 120b may be attached to a part of the working device 120a. Here, the photographing device 120b may include one camera or a plurality of cameras. Receiving a command from the user device 110 and transmitting an image to the user device 110 are performed by at least one of the work device 120a and the photographing device 120b, or by a separate third relay device. can be performed For convenience of description below, a camera included in the photographing device 120b may be referred to as a 'remote camera'.

The server 130 provides a management function for providing services according to the proposed technology. The server 130 performs functions for supporting interconnection between the user device 110 and the work device 120a. For example, the server 130 processes registration of the work device 120a or the controlled device module 120 and manages information on a plurality of work devices including the work device 120a. In addition, the server 130 may provide information on the work device 120a or the controlled device module 120 according to a request of the user device 110 .

2 shows a structure of a user device 110 according to an embodiment of the present invention.

Referring to FIG. 2 , the user device 110 includes an input/output unit 210, a communication unit 220, a sensor unit 230, a storage unit 240, and a control unit 250.

The input/output unit 210 includes means for displaying a screen and means for detecting input from the outside. Here, the means for displaying the screen may be implemented as a display such as a liquid crystal screen. Means for detecting an input include a touch pad, a hardware button, and the like. The input/output unit 210 may include a touch screen in which a liquid crystal screen and a touch pad are combined. In some cases, the input/output unit 210 may be interpreted as a set of two or more physically separated hardware modules.

The communication unit 220 connects to an external communication network and performs functions for transmitting and receiving data/signal/information. For example, the communication unit 220 may perform a function of accessing an Internet network, a wireless communication network, or a mobile communication network. To this end, the communication unit 220 performs functions such as modulation/demodulation, amplification, and filtering of signals. The communication unit 220 may support communication technologies such as wireless LAN communication (eg, Wi-Fi) and cellular communication (eg, LTE, 5G, 6G).

The sensor unit 230 detects external physical phenomena. The sensor unit 230 may include at least one of an acceleration sensor, a gyro sensor, a tilt sensor, an optical sensor, a geomagnetic sensor, a gravity sensor (G-sensor), a temperature sensor, a biosensor, and a position sensor. there is. Here, the gyro sensor may measure the angular velocity of at least one of three axes defined in space, and the acceleration sensor may measure acceleration acting on at least one of the three axes. The sensor unit 230 may sense motion, light, tilt, orientation, and the like, and generate an electrical signal representing the detection result.

The storage unit 240 stores data, programs, microcodes, applications, and the like necessary for the operation of the user device 110 . The storage unit 240 may be implemented as a temporary or non-transitory storage medium. Also, the storage unit 240 may be fixed to the device or implemented in a detachable form.

The controller 250 controls overall operations of the user device 110 . The controller 250 may include at least one of a processor and a microprocessor. The control unit 250 may visually express information to the user through the input/output unit 210 or detect the user's input. The controller 250 may execute programs stored in the storage 240 and transmit or receive data with an external device through the communication unit 220 . In addition, the controller 250 controls the camera unit to obtain a still image or video, controls the audio unit 230 to output audio or detect sound, and obtains sensing data through the sensor unit 230. can do. In particular, the controller 250 controls the user device 110 to operate according to various embodiments described below.

Although not shown in FIG. 2 , the user device 110 may further include a camera unit. In this case, the camera unit may perform functions such as photographing and video recording. The camera unit may include an image sensor and a lens. As the image sensor, at least one of a charged coupled device (CCD) and a complementary metal-oxide semiconductor (CMOS) may be used. For example, the camera unit recognizes light input through a lens with the image sensor and converts the image recognized by the image sensor into digital data.

3 shows the structure of a work device 120a according to an embodiment of the present invention.

Referring to FIG. 3 , the work device 120a includes a communication unit 310, a storage unit 320, a driving unit 330, a structure unit 340, and a control unit 350.

The communication unit 310 performs a function for communicating with an external device. For example, the communication unit 310 may support wireless/wired communication. The storage unit 320 stores data, programs, microcodes, and applications necessary for the operation of the work device 120a. The storage unit 320 may be implemented as a temporary or non-transitory storage medium. Also, the storage unit 320 may be fixed to the device or implemented in a detachable form.

The driving unit 330 generates power for movement of the work device 120a. That is, the driving unit 330 generates an kinetic force corresponding to the electrical signal. For example, the driving unit 330 may include a motor providing rotational motion, a servomotor and a guide for linear motion, and the like.

The structure part 340 includes structures forming an exterior for applying the physical force of the work device 120a to the outside. For example, the structure 340 includes joints, skeletons, housings, and the like. The structure unit 340 includes a plurality of components and forms the movement of the working device 120a according to the kinetic force generated by the driving unit 330 .

The controller 350 controls the overall operation of the work device 120a. The controller 350 may include at least one of a processor and a microprocessor. The controller 350 may execute programs stored in the storage 320 and transmit or receive data with an external device through the communication unit 310 . Also, the control unit 350 may generate and supply a signal for operating the driving unit 330 . In particular, the controller 350 controls the work device 120a to operate according to various embodiments described below.

4 illustrates a structure of a photographing device 120b according to an embodiment of the present invention.

Referring to FIG. 4 , the photographing device 120b includes a communication unit 410, a storage unit 420, a camera unit 430, and a controller 440.

The communication unit 410 performs a function for communicating with an external device. For example, the communication unit 410 may support wireless/wired communication. The storage unit 420 stores data, programs, microcodes, applications, and the like necessary for the operation of the photographing device 120b. The storage unit 420 may be implemented as a temporary or non-transitory storage medium. Also, the storage unit 420 may be fixed to the device or implemented in a detachable form.

The camera unit 430 may perform functions such as photographing and video recording. The camera unit 430 may include an image sensor and a lens. As the image sensor, at least one of CCD and CMOS may be used. For example, the camera unit 430 recognizes light input through a lens with the image sensor, and converts the image recognized by the image sensor into digital data.

The controller 440 controls overall operations of the photographing device 120b. The controller 440 may include at least one of a processor and a microprocessor. The control unit 440 may execute programs stored in the storage unit 420 and transmit or receive data with an external device through the communication unit 410 . In addition, the controller 440 may obtain an image by controlling the camera unit 430 and generate streaming data including the image. In particular, the controller 440 controls the photographing device 120b to operate according to various embodiments described below.

5 shows the structure of the server 130 according to an embodiment of the present invention.

Referring to FIG. 5 , the server 130 includes a communication unit 510, a storage unit 520, and a control unit 530.

The communication unit 510 performs a function for communicating with an external device. For example, the communication unit 510 may support wired communication. The storage unit 520 stores data, programs, microcodes, applications, and the like necessary for the operation of the server 130 . The storage unit 520 may be implemented as a temporary or non-transitory storage medium. Also, the storage unit 520 may be fixed to the device or implemented in a detachable form.

The controller 530 controls overall operations of the server 130 . The controller 530 may include at least one of a processor and a microprocessor. The controller 530 may execute programs stored in the storage 520 and transmit or receive data with an external device through the communication unit 510 . In particular, the controller 530 controls the server 130 to operate according to various embodiments described later.

6 illustrates a concept of a remote control service according to an embodiment of the present invention.

Referring to FIG. 6 , the remote control service includes operations performed at a work site 610 and operations performed at a user site 620 . At the work site 610, a photographing operation 612 of the work device using a remote camera is performed. An image obtained by the photographing operation 612 is provided to the user site 620 through real-time streaming. At the user site 620, an output operation 622 of the streamed video is performed. Accordingly, the user recognizes the situation through the streaming screen. At the same time, an operation 624 for recognizing the posture of the user's hand and the inclination of the user device is performed based on the image of the user's hand. Subsequently, a control signal calculation operation 626 of converting the coordinate values of the hand and/or arm obtained through the recognition operation 624 and the information on the tilt of the device into a control signal for controlling the working device and the remote camera is performed. do. The control signal is transmitted to the work site 610 again, and a control operation 614 for the work device and the remote camera is performed based on the control signal.

According to the service shown in FIG. 6 , the user can remotely control the work device in real time. In this case, the user may move a part of his/her body (eg, a hand) and view a real-time streaming image of the working device moving in response thereto. 6 illustrates the hand as a body part, the work device may be controlled using other body parts on the hand. In addition, a remote camera taking pictures of the work device may be controlled according to the posture of the user's camera. In this way, if the work device and the remote camera are controlled in response to the movement of a part of the user's body and the movement of the user device, the user can obtain a user experience (UX) as if the work device is a part of the user's body. will be. That is, the service according to various embodiments of the present disclosure provides a very intuitive work device and camera control method to the user.

7 illustrates an operating procedure for controlling a work device and a camera according to an embodiment of the present invention. 7 shows operations performed by user device 110 .

Referring to FIG. 7 , in step 701, the user device 110 displays a real-time video of the work device 120a. A real-time video is generated by a photographing device 120b disposed near the work device 120a. The user device 110 receives video data generated by a remote camera that captures the work device 120a, restores video information by decoding the video data, and outputs a real-time video using the provided display. At this time, according to various embodiments, in addition to the real-time video of the work device 120a, additional information or interface required for controlling at least one of the work device 120a and the photographing device 120b may be displayed together. .

In step 702, the user device 110 remotely controls the work device 120a based on the movement of a part of the user's body. Specifically, the user's camera recognizes the movement of at least a part of the user's body and generates at least one command for controlling the work device 120a. In addition, the user device may remotely control the work device 120a by transmitting at least one command to the work device 120a. Accordingly, although not shown in FIG. 7 , the user device may display the appearance of the work device 120a controlled according to at least one command as a real-time video again.

In step 703, the user device 110 remotely controls a remote camera that takes pictures of the work device 120a based on the tilt of the user device 110. Specifically, the user device 110 generates at least one command for controlling a photographing angle of a remote camera according to the tilt of the user device 110, and sends the generated at least one command to the photographing device 120b. can be sent For this reason, as the tilt of the user device 110 changes, the shooting angle of the remote camera changes, and the screen displayed through streaming may change.

In the embodiment described with reference to FIG. 7 , movement of a body part is recognized. Movement of a body part can be recognized by various technologies. According to an embodiment, movement of a body part may be recognized using an image obtained by capturing the body part. In this case, at least one command for controlling the work device 120a is a pose of a subject (eg, hand) detected in a photographed image, a distance between the subject and a camera lens for photographing, and an angle at which the subject is viewed. , It may be generated based on at least one of the location of the subject (eg, 3-dimensional coordinates). According to another embodiment, movement of a body part may be recognized using a wearable device worn on the body part. For example, a glove-type controller equipped with sensors, processors, and communicators may be used to recognize hand postures and movements.

8 illustrates an example of a structure of a camera for photographing a work device according to an embodiment of the present invention. 8 illustrates a structure of a remote camera included in the photographing device 120b. The external appearance of the remote camera illustrated in FIG. 8 is only an example, and the present invention is not limited to the external appearance illustrated in FIG. 8 .

Referring to FIG. 8 , the remote camera includes a lens unit 802, an elevation driving unit 812, an azimuth driving unit 814, and a base 816.

The lens unit 802 receives light and generates an image signal. For example, the lens unit 802 may include at least one lens that collects light and at least one sensor that generates a signal corresponding to the light. In addition, the lens unit 802 may further include a physical driving means for controlling the angle of view.

The elevation angle driver 812 changes the direction of the lens unit 802 to a vertical direction. The elevation driving unit 812 is fixed to the azimuth driving unit 814 and can rotate around an axis A-A parallel to the X axis. Accordingly, the elevation angle driving unit 812 may adjust the elevation angle in the direction the lens unit 802 faces, that is, in the photographing direction of the remote camera.

The azimuth driving unit 814 changes the direction of the lens unit 802 to a horizontal direction. The azimuth drive unit 814 is fixed to the base 816 and can rotate about an axis B-B parallel to the Z axis. Accordingly, the elevation angle driving unit 812 may adjust the azimuth angle of the direction in which the lens unit 802 faces, that is, the shooting direction of the remote camera.

Base 816 is a component that supports other components. The base 816 may function as a connection member for installing and fixing the remote camera to a designated location. For example, the base 816 may be installed on a specific component of the work device 120a, or may be fixed at a position spaced apart from the work device 120a, suitable for photographing the work device 120a. there is.

As in the example of FIG. 8 , the photographing direction of the remote camera may be specified as a combination of an elevation angle and an azimuth angle. That is, the photographing direction of the remote camera according to an embodiment may be determined by rotation of two axes. In this case, the photographing direction of the remote camera can be adjusted by controlling elevation and azimuth. According to another embodiment, the remote camera may have a different structure, and depending on the structure, the shooting direction of the remote camera may be determined by rotation of three or more axes. That is, according to various embodiments, a photographing direction of the remote camera may be determined by rotation of at least one axis. The angle of at least one axis that determines the photographing direction of the remote camera may be controlled by commands generated by the user device 110, and an example of the procedure is shown in FIG. 9 below.

9 illustrates an operating procedure for generating a signal for controlling a camera taking pictures of work equipment according to an embodiment of the present invention. 9 shows operations performed by user device 110 .

Referring to Figure 9, in step 901. The user device 110 measures the inclination of the user device 110 . According to an embodiment, the inclination may be measured using at least one of a gyro sensor, an acceleration sensor, and a tilt sensor. According to an embodiment, when the user device 110 includes a camera, the tilt of the user device 110 may be understood as the tilt of the user's camera.

In step 902, the user device 110 generates a control command for the remote camera based on the tilt. The controlled remote camera is taking pictures of the working device 120a. The user device 110 checks how much angular change in which axis the inclination measured in step 901 includes, and calculates the amount of angular change in at least one axis of the remote camera based on the confirmed axis and angle change. Decide. In other words, the user device 110 may convert the inclination into an angular change amount in at least one rotation axis (eg, X axis, Z axis) supported by the remote camera. Then, the user device 110 generates at least one command indicating the determined angular change amount. Here, the at least one command may indicate at least one angular value relative to the synchronized reference direction or an angular change amount relative to an angle transmitted immediately before. At this time, according to various embodiments, the generation of the control command may be generated at regular time intervals or when a tilt change of the user device 110 occurs and the change is stopped for a predetermined threshold time or longer.

In step 903, the user device 110 transmits a control command to the photographing device 120b. When the photographing device 120b and the working device 120a are implemented as one device, it may be understood that the control command is transmitted to the working device 120a. The user device 110 may generate a control signal including a control command according to a set communication protocol and transmit the generated control signal through a communication network.

10 illustrates an example of controlling a remote camera according to an inclination of a user device according to an embodiment of the present invention. 10 illustrates a case where the user device is a VR device, a remote camera can be similarly controlled even when the user device is a device of another type (eg, a smartphone).

Referring to FIG. 10 , the inclination of the user device 120a may be interpreted as rotation in any one of four rotational directions 1001 to 1004 or a combination of two or more. According to an embodiment, when the user device 120a is tilted, the tilt may be expressed as a rotation angle combination of four rotation directions 1001 to 1004 . For example, if the user turns his/her head 15° to the left while wearing the user chair 120a, since this movement is a 15° rotation in the second direction 1002, {0,15°,0,0} can be expressed as As another example, if the user turns his/her head in the lower left oblique direction corresponding to 10° left and 10° downward while wearing the user device 120a, this movement is a 10° rotation in the second direction 1002. , since it is a rotation of 10° in the third direction 1003, it can be expressed as {0,10°,10°,0}. Correspondingly, at least one of the elevation driving unit 812 and the azimuth driving unit 814 of the remote camera may rotate. According to an embodiment, when rotation in the first direction 1001 or the third direction 1003 is recognized, the elevation driving unit 812 will rotate about the axis A-A. For example, when a rotation of 10° is detected in the first direction 1001, the elevation driving unit 812 may rotate +10° about the axis A-A. According to an embodiment, when rotation in the second direction 1002 or the fourth direction 1004 is recognized, the azimuth driving unit 814 will rotate about the axis B-B. For example, when a 10° rotation is detected in the fourth direction 1004 , the azimuth driving unit 814 may rotate -10° about the axis B-B.

As in the above-described embodiment, an inclination of the user device 110 may be converted into a rotation angle of at least one axis of the remote camera. In this case, if axes for recognizing the tilt in the user device 110 and rotation axes supported by the remote camera are the same, a rotation angle applied to the remote camera may be directly obtained from the angle of the recognized tilt. However, if the axes recognizing the tilt of the user device 110 and the rotation axes supported by the remote camera are different from each other, an algorithm for converting the rotation angle in consideration of the difference in rotation axes may be required. Accordingly, the user device 110 may have a plurality of transformation algorithms, select and apply one suitable transformation algorithm according to the structure of rotation axes supported by the remote camera. In this case, information on the structure of rotational axes supported by the camera may be provided from the server 130 in the process of searching for a work device.

As described with reference to FIG. 9 , a photographing direction of a remote camera may be controlled according to an inclination of the user device 110 . For example, when the user camera is a component of the user device 110, the tilt of the user device 110 means the tilt of the user camera. In this case, the photographing direction of the image photographed by the user camera will also change due to the change in the tilt of the user device 110 . Accordingly, according to another embodiment, the slope may be measured based on an image captured by a user's camera rather than a sensor. In this case, in step 901, the user device 110 may recognize a change in the shooting direction of the user camera by analyzing the image captured by the user camera, and measure the tilt of the user device 110 based on this.

Also, when the user camera is a component of the user device 110, the tilt of the user device 110 means the tilt of the user camera. Accordingly, as a result, a change in tilt of the user's camera causes a change in the photographing direction of the remote camera. At this time, if the angle change of a specific axis converted from the tilt change of the user camera is converted to the angle change of the specific axis of the remote camera at a 1:1 ratio, the change of the user camera's shooting angle and the remote camera's shooting angle change are understood to be in a synchronized state. It can be. In this case, the user will experience a user experience as if directly controlling the remote camera by tilting the user device 110, and this will give the user a sense of realism as if the user is directly photographing the work device with a camera in the user's field.

According to an embodiment, a user camera may not be a component of the user device 110 . In this case, the operation of measuring the tilt of the user device 110 in step 901 may be replaced with an operation of measuring the tilt of another device including the user's camera. However, according to another embodiment, even if the user camera is not a component of the user device 110, the shooting direction of the second camera may still be controlled based on the tilt of the user device 110.

11 illustrates an operating procedure for generating a signal for controlling a work device according to an embodiment of the present invention. 11 shows operations performed by user device 110 . 11 illustrates a case of generating a control message using a hand, the present invention can be extended to a case of using other body parts (eg, arms, legs, feet, head, face, etc.).

Referring to FIG. 11 , in step 1101, the user device 110 recognizes the posture of the hand. Here, the posture of the hand is meant to include positions of joints included in the hand, positions and angles of each part of the hand, angles formed by two parts connected to the joint, and the like. According to various embodiments, the user device 110 may determine the posture of the hand based on sensing data acquired using at least one sensor or based on an image of the hand.

In step 1102, the user device 110 generates a control command for the work device 120a based on the posture of the hand. The controlled working device 120a has a shape imitating a human hand or arm, each of the joints of the working device 120a corresponds to at least one joint of the hand, and the angle of the corresponding joint of the hand corresponds to the working device 120a. It is converted into the angle of the joint of (120a). According to various embodiments, the user device 110 may generate a command to control the work device 120a periodically according to a predetermined time interval or using the recognized hand postures at a timing according to a set condition. Here, the command may indicate absolute values of angles of the currently detected joint or change values compared to angles transmitted immediately before.

In step 1103, the user device 110 transmits a control command to the work device 120a. The user device 110 transmits a control command to the work device 120a. The user device 110 may generate a control signal including a control command according to a set communication protocol and transmit the generated control signal through a communication network.

In the embodiment described with reference to FIG. 11 , a command for controlling the operation of the work device is generated based on the posture of the hand. To this end, awareness of the hand posture is required. According to various embodiments, the posture of the hand may be recognized using a hand-worn controller equipped with at least one sensor or may be recognized using an image of the hand.

According to an embodiment, the user may move his or her hand to control the work device 120a while wearing a glove-type wearable device capable of tracking the posture of the hand. The user device 110 may establish a communication link with the wearable device and recognize the posture of the user's hand based on sensing data received through the communication link and measured by the wearable device.

According to another embodiment, the user may move his hand to control the work device 120a while photographing his hand using a camera provided in the user device 110 . Accordingly, the user device 110 acquires images of the user's hand. The camera captures the user's hand, and images are generated to check the shape of the hand. The user device 110 may recognize the posture of the hand using an image analysis algorithm or an artificial intelligence algorithm. Furthermore, since the images constitute a moving picture, the user device 110 can determine not only the posture of the hand in each image, but also a change in the posture of the hand. For example, the user device 110 moves the work device 120a using postures of the hands determined based on images periodically extracted from consecutive images at predetermined time intervals or images selected according to set conditions. You can create commands to control. In this case, since the control command for the work device is determined based on the image of the hand, it is possible to generate a control command based on the user's body without additional hardware (e.g., wearable device) tracking the posture of the hand. .

In the case of using an image captured using a camera, the information obtained through the result of recognizing the posture of the hand in the 2D image is obtained in the form of 2D coordinates. Accordingly, the user device 110 calculates 3D coordinates from 2D coordinates in consideration of the angle of view of a camera used to photograph the work device. In general, a time of flight (ToF) sensor is required to sense a two-dimensional depth using one sensor. However, the proposed technology of the present invention enables 3D coordinate recognition using only a general-purpose camera sensor without a ToF sensor.

As described above, sensing data or images may be used to recognize the posture of the hand. In this case, sensing data and images are continuously generated, but all or some of the frames of the sensing data or images may be used to generate a control command. Hereinafter, frames of sensing data and images are referred to as 'samples'.

According to an embodiment, samples may be periodically extracted according to a predetermined time interval. In this case, the user device 110 extracts a sample every time a predetermined time elapses, and detects the posture of the hand based on the extracted sample. At this time, the time interval for sample extraction may be set to an appropriate value through experiments. Alternatively, the time interval may be adaptively adjusted based on various environmental factors, such as the type of work device, the purpose of use of the work device, and delay of a communication network.

According to another embodiment, some samples may be extracted according to detection of a designated event. For example, an event may be defined based on a degree of hand movement. Specifically, the event may be defined as a movement of the hand stopping for a certain period of time or more or an amount of movement less than a threshold value. That is, since the user intends to move the hand in a specific posture of the work device, the user device 110 may command and deliver only the final posture of the hand. To this end, the user device 110 may recognize a situation in which the movement of the hand temporarily stops or slows down, and may generate a new command in the corresponding situation.

In the procedure described above, the posture of the hand may be determined using an artificial intelligence algorithm. The artificial intelligence algorithm may be a convolutional neural network (CNN)-based image processing artificial intelligence model, and may be used to determine a hand posture in a pre-trained state. More specifically, the artificial intelligence model may include a two-step neural network pipeline with single-shot detector using a single-shot detector, and a regression model operating in a cropped region from an image. running　on　the　cropped　region) can be used. In addition to this, various algorithms may be used to detect the hand posture in the present invention.

The relationship between the posture of the hand detected by the artificial intelligence algorithm or other algorithm and the command for controlling the work device can be described as shown in FIG. 12 below. 12 shows a driving model of a work device according to an embodiment of the present invention. 12 illustrates a user's hand 1210 and work device 1220 . Referring to FIG. 12 , 21 points are detected on the hand 1210 . The detection points correspond to the tips of fingers, knuckles, and the like, and have unique identifiers. Here, the number and positions of detected points may vary depending on an algorithm used to detect the hand posture. At least some of the detected points are mapped to specific points of the work device 1220 . For example, three joints 1211 to 1213 of the index finger may be mapped to three joints 1221 to 1223 of an arm of the work device 1220 (eg, a forklift). As shown in FIG. 12 , when points are detected on the hand 1210, an angle formed by lines connected to adjacent points at each point may be calculated, and the angle is an angle applied to a corresponding joint of the work device 1220. do.

In the above example, the rest of the joints other than the three joints 1211 to 1213 of the index finger are not mapped to any joint of the working device 1220 . The remaining joints correspond to non-mapped joints. However, if the mapping with the joints of the work device 1220 is changed, non-mapped joints may be changed. For example, in the case of FIG. 12 , each of the three joints 1221 to 1223 of the work device 1220 may be mapped to joints of different fingers. Such mapping may be predefined or determined through a separate mapping setting procedure. According to an embodiment, during a mapping setting procedure, the user device 110 repeatedly requests the user to move a body part to perform a specific motion related to a specific joint or a combination of joints of the work device 120b, and is confirmed. After selecting the optimal motion among the motions, the joints used in the optimal motion may be mapped to the joints of the working device 120b. If a separate mapping setting procedure is performed, mapping optimized for the user's personal characteristics may be provided.

When the work device 1220 is not a heavy machine as shown in FIG. 12 but a robot having a shape imitating a human body, mapping can be performed more intuitively. For example, when the work device 1220 is a robot hand, most of the joints of the hand 1210 may be directly mapped to the joints of the robot hand. However, the present invention can be applied not only to heavy equipment, but also to work devices having the shape of tools of various structures (eg, tongs, scissors, cutters, etc.). In this case, an operation of appropriately mapping between the joints of the work device and detection points (eg, joints and extremities) of the body is required. For example, only some of the detection points on the body may be used for control of the work device. Specifically, in the case of a tongs-shaped or cutter (eg, scissors)-shaped work device having only two fingers, only detection points on the user's index finger and thumb may be used. As another example, at least one of the detection points of the body may be mapped to a plurality of joints of the work device. In this case, an operation of designating one of the plurality of joints of the working device as a control target according to a user's selection may be performed to individually control the plurality of joints of the working device mapped to one detection point.

13 illustrates a signal exchange procedure for real-time control of a work device according to an embodiment of the present invention. 13 shows an interaction between the user device 110, the server 130, and the controlled device module 120.

Referring to FIG. 13 , in step 1301, the controlled device module 120 is powered on. Here, the controlled device module 120 is an object including the work device 120a and the photographing device 120b, and the controlled device module 120 in FIG. 13 is the work device 120a or the photographing device 120b. , or may be a third device linked to the working device 120a and the photographing device 120b. That is, in step 1301, at least one of the work device 120a and the photographing device 120b is turned on. Accordingly, the operating system of the controlled device module 120 is booted and becomes an operable state.

In step 1302, the controlled device module 120 accesses the server 130 and performs a registration procedure. The controlled device module 120 accesses the server 130 using the designated address information and then registers the work device 120a with the server 130 . Communication between the controlled device module 120 and the server 130 may be based on socket communication. Here, registration is a procedure of storing information on the work device 120a in the server 130 and notifying that the remote control of the work device 120a will be allowed according to the request of the user device 110 . For example, information stored through registration may include a variety of information such as an identifier of a work device, a type of work device, a control protocol, and a view angle of a camera that captures the work device. Accordingly, the user device 110 is in a state where it can access information about the work device 120a. Access and registration may be started by installing software or an add-on module for the proposed technology of the present invention in the work device 120a or according to an operator's command after installing the software or add-on module.

In step 1303, the user device 110 accesses the server 130 and searches for available work devices. The user device 110 may execute software or applications for the proposed technology of the present invention, check user information (eg, log-in information), and then access the server 130 according to the user's command. Thereafter, the user device 110 transmits a request message requesting information on available work devices, and the server 130 transmits information on registered work devices including the work device 120a to the user device 110. ) is sent to Accordingly, the user device 110 may display a list of work devices and confirm the user's selection. That is, the user device 110 displays information on the work devices (eg, work device identifier, type of work device, conditions necessary for controlling the work device, billing information, etc.), and determines which work device the user selects from the list. You can check. At this time, the server 130 may further transmit information about the remote camera included in the controlled device module 120 . Since the remote camera is also controlled by the user device 110, information on the number and direction of rotatable axes and a movable range (eg, rotatable angle range) may be required.

In step 1304, the user device 110 transmits a video streaming request message to the controlled device module 120. The video streaming request message may be transmitted directly to the controlled device module 120 or via the server 130 . For example, a request message for video streaming may be transmitted through socket communication. Here, the request message for video streaming may also be interpreted as a message requesting control authority for the work device 120a.

In step 1305, the controlled device module 120 transmits a video signal to the user device 110. For example, video streaming may be performed in a web real-time communication (WebRTC) method. Although not shown in FIG. 13 , a connection setting procedure for video streaming between the user device 110 and the controlled device module 120 may be performed first. Accordingly, the user device 110 displays a real-time image captured by the work device 120a, and the user can check the movement of the work device 120a through the image. The video signal may be continuously transmitted for real-time streaming.

In step 1306, the user device 110 transmits a work device/camera control signal to the controlled device module 120. The user device 110 captures a part of the user's body (eg, a hand) using a camera while displaying an image of the work device 120a streamed in real time from the controlled device module 120, and captures the captured image. At least one command for controlling the work device 120a may be generated based on and a control signal including the generated command may be transmitted. In addition, the user device 110 measures the tilt of the user device 110, generates at least one command for controlling the photographing direction of the remote camera based on the measured tilt, and generates a control signal including the generated command. can send A work device/camera control signal may be continuously generated and transmitted according to generation of a command.

According to the embodiment described with reference to FIG. 13 , a command for controlling the work device 120a (hereinafter referred to as 'work device control command') and a command for controlling a remote camera (hereinafter referred to as 'camera control command') are generated and transmitted. It can be. That is, two types of commands with different control targets may be transmitted. According to an embodiment, the work device control command and the camera control command may be transmitted through separate control links. According to another embodiment, the work device control command and the camera control command may be transmitted through the same control link and may include identifiers for distinguishing types of commands.

14 illustrates an example of an interface for controlling a work device according to an embodiment of the present invention. 14 illustrates a screen displayed through a user device while remotely controlling a work device.

Referring to FIG. 14 , the screen includes a setting menu 1410, a real-time image window 1420 of a work device, a first joystick 1430a, a second joystick 1430b, and a body part graphic 1440. Depending on the implementation or state, either the first joystick 1430a or the body part graphic 1440 may not be displayed.

The setting menu 1410 is an item for inputting various settings necessary for controlling a work device. For example, the setting menu 1410 may be used for settings related to cameras in a work site, settings related to a control method of work devices, and the like. Specifically, the setting menu 1410 is an input for selecting one of a plurality of cameras disposed at the work site, an input for zoom-in/zoom-out of the camera at the work site, and selection of a control method for the work device. It can be used for input, etc. Here, the selectable control methods may include a control method using a body image and a control method using the joystick 1413 .

The working device real-time image window 1420 is an area displaying a real-time image of a working device photographed at a work site. The user device displays a real-time streamed image of the work device through the work device real-time image window 1420 . Through this, the user can check the appearance of the work device in real time.

The first joystick 1430a is an item used to input a command for controlling the work device. As in the various embodiments described above, the work device may be controlled based on the posture of the body to be photographed, but in parallel, control using the first joystick 1430a may also be supported. The first joystick 1430a may include an item for moving or rotating a control point of the work device, an item for selecting a control point, and the like. According to an embodiment, the first joystick 1430a may be displayed translucently and may be moved to another position on the screen by a designated input (eg, long press or drag).

The second joystick 1430b is an item used to input a command for controlling the shooting direction of the remote camera. As in the various embodiments described above, the remote camera may be controlled based on the tilt of the user device 110, but in parallel, control using the second joystick 1430b may also be supported. The second joystick 1430b may include at least one of an item for selecting a rotation axis for a photographing direction of the remote camera and an item for inputting an amount of rotation angle. According to an embodiment, the second joystick 1430b may be displayed translucently, and may be moved to another position within the screen by a designated input (eg, long press and drag).

The body part graphic 1440 is a graphic representing a recognition result (eg, posture) of a user's body part used for controlling the work device 120a. The body part graphic 1440 may be a body part extracted from a photographed image or a graphic generated based on a recognition result. Additionally, the body part graphic 1440 may express detection points identified by posture recognition and a line connecting the detection points together.

Through the interface as shown in FIG. 14, the user can more conveniently control the work device. In particular, through the body part graphic 1440, the user may check a result of recognizing a part of the body for controlling the work device. Additionally, the interface may further include a warning window or warning indicator for a shooting state at the user's site. A warning window or warning indicator may be temporarily displayed to notify the user when a body part required to control the work device is not sufficiently photographed. For example, if the hand's posture is recognized using an image and the hand is out of the camera's field of view, the user device may display a warning window or a warning indicator notifying that the hand is out of the shooting range. As another example, when the user's hand is used to control the work device, and the hand is within the camera's field of view but the shooting angle is not appropriate for recognizing the hand posture, the user device requests to change the relative angle of the hand and the camera. A warning window or warning indicator may be displayed.

In addition, the interface may further include a warning window or warning indicator about the limit of the movable range of the remote camera. When the user device 110 rotates further on the corresponding axis even though the remote camera has rotated by the limit of possible rotation angles on a specific axis, a warning window or warning indicator indicating that the remote camera cannot rotate any more may be displayed. . Through this, the user will recognize that the remote camera has rotated to the limit on the corresponding axis, and will not make unnecessary attempts to further rotate the remote camera.

The method according to various embodiments of the present invention described above may be implemented as a computer program or a mobile application and stored in a medium to be executed by being coupled with a computer that is hardware. Alternatively, steps of a method or algorithm described in relation to the embodiments of the present invention may be directly implemented as hardware, implemented as a software module executed by hardware, or a combination thereof. A software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art to which the present invention pertains. In addition, the algorithm is produced in the form of an installation file and provided as an online download, and for this purpose, it can be stored on a server accessible through an online software market.

Meanwhile, in the detailed description of the present invention, specific embodiments have been described, but various modifications are possible without departing from the scope of the present invention. Therefore, the scope of the present invention should not be limited to the described embodiments and should not be defined, but should be defined by not only the scope of the claims to be described later, but also those equivalent to the scope of these claims.

Claims (5)

A method of operating a second device for remotely controlling a first device, comprising:
A first device that accesses a server that registers and manages a plurality of remote devices, transmits a first request message requesting information on available work devices to the server, and is included in the controlled device module from the server; and establishing a connection with the controlled device module located at a work site based on support from a server by receiving information about a first camera;
repeatedly requesting movement of a body part to perform an operation related to the joint of the first device from the user;
checking a motion generated in response to the motion request;
mapping at least one joint of the user used in the identified movement to the joint of the first device;
receiving video data of a first device included in the controlled device module, which is captured using a first camera included in the controlled device module;
displaying a real-time video for the first device based on the video data;
generating at least one device control command for controlling the first device based on the movement of the body part of the user;
generating at least one camera control command for controlling the first camera based on the tilt change of the second device; and
sending the at least one device control command and the at least one camera control command to control the first device and the first camera;
The device control command is a value generated based on an angle of a joint of the user's body corresponding to the joint of the first device, based on an angle of at least one joint of the user used in the identified movement. is generated and includes a value indicating the angle of the joint of the first device;
The camera control command is performed by the first camera among a plurality of transformation algorithms supported by the second device according to the difference between the axes recognizing the tilt in the second device and the rotation axes supported by the first camera. It is created based on one conversion algorithm corresponding to the structure of the rotation axes supported,
The information on the structure of the rotational axes supported by the first camera is provided from the server.
delete The method of claim 1,
The step of establishing a connection with the controlled device module,
accessing the server for registering and managing the plurality of remote devices;
sending, to the server, the first request message requesting information on available work devices;
receiving, from the server, information about a plurality of work devices including information about the first device and the first camera;
displaying a list of controlled devices including the first device;
confirming the user's selection of the first device among the plurality of work devices;
Transmitting a second request message requesting video streaming and control rights of the first device;
The information on the first camera includes information on the number and direction of rotational axes supported by the first camera, and information on a movable range;
The information on the plurality of work devices includes an identifier of each work device, a type of each work device, a condition necessary for controlling each work device, and billing information for each work device.
The method of claim 1,
In addition to the real-time video, the method further includes displaying on a display a graphic representing the part of the body photographed using a second camera provided in the second device and used to generate the at least one device control command. do,
Wherein the graphic includes dots indicating positions of joints detected in the body part.
An application program stored in a recording medium to execute the method according to any one of claims 1, 3, and 4 when operated by a processor.

Images