KR102005196B1 - Robot that manipulate object by remote control - Google Patents

Abstract

According to an embodiment, a robot operated based on the remote control of a manager terminal includes: a communication module communicating with the manager terminal; a manipulation device for physically manipulating a physical object; a camera module generating an image by photographing the object; a moving module moving the manipulation device and the camera module; a processor; and a memory storing instructions able to be executed by the processor. The processor loads the image into the memory and transmits the image to the manager terminal, and synchronizes at least one object generated from the image loaded into the memory with at least one object generated from the image loaded into the manager terminal; receives information about a specific object selected by a manager from the manager terminal; calculates a (X, Y, Z) value which is a three-dimensional position of a subject which is a part of a target corresponding to the specific object; and manipulates the subject corresponding to the specific object through the manipulation device by moving the manipulation device based on the (X, Y, Z) value.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a robot for operating an object by remote control,

The present invention relates to a robot that is remotely controlled, and more particularly, to a robot that can physically manipulate an object such as a machine tool by remote control and an operation method of the robot.

With the advancement of robot technology, it is possible to remotely control the robot, thereby allowing the robot to operate the machine tool or to remove explosives with the robot. For this purpose, the robot generally has a camera. The robot transmits an image photographed by the camera to the terminal of the manager at the remote site, and the manager controls the robot while viewing the photographed image displayed on the terminal. In this way, since the manager directly controls the robot while viewing the shot image remotely without looking directly at the machine tool, the physical manipulation of the machine tool is often not precise. If the manager is not proficient in robot control, mistakes may occur.

The present invention enables a robot to automatically analyze an actual three-dimensional position of an object and precisely manipulate the object by simply selecting an object to be manipulated from a photographed image, And to provide a method of operating a robot and a robot.

A robot that operates by remote control of an administrator terminal according to one aspect includes: a communication module that communicates with the administrator terminal; An operating device capable of physically manipulating a physical object; A camera module for photographing the object and generating a shot image; A movement module for moving the operation device and the camera module; A processor; And a memory for storing instructions executable by the processor; Wherein the processor loads at least one object created in the shot image loaded into the memory and at least one object created in the shot image loaded in the manager terminal, (X, Y, Z) of a subject, which is a part of the object corresponding to the specific object, and calculates (X, Y, Z) X, Y, Z) of the object, and operates the object corresponding to the specific object with the operation device.

Wherein the processor moves the camera module such that the distance between the subject and the camera module corresponding to the specific object is the shortest photographing distance, Calculating a Z value that is a distance from the camera module to the subject using the shortest photographing distance and the moving distance of the camera module, and calculating a Z value based on the (X, Y, Z) The operating device can be moved.

In calculating the Z value, the processor may determine a value obtained by adding the moving distance to the shortest photographing distance as the Z value.

Wherein the processor is configured to position the camera module at a first position that is a distance from the subject to a shortest photographing distance and then focus the camera module at a shortest photographing distance, The moving distance of the subject is determined as the moving distance until the subject is focused on the subject.

The processor may determine the moved distance when the camera module is moved away from the subject from a position at which the distance between the subject and the camera module is the shortest photographing distance as the moving distance.

Wherein the processor calculates a value of (x1, y1) which is a distance between a center and a center of the subject in the shot image and calculates a value of (x1, y1) By using the focal length, the (X, Y) value can be calculated.

Wherein the memory further stores a Z value which is a distance to a subject corresponding to the reduction of the focal distance based on a shortest photographing distance of the camera module, A Z value corresponding to the distance is checked in the memory and the camera module is moved so that the distance between the subject and the camera module is the shortest photographing distance. Then, the distance between the center and the subject in the subject plane (X, Y) can be calculated.

Wherein the camera module includes two cameras, and the processor calculates an angle formed by a line segment connecting the second principal point of the camera and the subject on the imaging plane to the central axis of the camera for each camera, Calculating a vertical distance between the second principal point and a subject by using a distance between two cameras and calculating a Z value by adding the distance between the second principal point and the imaging plane to the vertical distance, (X, Y) values can be calculated by using the above-described vertical distances between the subject and the subject and the vertical distance between the subject and the subject.

Wherein the camera module includes a depth sensor using a laser and the processor calculates an angle formed by a line segment connecting the second principal point of the camera and the subject on the imaging surface to the center axis of the camera, After moving to the position of the two taverns Calculating a depth value by causing the laser of the depth sensor to be directed to the subject based on the angle and calculating a depth value based on the calculated depth value, the angle, the distance from the center of the sensing surface to the second trough, (X, Y, Z) values of the subject can be calculated using the distance value of the subject in the x-axis direction and the distance value in the y-axis direction of the subject in the image sensing plane.

The processor divides the photographed image and the subject surface into a plurality of gratings corresponding to each other, and then calculates (X, Y, Z) values of the centers of the gratings of the subject surfaces corresponding to the centers of the gratings of the photographed images (X, Y, Z) values of the grid in which the specific object is located can be read and used in the memory.

The processor may cause the camera module to approach the subject after stopping the camera module so as to focus on the shortest shooting distance, and then, when the camera module is retracted while the focus remains unchanged, And the camera module is recorded with the Z value for each of the different subjects when the respective images are captured and the camera module is moved such that the distance between the other objects and the camera module is the shortest shooting distance, (X, Y), which is the distance between the center of the object plane and the other objects in the object plane, can be calculated for each different object.

The processor may record the process of object selection by an administrator in the memory, and may automatically process an object selection process subsequent to selection of a specific object based on the recorded object selection process.

Wherein the object is a crop, and the processor averages position values of a vertex, which is an object selected by the administrator as a cropping portion, based on a fruit portion, which is an object selected by the administrator as an object to be harvested, The fruit and the nipple can be separately displayed on the photographed image based on the recorded position value.

A method of operating a robot operated by remote control of an administrator terminal according to another aspect includes the steps of loading the captured image captured by a camera into a memory and transmitting the loaded image to an administrator terminal; Synchronizing at least one object created in the shot image loaded in the memory and at least one object generated in the shot image loaded in the manager terminal; Receiving information of a specific object selected by an administrator from the administrator terminal, and calculating (X, Y, Z) a three-dimensional position of a subject, which is a part of the object corresponding to the specific object; And moving the operation device based on the (X, Y, Z) value to operate the subject corresponding to the specific object with the operation device.

According to an embodiment of the present invention, the administrator merely needs to view a shot image sent from a robot at a remote place and select an object to be operated on the shot image. The robot analyzes the physical three-dimensional position of the object part corresponding to the object selected by the manager and moves the operation device to manipulate the part of the actual object, thereby reducing the mistake of the manager.

According to an embodiment of the present invention, the robot analyzes the three-dimensional position of portions of an actual object by software using a camera, thereby reducing the manufacturing cost of the robot for precision control and increasing the operation speed.

According to an embodiment of the present invention, a robot can learn a process of object selection by an administrator, and the robot can artificially manipulate the object by itself, thereby achieving the task automation.

1 is a block diagram of a robot remote control system according to an embodiment of the present invention.
Fig. 2 is a view showing a configuration of the robot shown in Fig. 1. Fig.
Fig. 3 is a diagram showing a configuration of the control apparatus of Fig. 1. Fig.
4 is a flow diagram of one embodiment for explaining how to remotely control a robot in a system of FIG. 1 to depress a button of a machine tool.
5 is a diagram showing a relationship between an imaging surface and a subject surface according to an embodiment of the present invention.
FIG. 6 is a diagram showing only the X-axis extracted in FIG.
7 is a view showing a positional relationship between a subject and a camera according to an embodiment of the present invention.
8 is a diagram illustrating the X-axis of two cameras according to an exemplary embodiment of the present invention.
9 is a view showing an image sensing plane divided into n × n lattices according to an embodiment of the present invention.

The foregoing and other objects, features and advantages of the present invention will become more apparent from the following detailed description of the present invention when taken in conjunction with the accompanying drawings, in which: There will be. In the following description, well-known functions or constructions are not described in detail since they would obscure the invention in unnecessary detail. Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

FIG. 1 is a view showing a robot remote control system according to an embodiment of the present invention. FIG. 2 is a diagram showing the configuration of the robot 130 of FIG. 1, Fig. 1, a plurality of machine tools (for example, a CNC (Computer Numerical Control) 110) are arranged laterally at regular intervals, a linear rail 120 is installed in parallel with the machine tool, A robot 130 capable of controlling the machine tools 110 is installed on the robot 120. The robot 130 can control each machine tool 110 while moving along the rail 120. The rail 120 is an example for easily explaining the present invention. If the robot 121 has a separate moving function such as a wheel, it is not necessary to install it separately. The machine tool 110 may include a touch display or a physical button on a front surface thereof for displaying a button for driving the machine tool 110 or the like.

The robot 130 communicates with the remote control device 140. The robot 130 loads a photographed image of an operation panel of the machine tool 110 as an object on an internal application program and transmits the loaded image to the control device 140. The robot 130 generates an object synchronized with the shot image loaded on the internal application program according to the object information received based on the shot image from the control device 140. [ Or the robot 130 sets a specific part of the shot image loaded on the application program as an object and transmits the shot image and object information to the control device 140 and then transmits the modified image and the object information from the control device 140 to the controller 140, And the like, and creates an object in the captured image loaded on the internal application program according to the received object information. That is, the objects of the shot image loaded on the internal application program of the robot 130 and the objects of the shot image displayed on the control device 140 are synchronized with each other. The robot 130 physically controls the operation panel of the machine tool 110 according to the command when a command for a specific object is input by the controller in the controller 140. [ Here, the object information may include, for example, an object type (e.g., button, switch), action information (e.g., push, rotation, etc.), position coordinates, and the like.

2, the robot 130 includes a camera module 210, a communication module 220, a processor 230, a memory 240, a movement module 250, and a manipulation device 260. As shown in FIG.

The camera module 210 includes a camera lens and an image sensor and photographs the machine tool 110, more specifically, the operation panel of the machine tool 110 under the control of the processor 230. The communication module 220 may perform wireless or wired communication. The communication module 220 converts electrical signals to electromagnetic waves and vice versa and is capable of communicating with the communication network, other mobile gateways, and communication devices through the electromagnetic waves. The communication module 220 includes, for example, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module And may include well-known circuits for performing such functions without limitation. The communication module 220 may be any type of communication module such as the Internet, called the World Wide Web (WWW), a network such as an intranet and / or a cellular telephone network, a wireless LAN and / or a metropolitan area network (MAN) It can communicate with other devices by communication.

Memory 240 may include high speed random access memory and may also include one or more magnetic disk storage devices, non-volatile memory such as flash memory devices, or other non-volatile semiconductor memory devices. In some embodiments, the memory may be a storage device, e.g., a communication circuit, located remotely from one or more processors, and may be connected to the Internet, an intranet, a local area network (LAN), a wide area network (WLAN) (Not shown), such as a computer readable storage medium, or any suitable combination thereof. As a software component, an operating system and an application program (instruction set) can be mounted (installed) in the memory. The operating system may be an embedded operating system such as, for example, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS or VxWorks, Android, etc. and may include general system tasks Device control, power management, etc.), and facilitates communication between various hardware and software components.

Processor 230 is a processor configured to perform an operation associated with robot 130 and to perform instructions and instructions to retrieve input and output data between components of robot 130 using, for example, Reception and operation can be controlled. The processor 230 may load the captured image photographed by the camera module 210 into an application program and simultaneously transmit the captured image to the controller 140 of the administrator through the communication module 220. [ The processor 230 may generate an object in the shot image loaded on the internal application program according to the object information received based on the shot image from the control device 140. [ Alternatively, the processor 230 may set a specific part of the shot image loaded on the application program as an object, transmit the shot image and object information to the control device 140, and then modify the object from the control device 140, Addition, and the like, and generate an object in the captured image loaded on the internal application program according to the received object information. The processor 230 physically controls the operation panel of the machine tool 110 according to the instruction when a command for a specific object is input by the controller in the controller 140. [

The movement module 250 may include various manipulation means suitable for manipulating the machine tool 110 such as a finger-like manipulator capable of pressing the button of the machine tool 110 or rotating the switch of the machine tool 110 And means for moving the operation means, or the operation means having a scissor function suitable for harvesting fruits or vegetables, along the X, Y, and Z axes. In addition, the movement module 250 may include a motor and wheels for moving the robot 130. In addition, the movement module 250 may include a motor, a chain, and the like that can move the camera module 210 back and forth. Hereinafter, the backward and forward movement of the camera module 210 may be any one of moving the robot 130 or moving the camera module 210 only.

The operation device 260 includes the aforementioned finger-shaped operation means, operation means having a scissors function, and the like, and is coupled with the movement module 250 to move on the X, Y, and Z axes. Preferably, the movement module 250 is moved in parallel to the X and Y axes and the Z axis of the imaging plane of the camera module 210, respectively, .

1, the controller 140 displays a shot image received from the robot 130 on the screen, and transmits the object information generated by the manager to the robot 130 based on the shot image. Here, the object information is information of a specific part (e.g., a button) selected by the administrator in the captured image, for example, information such as the type of object (e.g., button, switch), action information Coordinates, and the like. When a specific part of the photographed image is already set and received as an object in the robot 130, the control device 140 may receive object information related to modification or addition of the object from the manager and transmit the object information to the robot 130. In addition, the control device 140 may receive an administrator's command (e.g., push, switch ON, etc.) for a specific object of the shot image and transmit the command to the robot 130.

3, the control device 140 includes a memory 310, a memory controller 321, one or more processors (CPU) 322, a peripheral interface 323, an input / output (I / O) An input device 330, a display device 341, an input device 342, and a communication module 352. These components communicate through one or more communication buses or signal lines. The various components shown in FIG. 3 may be implemented in hardware, software, or a combination of both hardware and software, including one or more signal processing and / or application specific integrated circuits. The control device 140 may be, for example, a smart phone, a tablet PC, or a notebook computer or a personal computer, but is not limited thereto.

Memory 310 may include high speed random access memory and may also include one or more magnetic disk storage devices, non-volatile memory such as flash memory devices, or other non-volatile semiconductor memory devices. In some embodiments, the memory 210 may include a storage device, e.g., a communication module 352, located remotely from the one or more processors 222, an Internet, an intranet, a Local Area Network (WLAN) , A Storage Area Network (SAN), or the like, or any suitable combination thereof, via a network (not shown). Access to the memory 310 by other components of the control device 140 such as the processor 322 and the peripheral interface 323 may be controlled by the memory controller 321. [ The memory 310 may store various types of information and programs of the control device 140.

The peripheral interface 323 connects the input / output peripheral device of the control device 140 to the processor 322 and the memory 310. The one or more processors 322 perform various functions for the control device 140 and process data by executing various software programs and / or a set of instructions stored in the memory 310. [ In some embodiments, peripheral interface 323, processor 322 and memory controller 321 may be implemented on a single chip, such as chip 320. In some other embodiments, these may be implemented as separate chips 320. [

The I / O subsystem 330 provides an interface between the input / output peripheral of the control device 140, such as the display device 341, the input device 342, and the peripheral interface 323. The display device 341 may be a liquid crystal display (LCD) technology or a light emitting polymer display (LPD) technology. The display device 341 may be a capacitive type, a resistive type, . The touch display provides an output interface and an input interface between the control device 140 and the user. The touch display displays a visual output to the user. The visual output may include text, graphics, video, and combinations thereof. Some or all of the visual output may correspond to a user interface object. The touch display forms a touch sensitive surface that accommodates user input.

The processor 322 is a processor configured to perform the operations associated with the controller 140 and to perform the instructions and to control the operations of the controller 140 using, for example, the instructions retrieved from the memory 310, Reception and manipulation of data can be controlled. The processor 322 can perform the above-described reception of the shot image, setting of the object in the shot image, transmission of the object information, and transmission of the command to the object according to the application program 313 stored in the memory 310. [ The software components are installed in the memory 310 by an operating system 311, a graphics module 312 and an application program 313. The operating system 311 may be an embedded operating system such as, for example, Darwin, RTXC, LINUX, UNIX, OS X, WINDOWS or VxWorks, Android, Management, storage control, power management, etc.), and facilitates communication between the various hardware and software components. Graphics module 312 includes a number of well known software components for providing and displaying graphics on display device 341. The term "graphics" includes, without limitation, text, web pages, icons (e.g., user interface targets including soft keys), digital images, video, animations, . The application program 313 displays a graphic for controlling the robot 130 and communicates with the application program of the robot 130 through the communication module 352 under the control of the processor 322. [

The communication module 352 performs communication via an external port or communication by an RF signal. The communication module 352 converts electrical signals to RF signals and vice versa and is capable of communicating with the communication network, other mobile gateway devices, and communication devices via the RF signals. The communication module 352 may include, for example, an antenna system, an RF transceiver, one or more amplifiers, a tuner, one or more oscillators, a digital signal processor, a CODEC chipset, a subscriber identity module And may include well-known circuits for performing such functions. The communication module 352 may be any type of communication module such as the Internet, called the World Wide Web (WWW), a network such as an intranet and / or a cellular telephone network, a wireless LAN and / or a metropolitan area network (MAN) It can communicate with other devices by communication.

The input device 342 may receive user input via a physical input interface, such as a keyboard or touchpad or a mouse, a touch pen, or may receive user input in conjunction with a touch display of the display device 341 .

As described above with reference to FIGS. 1 to 3, the robot 130 loads the captured image of the operation panel of the machine tool 110 into the internal application program and simultaneously transmits the captured image to the controller 140 of the manager. The manager analyzes the photographed image displayed on the control device 140 and performs operations such as setting an appropriate part (e.g., a button on the photographed image) as an object and adding an attribute, and the control device 140 And transmits the object information to the robot 130. The robot 130 creates an object having the same properties as the objects created in the controller 140 on the captured image loaded in the internal application program based on the received object information. Alternatively, the robot 130 analyzes the shot image loaded in the internal application program, sets suitable parts as objects, transmits the shot image and object information to the controller 140 of the manager using the communication function, The object information may be received from the control device 140 when the object is added or modified. Through this series of processes, the objects of the shot image in the application program of the control device 140 and the objects of the shot image in the application program of the robot 130 cooperate with each other. That is, when the manager touches the A button object of the shot image displayed on the control device 140, the robot 130 recognizes that the A button object is also touched by the robot 130, and the robot 130 moves the movement module 250 The operation device 260 presses the A button actually present on the object, that is, the operation panel of the machine tool 110. [

On the other hand, in the application program of the controller 140 and the robot 130, the photographed image is positioned in a lower layer in a transparent state, and the objects created on the photographed image are positioned in a semi-transparent state on the upper layer, The photographed image photographed in real time by the robot 130 is displayed in real time. For example, when the manager touches the A button object on the captured image displayed on the control device 140, the operating device 260 of the robot 130 displays the real time A button on the operation panel of the machine tool 110 The scene image is transmitted in real time to the control device 140 and displayed in real time on the lower layer in the application program of the control device 140. [ Therefore, the administrator can confirm that the actual A button is pressed well.

The robot 130 learns the repeated operation of the administrator through the control device 140, stores the learning result in a storage device such as a memory, and uses the learning result to perform operations to be performed in the future by the control device 140 You can tell. For example, if the administrator pressed the 'A button' and then the 'B button' in the past, the robot 130 analyzes the record and if the manager presses the 'A button', the 'B button 'It is possible to display a certain display on the object, for example, to give a flicker by changing the color. The robot 130 can perform an autonomous operation regardless of the operation of the manager if the subsequent operation of the administrator is determined to be stochastically clear based on the accumulated learning result, and transmits the real-time image related to the autonomous operation to the control device 140 So that the administrator can view it.

4 is a flow diagram of one embodiment for explaining how to remotely control a robot in a system of FIG. 1 to depress a button of a machine tool.

Referring to FIG. 4, the robot 130 photographs the operation panel of the machine tool 110 using the camera module 210 (S401). Then, the robot 130 loads the captured image into the internal application program, analyzes the captured image, and generates each button in the captured image as an object (S402). For example, the photographed image is displayed in a transparent state and displayed on a lower layer, and an object for a button included in the photographed image is displayed in a translucent state on an upper layer. That is, a button object is superimposed on the captured image.

The robot 130 inserts the object information of the buttons generated in step S402 into the shot image (S402). The object information can be inserted into the shot image by replacing the pixel value that is not affected by the color in the shot image with the object information. The object information may include, for example, the type of object (button, switch, etc.), position coordinates of the object in the shot image, and the like. The robot 130 transmits the captured image including the object information to the control device 140 through the communication network (S404). In the present embodiment, it is described that object information is inserted and transmitted in a captured image, but object information and a captured image may be separately transmitted.

The control device 140 receives the captured image in which the object information is inserted, extracts the object information by analyzing the captured image, displays the captured image in a transparent state, displays the captured image on the lower layer, and displays the button object according to the object information on the upper layer . Accordingly, the button object is displayed superimposed on the designated position of the shot image (S405). It is possible to receive a correction input, such as changing the position of the button object whose position is misplaced among the button objects displayed on the control device 140, from the manager, An input for generating the button object is received from the manager, and the modified / added object information is generated (S406). The control device 140 transmits the modified and / or added object information generated in step S406 to the robot 130 via the communication network in step S407. The robot 130 modifies an object displayed on the captured image loaded in the internal application program according to the received modified and / or added object information, or creates a new object and displays it on an upper layer. Through the above process, the objects on the shot image loaded on the robot 130 and the objects on the shot image loaded on the control device 140 are interlocked with each other.

The control device 140 receives the push input of the manager for the specific button among the button objects on the displayed photographed image (S408). Here, the push input may be input through the touch display, or may be input through a mouse or the like. The control device 140 transmits a push input command for the specific button to the robot 130 via the communication network (S409). The push input command may include object information of the specific button, for example, position coordinates in the shot image, kind of object, and the like. Accordingly, the robot 130 drives the movement module 250 in response to the push input command, moves the operation device 260 to the actual button on the operation panel of the machine tool 110, and completes the act of pressing the corresponding button (S410).

Hereinafter, when the manager selects a specific object from the captured image of the control device 140, the robot 130 selects the actual object corresponding to the specific object on the operation panel of the machine tool, for example, Describes how to calculate the actual position of an object. Here, the actual position of the actual object is the value of (X, Y, Z), Z is the distance from the center of the imaging surface of the camera module 210 of the robot 130 to the object parallel to the imaging surface, And X and Y are X and Y coordinates when the object plane is assumed to be the XY plane. Hereinafter, a method of obtaining (X, Y, Z) values will be described in detail with reference to the drawings.

1. How to find the (X, Y) value

FIG. 5 is a view showing a relationship between an imaging surface and a subject surface according to an embodiment of the present invention, and FIG. 6 is a view showing only the X-axis in FIG.

In Fig. 5, reference symbol S1 denotes the object surface, and reference symbol S2 denotes the imaging surface. The axis of the imaging surface S2 (i.e., the image sensor) is represented by lowercase letters x, y, and z, and the axis of the subject surface S1 is represented by uppercase letters X, Y, The center point of the subject surface S1 is denoted by P1, and the center point of the imaging surface S2 is denoted by P2. And reference numeral Q2 denotes a second principal point as an optical center point of the camera lens. And distance to the Q2 represented by d ₀ from the center point P1 of the object surface (S1), from the distance to the center of the imaging surface (S2) P2 Q2 is denoted by d _i.

6, reference numeral X1 denotes the position of the subject on the subject plane S1 as the actual position of the subject. And reference numeral x1 denotes a position of the object on the sensing surface S2 of the sensing surface S2. 6, the distance from the center P1 to the subject X1 on the object plane S1 is denoted by A, and the distance from the center P2 to the subject x1 on the image sensing plane S2 is denoted by a. And the distance to the imaging surface (S2) from the object surface (S1) is represented by d _s.

5 and 6, the following relationship (1) is established first.

(1) w / W = h / H = d _i / d _o = a / A

In Equation (1), A is the X value at (X, Y) to be obtained, and A in (Equation 1) is expressed by Equation (2). The distance a from the center P2 to the point of the object x1 touched by the manager on the image sensing plane S2 and the center P2 of the image sensing plane S2 from the optical center point Q2 of the camera lens and a distance of d _i to, knowing the distance of d ₀ to the center P1 of the object surface (S1) from the optical center point Q2 of the camera lens, a value, that is, to know the X value of the subject in the object plane (S1) have. Therefore, we need to know each of a, d ₀ , and d _i .

&Quot; (2) " A = a x d ₀ / d _i

a, d ₀ , and d _i are obtained as follows.

(1) How to obtain a

First, a is the distance of the x-axis from the center P2 to the point of the object touched by the manager on the sensing plane S2. The physical length of the sensing plane S2 and the number of pixels in the horizontal and vertical directions of the captured image are used . For example, in the case of a full frame of a single lens having a focal length of 50 mm, the horizontal and vertical values of the imaging surface S2 are 36 mm and 24 mm, respectively. The number of horizontal pixels and vertical pixels of the output image is 3,600 pixels and 2,400 pixels, respectively. If the point touched by the manager is 900 pixels on the x-axis, a is calculated to be 9 mm according to the following relation (Equation 3).

(3) 3,600 pixels: 900 pixels = 36 mm: a

(2) d ₀ , d _i To  How to obtain

At least an administrator to match the specific object, that is, if touch the subject, then obtaining a a, as described above, the robot 130 is a camera sensor on a subject (X1) to obtain the d _0, d _i in the recorded image The camera module 210 approaches the subject X1 at the shortest photographing distance. The shortest photographing distance is a value provided by the manufacturer of the camera lens. The distance to the imaging surface (S2) from the object plane (S1) as described above, d _s is as follows (Equation 4). Access the camera module 210 in the object (X1) by placing at the shortest photographing distance is the distance d _s to the Minimum Distance soon imaging surface from the object side (S1) (S2). And the focal distance of the camera lens (the focal distance when the subject is at an infinite distance) is f, the following relation (5) is established. Substituting (Equation 4) into (Equation 5), a quadratic equation of (Equation 6) is generated. When the focal length f of the camera lens and the shortest photographing distance d _s are input to (Equation 6), d ₀ and d _i are obtained.

(4) d _o + d _i = d _s

(Equation 5) 1 / d _o + 1 / d _i = 1 / f

(6) d _i ² - d _s x d _i + f d _s = 0

For example, when the shortest imaging distance d _s of a single lens having a focal length f of 50 mm is 0.45 m, substituting this into Equation (6) yields d _i 57.3 mm. Substituting this into (Equation 4), d _o ? 392.7 mm is obtained.

Substituting a, d ₀ and d _i obtained in the above into the above equation (2), A, which is an X value on the object plane S1, is finally obtained.

The calculation of the Y value on the object plane S1 can be obtained by the same method as described above. When the distance of the y-axis from the center P2 of the imaging surface S2 to the point touched by the manager is substituted as the a value of the expression (2), the obtained A value is the Y value on the object surface S1.

2-1. (Z) 1st  Way

7 is a view showing a positional relationship between a subject and a camera according to an embodiment of the present invention. 7, reference numeral 710 denotes an actual object, for example, an operation panel of the machine tool 110. [ The robot 130 adjusts the focus so as to focus on the shortest shooting distance in a state where the imaging plane of the camera module 210 is present at the position of C In Fig. 7, the distance from C to C1 is the focal length adjusted to the shortest shooting distance. After the focus is adjusted so as to focus on the shortest shooting distance, the robot 130 moves the camera module 210 to the subject 710 so as to focus on the subject 710. That is, the image sensing plane of the camera module 210 in FIG. When the distance physically moved in this manner is added to the shortest photographing distance, the Z value of the subject 710 based on the position of C is derived.

Alternatively, after positioning the camera module 210 such that the camera module 210 is focused on the subject 710 after the camera module 210 is focused on the shortest shooting distance, the robot 130 The camera module 210 is retracted away from the subject 710 and moved to an arbitrary position, that is, the position of C. At this time, when the distance D - > C moved backward is added to the shortest distance, the Z value of the object 710 based on the position of the image sensing plane of the camera module 210, that is, C is derived.

When the specific object in the shot image is selected by the administrator, the robot 130 can focus the object 710 corresponding to the specific object, that is, the camera sensor on the operation panel of the machine tool 110 (X, Y) values based on the center of the object plane are obtained after the camera module 210 is approached to the subject 710 at the shortest photographing distance that is the minimum distance between the camera module 210 and the camera module 210, (Z) is obtained by adding the shortest shooting distance. The robot 130 then controls the moving module 250 based on the (X, Y, Z) value to physically manipulate the subject 710 through the operating device 260. [

Alternatively, the robot 130 may move the camera module 210 to the subject 710 with the shortest photographing distance, which is the minimum distance that the camera sensor can focus on the subject 710, that is, the operation panel of the machine tool 110 (X, Y) values based on the center of the object plane of each of the actual objects (e.g., buttons) corresponding to each object generated in the captured image are obtained and stored. Then, the robot 130 retracts the camera module 210 and moves it to an arbitrary position away from the subject 710. When the shortest imaging distance is added to the moving distance, the Z value from the center of the imaging surface to the center of the object surface is obtained. Since the robot 130 recognizes the values (X, Y, Z) of the subject 710, when the manager touches the specific object in the captured image, the controller 130 controls the movement module 250 so that the operation device 260 physically The subject 710 can be operated.

2-2. (Z) Second  Way

The focal length of the camera lens changes depending on the depth of the subject. That is, in the case of a single lens having a focal length of 50 mm, the focal distance is 50 mm when the subject is at an infinite distance, and the focal distance is about 57.3 mm when the subject is located at the shortest shooting distance of 0.45 m. By adjusting the focus ring of the lens, focusing from a distance of 0.45m to an infinitely-positioned subject, the focal length also decreases sequentially from 57.3mm to 50mm. Therefore, the robot 130 stores the Z value, which is the distance to the subject when the focal length is reduced by 0.1 mm, in advance in reference to the time when the focus is focused on the shortest shooting distance, and then focuses the object dynamically And the Z value corresponding to the focal distance can be confirmed in the memory when it is focused. When there are a plurality of subjects having different depths, the values of (X, Y, Z) can be automatically calculated and recorded in the memory each time the subject is focused. This record value can be referred to when an administrator touches a specific part of the shot image or when the robot 130 performs autonomous operation.

3. How to obtain (X, Y, Z) using a dual camera

The camera module 210 of the robot 130 may include two cameras. 8 is a diagram illustrating the X-axis of two cameras according to an exemplary embodiment of the present invention. As shown in Fig. 8, the shooting regions of the two cameras 810 and 820 overlap. The overlapping common photographing area is an area formed by the three points S1, S2, and M in Fig. The vertical distance from the imaging plane of each of the cameras 810 and 820 to M is designed to be less than or equal to the shortest imaging distance and the distance between the cameras 810 and 820 and the lens. The Z value of the subject outside the common photographing area can be replaced with the Z value of the subject located in the common photographing area of the same focal distance.

In FIG. 8, the actual position of the subject is indicated by X12. The distance from the second principal point Q1 to C of the first camera 810 is the distance from the center of the subject surface of the first camera 810 to the subject and is denoted by a. The distance from the second principal point Q2 to C of the second camera 820 is the distance from the center of the object plane of the second camera 820 to the object, Therefore, the interval c between the two cameras 810 and 820 is a + b. If you know a and b, you can find the (X, Y) value of the (X, Y, Z) values you want to find. a and b are obtained from below.

On the other hand, the angle formed between the central axis of the first camera 810 and the line segment Q1-x1 (x1 is the position of the subject on the sensing surface) is α, the center axis of the second camera 820 and the line segment Q2- (The position of the subject in the photographing direction) is represented by?,? And? Can be obtained by the following formula (7).

(7)

? = tan ^-1 (distance between a pixel value of the x1 point on the captured image and the image pickup surface of the first camera 810 / Q1)

? = tan ^-1 (the distance between the pixel value of the x2 point on the captured image and the image sensor size of the second camera 820 / Q2 and the distance between the imaging surface)

After obtaining α and β in this way, the vertical distance h between the second principal point Q1 or Q2 and the subject X12 can be obtained as follows. That is, tan (90 -?) = H / a, tan (90 -?) = H / b, and c = a + b. Solving these simultaneous equations, h can be obtained as in (Equation 8).

(8)

(90 -?) / tan (90 -?) + tan (90 -?)

A and b can be obtained by substituting the values h and tan (90 -?) = H / a and tan (90 -?) Y) can be obtained.

And the Z value can be obtained by adding the h value to the vertical distance between the second principal point (Q1 or Q2) of the camera 810 or 820 and the image sensing plane.

4. How to find (X, Y, Z) using depth sensor

The camera module 210 may include a depth sensor that includes a laser. The depth sensor may be one of SL (Structured Light) and ToF (Time of Flight). The SL-type depth sensor calculates the depth by analyzing the deformation information of the pattern according to the shape of the surface of the subject after emitting a laser of a specific pattern to the subject. The ToF method calculates the depth by measuring the time the laser returns from the subject. In this embodiment, the robot 130 calculates the Z value using the depth sensor, and the (X, Y) value can be calculated in the same manner as described above. The Z value may be calculated per pixel of the shot image, or may be calculated per n pixels or per n x n pixels. It is assumed that the Z value per n pixels is a Z value based on the center of n pixels, and that all the n pixels have the same Z value. It is also assumed that the Z value per n × n pixels is a Z value based on the center of the grid formed by the n × n pixels, and all the pixels in the grid have the same Z value. Hereinafter, a method of obtaining a Z value per n x n number of pixels will be described.

9 is a view showing an image sensing plane divided into n × n lattices according to an embodiment of the present invention. The image sensing plane is divided into n × n lattices, Are also divided into n × n lattices. Therefore, it can be assumed that the object plane S1 is also divided into n x n lattices in the same manner as the image sensing plane S2. Let A be the subject of the subject surface S1 and a be the subject of the corresponding imaging surface S2. An angle? (An X-axis angle and a Y-axis angle) formed by a line segment connecting the center of the grating on which the subject "a" is formed with the second principal point Q2 to the central axis on the image sensing plane S2 is used to obtain the Z value of the object A on the object plane S1. ). This means the direction and angle of the light reflected from the subject A. More specifically, the focal length f, which is the distance from the second principal point Q2 to the center point of the imaging surface S2, ax, the distance value in the x-axis direction of the subject a in the imaging plane S2, The angle a formed by the object a in the imaging plane S2 with the x-axis direction in the x-axis direction and the y-axis direction and the angle a formed in the xy-axis direction by the distance a in the y- (9) through (11).

(9)? X = tan ^-1 (ax / f)

(10)? Y = tan ^-1 (ay / f).

Α = tan (Equation ^{11) -1 ((√ ax 2} + ay 2) / f)

The robot 130 moves the depth sensor included in the camera module 210 to the position of the second principal point Q2, and directs the laser to the subject with reference to the angle?, And obtains the depth value of the subject A. Using the depth value of the subject A and the angle alpha formed by the subject a in the image sensing plane S2 obtained by the above formula (11) with the xy axis direction, the second principal point Q2 and the subject The distance Z _d to the center of the surface S1 can be obtained.

(12)

Z _d (distance from Q2 to P1) = D (distance from Q2 to A as depth value of subject A) x cos?

Next, the robot 130 calculates the distance (Z _d ) from the second principal point Q 2 to the center of the subject surface S 1 and the distance from the imaging plane S 2 to the second principal point Q 2 The Z value of the subject A is calculated. This is expressed by the following equation (13).

(13)

Z = Z _d + f (focal length, distance from the center of the imaging surface S2 to Q2)

When the Z value is calculated as described above, the robot 130 calculates the Z value using the angle? X formed by the object a with the x axis direction in the central axis and the angle? Y with the y axis direction, And (X, Y) values as in (15).

X = Z _d x tan (? X)

(15) Y = Z _d x tan (? Y)

Accordingly, if the administrator selects the subject a from the photographed image displayed on the control device 140, the value of (X, Y, Z) is calculated in the above manner.

The depth sensor is fixed by a support having a pan (pan) function and a tilt function (vertical rotation). The depth sensor rotates horizontally by the angle? X at the position Q2, And the depth is measured by aiming at the object A with an axis and an angle.

In the embodiment of FIG. 9, when the administrator selects an object from the shot image, the value (X, Y, Z) is obtained. However, the present invention is not limited to this. When the shot image is divided into nxn grid (X, Y, Z) values of the respective gratings of the object plane can be previously stored in the memory by obtaining all the Z values of the respective gratings of the object plane corresponding to the grating. When the manager selects a specific grid portion in the shot image, the user can manipulate the object by reading the (X, Y, Z) values of the grid from the memory and moving the operation device 260. This method can be applied not only to the use of the depth sensor but also to the method described above. In other words, after dividing the photographed image into a plurality of gratings, taking the center of the grating as a subject for each grating, the (X, Y, Z) values of the centers of the gratings on the object plane corresponding to the center of each grating were previously stored in the memory Can be used.

Also, when the administrator sequentially selects a plurality of objects in the shot image, the αx, αy, and α values of each object are sequentially stored in the memory, and the first to fourth quadrants And determines the rotation direction of the depth sensor support by applying the code value of the selected object to the values of? X and? Y.

Smart farm Example

Tomatoes are planted on the tomato farm in parallel with the movement path of the robot (130). The robot 130 maintains the position of the camera module 210 such that the distance between the camera and the tomato is at least the shortest photographing distance. The robot 130 transmits the photographed image photographed by the camera module 210 to the controller 140 of the administrator. The robot 130 and the control device 140 generate tomatoes as objects and synchronize them in the captured images.

The robot 130 displays the selected ripe tomatoes in a circular shape in the control unit 140 and displays information about the tomatoes in a short line segment, do. At this time, the control device 140 does not transmit the entire image, but can transmit only the center coordinate value of the circled handwritten graphic and the start and end coordinate values of the handwritten graphic handwritten.

The robot 130 controls the focus ring of the camera module 210 so as to focus on the shortest shooting distance and then moves the camera module 210 as far as possible in the tomato direction to stop the movement when the selected specific tomato is focused. The robot 130 calculates the (X, Y, Z) value of the specific tomato by calculating the (X, Y) value of the specific tomato in the photographed image and using the value obtained by adding the shortest photographing distance and the moving distance as the Y value do. The robot 130 moves the operating device 260 based on the (X, Y, Z) values to cut and harvest the tomatoes. At this time, the robot 130 cuts the scissors included in the manipulation device 260 using the starting point and the end point of the line segment.

The robot 130 may record the frequency distribution table of the color values of the selected tomatoes and store them in the memory as judged by the administrator. In addition, the robot 130 divides the selected portion of the faucet and the tomato fruit portion selected by the administrator into data and records the color value, and also records the average value of the position of the cut portion based on the shape of the tomato fruit portion . As the amount of data increases, it is numerically represented close to the view of the manager. The robot 130 displays a circle shape on the fruit portion of the tomato and a line shape on the stem portion of the tomato prior to the judgment of the manager, (140) and wait for the manager's judgment. If it is determined that there is no difference between the judgment of the robot 130 and the judgment of the robot 130, the manager presses the autonomic button, and the robot 130 autonomously judges the robot 130 without the administrator's decision, and the tomato is harvested.

In the above embodiment, the robot 130 moves the camera module 210 to focus on the shortest shooting distance, and then moves to the tomato selected by the administrator to calculate the values of (X, Y, Z) The Z value can be calculated by checking the Z value, and the (X, Y) value can be calculated in the same manner as described above. Also, the robot 130 may calculate the (X, Y, Z) values of the selected specific tomato using the two cameras when the manager selects a specific tomato. The robot 130 may also calculate the Z value of the tomato using the depth sensor described above.

Alternatively, the robot 130 closes the camera module 210 by approaching the camera module 210 after aligning the camera module 210 so as to focus on the shortest shooting distance. Then, the robot 130 retreats the camera module 210 while keeping the focus, and records images of various scenes and records the positions of the images when the images are captured. For example, when the camera module 210 is retracted and one tomato is focused, the position of the tomato at that point is forward by the shortest shooting distance. When the camera module 210 is further retracted by the first distance, the position of the tomato is the sum of the first distance and the shortest photographing distance. The Z values of the tomatoes thus focused are recorded, and the robot 130 transmits a plurality of photographed images to the control device 140. When the manager selects a specific tomato from a plurality of photographed images, X, Y), and the tomato can be harvested.

While the specification contains many features, such features should not be construed as limiting the scope of the invention or the scope of the claims. In addition, the features described in the individual embodiments herein may be combined and implemented in a single embodiment. Conversely, various features described in the singular < Desc / Clms Page number 5 > embodiments herein may be implemented in various embodiments individually or in combination as appropriate.

Although the operations have been described in a particular order in the figures, it should be understood that such operations are performed in a particular order as shown, or that all described operations are performed to obtain a sequence of sequential orders, or a desired result . In certain circumstances, multitasking and parallel processing may be advantageous. It should also be understood that the division of various system components in the above embodiments does not require such distinction in all embodiments. The above-described program components and systems can generally be implemented as a single software product or as a package in multiple software products.

The method of the present invention as described above can be implemented by a program and stored in a computer-readable recording medium (CD-ROM, RAM, ROM, floppy disk, hard disk, magneto optical disk, etc.). Such a process can be easily carried out by those skilled in the art and will not be described in detail.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. The present invention is not limited to the drawings.

110: Machine tools
120: rail
130: Robot
140: Control device
210: camera module
220: Communication module
230: Processor
240: Memory
250: Moving module
260: Operation device

Claims (15)

delete delete delete A robot that operates by remote control of an administrator terminal,
A communication module for communicating with the manager terminal;
An operating device capable of physically manipulating a physical object;
A camera module for photographing the object and generating a shot image;
A movement module for moving the operation device and the camera module;
A processor; And
A memory for storing instructions executable by the processor;
The processor comprising:
Loading the captured image into the memory and transferring the captured image to the manager terminal, synchronizing at least one object generated in the captured image loaded in the memory and at least one object generated in the captured image loaded in the manager terminal,
(X, Y, Z), which is a three-dimensional position of the object to be manipulated, which is a part of the object corresponding to the specific object, Y, Z) to move the operation device to operate the operation object corresponding to the specific object with the operation device,
(X, Y), which is the distance between the center and the object in the object plane,
The camera module is moved from the first position to the operation position in a state in which the camera module is positioned at a first position where the distance from the operation target is longer than the shortest photographing distance, And calculates a Z value that is a distance to the operation target from a value obtained by adding the shortest photographing distance and the distance moved by moving the subject in the subject direction until the subject is focused on the operation subject. A robot that operates by remote control of an administrator terminal,
A communication module for communicating with the manager terminal;
An operating device capable of physically manipulating a physical object;
A camera module for photographing the object and generating a shot image;
A movement module for moving the operation device and the camera module;
A processor; And
A memory for storing instructions executable by the processor;
The processor comprising:
Loading the captured image into the memory and transferring the captured image to the manager terminal, synchronizing at least one object generated in the captured image loaded in the memory and at least one object generated in the captured image loaded in the manager terminal,
(X, Y, Z), which is a three-dimensional position of the object to be manipulated, which is a part of the object corresponding to the specific object, Y, Z) to move the operation device to operate the operation object corresponding to the specific object with the operation device,
(X, Y), which is the distance between the center and the object in the object plane,
A moving distance of the camera module when the camera module is moved away from the operation target in a state in which the camera module is positioned so that the distance between the operation target and the camera module is the shortest shooting distance, And calculates a value obtained by adding the distance as a Z value that is a distance to the operation target. The method according to claim 4 or 5,
The processor comprising:
(X1, y1), which is the distance between the center and the specific object selected by the manager in the captured image,
Calculates the (X, Y) value using the (x1, y1) value, the shortest photographing distance, and the focal distance of the camera module. A robot that operates by remote control of an administrator terminal,
A communication module for communicating with the manager terminal;
An operating device capable of physically manipulating a physical object;
A camera module for photographing the object and generating a shot image;
A movement module for moving the operation device and the camera module;
A processor; And
A memory that stores a Z value that is a distance to the object corresponding to a decrease in focal length based on a shortest shooting distance of the camera module;
The processor comprising:
Loading the captured image into the memory and transferring the captured image to the manager terminal, synchronizing at least one object generated in the captured image loaded in the memory and at least one object generated in the captured image loaded in the manager terminal,
(X, Y, Z), which is a three-dimensional position of the object to be manipulated, which is a part of the object corresponding to the specific object, Y, Z) to move the operation device to operate the operation object corresponding to the specific object with the operation device,
The camera module is moved so that the distance between the operation target and the camera module becomes the shortest photographing distance, and the Z value corresponding to the focal length of the camera module when the operation object is in focus, (X, Y) which is the distance between the center and the object in the plane. delete delete delete delete A robot that operates by remote control of an administrator terminal,
A communication module for communicating with the manager terminal;
An operating device capable of physically manipulating a physical object;
A camera module for photographing the object and generating a shot image;
A movement module for moving the operation device and the camera module;
A processor; And
A memory for storing instructions executable by the processor;
The processor comprising:
Loading the captured image into the memory and transferring the captured image to the manager terminal, synchronizing at least one object generated in the captured image loaded in the memory and at least one object generated in the captured image loaded in the manager terminal,
(X, Y, Z), which is a three-dimensional position of the object to be manipulated, which is a part of the object corresponding to the specific object, Y, Z) to move the operation device to operate the operation object corresponding to the specific object with the operation device,
After the camera module is aligned with the shortest photographing distance, the camera module is stopped to approach the operation target, and when the camera module is retracted while the focus remains unchanged, when the other operation targets are focused, The position of the camera module when each image is captured is recorded as a Z value for each different operation object,
(X, Y) value, which is the distance between the center and the other operation objects in the object plane, after moving the camera module so that the distance between the other operation objects and the camera module is the shortest imaging distance For each different operation target. 13. A method according to any one of claims 4, 5, 7, or 12,
The processor comprising:
Wherein the process of selecting an object of an administrator is recorded in the memory and an administrator automatically processes an object selection process subsequent to selection of a specific object based on the recorded object selection process. 13. A method according to any one of claims 4, 5, 7, or 12,
The object is a crop,
The processor comprising:
The image processing method according to claim 1, further comprising: averaging a position value of a vertex, which is an object selected by the administrator as a truncated part, based on a fruit part, which is an object selected as an object to be harvested by an administrator, And displays the fruit and the nip portion separately. delete

Images