KR102567743B1 - System for providing vision based data synchronization service between collaborative robot - Google Patents

Abstract

A system for providing a data synchronization service between vision-based collaborative robots is provided, and includes at least one user terminal outputting synchronized real-time data between at least one collaborative robot and a collection unit collecting real-time data from the at least one collaborative robot. A photographing unit that photographs at least one cobot using a camera, a synchronization unit that synchronizes real-time data of at least one cobot based on image data captured by at least one camera, and transmits the synchronized real-time data to a user. and a synchronization service providing server including a forwarding unit for transmitting to the terminal.

Description

Data synchronization service provision system between vision-based collaborative robots {SYSTEM FOR PROVIDING VISION BASED DATA SYNCHRONIZATION SERVICE BETWEEN COLLABORATIVE ROBOT}

The present invention relates to a system for providing a data synchronization service between vision-based collaborative robots, and provides a system capable of securing high-quality data by performing data synchronization using a vision-based keypoint of a collaborative robot at each point in time.

Most artificial intelligence research is focused on high-quality refined open data, and guidelines for the process from unrefined raw data generated in the field to securing multimodal datasets necessary for CPPS (Cyber Physical Production System) construction this situation is lacking. Hardware for collecting data at industrial sites is prepared, but data pipelines that process and handle the pouring data are not prepared. In order for each step of the 5C (Connection, Conversion, Cyber, Cognition, and Configure) constituting the CPS (Cyber Physical System) to interact robustly, it is essential to collect operation data, and collect and merge sensor data from IoT equipment. The time delay problem between visual data acts as a major obstacle, and labeling work to improve the quality of datasets has caused a major bottleneck. Operation information is very important due to the nature of the manufacturing industry, but the precision limitations of the sensor itself Due to communication protocol problems, there is always a time difference between collected data.

At this time, a method of synchronizing data in the robot control system has been researched and developed. In this regard, prior art Korean Patent Publication No. 2022-0150700 (published on November 11, 2022) and Korean Patent Registration No. 10-1645260 ( Announced on August 4, 2016), acquires the status information and control information of the robot captured by the camera, outputs the simulation result of the virtual robot in the virtual environment based on the status information and control information, and re-creates it according to the simulation result. Synchronization signal to synchronize data between sensors placed in the robot by using TimeStamp in the configuration and server-client structure that performs synchronization between the real environment and the virtual environment when the error tolerance is exceeded when controlling the robot. After generating , a synchronization signal and a recording time are extracted, and a configuration using OS as a standard when the error range is greater than or equal to a reference value and using a synchronization signal when the error range is less than the reference value is disclosed.

However, in the former case, it is a configuration that matches the physical world and the virtual world of the digital twin, and is not a configuration that performs data synchronization between swarm robots or cooperative robots. It is not a configuration that solves the problem of inconsistent data synchronization between cobots as real-time data is loaded. Therefore, it is required to research and develop a method capable of synchronizing data in real time based on a vision-based keypoint of a cobot at a preset time point.

In one embodiment of the present invention, after photographing a collaborative robot in a smart factory with a camera, extracting keypoints for each preset time point, and then synchronizing real-time data at the time when the keypoint occurred based on image data, synchronization between data is achieved. It is possible to obtain high-quality data by elaborately adjusting the data synchronization between each collaborative robot based on vision, and to refine it so that it can be effectively applied to smart factory construction using data generated in actual industrial sites. It is possible to provide a data synchronization service provision system between vision-based collaborative robots that does not stop at simply extracting information from refined data, but sends decision-making back to the actual field so that it can be used for simulation to help production. there is. However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, an embodiment of the present invention collects real-time data from a user terminal outputting synchronized real-time data between at least one collaborative robot and at least one collaborative robot. a collection unit that captures at least one camera; a recording unit that photographs at least one cobot using at least one camera; a synchronization unit that synchronizes real-time data of at least one cooperative robot based on image data captured by at least one camera; and a synchronization service providing server including a delivery unit for delivering synchronized real-time data to a user terminal.

According to any one of the above-described problem solving means of the present invention, after photographing a collaborative robot in a smart factory with a camera, extracting keypoints for each preset time point, and then synchronizing real-time data at the time when the keypoint occurred based on the image data, data data Synchronization is performed between them, and data synchronization between each cobot is elaborately matched based on vision so that high-quality data can be obtained, and data generated in actual industrial sites can be used effectively to build a smart factory. It does not stop at simply extracting information from refined, well-refined data, but sends decisions about it back to the field so that it can be used in simulations that help production.

1 is a diagram for explaining a data synchronization service providing system between vision-based collaborative robots according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a synchronization service providing server included in the system of FIG. 1 .
3 and 4 are diagrams for explaining an embodiment in which a data synchronization service between vision-based collaborative robots is implemented according to an embodiment of the present invention.
5 is an operational flowchart illustrating a method of providing a data synchronization service between vision-based collaborative robots according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

As used throughout the specification, the terms "about", "substantially", etc., are used at or approximating that value when manufacturing and material tolerances inherent in the stated meaning are given, and do not convey an understanding of the present invention. Accurate or absolute figures are used to help prevent exploitation by unscrupulous infringers of the disclosed disclosure. The term "step of (doing)" or "step of" as used throughout the specification of the present invention does not mean "step for".

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware. On the other hand, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or configured to reproduce one or more processors. Thus, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, device, or device may be performed instead by a server connected to the terminal, device, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is the terminal's identifying data. can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a data synchronization service providing system between vision-based collaborative robots according to an embodiment of the present invention. Referring to FIG. 1 , a data synchronization service providing system 1 between vision-based collaborative robots may include at least one user terminal 100, a synchronization service providing server 300, and at least one camera 400. . However, since the system 1 for providing data synchronization service between vision-based collaborative robots of FIG. 1 is only one embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 .

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1 , at least one user terminal 100 may be connected to a synchronization service providing server 300 through a network 200 . Also, the synchronization service providing server 300 may be connected to at least one user terminal 100 and at least one camera 400 through the network 200 . In addition, at least one camera 400 may be connected to the synchronization service providing server 300 through the network 200 .

Here, the network refers to a connection structure capable of exchanging information between nodes such as a plurality of terminals and servers, and examples of such networks include a local area network (LAN) and a wide area network (WAN: Wide Area Network), the Internet (WWW: World Wide Web), wired and wireless data communications networks, telephone networks, and wired and wireless television communications networks. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi , Internet (Internet), LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth (Bluetooth) network, NFC ( A Near-Field Communication (Near-Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc. are included, but not limited thereto.

In the following, the term at least one is defined as a term including singular and plural, and even if at least one term does not exist, each component may exist in singular or plural, and may mean singular or plural. It will be self-evident. In addition, the singular or plural number of each component may be changed according to embodiments.

At least one user terminal 100 may be a terminal of a user or manager that outputs real-time data in which data is synchronized between collaborative robots using a web page, app page, program, or application related to a data synchronization service between vision-based collaborative robots. there is.

Here, at least one user terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. In this case, at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100 is, for example, a wireless communication device that ensures portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet ) may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

The synchronization service providing server 300 may be a server that provides a data synchronization service web page, app page, program, or application between vision-based collaborative robots. In addition, the synchronization service providing server 300 converts 2D data of image data of at least one camera 400 that has taken at least one collaborative robot into 3D data, and receives it from the collaborative robot based on key points for each viewpoint. It may be a server that synchronizes real-time data that is

Here, the synchronization service providing server 300 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser.

The at least one camera 400 may be a camera device that transmits image data to the synchronization service providing server 300 with or without using a web page, app page, program, or application related to a data synchronization service between vision-based collaborative robots. there is.

Here, at least one camera 400 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. In this case, at least one camera 400 may be implemented as a terminal capable of accessing a remote server or terminal through a network. The at least one camera 400 is, for example, a wireless communication device that ensures portability and mobility, and includes navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet) It may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

2 is a block diagram illustrating a synchronization service providing server included in the system of FIG. 1, and FIGS. 3 and 4 are an embodiment in which a data synchronization service between vision-based collaborative robots is implemented according to an embodiment of the present invention. It is a drawing for explaining an example.

Referring to FIG. 2 , the synchronization service providing server 300 includes a collection unit 310, a photographing unit 320, a synchronization unit 330, a transmission unit 340, a coordinate matching unit 350, and a keypoint unit 360. ), a time synchronization unit 370, an impulse extraction unit 380, a dimension conversion unit 390 and a protocol unit 391.

The synchronization service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one user terminal 100 and at least one camera 400 transmits data between vision-based collaborative robots. In the case of transmitting a synchronization service application, program, app page, web page, etc., at least one user terminal 100 and at least one camera 400, data synchronization service application, program, app page, You can install or open web pages, etc. Also, a service program may be driven in at least one user terminal 100 and at least one camera 400 by using a script executed in a web browser. Here, the web browser is a program that allows users to use the web (WWW: World Wide Web) service, and means a program that receives and displays hypertext described in HTML (Hyper Text Mark-up Language). For example, Netscape , Explorer, Chrome, and the like. In addition, an application means an application on a terminal, and includes, for example, an app running on a mobile terminal (smart phone).

Referring to FIG. 2 , a basic concept of converting 2D data into 3D data will be described before describing FIG. 2 . The description below is not repeated while describing FIG. 2 .

<Point cloud>

A point cloud is a set of points belonging to a certain coordinate system. In a 3D coordinate system, a point is usually defined as (X, Y, Z), and a 2D coordinate system can be converted to a 3D coordinate system by applying a point cloud to a depth image. First, in order to convert the depth image, which is a 2D coordinate system, into a 3D coordinate system, the focal length, the coordinates of the 2D depth image, and the depth value among the internal parameter information of the camera that took the depth image are used. Here, the focal length is the distance from the camera lens to the image sensor. For example, when a camera photographs a sphere, the light reflected from the sphere passes through the lens. Here, the light passing through the lens is reflected on the image sensor and a 2D depth image of the sphere is obtained. The 2D coordinate system should be converted from the 2D coordinate system to the 3D coordinate system using the 2D depth image. Here, you can simply convert using the Similar Triangles Rule. f is the focal length, u is the coordinate where the object is located in the depth image, the origin of the coordinate is the midpoint of the image, not the top left of the image, z is the depth value corresponding to the 2D coordinate in the 2D depth image, and x is the 3D Assuming that it is the X-axis coordinate of the object when converted to the coordinate system, by substituting this into Equation 1 below, the value obtained by dividing the distance (u) in the x-axis direction from the midpoint of the 2D image to the pixel in the x-axis direction by the focal length (f) is, Since it is equal to the value obtained by dividing the x-coordinate by the depth value in the 3D coordinate system, the x-coordinate can be estimated. Therefore, u of the 2D depth image can be converted into 3D coordinates on the X axis, and the 2D coordinate system can be converted into a 3D coordinate system by repeating the above method for the Y axis.

<Coordinate Conversion>

Coordinate transformation is the transformation of the location of one point into another location on a coordinate system. In computer vision, a matching relationship between multiple images can be modeled using rotation transformation, enlargement transformation, and translation transformation.

<3D Coordinate Conversion>

3D coordinate transformation is transformation of coordinate axes in 3D space. 3D coordinate transformation can be used by changing the 3D world coordinate system into the camera coordinate system or by converting the camera coordinate system into the 3D world coordinate system. Y1,Z1) is the purpose of finding the point (X,Y,Z) of the world coordinate system. In the 3D world coordinate system, the X-axis and Y-axis planes represent the ground, and the Z-axis is defined perpendicular to the plane. The camera coordinate system is defined as the Zc axis in the direction of the optical axis of the camera, the Xc axis in the vertical direction to the right of the optical axis, and the Yc axis in the vertical direction between the Xc axis and the Zc axis.

The rotation transformation rotates a point (X, Y, Z) in 3D space by θ around the X, Y, and Z axes in a counterclockwise direction based on the direction of each axis using a rotation matrix. Rotation matrices for rotating each axis are as shown in Equations 2, 3 and 4. Equation 2 is a matrix for rotating around the X axis, Equation 3 is a matrix for rotating around the Y axis, and Equation 4 is a matrix for rotating around the Z axis. An arbitrary 3D rotation like Equation 5 can be expressed by combining the rotation matrices of Equations 2 to 4.

The movement transformation is a rotation transformation to convert a point (XC, YC, ZC) on the camera coordinate system to a point (X, Y, Z) on the world coordinate system, and then the camera coordinate system is moved in the direction of the world coordinate system using Equation 6. convert

Referring to FIG. 2 based on the above-described basic concept, the collection unit 310 may collect real-time data from at least one cobot. At this time, real-time data may include various types of data, but, for example, ① position data, ② speed data, ③ torque data, and ④ end-effector data. Data), ⑤ other monitoring data (Monitoring Data), ⑥ device data (Device Data), ⑦ joint data (Joint Data), ⑧ sensor data (Sensor Data), etc.

① Position data refers to the position of each joint of the cobot. Depending on the point of view, it is largely divided into angular position data and coordinate position data. While angular position data indicates the angle at which the motor is twisted by motion, coordinate position data means that the position of the axis to be observed is expressed as a 3-dimensional coordinate value, with the position of axis 1 where the cobot is mounted as the origin. . This location data is used as basic data because it is easy to check the movement of each joint of the cobot. ② Speed Data means the differential value of the position data analyzed in ① according to the time axis. Therefore, when the angular position is differentiated, the instantaneous angular velocity (°/sec or rad/sec) is obtained, and when the coordinate position is differentiated, the instantaneous velocity value (x/sec, y/sec, z /sec) can be obtained. In addition, when the velocity data is differentiated once again along the time axis, acceleration data can be obtained. In the case of such speed and acceleration data, the time taken to reach the same position can be compared and analyzed, and sensitive differences that occur when there are defects can be detected, so they are utilized in various fields.

③ Torque Data performs the motion of each joint of the cobot and represents the rotational force applied to the joint. According to the measurement method, it is divided into current-based torque and sensor-based torque data. Current-Based Torque represents a rotational force obtained by multiplying a torque constant based on a current value applied to a motor, and represents a force applied inside a joint. On the other hand, sensor-based torque means a torque value sensed from a joint by installing a torque sensor outside the joint. When an external force acts on such a sensor-based torque, the value varies greatly and is sometimes used to detect a collision. Lastly, in the case of vibration data, it represents data measured by mounting a vibration sensor outside the joint, similar to the sensor-based torque. However, in the case of vibration data, it has a limitation that it is sensitive to external forces and is vulnerable to noise. ④ End-effector data refers to data that can be collected according to the end-effector mounted on the end-axis of the cobot. Types of end effectors include grippers that can perform pick and place operations, vacuum pumps, and the like. Accordingly, when each task is performed, there may exist weight and pressure sensor data that need to be measured. Such data is used as basic sensor data to check the performance evaluation of the collaborative robot.

⑤ Other monitoring data refers to data that does not directly affect the cobot's motion, but requires continuous monitoring. For example, in the case of temperature, although it does not have a close relationship with operation, it is very likely that high temperature due to overheating of the motor will be measured if unreasonable execution or an error situation occurs. In addition, in the case of power, a high power load situation can be measured with the power value applied to the motor or brake. In other words, data such as temperature and power are less closely related to cobot operation, but can be directly related to faults or failure situations. Therefore, continuous monitoring is required, and most cobots are equipped with temperature and power measurement equipment for safety and risk management.

Collaborative robots have 6 or 7 joints according to degree of freedom (DoF), and various sensors such as torque, speed, and position sensors are built-in for each joint. The detailed schema of the sensor data model is largely divided into ⑥ devices, ⑦ joints, and ⑧ sensors. ), device name (Name), axis degree of freedom (DOF), manufacturer (Manufacturer), production date (Manufacturing Date), and development version (Version) information are included. ⑦ Looking at the joint data, each joint of the cobot has a defined physical range that can be driven. Joint identification number (ID), load (Payload), maximum speed (MaxSpeed), radius of motion (Moving Range), and maximum torque (Peak Torque) are included. ⑧ Looking at sensor data, each joint can have various sensors such as position, speed, torque, temperature, pressure, and vision. Sensor data may be generated according to characteristics of each type of sensor, and includes a sensor identification number (ID), a sensor name (Name), and a measurement value (Measurement). In this case, the measured value field is a field in which a value sensed in real time is recorded. Although these sensor data can be collected in real time according to the execution of the cobot, it is difficult to check operation information on the robot's work, program, and motion. Therefore, a log (operation information, program execution information, motion execution information, error information, etc.) schema that can be interpreted by linking the operation and operation information of the cobot with respect to sensor data is vision-based synchronization according to an embodiment of the present invention. available for Log information may be non-periodically (intermittently) generated when events such as operation, execution of programs and motions occur.

<data interface>

An interface refers to a shared boundary that exchanges information between two or more independent system components. Specifically, it means a device or program that supports communication between components. Among them, Data Interface refers to a data property template used to create a process for reading or writing data from a data source. For example, the data collected while performing one unit task can be divided into robot operation units (from BootOn to Off), executed program units (from ProgramOn to Off), and executed motion units (from MotionOn to Off). can This process is a process of precisely interpreting the collected data, and is a necessary attribute to classify data in the same execution situation in a cobot. At this time, in an embodiment of the present invention, a motion data interface may be used by performing synchronization based on the position analyzed based on vision. The motion data interface is a data attribute for identifying execution motion information of the cobot, and represents, for example, templates such as motion execution time, motion name, current position, target position, and motion end time. Position, speed, torque, end effector, and monitoring data may be collected in the data collection layer for collecting data from sensors built into the collaborative robot, such as gyro sensors, current sensors, pressure sensors, and voltage sensors.

The photographing unit 320 may photograph at least one cooperative robot using at least one camera 400 . In this case, the camera 400 may be a general camera, but may also be a depth camera or a stereo camera.

The synchronizing unit 330 may synchronize real-time data of at least one cobot based on image data captured by at least one camera. A detailed description thereof will be described later.

The delivery unit 340 may deliver the synchronized real-time data to the user terminal 100 . The user terminal 100 may output synchronized real-time data between at least one collaborative robot.

The coordinate matching unit 350 detects an object for each preset viewpoint of at least one cobot captured by at least one camera, and sets the XYZ coordinates of the detected object to the XYZ coordinates in real-time data collected from at least one cobot. After extracting the coincident point in time, real-time data of at least one cobot may be synchronized with a preset point in time. In this case, the XYZ coordinates in the real-time data collected from the cobot may be position data, and the position data means the position of each joint of the cobot. Broadly, it is divided into angular position data and coordinate position data. Angular position data indicates the angle at which the motor is twisted by motion. On the other hand, the coordinate position data means that the position of the axis to be observed is expressed as a three-dimensional coordinate value with the position of axis 1 on which the collaborative robot is mounted as the origin. Since this positional data is easy to check the movement of each joint of the cobot, it is set as real-time data of [video data-real-time data] corresponding to [reference value-comparison value].

reference value video data comparative value real-time data

The keypoint unit 360 may detect an object centered on a keypoint, which is a preset part of at least one collaborative robot, when detecting an object for each preset viewpoint. At this time, general keypoint detection comes from a study in which a computer understands human behavior through keypoint detection in a task that was directly judged by human eyes to understand human behavior in images or videos. An attempt was made to apply deep neural networks to the field of keypoint detection. As a result, it is possible to detect 2D keypoints with high accuracy and speed using RGB images. In this case, a key point is defined as a connection part of each joint in the case of a person, for example, a knee, an elbow, a neck connecting a face and a body, a foot, and the like. When this is applied to a collaborative robot, it can correspond to an axis (joint) where each motion occurs. Of course, other keypoints can be set even for each axis. At this time, 2D keypoint detection detects (X,Y) of each keypoint when detecting keypoints, while 3D keypoint detection detects (X,Y,Z) coordinates of each keypoint in 3D space when detecting keypoints. do. Detecting (X,Y,Z) of each keypoint using 3D keypoint detection can detect keypoints in more detail than estimating (X,Y) for each keypoint of 2D keypoint detection. The keypoint unit 360 may detect a keypoint based on a marker of at least one cobot. For example, when obtaining a feature point of a face, points such as both ends of the lips, both ends of the eyes, and the end point of the width of the nose can be determined. Although is specified, since it is analyzed based on the image, markers can be attached to each joint part, such as AR (Augmented Reality) markers, in order to find them more quickly and easily with less computational effort in the image.

The time synchronization unit 370 drives at least one cobot to adjust the difference between the recording start time of image data received from at least one camera and the collection start time of real-time data collected from at least one cobot. By setting a predetermined trajectory to be drawn at the start of the process, a point in time of drawing a preset trajectory within real-time data can be found and matched with the collection start time. For example, a synchronization point between real-time data (sensor data) and video data can be created, such as synchronizing each camera by clapping at the start of each scene in a broadcast. For example, when the cobot rapidly rotates left and right, the speed collected from the acceleration sensor may draw an impulse shape. Based on this impulse shape, synchronization between real-time data and video data can be performed. The impulse extraction unit 380 may extract sensor data from among real-time data in which an impulse is generated according to a predetermined trajectory, and may synchronize the real-time data of the cobots based on the impulse.

The dimension conversion unit 390 may convert 2D data, which is image data captured by at least one camera, into 3D data to obtain trajectory data in 3D. The dimension conversion unit 390 may use a rotation matrix and a projection matrix when converting 2D data into 3D data. Between the real space and the image plane obtained by shooting with a camera, there exists a correspondence that can be explained by rotation transformation, movement transformation, and projection transformation. Knowing the relationship between four points in space and four points on the image, one can determine the transformation required to describe the correspondence. That is, if only the coordinates of the camera 400 and the four points on the object are known, the 3D positioning of the collaborative robot is possible, and path data and trajectory data can be converted to XYZ coordinates, that is, 3D, without expensive lasers, lidars, depth cameras, etc. can be extracted.

First, a point in a 3D space in the camera model of the camera 400 and a point in an image obtained by the camera 400 are explained by the relationship shown in Equation 7.

First, coordinates on the world coordinate system are converted to coordinates on the camera coordinate system by sequentially applying scale transformation, rotation transformation, and translation transformation. Through this process, the matrix [R|t] of Equation 7 can be obtained.

Since the scale is not different between the world coordinate system and the camera coordinate system, the scale transformation matrix becomes the identity matrix. Next, in the case of the rotation matrix, even if the same angle is rotated, the result of the rotation matrix varies depending on which axis is rotated first, so assuming that the XYZ axes are rotated by φ, θ, and ψ in order, the rotation matrix as shown in Equation 8 is expressed In the case of movement transformation, the vector heading from the origin of the camera coordinate system to the origin of the world coordinate system is added in the 4-dimensional homogeneous coordinate system. When a linear matrix is obtained by synthesizing these information, it can be seen that it can be expressed as [R|t] in Equation 7. H in Equation 7 serves to make a 4-dimensional homogeneous coordinate system into a 3-dimensional real coordinate system. Since the homogeneous coordinate system is projected from the real space to the real space of the normalized image plane, it can be expressed as the matrix F of Equation 7.

In order to determine the position and attitude of the camera 400, it is necessary to know the focal length of the camera, the size of one pixel on the image sensor, the three-dimensional coordinates of four points in the world coordinate system, and the corresponding two-dimensional coordinates in the image. . If the focal length is known, matrix F in Equation 7 can be determined. Since the coordinates on the image are expressed in pixels, multiply the pixel size to enter the coordinates in m units. To express the rotation matrix, three components of the rotation reference vector, rotation angle, and at least four pieces of information are required. Therefore, a total of 7 elements must be obtained from the three elements tx, ty, and tz of the translation matrix and the four elements of the rotation matrix. Since two equations can be obtained for each coordinate pair consisting of world coordinates and corresponding pixel coordinates, if the four coordinates of the world coordinate system correspond to the image, eight equations can be obtained, which is sufficient to solve the equations. By solving the equation, the matrix [R|t] of Equation 7 can be obtained. As a result of solving the equation, all values of Equation 7 can be found. The position of the camera relative to the object can be known through the coordinates of the origin of the camera coordinate system on the world coordinate system. The attitude of the camera can be obtained as in Equation 10 from the matrix of Equation 9. Accordingly, the position and attitude of the camera with respect to the world coordinate system set for the object can be obtained.

In addition to this, it goes without saying that 2D data can be converted into 3D data in various ways. If more precise coordinate measurement is required, additional laser equipment can be used.

The protocol unit 391 may use at least one time synchronization protocol for building a smart factory with respect to real-time data collected from at least one collaborative robot. At this time, at least one time synchronization protocol includes ① OPC UA Server/Client, ② TSN real-time communication technology, ③ NTP & PTP time synchronization protocol, ④ field bus protocol, and ⑤ synchronization protocol for data transmission delay. It can, but is not limited to those listed and not excluded for reasons not listed. ① OPC UA (Open Platform Communications-Unified Architecture) Server/Client uses the OPC UA protocol to enable independent configuration that is not dependent on the platform and provides a stable communication mechanism implemented inside the protocol, so Industry 4.0 can be implemented. It is the most ideal communication protocol for Functions implemented in the OPC UA protocol to build a platform environment required by industrial sites may be shown in Table 2 below.

No. Item Specification One Core Specification Part 1: Concepts Part 2: Security Model Part 3: Address Space Model Part 4: Services Part 5: Information Model Part 6: Service Mapping Part 7: Profiles 2 Access Type Specification Parts Part 8: Data Access Part 9: Alarms and Conditions Part 10: Programs Part 11: Historical Access 3 Utility Type Specification Parts Part 12: Discovery Part 13: Aggregates

② TSN (Time-Sensitive Networking) real-time communication technology is a technology that provides deterministic service with low delay, low delay variation, and low packet loss based on Layer 2 (L2) Ethernet. By adopting OPC UA, functions such as accurate timing sharing between high and low speed devices, synchronization of equipment through minimization of transmission delay, synchronization through minimization of communication delay and loss, etc. ③ NTP (Network Time Protocol) & PTP (Precision Time Protocol) Time synchronization protocol is a protocol implemented to synchronize time between devices when multiple embedded systems are connected through a network. It operates as a daemon in the operating system and performs synchronization with other devices through the network. It follows the IEEE1588 standard and synchronizes periodically for precise time synchronization. Synchronize. For more precise time synchronization, a master operation module and a slave operation module are required, and BMC (Best Master Clock) algorithm can be introduced and implemented to correct the time error according to the clock offset. After the tracking path for the Master-Slave layer is established, synchronization is performed, and event messages such as Sync, Delay_Req, and Delay_Resp are exchanged to implement frequency or time synchronization, and the calculated temporal compensation value is applied to the system. It is a method of synchronizing time.

④ In order to convert various protocols used in industrial sites into OPC UA protocols, conversion modules for various protocols such as Modbus TCP, Modbus RTU (Remote Terminal Unit), Genibus, Mitshbishi MC, OPC UA can be used. The Field Bus protocol is converted to the OPC UA protocol and interworks with the management system through a conversion module for data exchange with other PLC (Programmable Logic Controller) devices or industrial devices. Although each PLC module uses a different industrial protocol, data can be exchanged in the mutual FieldBus Network through the conversion module. ⑤ On the other hand, RAM and flash memory for data storage are used to provide a solution to data transmission delay and data loss due to network errors or failure to connect to the server due to a very large problem of communication devices. Therefore, a data storage technology in the form of an efficient message in volatile/non-volatile memory has been developed. A storage technology and synchronization protocol for data transfer delay can support both transactions and distributed transactions using a Message Queue messaging system. . In addition, data generated from sensors and devices is temporarily stored in an extended memory area by attaching a tag of the date and time of creation, and data loss is minimized by storing data in non-volatile memory in preparation for long-term data transmission delays, and server connection This is a synchronization protocol that minimizes loss by transmitting data according to the retransmission and synchronization protocol when this is detected. In a situation where the clocks are synchronized between the OPC UA server and the client and data is being exchanged through the mutual interface, a problem occurs in the network, for example, the power is turned off or a problem occurs in the communication line. In order to realize the function of saving the latest 100 log contents in real time in case of interruption due to physical external factors of the system and restoring them to restore them to the same situation as the conventional network environment, the log data format of the flash memory is changed. to provide.

In addition, in one embodiment of the present invention, a data pipeline (PipeLine) can be further used to process data pouring from the smart factory. The reason for synchronization between data in an embodiment of the present invention is to generate meaningful information and then use it, so real-time data is collected (Acquisition), automatic labeling (Auto Labeling), compression (Compression), and simulation (Simulation). ), a further four-stage data pipeline is available. In the first step, the collection step, operation data is indirectly collected during process operation and synchronized with the sensor data collected through the aforementioned OPC-UA). In the second step, the automatic labeling step, when sensor data is collected in the future, the classification algorithm is learned so that operation data, equipment conditions, and processes can be automatically synchronized and labeled. The third step, the compression step, effectively compresses the collected data based on deep learning, and then, when the fourth step, the simulation operation, generates an expected profile based on various process plans to search for optimal conditions. there is.

<Collection step>

When the machine is in operation, it always moves according to the commands of the predetermined program and achieves its purpose by taking different actions for each command. Therefore, rather than simply collecting sensor data, the quality of data can be further improved if the cause information of the generated sensor data is stored together by converting the operation command that generated the sensor data into data. Machines can secure this operation information by converting it into data through the provided open channel, but in many cases, such a channel does not exist for old equipment, and when the machine manufacturer intentionally does not support open protocol even when using new equipment There are many. In this situation, retrofitting method can be used to obtain operation information. Retrofitting is a method of indirectly connecting and managing equipment information in a situation where data cannot be obtained because the communication protocol is not provided in the machine itself. It transmits information that could not be obtained by attaching sensors and communication equipment to the machine. The method commonly used for CPPS that makes it possible also belongs to a wide category of retrofitting.

Accordingly, in order to collect operation data, a human machine interface (HMI) screen in which communication is impossible may be recorded and stored as video data. Then, a method of extracting continuous image data listed in chronological order and applying an OCR (Optical Character Recognition) algorithm to obtain operation data in a string format can be used. Different criteria may be used for each moment used to apply the OCR algorithm, which may cause problems with accuracy and speed. By predicting the information, we need an algorithm to solve the accuracy and speed problems that will arise during the collection phase.

<Auto Labeling>

In the automatic labeling step, a classification model can be used to solve this problem. The dataset built in the automatic labeling step is formed in such a way that operation information is paired with continuous sensor data. Since operation information can be predicted, a multimodal dataset can be built by omitting the operation information collection process and merging process. Therefore, in the automatic labeling step, when sensor data is input using the dataset generated in the collection step, a classification model that returns whether the machine is operating or not and a classification model that returns operation information can be used.

In most facilities, the physical quantity generated when an operation is commanded does not show a clear correlation with the operation. This is because there are not many users or administrators who are aware of the entire complex mechanism inside the machine, and it is affected by various factors that are difficult for humans to understand other than the input operation. Therefore, deep learning can be used as a classification model to find hidden correlations that cannot be known by classical statistical techniques. Deep learning is specialized in the role of identifying correlations that are difficult for humans to grasp between given data and labels matched to the data through artificial neural networks made by imitating human neural networks. In particular, unlike classical classification models, the model can be flexibly adjusted according to the shape of input and output data and shows high accuracy.

In the automatic labeling step, a model that uses sensor data as input data and whether or not a machine is operated during the input sensor time period as output data and a model that uses commanded operation output data can be used. The sensor data is time-series data in which the meaning of the values of the current time zone changes according to the values of the previous time zone. Accordingly, CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) based deep learning can be used to well understand this temporal correlation. Hyperparameters such as the number of layers, number of filters, and activation function can be adjusted, and various evaluation scales can be compared and analyzed to find the optimal deep learning model for each layer type. The trained and verified deep learning model plays a role of automatically generating operation information when sensor data is input later, enabling the construction of a multimodal dataset that can be analyzed more quickly and with various high values.

<compression step>

In the manufacturing field, a huge amount of data is produced every hour. In particular, as more and more types of precise sensors are used, the amount of data increases proportionally. Currently, many companies lack data processing capabilities such as server expansion to keep pace with the development of IoT hardware such as sensors. Therefore, storage space is secured by deleting data or lowering the data resolution after a certain period of time. However, in the end, this method leads to a loss of information, and significantly lowers the quality along with the amount of data, so that it cannot bring good results when applied to the algorithm required for CPPS construction in the future. Therefore, it is possible to use a method of reducing only the volume of data while maximizing the characteristics of the data.

In the automatic labeling step, the deep learning model received the sensor data as a problem, and the operation data served as a ground truth that helped the deep learning model learn the characteristics of the sensor data. In this way, the case where a label serving as an answer sheet in a machine learning algorithm is given is supervised learning, while the case where the characteristic is learned through an algorithm only with input data without a label is classified as unsupervised learning. The algorithm learned through supervised learning plays a role in predicting the corresponding answer when input data is entered later, and unsupervised learning identifies the characteristics of the input data itself and performs clustering or recognizing patterns and extracting features. play a role

In the compression step, an algorithm that identifies a pattern of sensor data using the characteristics of unsupervised learning and extracts characteristics may be used. Among deep learning algorithms, the representative model that plays this role is an autoencoder. The autoencoder aims to learn the characteristics of the data by putting the same data as the input and output of the model, and then to generate the same data as similarly as possible through the learned characteristics. At this time, if the middle part of the stacked deep learning layer is designed to be output in a lower-dimensional form than the input, it can intentionally add an information bottleneck and play a role in compressing information. Utilizing these characteristics of autoencoder, it can be used to compress data.

Each layer of the autoencoder model is stacked in such a way that the dimension gradually decreases and returns to the original dimension after the intermediate point. At this time, the part before the midpoint is called an encoder, the part after the midpoint is called a decoder, and the compressed data output at the midpoint is called a latent vector. Therefore, since this latent vector is not a result compressed by simply deleting the amount of information, but a result compressed with the characteristics maintained by a deep learning model, it can be used as effective compressed data to replace existing data, and the storage used It can serve as a space saver. At this time, not only the compressed data is stored, but also the weight of the learned decoder part must be stored together so that the data can be restored to its original size through the decoder part later and used together with the data collected in real time.

<Simulation step>

If the physical quantity sensed by the sensor can be simulated in advance, the increase in process efficiency can be expected and the occurrence of abnormalities can be predicted. In addition, these simulation functions can assist field workers when performing machine operation. Therefore, in the simulation step, when a machine executes a specific program and inputs a list of operations to be commanded in sequence, a model predicting what value the sensor will return can be used after training. The simulation step can also use a deep learning model like the automatic labeling step and compression step. In the automatic labeling step, if a classification model was learned and used with sensor data, which is continuous numerical data, as input, and operation data, which is discrete categorical data, as output, operation data is queried as input in the simulation step, on the contrary, and sensor data is queried as input. is used as the output, and the model can use a regression model that solves the problem of predicting continuous numerical data.

Hereinafter, an operation process according to the configuration of the above-described synchronization service providing server of FIG. 2 will be described in detail with reference to FIGS. 3 and 4 as examples. However, it will be apparent that the embodiment is only any one of various embodiments of the present invention, and is not limited thereto.

Referring to FIG. 3 , (a) a synchronization service providing server 300 receives image data from at least one camera 400 and collects real-time data from at least one collaborative robot. (b) And the synchronization service providing server 300 extracts key points for each viewpoint in the video data, compares the real-time data, and when the time delay (Δt) is + as shown in (c), -Δt is Data synchronization between real-time data can be performed by sorting the data as much as possible. In addition, as shown in (a) of FIG. 4, for synchronization between the video recording time and the real-time data collection time, the collaborative robot is operated or the environment is changed so that the impulse appears in the real-time data, and the place where the impulse appears is based on the image. Synchronization can also be performed with As in (b), 2D data, which is image data, can be collected as 3D data through coordinate transformation, and data sync can be matched between each IoT device or collaborative robot existing in the smart factory using a protocol as in (c). .

Matters not described in the method for providing data synchronization services between vision-based collaborative robots of FIGS. 2 to 4 are the same as those described for the method for providing data synchronization services between vision-based collaborative robots through FIG. 1 or described above. Since it can be easily inferred from the contents, the following description will be omitted.

5 is a diagram illustrating a process of transmitting and receiving data between components included in the system for providing data synchronization service between vision-based collaborative robots of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process of transmitting and receiving data between each component will be described through FIG. 5, but the present application is not limited to such an embodiment, and according to various embodiments described above, It is obvious to those skilled in the art that a process of transmitting and receiving data may be changed.

Referring to FIG. 5 , the synchronization service providing server collects real-time data from a user terminal outputting synchronized real-time data between at least one collaborative robot and at least one collaborative robot (S5100).

In addition, the synchronization service providing server photographs at least one cobot using at least one camera (S5200), and synchronizes the real-time data of the at least one cobot based on the image data captured by the at least one camera. It is performed (S5300).

In addition, the synchronization service providing server transmits the synchronized real-time data to the user terminal (S5400).

The order between the above-described steps (S5100 to S5400) is only an example, and is not limited thereto. That is, the order of the above-described steps (S5100 to S5400) may be mutually changed, and some of the steps may be simultaneously executed or deleted.

Matters not described in the method for providing data synchronization services between vision-based collaborative robots in FIG. 5 are the same as those described for the method for providing data synchronization services between vision-based collaborative robots through FIGS. 1 to 4, or described above. Since it can be easily inferred from the contents, the following description will be omitted.

The method for providing a data synchronization service between vision-based collaborative robots according to an embodiment described with reference to FIG. 5 is also implemented in the form of a recording medium containing instructions executable by a computer, such as an application or program module executed by a computer. It can be. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The method for providing a data synchronization service between vision-based collaborative robots according to an embodiment of the present invention described above is based on an application basically installed in a terminal (this may include a program included in a platform or operating system basically installed in the terminal) It can also be executed by an application (i.e., a program) that a user directly installs in the master terminal through an application providing server such as an application store server, an application or a web server related to the corresponding service. In this sense, the above-described method for providing data synchronization service between vision-based collaborative robots according to an embodiment of the present invention is implemented as an application (i.e., a program) that is basically installed in a terminal or directly installed by a user, and is implemented as a computer can be recorded on a readable recording medium.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (10)

A user terminal outputting synchronized real-time data between at least one collaborative robot; and
A collection unit that collects real-time data from the at least one cobot, a photographing unit that photographs the at least one cobot using at least one camera, and transmits real-time data of the at least one cobot to the at least one camera. A synchronization unit performing synchronization based on captured image data, a delivery unit transmitting the synchronized real-time data to the user terminal, and detecting an object for each preset viewpoint of at least one collaborative robot captured from the at least one camera. and extracting a time point at which the XYZ coordinates of the detected object in the real-time data collected from the at least one cobot coincide with the XYZ coordinates of the detected object, and then synchronizing the real-time data of the at least one cobot with the preset time point. a synchronization service providing server including a unit;
A data synchronization service providing system between vision-based collaborative robots.
delete According to claim 1,
The synchronization service providing server,
a keypoint unit configured to detect the object around a keypoint, which is a preset part of the at least one cooperating robot, when detecting an object for each preset viewpoint;
A data synchronization service providing system between vision-based collaborative robots, characterized in that it further comprises.
According to claim 3,
The key point part,
A system for providing a data synchronization service between vision-based collaborative robots, characterized in that the keypoint is detected based on a marker of the at least one collaborative robot.
According to claim 1,
The synchronization service providing server,
In order to adjust the difference between the recording start time of the image data received from the at least one camera and the collection start time of the real-time data collected from the at least one cobot, when the at least one cobot starts driving, a predetermined setting is set. a time synchronizing unit for setting a trajectory to find a point in time of drawing a preset trajectory within the real-time data and matching it with the collection start time;
A data synchronization service providing system between vision-based collaborative robots, characterized in that it further comprises.
According to claim 5,
The synchronization service providing server,
an impulse extraction unit extracting sensor data among real-time data in which an impulse is generated according to the preset trajectory and synchronizing the real-time data of the cobot based on the impulse;
A data synchronization service providing system between vision-based collaborative robots, characterized in that it further comprises.
According to claim 1,
The synchronization service providing server,
a dimensional conversion unit converting 2D data, which is image data captured by the at least one camera, into 3D data to acquire trajectory data in 3D;
A data synchronization service providing system between vision-based collaborative robots, characterized in that it further comprises.
According to claim 7,
The dimension conversion unit,
A data synchronization service providing system between vision-based collaborative robots, characterized in that using a rotation matrix and a projection matrix when converting the 2D data to 3D data.
According to claim 1,
The synchronization service providing server,
A protocol unit for using at least one time synchronization protocol for building a smart factory with respect to real-time data collected from the at least one collaborative robot;
A data synchronization service providing system between vision-based collaborative robots, characterized in that it further comprises.
According to claim 9,
The at least one time synchronization protocol,
OPC UA Server/Client, TSN real-time communication technology, NTP & PTP time synchronization protocol, Field Bus protocol, and data synchronization service provision system between vision-based collaborative robots, characterized by including synchronization protocol for data transmission delay .

Images