KR20240032260A - Anti-spilling system for liquid delivery robot - Google Patents

Abstract

The present invention relates to a liquid movement detection system for a delivery robot, and a liquid delivery robot having the detection system. More specifically, it is a system provided on a robot that delivers liquid and detects the liquid state during delivery. at least one liquid container placed on a tray; At least one camera that acquires an image by photographing the liquid container; It relates to a liquid movement detection system for a delivery robot, comprising a liquid state analysis unit that analyzes the image to detect the liquid state.

Description

Liquid movement detection system for delivery robot, and liquid delivery robot having the detection system {Anti-spilling system for liquid delivery robot}

The present invention relates to a liquid movement detection system for a delivery robot, and a liquid delivery robot having the detection system.

Restaurant service robots provide food delivery services. Using LiDAR and camera sensors, food is transported autonomously to the location of the table where it needs to be transported. It also has a function to stop when an obstacle occurs in the path. When a person moves food to the table, the weight of the tray decreases and it also has a return function that automatically returns to its original position after detecting that food delivery is complete. However, since there is no detection function for liquid sloshing, liquid may spill.

Although not currently being applied to restaurant service robots, liquid state detection systems are being developed research-wise. However, existing liquid state detection systems indirectly estimate the degree of liquid sloshing using distance sensors and accelerometers, so they have the limitation of being usable only in simple environments.

Current commercial service robots can be used in limited spaces where mapping has been completed. The focus is on performing missions while moving autonomously, avoiding obstacles within predictable terrain, while moving at a slow speed of about 1 m/s on a flat surface without thresholds. Therefore, it does not have a control function to prevent the food or drinks being transported from spilling.

Therefore, current food delivery robots spill liquid when there is a sudden change in movement speed. To prevent this, a simple control method is used to limit the robot's moving speed. In addition, even if the floor of the restaurant is uneven and the liquid sloshes around, there is a problem in that the liquid spills because the control does not actively change. Additionally, when service robots are used for outdoor food delivery, they must move over numerous obstacles and uneven surfaces. This makes it impossible to maintain a slow, steady movement speed, making liquid food more likely to spill.

Existing systems that detect the state of liquid indirectly estimate the degree of liquid sloshing using distance sensors and accelerometers, but this has the limitation that it is difficult to use to prevent spillage because it oversimplifies and estimates the complex movement of liquid. In addition, because the location of the liquid container must be determined considering the sensor location, the location of the liquid container must be limited to a specific location, and when there are multiple liquid containers, it has the disadvantage of having to use multiple sensors.

In other words, the existing liquid motion detection system estimates the sloshing of the liquid using a distance sensor or acceleration sensor, but this has limitations in detecting complex liquid movement.

Republic of Korea registered patent 10-2391562 Republic of Korea Public Patent No. 10-2019-0009766 Republic of Korea Open Patent No. 10-2016-0126391 Republic of Korea registered patent 10-2242380

Accordingly, the present invention was devised to solve the above-described conventional problems. According to an embodiment of the present invention, the purpose of the present invention is to provide a system for controlling a robot carrying a container containing liquid to prevent the liquid from spilling.

According to an embodiment of the present invention, the purpose is to improve liquid spill prevention performance by more effectively detecting complex movements of liquid using image information.

According to an embodiment of the present invention, by using vision, even if the containers carried by the robot are randomly positioned, individual sloshing and overall sloshing within the container can be recognized, making it possible to detect movement of the liquid surface in more diverse and complex situations. The purpose of this is to provide the configuration and method of a system that implements the necessary control over the liquid surface state.

In addition, according to an embodiment of the present invention, the information on the height difference between the container placed on the tray and the liquid surface and the sloshing of the liquid inside are estimated in real time and the robot is controlled accordingly to control the degree of sloshing of the liquid and prevent liquid spilling. The purpose is to provide a liquid movement detection system for a robot, and a liquid delivery robot having the detection system.

And according to an embodiment of the present invention, it can be applied to transparent containers and liquids, and through the design of this system, even in situations where the speed is adjusted rapidly due to various unexpected situations during the operation of the service robot or when it is transported on uneven ground. The purpose is to provide a liquid movement detection system for a delivery robot that can safely transport liquid cargo without spilling, and a liquid delivery robot with the detection system.

Meanwhile, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly apparent to those skilled in the art from the description below. It will be understandable.

A first object of the present invention is a system provided on a robot that delivers liquid to detect the state of the liquid upon delivery, comprising: at least one liquid container placed on a tray of the delivery robot; At least one camera that acquires an image by photographing the liquid container; It can be achieved as a liquid movement detection system for a delivery robot, comprising a liquid state analysis unit that analyzes the image to detect the liquid state.

In addition, the camera may be installed to photograph an inlet edge of the liquid container and the surface of the liquid stored in the liquid container.

In addition, the liquid state analysis unit may be characterized in that it detects the border of the liquid container inlet using an image-based object tracking algorithm and detects the movement of the liquid within the area using an image.

And the image-based object tracking algorithm detects the border of the liquid container entrance in the image through object detection, detects the border of the liquid surface that appears to be the container entrance through edge detection, and calculates the height difference in the image between the container entrance and the liquid surface. It can be characterized as:

Additionally, the camera may be configured as a DApps camera and convert the height difference in the image into an actual height difference.

In addition, a marking may be displayed on the tray, a size ratio between the marking on the image and the actual marking may be calculated, and the height difference on the image may be converted into an actual height difference based on the size ratio.

And the camera includes a reference camera and a second camera, and each of the reference camera and the second camera acquires images by photographing the inlet edge of the liquid container and the surface of the liquid stored in the liquid container from different directions, , the liquid state analysis unit detects the container inlet border as a bounding box in each image, calculates the size ratio of the size on the image and the actual size through the position difference between the two bounding boxes, and based on the size ratio, It may be characterized by converting the height difference in the image into an actual height difference.

A second object of the present invention is a method for detecting the state of liquid upon delivery by being installed on a robot that delivers liquid, comprising the steps of using a camera to capture an image of the edge of the entrance to a liquid container and the surface of the liquid stored in the liquid container; And a liquid state analysis unit detecting the liquid state by analyzing the image. The step of detecting the liquid state includes detecting the border of the liquid container inlet in the image through object detection, and detecting the liquid surface visible as the container inlet. Detecting the border through edge detection, calculating the height difference in the image between the container inlet and the liquid surface, converting the height difference in the image to the actual height difference, and the height difference exceeding or changing the set height difference. This can be achieved as a method of detecting liquid movement of a delivery robot, which includes the step of determining that it is a control requirement when the set change amount is exceeded.

A third object of the present invention is a liquid delivery robot that delivers liquid, comprising: a frame; Running part; A tray on which at least one liquid container storing liquid is placed; A detection system according to the first purpose mentioned above; and a control unit that controls the driving of the traveling unit based on the liquid state determined by the detection system. It can be achieved as a liquid delivery robot with a liquid movement detection system.

And an angle adjustment unit for adjusting the angle of the tray; further comprising, when the height difference calculated by the detection system exceeds the set height difference or the change amount exceeds the set change amount, the control unit controls the traveling unit to adjust the moving speed. The tray angle may be adjusted by adjusting or controlling the angle adjustment unit.

In addition, the control unit may be characterized in that it controls at least one of the moving speed of the robot and the angle of the tray using a reinforcement learning model that has the maximum speed at which liquid is moved without spilling as a compensation value.

It may further include an IMU sensor that measures the posture of the liquid delivery robot, and the control unit may control the angle adjustment unit so that the tray is maintained horizontal based on tilt data measured by the IMU sensor. .

A fourth object of the present invention is a control method of a liquid delivery robot, comprising the steps of loading and traveling at least one liquid container storing liquid on a tray; Acquiring an image by using a camera to photograph the border of the liquid container entrance and the surface of the liquid stored in the liquid container; The liquid state analysis unit detects the border of the liquid container entrance in the image through object detection, and detects the border of the liquid surface visible as the container entrance through edge detection; Calculating a height difference in the image between the container inlet and the liquid surface, converting the height difference in the image into an actual height difference; Determining that control is required when the height difference exceeds a set height difference or the change amount exceeds the set change amount; and a control unit controlling the moving speed by controlling the traveling unit, or controlling the angle adjusting unit to adjust the tray angle. It can be achieved as a control method of a liquid delivery robot having a liquid movement detection system. there is.

According to the liquid movement detection system of the delivery robot according to an embodiment of the present invention, and the liquid delivery robot having the detection system, a robot carrying a container containing liquid can be controlled so as not to spill the liquid.

According to the liquid movement detection system of the delivery robot according to an embodiment of the present invention, and the liquid delivery robot having the detection system, the liquid spill prevention performance can be improved by more effectively detecting the complex movement of the liquid using image information. .

And according to the liquid movement detection system of the delivery robot according to an embodiment of the present invention, and the liquid delivery robot having the detection system, by using vision, even if the containers carried by the robot are located randomly, individual sloshing and overall sloshing within the containers are prevented. Since it is possible to recognize the movement of the liquid surface in more diverse and complex situations, it is possible to implement the necessary control over the liquid surface condition.

In addition, according to the liquid movement detection system of the delivery robot according to an embodiment of the present invention, and the liquid delivery robot having the detection system, information on the height difference between the container placed on the tray and the liquid surface and the sloshing of the liquid inside are estimated in real time and accordingly. By controlling the robot, it has the effect of controlling the degree of liquid sloshing and preventing liquid spilling.

And according to the liquid movement detection system of the delivery robot according to an embodiment of the present invention, and the liquid delivery robot having the detection system, it can be applied to transparent containers and liquids, and through the design of this system, various It has the effect of safely transporting liquid cargo without spilling it, even in situations where the speed is suddenly adjusted due to an unexpected situation or transport is carried on uneven ground.

Meanwhile, the effects that can be obtained from the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below. You will be able to.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and serve to further understand the technical idea of the present invention along with the detailed description of the invention, so the present invention is limited only to the matters described in such drawings. It should not be interpreted as such.
1 is a perspective view of a liquid delivery robot having a liquid movement detection system according to an embodiment of the present invention;
Figure 2 is a configuration diagram of a liquid delivery robot having a liquid movement detection system according to an embodiment of the present invention;
Figure 3 is a partial perspective view of a tray, a liquid container, two cameras, and an angle adjustment unit according to an embodiment of the present invention;
Figure 4 is a block diagram of a liquid delivery robot with a liquid movement detection system showing signal flow according to an embodiment of the present invention;
5 is a flowchart of a liquid movement detection method according to an embodiment of the present invention;
Figure 6 shows a schematic diagram of an image acquired according to an embodiment of the present invention.
Figure 7 shows a flowchart of a liquid movement detection method according to another embodiment of the present invention.

The above objects, other objects, features and advantages of the present invention will be easily understood through the following preferred embodiments related to the attached drawings. However, the present invention is not limited to the embodiments described herein and may be embodied in other forms. Rather, the embodiments introduced herein are provided so that the disclosed content will be thorough and complete and so that the spirit of the present invention can be sufficiently conveyed to those skilled in the art.

In this specification, when an element is referred to as being on another element, it means that it may be formed directly on the other element or that a third element may be interposed between them. Also, in the drawings, the thickness of components is exaggerated for effective explanation of technical content.

Embodiments described herein will be explained with reference to cross-sectional views and/or plan views, which are ideal illustrations of the present invention. In the drawings, the thicknesses of films and regions are exaggerated for effective explanation of technical content. Therefore, the shape of the illustration may be changed depending on manufacturing technology and/or tolerance. Accordingly, embodiments of the present invention are not limited to the specific form shown, but also include changes in form produced according to the manufacturing process. For example, an area shown as a right angle may be rounded or have a shape with a predetermined curvature. Accordingly, the regions illustrated in the drawings have properties, and the shapes of the regions illustrated in the drawings are intended to illustrate a specific shape of the region of the device and are not intended to limit the scope of the invention. In various embodiments of the present specification, terms such as first and second are used to describe various components, but these components should not be limited by these terms. These terms are merely used to distinguish one component from another. Embodiments described and illustrated herein also include complementary embodiments thereof.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, 'comprises' and/or 'comprising' does not exclude the presence or addition of one or more other elements.

In describing specific embodiments below, various specific details have been written to explain the invention in more detail and to aid understanding. However, a reader with sufficient knowledge in the field to understand the present invention can recognize that it can be used without these various specific details. In some cases, it is mentioned in advance that when describing the invention, parts that are commonly known but are not significantly related to the invention are not described in order to prevent confusion without any reason in explaining the invention.

Hereinafter, the configuration and function of a liquid delivery robot with a liquid movement detection system according to an embodiment of the present invention will be described. The explanation will focus on the liquid movement detection system and detection method.

The purpose of the present invention is to develop a system that detects the sloshing of the container and the liquid in real time for each container containing liquid placed on the service robot tray and controls the moving speed of the service robot and the position and posture of the tray to prevent the liquid from overflowing. am.

Figure 1 shows a perspective view of a liquid delivery robot with a liquid movement detection system according to an embodiment of the present invention. Figure 2 shows the configuration of a liquid delivery robot with a liquid movement detection system according to an embodiment of the present invention. Figure 3 also shows a partial perspective view of a tray, a liquid container, two cameras, and an angle adjusting unit according to an embodiment of the present invention. And Figure 4 shows a block diagram of a liquid delivery robot with a liquid movement detection system showing signal flow according to an embodiment of the present invention.

The liquid delivery robot 100 according to an embodiment of the present invention includes a traveling part 40, a main body 1, a body 10 on which a tray 11 is installed, and a specific interval between the tray 11 and the upper side. It is configured to include a housing 20 having an upper frame 21 that is spaced apart from each other.

At least one liquid container 3 storing liquid is placed on the tray 11.

The detection system includes at least one camera 30 that acquires an image by photographing the liquid container 3, and a liquid state analysis unit 50 that analyzes the image to detect the liquid state.

That is, a single or multiple RGB cameras (or RGB-D cameras) 30 that can photograph the liquid container 3 from different directions are installed. Fix the liquid surface at an angle that can be photographed through the rim and entrance of the container. (Although it is possible to detect liquid sloshing with a single camera, multiple cameras can be used in case it is not visible in the image because it is obscured by another container. .). These cameras 30 may be installed on the inner surface of the upper frame 21, spaced apart from each other in the width direction.

Additionally, the control unit 60 is configured to control the driving of the traveling unit 40 based on the liquid state determined by the detection system.

The liquid state analysis unit 50 detects the entrance border of the liquid container 3 using an image-based object detection algorithm and detects the movement of the liquid within the area using the image. Based on the detected motion information of the liquid, the control unit 60 controls the speed of the service robot or the posture of the tray to prevent liquid spilling.

Hereinafter, a method for detecting the liquid state in the liquid state analysis unit according to an embodiment of the present invention will be described. This liquid state is determined based on the height difference between the container inlet rim and the liquid surface inside the container. Below, a specific method for calculating the height difference will be described.

Figure 5 shows a flowchart of a liquid movement detection method according to an embodiment of the present invention. Figure 6 also shows a schematic diagram of an image acquired according to an embodiment of the present invention.

When an image is acquired from the camera 30 (S1), the state of the liquid surface can be recognized in various ways using the camera image. One of them is using object detection and edge detection. The liquid container entrance border is detected in the image using object detection. Afterwards, edge detection can be performed on the liquid that appears to be the container inlet (S2) and the height difference between the container inlet and the liquid surface can be calculated (S3). Using this height difference, the possibility of liquid overflow can be predicted (the height of the liquid can be observed through the side of a transparent container, but this is not possible in an opaque container.)

The height difference value between the container entrance and the liquid surface obtained through the image is the pixel value on the image. Therefore, the height difference in these images must be converted to the actual height difference value.

To obtain the actual length of this height difference, you can use a tray (11) marked (5) like a mosaic tag or a stereo camera. By using the marked tray (11) (5), the actual position and size of the container can be easily seen, making it possible to know the ratio between the pixel value on the image and the actual length (S4). By using this ratio value to obtain the actual height difference between the container inlet and the liquid surface (S5), the possibility of liquid overflow can be more accurately predicted. That is, the marking 5 is displayed on the tray 11, the size ratio of the marking 5 on the image and the actual marking 5 is calculated, and the height difference in the image is converted to the actual height difference based on this size ratio. It is to convert. This method can be applied even when the liquid is transparent or the liquid container is transparent.

The second method involves using a depth camera 30. A depth camera can extract the 3D coordinates of an object using the TOF method or stereo vision method. Therefore, the previously calculated object detection pixel results for the liquid container entrance and edge detection pixel results for the liquid surface can be converted into actual coordinate values. However, both TOF and stereo vision methods cannot be used when the liquid container is transparent or the liquid is transparent water.

Figure 7 shows a flowchart of a liquid movement detection method according to another embodiment of the present invention. The third method is to obtain the actual height difference value by comparing the object detection results of the stereo image.

Specifically, the difference in the position of the bounding box for object detection with respect to the border of the container entrance predicted in the two images collected by the stereo camera (30) is used. Using the position difference of this bounding box, the ratio between the size on the image and the actual size of the liquid container can be calculated, and the actual height difference value can be calculated using this.

That is, the camera 30 includes a reference camera and a second camera, and each of the reference camera and the second camera photographs the liquid container inlet border and the liquid surface stored in the liquid container 3 from different directions to produce an image. acquisition, detect the container entrance border as a bounding box in each image (S10), calculate the Dapps value through the position difference between the two bounding boxes (S20), and calculate the size on the image and the actual size based on the calculated Dapps value. The size ratio is calculated (S30), and the height difference in the image is converted to the actual height difference based on this size ratio (S40). This method can be applied even when the liquid is transparent or the liquid container is transparent.

Additionally, as shown in Figure 3, it can be seen that the embodiment of the present invention may include an angle adjusting unit 13 for adjusting the angle of the tray and a position adjusting unit 12 for adjusting the position of the tray.

Therefore, when the height difference calculated by the detection system exceeds the set height difference or the change amount exceeds the set change amount, the control unit 60 controls the traveling unit 40 to adjust the moving speed or the angle adjusting unit 13. The angle of the tray 11 can be adjusted by controlling.

In other words, the status information is used as a controller input value of the control unit 60, and when it exceeds a preset control requirement standard, a control command is output. For example, if liquid sloshes above the standard height, it can be recognized as a situation requiring control and the moving speed can be reduced. And even if the liquid does not slosh above the standard height, if there is a large instantaneous change in the height of the liquid, it is recognized as a situation requiring control and the moving speed can be controlled.

Additionally, reinforcement learning can be used for more precise robot control. The robot movement speed and tray angle can be controlled using a reinforcement learning model whose reward is moving quickly without spilling liquid.

And according to an embodiment of the present invention, as shown in FIG. 1, it may include an IMU sensor 2 that measures the posture of the liquid delivery robot, and the control unit 60 determines the tilt measured by the IMU sensor 2. Based on the data, the angle adjusting unit 13 can be controlled so that the tray 11 is maintained horizontally.

That is, the service robot's tilt angle information is acquired using the IMU sensor 2 so that the tray can maintain a flat angle even when moving on a tilted surface, and the tray 11 is controlled so that it is not tilted.

In addition, the apparatus and method described above are not limited to the configuration and method of the above-described embodiments, but all or part of each embodiment can be selectively combined so that various modifications can be made. It may be composed.

1:Body
2:IMU sensor
10:Body
11:Tray
12: Position control unit
13: Angle adjustment unit
20: Housing
21: Upper frame
30:Camera
40: Driving department
50: Liquid state analysis unit
60: Control unit
100: Liquid delivery robot with liquid movement detection system

Claims (15)

A system provided on a robot that delivers liquid to detect the liquid state during delivery,
At least one liquid container placed on the tray of the delivery robot;
At least one camera that acquires an image by photographing the liquid container;
A liquid movement detection system for a delivery robot, comprising a liquid state analysis unit that analyzes the image to detect the liquid state.
According to clause 1,
A liquid movement detection system for a delivery robot, characterized in that the camera is installed to photograph an inlet edge of the liquid container and the surface of the liquid stored in the liquid container.
According to clause 2,
The liquid state analysis unit,
A liquid movement detection system for delivery robots that detects the border of the liquid container entrance using an image-based object tracking algorithm and detects the movement of liquid within that area using images.
According to clause 3,
The image-based object tracking algorithm is,
The liquid movement of the delivery robot is characterized by detecting the border of the liquid container entrance in the image using object detection, detecting the border of the liquid surface that appears to be the container entrance using edge detection, and calculating the height difference in the image between the container entrance and the liquid surface. Detection system.
According to clause 4,
A liquid movement detection system for a delivery robot, wherein the camera is composed of a DApps camera and converts the height difference in the image into an actual height difference.
According to clause 4,
A marking is displayed on the tray,
A liquid movement detection system for a delivery robot, characterized in that the size ratio of the marking on the image and the actual marking is calculated, and the height difference on the image is converted to the actual height difference based on the size ratio.
According to clause 4,
The camera includes a reference camera and a second camera,
The reference camera and the second camera each acquire images by photographing the lip of the liquid container inlet and the surface of the liquid stored in the liquid container from different directions,
The liquid state analysis unit detects the container inlet border as a bounding box in each image, calculates the size ratio of the size on the image and the actual size through the position difference between the two bounding boxes, and calculates the size ratio of the size in the image to the actual size based on the size ratio. A liquid movement detection system for a delivery robot that converts the height difference between the images into an actual height difference.
A system provided on a robot that delivers liquid to detect the liquid state during delivery,
Acquiring an image by using a camera to photograph the border of the liquid container entrance and the surface of the liquid stored in the liquid container; and
A liquid state analysis unit detects the liquid state by analyzing the image,
The step of detecting the liquid state is,
Detecting the border of the liquid container entrance in the image using object detection, detecting the border of the liquid surface that appears to be the container entrance using edge detection, calculating the height difference in the image between the container entrance and the liquid surface, and calculating the height difference in the image. A method for detecting liquid movement in a delivery robot, comprising converting the height difference into an actual height difference, and determining a control requirement when the height difference exceeds a set height difference or the change amount exceeds the set change amount.
As a liquid delivery robot that delivers liquid,
frame;
Running part;
A tray on which at least one liquid container storing liquid is placed;
A detection system according to any one of claims 1 to 7; and
A liquid delivery robot with a liquid movement detection system, comprising a control unit that controls the driving of the traveling unit based on the liquid state determined by the detection system.
According to clause 9,
It further includes an angle adjusting unit that adjusts the angle of the tray,
When the height difference calculated by the detection system exceeds the set height difference or the change amount exceeds the set change amount, the control unit controls the traveling unit to adjust the moving speed or controls the angle adjustment unit to adjust the tray angle. A liquid delivery robot with a liquid movement detection system.
According to clause 10,
The control unit controls at least one of the moving speed of the robot and the angle of the tray using a reinforcement learning model that has the maximum speed at which the liquid moves without spilling as a compensation value. Delivery robot.
According to clause 10,
Further comprising an IMU sensor that measures the posture of the liquid delivery robot,
The control unit is a liquid delivery robot having a liquid movement detection system, wherein the control unit controls the angle adjustment unit so that the tray is maintained horizontal based on the tilt data measured by the IMU sensor.
As a control method for a liquid delivery robot,
Loading and driving at least one liquid container storing liquid on a tray;
Acquiring an image by using a camera to photograph the border of the liquid container entrance and the surface of the liquid stored in the liquid container;
The liquid state analysis unit detects the border of the liquid container entrance in the image through object detection, and detects the border of the liquid surface visible as the container entrance through edge detection;
Calculating a height difference in the image between the container inlet and the liquid surface, converting the height difference in the image into an actual height difference;
Determining that control is required when the height difference exceeds a set height difference or the change amount exceeds the set change amount; and
A method of controlling a liquid delivery robot having a liquid movement detection system, comprising a step where a control unit controls the traveling unit to adjust the moving speed or a tray angle by controlling the angle adjustment unit.
According to clause 13,
The adjustment step
Liquid having a liquid movement detection system, wherein the control unit controls at least one of the moving speed of the robot and the tray angle using a reinforcement learning model that has the maximum speed at which the liquid moves without spilling as a compensation value. Control method of delivery robot.
According to clause 13,
Further comprising an IMU sensor that measures the posture of the liquid delivery robot,
A method of controlling a liquid delivery robot having a liquid movement detection system, wherein the control unit controls the angle adjustment unit so that the tray is maintained horizontal based on the tilt data measured by the IMU sensor.

Images