RU2814860C1 - Software and hardware complex for automatic sorting of solid wastes - Google Patents

Abstract

FIELD: processing and recycling of wastes.

SUBSTANCE: group of inventions relates to a software and hardware system for automatic sorting of solid wastes and a method for automatic sorting of solid wastes using said complex. Hardware and software system comprises a circular ring-shaped conveyor with solid wastes arranged on the circular table. Hardware and software system includes a robotic garbage capture device configured to move above a semicircle of an annular conveyor along a zero line, which represents a projection of the line of movement of the garbage gripper on the plane of the circular table and passes from the center to the end of the circular table. Hardware and software system also includes a video camera configured to obtain a video stream containing images of solid waste objects, and a reading unit, configured to select each n image frame from the obtained video stream and transmit the selected images to the recognition unit. Recognition unit is configured to process the obtained images using a neural network, to recognize objects on said images, and for each recognized object to obtain its coordinates and type of recognized object. Hardware-software system also contains a post-processing unit, made with the possibility to calculate the circular conveyor table rotation angle based on the recognized object coordinates obtained using the neural network, which is the angle between the zero line and the line of the object, and the distance by which the robotic garbage capture device must move, and to transmit to the robotic garbage gripping device the type of the recognized object, the calculated angle of rotation of the circular conveyor table, the distance from the center of the circle, on which the garbage gripper should move along its line of movement from the center of the circle, so that as a result, the garbage capture device is above the center of the recognized object and captures the recognized object using the robotic garbage capture device to move the captured object depending on the recognized type of object into the corresponding container.

EFFECT: providing automatic sorting of solid wastes using a robotic device "manually" in a limited space.

8 cl, 2 dwg

Description

TECHNICAL FIELD

The invention relates to the field of solid waste sorting, in particular, to a hardware and software complex for automatic sorting of solid waste.

The present invention can be used, at least, for sorting various solid wastes for the purpose of their further processing, including in the courtyards of residential buildings, in residential buildings.

BACKGROUND OF THE ART

Patent RU 2731052 C1, publication date 08.28.2020, discloses a robotic complex, which includes a machine vision system with a digital camera and a computer with software based on a convolutional neural network, a conveyor belt, and a robot with a gripping and moving system. The optical sensor unit is installed above the conveyor belt behind the object recognition area of the digital camera and includes emitters and recording cameras and an automated control system. The central server of the automated control system is connected via a local communication line to the machine vision system computer, synchronization module, robot controller, and conveyor belt speed measurement sensor. The synchronization module of the automated control system is connected via a local communication line to the camera of the machine vision system, the optical sensor unit and the central server. The computer of the machine vision system is connected via a local communication line to a digital camera, a unit of optical sensors, and a central server of the automated control system. The robotic complex allows you to increase the speed and quality of the selection of fractions of solid municipal waste.

However, this patent does not contain the ability to sort solid waste using a robotic device, such as a robotic arm, “manually,” that is, when the robotic arm imitates human actions. In addition, there is no possibility to place the robotic complex in a limited space, for example, in the courtyard of a residential building, in a residential building.

The technical objective of the present invention is to develop a software and hardware complex for sorting solid waste, which would allow sorting solid waste using a robotic device “manually” in a limited space.

SUMMARY OF THE INVENTION

The technical result of the claimed invention is to provide automatic sorting of solid waste using a robotic device “manually” in a limited space, which allows automatic sorting of solid waste directly in the courtyards of residential buildings.

The specified technical result is achieved due to the fact that:

Hardware and software complex for automatic sorting of solid waste, contains:

a circular circular conveyor, wherein solid waste is placed on the circular table of the circular conveyor;

a robotic waste pick-up device, wherein the robotic waste pick-up device moves over the semicircle of the ring conveyor along a zero line, wherein the zero line represents a projection of the line of movement of the waste pick-up device onto the plane of the circular table and extends from the center to the end of the circular table;

a video camera, and with the help of the specified video camera, a video stream is obtained containing images of solid waste objects located on a semicircle of a circular table, where a robotic waste pickup device moves;

a reading unit, whereby the reading unit selects every n image frames from the received video stream and transmits the selected images to the recognition unit;

a recognition block, whereby by means of a recognition block, the received images are processed using a neural network and, as a result of processing, objects in the specified images are recognized, and for each recognized object, the coordinates of the recognized object and the type of the recognized object are obtained;

post-processing block, and using the post-processing block, based on the coordinates of the recognized object obtained using a neural network, the angle of rotation of the ring conveyor table is calculated, which is the angle between the zero line and the object line, while the object line represents the projection onto the plane of the circular table of the line passing from the center the circular table to the center of the recognized object, and the distance by which the robotic debris picking device must move, which is the distance from the center of the circular table to the center of the recognized object; And

transmitting to the robotic garbage pickup device the type of recognized object, the calculated rotation angle of the ring conveyor table, the distance from the center of the circle by which the garbage pickup device should move along its line of movement from the center of the circle, so that as a result of the specified rotation of the circle and the specified movement, the garbage pickup device appears above the center of the recognized object; and grasping the recognized object using a robotic debris picking device to move the captured object, depending on the recognized object type, into an appropriate container; Moreover, the process of capturing solid waste is repeated as many times as the number of objects were recognized on the circle at a particular point in time.

In one embodiment of the hardware and software complex, the object type is at least plastic, non-ferrous metals, glass, paper.

In one embodiment of the hardware and software complex, the robotic device for picking up garbage is a robotic arm.

In one embodiment of the hardware and software complex, the neural network is a segmentation neural network.

In one embodiment of the hardware and software complex, if not a single object was recognized on the semicircle at a particular moment in time, then the command to rotate the table clockwise by N degrees is executed.

In one embodiment of the software and hardware complex, after completing a full circle clockwise M times, where M is equal to or greater than 1, the side flap opens and the table begins to rotate 360 degrees counterclockwise so that the remaining garbage falls into the container for unrecognized garbage.

In one embodiment of the hardware and software complex, containers for sorting objects are located under the small opening of the sorting circle, under the extreme position of the robotic arm.

In one embodiment, the software and hardware complex additionally contains an overview video camera to obtain a general view of the software and hardware complex for automatic sorting of solid waste for remote monitoring of the operation of the entire software and hardware complex as a whole.

In one embodiment of the hardware and software complex, the video camera is located vertically above the table of the ring conveyor.

Also, the specified technical result is achieved due to the fact that:

The method for automatic sorting of solid waste contains the following steps:

a) receive a video stream containing images of solid waste objects located on a semicircle of a circular table of a ring conveyor, where a robotic garbage pickup device moves, and the robotic garbage pickup device moves over the semicircle of a ring conveyor along the zero line, wherein the zero line represents the projection of the line of movement of the grabber device debris onto the plane of the circular table and runs from the center to the end of the circular table;

b) select every n image frames from the received video stream and transmit the selected images for recognition to the neural network;

c) using a neural network, the resulting images are processed and, as a result of processing, objects in the specified images are recognized, and for each recognized object, the coordinates of the recognized object and the type of the recognized object are obtained;

d) based on the coordinates of the recognized object obtained using a neural network, the angle of rotation of the ring conveyor table is calculated, which is the angle between the zero line and the object line, while the object line represents the projection onto the plane of the circular table of the line passing from the center of the circular table to the center of the recognized object , and the distance to which the robotic debris picking device must move, which is the distance from the center of the circular table to the center of the recognized object; And

e) transmit to the robotic garbage pickup device the type of recognized object, the calculated rotation angle of the ring conveyor table, the distance from the center of the circle by which the garbage pickup device should move along its line of movement from the center of the circle, so that as a result of the specified rotation of the circle and the specified movement, the pickup device the debris appears above the center of the recognized object;

f) carry out the capture of the recognized object using a robotic device for picking up garbage to move the captured object, depending on the recognized type of object, into the appropriate container;

g) the process of capturing solid waste is repeated as many times as the number of objects were recognized on the circle at a specific point in time;

h) if not a single object was recognized on the semicircle at a particular moment in time, execute the command to rotate the table clockwise by N degrees and repeat steps a) - g).

The technical result is achieved due to the fact that the coordinates of recognized objects in the images are transformed in such a way as to ensure that the robotic arm moves over the circle of the ring conveyor in such a way that the robotic arm imitates the action of a human hand and waste sorting is carried out automatically “by hand” in a limited space. The circular shape of the conveyor allows you to calibrate and convert the coordinates of the recognition system into real coordinates. Since the claimed hardware and software complex is equipped with a circular circular conveyor, which is compact, it can be placed in a limited space and sort waste directly in the courtyards of residential buildings.

DESCRIPTION OF DRAWINGS

The implementation of the invention will be described further in accordance with the accompanying drawings, which are presented to explain the essence of the invention and in no way limit the scope of the invention.

The claimed invention is illustrated in figures 1-2, which depict:

Fig. 1 – illustrates an example of a circular circular conveyor with a robotic waste pickup device.

Fig. 2 illustrates an example of object recognition in the resulting image of a ring conveyor using a neural network.

DETAILED DESCRIPTION OF THE INVENTION

The following detailed description of the invention sets forth numerous implementation details designed to provide a clear understanding of the present invention. However, it will be apparent to one skilled in the art how the present invention can be used with or without these implementation details. In other cases, well-known methods, procedures and components have not been described in detail so as not to unduly obscure the features of the present invention.

In addition, from the above discussion it will be clear that the invention is not limited to the above implementation. Numerous possible modifications, alterations, variations and substitutions, while retaining the spirit and form of the present invention, will be apparent to those skilled in the art.

The claimed software and hardware complex for automatic sorting of solid waste contains a circular circular conveyor. On the circular table of the ring conveyor, solid waste of various types (for example, metal, paper, plastic, glass, etc.) is located. A robotic waste picking device (for example, a robotic arm) moves over a circle of a ring conveyor along the zero line. The zero line is a projection of the line of movement of the waste pickup device onto the plane of the circular table and runs from the center to the end of the circular table. In FIG. 1 zero line (1) - projection of the line along which the robotic arm moves (2) onto the table plane (3), the line of movement of the robotic arm (grip line) (4) starts from the center of the metal rail (5), through which the axis of the ring conveyor passes (6), and ends with the grip attachment point (7). The resulting zero line (1) will run from the center of the small circle (8) to the end of the table parallel to the grip line (4). The video camera is located vertically above the table of the ring conveyor in such a way as to receive, using the specified video camera, a video stream containing images of solid waste objects located on the semicircle of the table, where the robotic waste pickup device moves, and transmit it to the reading unit. The reading unit selects every n image frames from the received video stream and transmits the selected images to the recognition unit. The recognition unit, using a neural network, processes the received images and, as a result of processing, recognizes objects in the specified images, and for each recognized object receives the coordinates of the recognized object, the type of the recognized object (Fig. 2, Plastic, Glass, Metal, Paper - Paper). The post-processing unit, based on the coordinates of the recognized object obtained using a neural network, calculates the angle of rotation of the ring conveyor table, which is the angle between the zero line and the object line, while the object line represents the projection of the line passing from the center of the circular table to the center of the recognized object onto the plane of the circular table, and calculates the distance that the robotic garbage picker should move, which is the distance from the center of the circular table to the center of the recognized object. Next, the type of recognized object, the calculated rotation angle of the ring conveyor table and the distance from the center of the circle by which the garbage pickup device should move along its line of motion are transmitted to the robotic garbage pickup device so that as a result of the specified rotation of the circle and the specified movement, the garbage pickup device is above the center recognized object. And the recognized object is captured using a robotic garbage pickup device to move the captured object, depending on the recognized object type, into the appropriate container. The process of capturing solid waste is repeated as many times as objects were recognized on the circle at a particular point in time.

The method for automatic sorting of solid waste contains the following steps:

a) receive a video stream containing images of solid waste objects located on a semicircle of a circular table of a ring conveyor, where a robotic garbage pickup device moves, and the robotic garbage pickup device moves over the semicircle of a ring conveyor along the zero line, wherein the zero line represents the projection of the line of movement of the grabber device debris onto the plane of the circular table and runs from the center to the end of the circular table;

b) select every n image frames from the received video stream and transmit the selected images for recognition to the neural network;

c) using a neural network, the resulting images are processed and, as a result of processing, objects in the specified images are recognized, and for each recognized object, the coordinates of the recognized object and the type of the recognized object are obtained;

d) based on the coordinates of the recognized object obtained using a neural network, the angle of rotation of the ring conveyor table is calculated, which is the angle between the zero line and the object line, while the object line represents the projection onto the plane of the circular table of the line passing from the center of the circular table to the center of the recognized object , and the distance to which the robotic debris picking device must move, which is the distance from the center of the circular table to the center of the recognized object; And

e) transmit to the robotic garbage pickup device the type of recognized object, the calculated rotation angle of the ring conveyor table, the distance from the center of the circle by which the garbage pickup device should move along its line of movement from the center of the circle, so that as a result of the specified rotation of the circle and the specified movement, the pickup device the debris appears above the center of the recognized object;

f) carry out the capture of the recognized object using a robotic device for picking up garbage to move the captured object, depending on the recognized type of object, into the appropriate container;

g) the process of capturing solid waste is repeated as many times as the number of objects were recognized on the circle at a specific point in time;

h) if no object was recognized on the semicircle at a particular moment in time, execute the command to rotate the table clockwise by N degrees and repeat steps a) - g).

The first camera, which is located vertically above the conveyor, films the circular conveyor table. In addition, the hardware and software complex contains a second video camera, which films a general view of the conveyor. The second camera is needed for remote monitoring of the operation of the entire system as a whole. The cameras can be USB cameras that are connected to a portable mini-computer via 2 USB cables. Using a mini-computer, garbage objects are recognized on a video stream. Recognition occurs using a neural network. Images are read from the camera video stream, in which objects are recognized using a neural network. As a result of the work of the neural network, the coordinates and types of recognized objects are obtained.

An example of the process of recognizing garbage objects.

The first video camera vertically films the circular table of the ring conveyor on which household waste objects are located. A video camera films a semicircle of a ring conveyor. The robotic garbage pickup device moves in a semicircle, which is recorded by a video camera.

The process of moving the ring conveyor circle and matching it with the gripping system occurs as follows. At the moment the system begins to operate, coinciding with the moment the device for tearing bags of garbage ends, when the garbage is already lying on the table from the torn bag (does not apply to the recognition system), a command comes to activate the recognition system. After this, the recognition of garbage objects by the neural network begins on a semicircle that falls within the field of view of the video camera.

The system supports 3 main commands:

- rotate the circle by N degrees (for example, 20 degrees) clockwise until it passes a full circle clockwise ( M * 360 degrees, where M is a specified parameter, number of circles M ≥ 1, default M = 1);

- performing a rotation by θ degrees clockwise + grabbing an object at a distance of R mm from the robotic hand to the object, θ and R are calculated by the system if the neural network detected objects in a semicircle;

- cleaning the table after passing a full circle clockwise, the side flap opens and the table begins to rotate 360 degrees, but counterclockwise, so all the remaining garbage falls into the container for unrecognized garbage.

If not a single object was recognized on the semicircle at a particular moment in time (frame), the system considers that there is no debris on this part of the circle and begins to execute the command to rotate the table clockwise by 20 degrees.

Thus, after completing the turn, the circle stops for a moment (the time from completion of the circle rotation by 20 degrees until the completion of processing of the frame by the neural network received after the rotation is about 0.5 seconds), the neural network scans the semicircle, if it does not find the target garbage objects, the rotation is performed again by another 20 degrees; if the target object is found, the object closest (by angle) to the zero line is captured.

In the absence of commands from the recognition system, the circle will always be in a static position and will not rotate

During the recognition process from a video stream, every n frames, for example, every 5 frames out of 40 per second, images are selected and transmitted to the neural network. The neural network, having processed the resulting image, outputs the coordinates of objects, their type, normalized coordinates to the circular axis of the conveyor, and the angle of rotation for the capture system.

For object detection/segmentation, neural network architectures such as centermask [1] and YOLOv5 / YOLOv6 [2] can be used.

An example of training a neural network YOLOv5.

1. In the basic architecture of a neural network, when training a model for the stated solution, the last layer is changed into 4 classes (Plastic, Glass, Metal, Paper), and the layers before that are filled with the basic values of the model.

2. Next, a dataset of garbage objects is created - a data set of the form image, pre-defined markup, which contains information about the location of the garbage object (bbox) and its type (Plastic, Glass, Paper, Metal). In the stated solution, marking data was taken from [3], as well as photographs were taken and the coordinates and types of various garbage objects were manually marked for each image.

3. Next, a network object is created in which the weights are initialized in accordance with point 1.

4. Next, “batches” are iteratively created - mini-sets of images randomly selected from the entire data set. In this case, we get a vector of n=16 images of size (3, 640, 640) -> (16, 3, 640, 640). Next, the batch goes through all the operations of multiplication, sum, etc., embedded in the architecture of the previously created network (this is called forwardpass), the output is a vector of dimension [n, x, y, w, h, p, c], where x , y, w, h are parameters that specify the location of the “object square” in the image, p is the confidence of the prediction, c is the serial number of the predicted object type (Plastic/Glass/Paper/Metal).

5. Next, the predictions are transferred to the loss function, which has the formulation [4], [5].

6. Next, the gradient from the complex function along the model weights is calculated from the loss function (how much the weights need to be changed in order to minimize the loss function). And then the weights are updated according to the following rule:

w_new = w_old -mu * d (Loss) / d (w_old)

7. This update occurs a fixed number of times (k * len(data) // 16), where k is the parameter, len(data) is the number of images in the entire data set, 16 is the parameter n - the number of images in the batch.

8. In this way, new weights are obtained and the predictions are iteratively changed, gradually adapting to the real markings in the training set.

9. After the processes described above, predictions are obtained based on new weights for the test sample (the one that the model has not seen), it is needed in order to track the moment when the model begins to learn the training data that is shown to it and deteriorate its generalizing ability by data she hasn't seen.

An image is transmitted to the trained neural network in the form of pixels of dimension: (1, 3, 640, 640), as a result of the work of the neural network, the coordinates of boxes of dimension (1, N, 4) are obtained, where N is the number of predicted boxes, the box is a square image of dimension 4 (xmin, xmax, ymin, ymax) - minimum\maximum coordinates in pixels along the x and y axis, respectively. In Fig. Figure 2 shows an example of object recognition with drawn coordinates.

Having values of the form: (xmin, xmax), (ymin, ymax) for each object, we can find the centers of the objects in pixels. They can be obtained by the following transformation:

bbox_center_x = xmin + (xmax - xmin) / 2

bbox_center_y = ymin + (ymax - ymin) / 2

After receiving the coordinates of each object in the image, post-processing starts. Having the coordinates of objects from the image, the post-processing unit calculates the angle between the line from the center of the small circle (the center of the ring conveyor circle) to the center of the object and the zero line (this is the angle by which the table must be rotated so that the object is located directly under the zero line of the robotic arm).

The rotation angle is calculated using the formula:

θ = atan2(A.y, A.x) - atan2(B.y, B.x), (1)

Where A.y, A.x are the coordinates of the center of the object

B.y, B.x - coordinates of the end of the point on the zero line (virtual point of the center of the small circle of the ring conveyor).

Next, the distance in pixels is calculated by which the robotic arm should move along the zero line, starting from the center of the small circle and ending with the center of the object. To do this, we need normalized coordinates, thanks to which the movement distance is calculated as follows:

where bbox_x, bbox_y are the coordinates of the center of the object,

cx, cy - coordinates of the center of the small circle in pixels

Then there is a transition (normalization of coordinates) from pixel distances to real ones through the following transformations:

mm_per_pxl = (R_MAX - R_MIN) / (outer_r - inner_r), (3)

where R_MAX, R_MIN are the constants of the maximum and minimum distance of movement of the robotic arm in mm,

outer_r, inner_r - pixel radii of the outer and inner circles (Fig. 2).

Next we get the distance in mm:

R = R_MAX - (pix_R - inner_r) * mm_per_pxl (4)

After this, the round table is rotated through the calculated rotation angle θ, the robotic arm moves along the zero line by R mm from the center of the small circle and captures the garbage object underneath it; after capturing, the system throws the object into the appropriate container depending on the recognized type of object.

Recognition of 4 types of objects is supported:

- Plastic

- Non-ferrous metals (aluminum cans, etc.)

- Glass (glass bottles, etc.)

- Paper

Containers for sorting objects are located under the small hole of the sorting circle, under the extreme position of the robotic arm, under the flap of the ring conveyor. For example, the first container is under the small hole of the sorting circle, into which plastic is thrown into which the robotic arm is thrown, the second - under the extreme position of the robotic arm (see Fig. 1), metal is dumped into the second container, the third - under the flap of the ring conveyor (the flap is located in the lower right corner in Fig. 1, it opens when the command to clean the table is executed, all remaining garbage is collected in the third container. The angle of rotation and the distance that the hand needs to travel to the nearest garbage object is calculated using formulas (1), (2), (3), (4). After the objects are recognized by the neural network in the frame, a command is sent to the robotic arm to grab the object, containing: the angle of rotation of the table to the object, the distance that the hand needs to travel to the object after rotation by the calculated angle, the type of container (1 - container for plastic, 0 - container for metal), 1 or 0 is determined based on the type of object determined by the neural network. The robotic arm can move from the position Rmin to Rmax, Rmin<Rmax, where Rmin by default = 0 mm - corresponds to the extreme right position of the arm, when the virtual axis of the circular conveyor (see Fig. 1) will pass through the robotic arm when the robotic arm is above the first container ( plastic) Rmax - the extreme left position of the hand when it is above container number 2 (metal). For example, the neural network received the calculated values of the rotation angle d = 14 degrees, the distance to the object R = 425 mm (Rmin<R<Rmax), the type of container (1 - container for plastic, if plastic is detected, 0 - container for metal, if detected metal), for example, 0 will be a metal can (metal). The table is rotated clockwise by d = 14 degrees, after turning the table, the robotic arm opens from the closed position and moves to a distance R = 425 mm, lowers vertically down to the table, having dropped, it closes , grabbing a metal can, then the hand rises vertically upward, then moves to a position corresponding to Rmax = 700 mm from the center of the circular conveyor, having moved, the robotic arm opens and releases the metal can, the can falls into the second container (for metal), then the robotic arm returns to the Rmin position, corresponding to the center of the circular conveyor, the center of the small circle. If plastic had been recognized instead of a metal can, the robotic arm would have dropped it into container number 1 (plastic) in position Rmin and then would have remained in Rmin.

The process occurs iteratively until the system is turned off by command or manually. The capture process is repeated as many times as objects were recognized on the table at a particular point in time. Waste enters the circle almost uninterruptedly or in parts, which allows us to call the process of recognition and capture repetitive.

It will be apparent to one skilled in the art that all of the data processing operations of the present invention can be implemented using at least one computing device. The computing device contains at least one processor, memory and instructions stored in the memory and executed by the processor, with the help of which data is processed in the automatic garbage collection hardware and software complex. Data processing can be centralized, such as using a single computing device, or distributed, such as using multiple computing devices distributed over a network.

In the general case, a computing device that provides data processing necessary to implement the claimed solution contains components such as: one or more processors, at least one memory, data storage means, input/output interfaces, I/O means, network communication means .

The device processor performs basic computing operations necessary for the operation of the device or the functionality of one or more of its components. The processor executes the necessary machine-readable instructions contained in the RAM. Memory, as a rule, is made in the form of RAM and contains the necessary program logic that provides the required functionality. The data storage medium can be in the form of HDD, SSD drives, raid array, network storage, flash memory, optical storage devices (CD, DVD, MD, Blue-Ray disks), etc. The tool allows for long-term storage of various types of information. Interfaces are standard means for connecting and working with the server part, for example, USB, RS232, RJ45, LPT, COM, HDMI, PS/2, Lightning, FireWire, etc. The choice of interfaces depends on the specific design of the device, which can be a personal computer, mainframe, server cluster, thin client, smartphone, laptop, etc. The keyboard is used as data I/O. In addition to the keyboard, I/O data tools can also include: joystick, display (touch display), projector, touchpad, mouse, trackball, light pen, speakers, microphone, etc. Network communication means are selected from devices that provide network reception and transmission of data, for example, an Ethernet card, WLAN/Wi-Fi module, Bluetooth module, BLE module, NFC module, IrDa, RFID module, GSM modem, etc. Using the tools, it is possible to organize data exchange via a wired or wireless data transmission channel, for example, WAN, PAN, LAN, Intranet, Internet, WLAN, WMAN or GSM, 3G, 4G, 5G, 6G, etc. The components of a computing device are interconnected via a common data bus.

These application materials provide a preferred disclosure of the implementation of the claimed technical solution, which should not be used as limiting other, private embodiments of its implementation, which do not go beyond the scope of the requested scope of legal protection and are obvious to specialists in the relevant field of technology.

It should be clear to a person skilled in the art that various variations of the proposed method and system do not change the essence of the invention, but only determine its specific embodiments and applications.

Sources

[1] https://arxiv.org/abs/1911.06667

[2] https://arxiv.org/abs/2209.02976

[3] http://tacodataset.org/

[4] https://github.com/ultralytics/yolov5/discussions/6276#discussioncomment-1954916

[5] http://ncbi.nlm.nih.gov/pmc/articles/PMC8610656/(chapter 2.4)

Claims (22)

1. Hardware and software complex for automatic sorting of solid waste, containing: a circular circular conveyor, wherein solid waste is placed on the circular table of the circular conveyor; a robotic device for picking up waste, wherein the robotic device for picking up waste is configured to move over the semicircle of the ring conveyor along the zero line, wherein the zero line represents the projection of the line of movement of the waste pick-up device onto the plane of the circular table and runs from the center to the end of the circular table; a video camera, wherein the video camera is configured to receive a video stream containing images of solid waste objects located on a semicircle of a circular table, where a robotic waste pickup device moves; a reading unit, wherein the reading unit is configured to select each n image frame from the received video stream and transmit the selected images to the recognition unit; a recognition block, wherein the recognition block is configured to, using a neural network, process the received images and, as a result of processing, recognize objects in the specified images, and for each recognized object, obtain the coordinates of the recognized object, the type of the recognized object; a post-processing block, wherein the post-processing block is configured to, based on the coordinates of the recognized object obtained using a neural network, calculate the angle of rotation of the circular conveyor table, which is the angle between the zero line and the object line, wherein the object line represents a projection onto the plane of the circular table of the line, passing from the center of the circular table to the center of the recognized object, and the distance by which the robotic garbage pickup device must move, which is the distance from the center of the circular table to the center of the recognized object, and transmit to the robotic garbage pickup device the type of recognized object, the calculated rotation angle of the circular table conveyor, the distance from the center of the circle by which the garbage pickup device must move along its line of movement from the center of the circle, so that as a result of the specified rotation of the circle and the specified movement, the garbage pickup device is above the center of the recognized object, and the recognized object is captured using a robotic capture device garbage to move the captured object, depending on the recognized object type, to the appropriate container. 2. Hardware and software complex according to claim 1, characterized by the fact that the object type is at least plastic, non-ferrous metals, glass, paper. 3. Hardware and software complex according to claim 1, characterized by the fact that A robotic device for picking up garbage is a robotic arm. 4. Hardware and software complex according to claim 1, characterized by the fact that neural network is a segmentation neural network. 5. Hardware and software complex according to claim 1, characterized in that Containers for sorting objects are located under the opening of the sorting circle, under the extreme position of the robotic arm. 6. Hardware and software complex according to claim 1, characterized in that the system additionally contains an overview video camera to obtain a general view of the hardware and software complex for automatic sorting of solid waste for remote monitoring of the operation of the entire hardware and software complex as a whole. 7. Hardware and software complex according to claim 1, characterized in that The video camera is located vertically above the ring conveyor table. 8. A method for automatically sorting solid waste, containing the following steps: a) receive a video stream containing images of solid waste objects located on a semicircle of a circular table of a ring conveyor, where a robotic garbage pickup device moves, and the robotic garbage pickup device moves over the semicircle of a ring conveyor along the zero line, wherein the zero line represents the projection of the line of movement of the grabber device debris onto the plane of the circular table and runs from the center to the end of the circular table; b) select every n image frames from the received video stream and transmit the selected images for recognition to the neural network; c) using a neural network, the resulting images are processed and, as a result of processing, objects in the specified images are recognized, and for each recognized object, the coordinates of the recognized object and the type of the recognized object are obtained; d) based on the coordinates of the recognized object obtained using a neural network, the angle of rotation of the ring conveyor table is calculated, which is the angle between the zero line and the object line, while the object line represents the projection onto the plane of the circular table of the line passing from the center of the circular table to the center of the recognized object , and the distance to which the robotic debris picking device must move, which is the distance from the center of the circular table to the center of the recognized object; And e) transmit to the robotic garbage pickup device the type of recognized object, the calculated rotation angle of the ring conveyor table, the distance from the center of the circle by which the garbage pickup device should move along its line of movement from the center of the circle, so that as a result of the specified rotation of the circle and the specified movement, the pickup device the debris appears above the center of the recognized object; f) carry out the capture of the recognized object using a robotic device for picking up garbage to move the captured object, depending on the recognized type of object, into the appropriate container; g) the process of capturing solid waste is repeated as many times as objects were recognized on the circle; h) if no object was recognized on the semicircle, execute the command to rotate the table clockwise by N degrees and repeat steps a) - g).