KR102635727B1 - Apparatus, system, and method for recyclable material classification based on artificial intelligence - Google Patents

Abstract

The recycling classification system according to an embodiment of the present disclosure includes a radar sensor that radiates electromagnetic waves to a plurality of recyclables and classifies the plurality of recyclables into undamaged normal recyclables and damaged recyclables, and first image data taken of the normal recyclables. and a camera for generating second image data capturing the damaged recyclables, classifying the normal recyclables by type according to the type of the normal recyclables indicated by the first image data, and classifying the recyclables classified by type according to color. and a recyclable sorting device for sorting the damaged recyclables indicated by the second image data according to color, a first conveyor belt on which the normal recyclables are transported, and a second conveyor belt on which the damaged recyclables are transported, The recyclables sorting device provides a first control signal for controlling the first conveyor belt and a second control signal for controlling the second conveyor belt, based on the sorting results of the normal recyclables and the sorting results of the damaged recyclables. Can be printed.

Description

Apparatus, system, and method for sorting recyclables based on artificial intelligence {APPARATUS, SYSTEM, AND METHOD FOR RECYCLABLE MATERIAL CLASSIFICATION BASED ON ARTIFICIAL INTELLIGENCE}

The present disclosure relates to image processing based on artificial intelligence, and more specifically, to devices, systems, and methods for sorting recyclables based on artificial intelligence.

The amount of waste generated worldwide continues to increase, and as the importance of waste disposal and recycling selection increases, the related market is growing rapidly. Technology for automatically selecting and sorting recyclables from waste using artificial intelligence has been extensively researched and commercialized both at home and abroad, but this mainly applies to low-specific wastes such as ferrous metals (e.g., aluminum, iron) or plastics (e.g., This is to select recyclables from plastic bottles, wires, vinyl, etc.).

Among recyclables, in the case of high-specific gravity waste such as glass bottles, technology to distinguish the color or size of glass bottles using optical technology such as Near-Infrared Spectroscopy (NIRS) has been commercialized, but artificial intelligence is used to distinguish glass bottles. Technology to automatically classify recyclable glass bottles by recognizing their appearance, color, and damage has not yet been actively commercialized, and recyclable glass bottles are still sorted and sorted manually. In particular, in the case of Korea, the recycling method of glass bottles varies by local government and type of residence, and they are generally collected mixed without distinction of color or type. Therefore, in order to eliminate the inconveniences associated with manually sorting and sorting recycled glass bottles and improve sorting yield, the introduction of technology to automatically sort recycled glass bottles using artificial intelligence is required.

Registered Patent Publication No. 10-2222176 (registered on February 24, 2021)

The technical problem that this disclosure aims to solve is to first classify recyclables by type according to the shape of the boundary line forming the recyclables, and then perform detailed classification using color information, so that the classification process is automated, the amount of calculation is reduced, and , to provide devices, systems, and methods for sorting recyclables based on artificial intelligence, which can speed up sorting.

Another technical problem that the present disclosure aims to solve is classification of recyclables based on artificial intelligence, where the learned artificial intelligence model can be used as is to generate a sketch map, without the need to learn a new artificial intelligence model every time a new type of recyclable is added. To provide devices, systems, and methods for.

In order to solve the above technical problem, a recycling classification system according to an embodiment of the present disclosure includes a radar sensor that radiates electromagnetic waves to a plurality of recyclables and classifies the plurality of recyclables into undamaged normal recyclables and damaged recyclables, the normal recyclables A camera that generates first image data of recyclables and second image data of damaged recyclables, classifies the normal recyclables by type according to the type of the normal recyclables indicated by the first image data, and classifies the normal recyclables by type A recyclable sorting device for sorting sorted recyclables according to color and sorting the damaged recyclables indicated by the second image data according to color, a first conveyor belt on which the normal recyclables are transported, and a second conveyor belt on which the damaged recyclables are transported. 2 comprising a conveyor belt, wherein the recyclables sorting device provides a first control signal for controlling the first conveyor belt and the second conveyor belt based on the sorting result of the normal recyclables and the sorting result of the damaged recyclables. A second control signal for control may be output.

In one embodiment, the recycling classification device includes a preprocessing module that preprocesses the first image data and the second image data, extracting a boundary line of the normal recycling indicated by the preprocessed first image data, and generating a sketch map. A sketch map generation module, a sketch similarity calculation module for calculating similarity between the sketch map and each sketch map of recycled product images stored in an image database, and the normal recyclable product and the preprocessed product indicated by the preprocessed first image data. 2 It may include a color similarity calculation module that calculates the similarity between the color of the damaged recyclable product indicated by image data and the colors of each of the recyclable product images stored in the image database.

In one embodiment, the sketch similarity calculation module may determine the type corresponding to the recycling product image for which the similarity is calculated to be the highest among the recycling product images stored in the image database as the type of the normal recycling product.

In one embodiment, the sketch map generation module includes a first encoder that extracts first features by performing first filtering on the pre-processed first image data, and a second filtering on the pre-processed first image data. It may include a first encoder that extracts a second feature with higher sparsity than the first feature, and a decoder that generates the sketch map based on a combined feature of the first feature and the second feature.

In one embodiment, the first filtering may be Gabor filtering, and the second filtering may be Laplacian of Gaussian (LoG) filtering.

In order to solve the above technical problem, a method of sorting recycled glass bottles according to another embodiment of the present disclosure includes generating first image data of a normal glass bottle and second image data of a broken glass bottle, the first 1 Classifying the normal glass bottles into types according to the shape of the normal glass bottles indicated by image data, classifying the glass bottles classified by type according to the color of the normal glass bottles, and the second image data It may include the step of classifying the broken glass bottles according to color.

In one embodiment, the step further includes preprocessing the first image data and the second image data, and the step of classifying the normal glass bottle by type includes performing first filtering on the first image data. Extracting a first feature, performing second filtering on the second image data to extract a second feature with higher sparsity than the first feature, and combining the first feature and the second feature It may include generating a sketch map using , and calculating similarity between the sketch map and each sketch map of glass bottle images stored in an image database.

In one embodiment, the step of classifying the normal glass bottle by type includes determining the type corresponding to the glass bottle image with the highest similarity among the glass bottle images stored in the image database as the type of the normal glass bottle. Additional steps may be included.

In one embodiment, the first filtering may be Gabor filtering, and the second filtering may be Laplacian of Gaussian (LoG) filtering.

In order to solve the above technical problem, a method of classifying recyclables according to another embodiment of the present disclosure includes the steps of classifying a plurality of recyclables into normal recyclables and damaged recyclables using a radar sensor, and a first image taken of the normal recyclables. generating data and second image data of the damaged recyclables; classifying the normal recyclables by type according to the type of the normal recyclables indicated by the first image data; coloring the recyclables classified by type; classifying according to color, classifying the damaged recyclables indicated by the second image data according to color, controlling a first conveyor belt on which the normal recyclables are transported according to the classification result of the normal recyclables, and the damaged recyclables. It may include controlling a second conveyor belt through which the damaged recyclables are transported according to the classification results.

1 is a block diagram showing an exemplary configuration of a recycling classification system according to an embodiment of the present disclosure.
Figure 2 is a block diagram showing an exemplary configuration of a recycling classification device according to an embodiment of the present disclosure.
FIG. 3 is a block diagram showing an example configuration of an artificial intelligence model included in the sketch map creation module of FIG. 2.
Figures 4 to 8 show examples of sketch maps generated through the operation of the artificial intelligence model of Figure 3 in an embodiment in which the recyclable product to be classified is a glass bottle.
Figure 9 conceptually shows the overall operation of the recycling classification system in an embodiment in which the recycling product to be classified is a glass bottle.
Figure 10 is a flowchart showing a method for classifying recyclables according to an embodiment of the present disclosure.
FIG. 11 is a flowchart showing in more detail the steps of classifying the normal recyclables of FIG. 10 by type according to shape.
FIG. 12 is an exemplary hardware configuration diagram when the recycling classification device of FIG. 2 is implemented as a computing device.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the attached drawings. The advantages and features of the present disclosure and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present disclosure is not limited to the embodiments described below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the present disclosure is complete, and those of ordinary skill in the art to which the present disclosure pertains are not limited to the present disclosure. It is provided to fully inform the scope of the invention, and the present disclosure is only defined by the scope of the claims.

When adding reference signs to components in each drawing, it should be noted that the same components are given the same reference numerals as much as possible even if they are shown in different drawings. Additionally, in describing the present disclosure, if it is determined that a detailed description of a related known configuration or function may obscure the gist of the present disclosure, the detailed description will be omitted.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which this disclosure pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined. The terms used in this specification are for describing embodiments and are not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context.

Additionally, in describing the components of the present disclosure, terms such as first, second, A, B, (a), and (b) may be used. These terms are only used to distinguish the component from other components, and the nature, sequence, or order of the component is not limited by the term. When a component is described as being “connected,” “coupled,” or “connected” to another component, that component may be directly connected or connected to that other component, but there is another component between each component. It will be understood that elements may be “connected,” “combined,” or “connected.”

As used in the specification, “comprises” and/or “comprising” refers to the presence of one or more other components, steps, operations and/or elements. or does not rule out addition.

1 is a block diagram showing an exemplary configuration of a recycling classification system 10 according to an embodiment of the present disclosure. The recycling classification system 10 divides a plurality of recyclables (e.g., glass bottles, plastic bottles, plastic containers, etc.) (1) into undamaged recyclables (e.g., unbroken glass bottles, hereinafter referred to as normal recyclables). After first sorting into recyclables and damaged recyclables (for example, broken glass bottles), normal recyclables can be sorted by type based on their respective types based on artificial intelligence. For example, if the recyclable product is a glass bottle, the type of recyclable product may mean different types of glass bottles such as beer bottles, soju bottles, and beverage bottles. If the recyclable product is a plastic bottle, the type of recyclable product may refer to types of PET bottles with different capacities. It can mean.

Furthermore, the recyclables classification system 10 can recognize the appearance of recyclables classified by type and classify them in more detail (for example, according to the company indicated on the attached label or according to color), and even damaged recyclables can be colored. It can be classified according to. The recycling classification system 10 can control the conveyor belts 14 and 15 according to the classification results. Referring to Figure 1, the recycling classification system 10 may include a radar sensor 11, a camera 12, a recycling classification device 13, a first conveyor belt 14, and a second conveyor belt 15. You can.

The radar sensor 11 radiates electromagnetic waves to a plurality of recyclables 1 on the first conveyor belt 14 and can classify them into normal recyclables and damaged recyclables. For example, the frequency of the wave reflected back from a normal recyclable product and the wave reflected back from a damaged recyclable product may be different. Damaged recyclables (2) can be moved to proceed on the second conveyor belt (15) and normal recyclables can continue to proceed on the existing first conveyor belt (14).

The camera 12 may photograph normal recyclables on the first conveyor belt 14 and provide an image of each normal recyclable as first image data (IDAT1) to the recyclables sorting device 13. The recyclables sorting device 13 may classify normal recyclables included in the first image data (IDAT1) into types (e.g., in the case of glass bottles, beer bottles, soju bottles, beverage bottles, medicine bottles, etc.) according to their shape. . For example, the shape of the recyclable product may be the shape of the edge that forms the recyclable product. Specifically, since recyclables have a unique boundary shape for each type and the boundary shapes of different types of recyclables are different, classification by type may be possible only by the shape of the boundary line. Afterwards, the recycling classification device 13 can recognize the color of the recycling products classified by type and classify them in more detail. In this way, the recycling classification device 13 according to the embodiment of the present disclosure first performs classification by type according to shape, and then performs classification according to color on the results classified by type, compared to when only classification according to color is used. The amount of calculation required is small and classification speed can be improved.

Additionally, the camera 12 may photograph damaged recyclables on the second conveyor belt 15 and provide an image of each damaged recyclable as second image data (IDAT2) to the recyclables sorting device 13. The recyclables sorting device 13 may classify damaged recyclables included in the second image data (IDAT2) according to color. The recyclables sorting device 13 may include an artificial intelligence model for performing the above-described sorting operations, and the artificial intelligence model used may vary depending on the recyclables to be sorted. For example, the recyclables sorting device 13 can download an artificial intelligence model for sorting plastic bottles from the web if the recyclables to be sorted are plastic bottles, and if the recyclables to be sorted are glass bottles, the recyclables can be downloaded from the web to classify glass bottles. You can download the artificial intelligence model. Through this, the recyclables classification system 10 can perform a classification operation for various recyclables simply by replacing the artificial intelligence model without replacing specific equipment.

In performing the above-described sorting operations, the recyclables sorting device 13 may communicate with the image database 20 in which images of various recyclables are stored. The recycling classification device 13 performs a classification operation by calculating the similarity between the first and second image data (IDAT1, IDAT2) provided from the camera 12 and the images stored in the image database 20 using an artificial intelligence model. It can be done. The recycling classification device 13 generates a first control signal (CTRL1) for controlling the first conveyor belt 14 and a second control signal (CTRL2) for controlling the second conveyor belt 15 according to the classification result. can do. The configuration and operation of the recycling sorting device 13 are described in more detail with reference to FIG. 2 .

The first conveyor belt 14 transports a plurality of recyclables 1, and the plurality of recyclables 1 are classified into normal recyclables and damaged recyclables 2 according to the detection result of the radar sensor 11. You can. Normal recyclables can proceed as is on the first conveyor belt 14, and broken glass bottles 2 can be moved to proceed on the second conveyor belt 15. The first conveyor belt 14 and the second conveyor belt 15 may be controlled based on the first control signal CTRL1 and the second control signal CTRL2, respectively. For example, the control signals CTRL1 and CTRL2 may control the conveyor belts 14 and 15 to transport recyclables to a sorting bin (not shown) corresponding to the sorting result.

Although the recycling classification system 10 in FIG. 1 is shown as including two conveyor belts 14 and 15, the present disclosure is not limited thereto, and the recycling classification system 10 includes two or more conveyor belts 14 and 15 depending on the classification method. It may include conveyor belts, and the recycling sorting device 13 may be implemented to generate control signals for controlling each of the conveyor belts. Additionally, the recyclables classification system 10 may be implemented to classify only normal recyclables and not perform a classification operation for damaged recyclables. Hereinafter, for clear explanation, embodiments will be described based on the recycling classification system 10 as shown in FIG. 1.

At least some components of the recycling sorting system shown in Figure 1 may communicate over a network. For example, networks include all types of wired/wireless networks such as Local Area Network (LAN), Wide Area Network (WAN), mobile radio communication network, Wibro (Wireless Broadband Internet), etc. It can be implemented as: For example, through the above-mentioned network, the recycling product sorting device 13 can communicate with the image database 20 and download an artificial intelligence model corresponding to the type of recycling product.

Figure 2 is a block diagram showing an exemplary configuration of a recycling classification device 100 according to an embodiment of the present disclosure. For example, the recyclables sorting device 100 of FIG. 2 may be the recyclables sorting device 13 of FIG. 1 . As explained with reference to FIG. 1, the recyclables sorting device 13 processes image data (IDAT1, IDAT2) based on artificial intelligence to classify normal recyclables and damaged recyclables to convey the recyclables on the conveyor belt (for example, For example, control signals (CTRL1, CTRL2) for controlling 14 and 15 in FIG. 1 can be generated. Referring to FIG. 2, the recycling classification device 100 may include a preprocessing module 110, a sketch map generation module 120, a sketch similarity calculation module 130, and a color similarity calculation module 140. It may include one or more artificial intelligence models to perform intelligence-based classification operations. For example, in some embodiments, the recycling sorting device 100 may be implemented as a computing device, and such implementation examples are described in detail with reference to FIG. 12 .

The preprocessing module 110 may preprocess the first image data (IDAT1), which is a photograph of a normal recyclable product, and the second image data (IDAT2), which is a photograph of a damaged recyclable product, so that it is suitable for input into an artificial intelligence model. For example, the preprocessing module 110 performs resizing, scaling, cropping, and normalization on images included in the first image data (IDAT1) and the second image data (IDAT2). , color space conversion, etc. can be performed for preprocessing.

In addition, the pre-processing module 110 labels the first image data (IDAT1) as an image of a normal recyclable glass bottle and labels the second image data (IDAT2) as an image of a damaged recyclable product so that it can be used as one of the features in a later classification operation. Can be labeled. The pre-processed first image data (i.e., images of normal recyclables) are provided to the sketch map creation module 120 for classification operation according to shape, and the pre-processed second image data (i.e., images of damaged recyclables) are directly color coded. It may be provided to the color similarity calculation module 140 for classification operation according to. In the following descriptions, unless otherwise stated, both the first image data IDAT1 and the second image data IDAT2 represent data preprocessed through the preprocessing module 110.

The sketch map creation module 120 may generate a sketch map (SMAP) by extracting the boundary line of the normal recyclable product indicated by the preprocessed first image data (IDAT1). The sketch map creation module 120 may include an artificial intelligence model for extracting the boundary line of a normal recycling image, and the sketch map (SMAP) may correspond to a feature map of the artificial intelligence model. Specifically, the artificial intelligence model of the sketch map creation module 120 may extract features by filtering images of received normal recycling products and then generate a sketch map (SMAP) based on the extracted features. For example, the sketch map generation module 120 according to an embodiment of the present disclosure may perform Gabor filtering and Laplacian of Gaussian (LoG) filtering to extract boundary lines with high accuracy, The artificial intelligence model may include two encoders for performing each filtering and a decoder for generating the sketch map. An exemplary configuration of the artificial intelligence model included in the sketch map creation module 120 is described in more detail with reference to FIG. 3, and examples of sketch maps (SMAP) are shown in FIGS. 4 to 8.

The sketch similarity calculation module 130 can calculate the similarity between the sketch map (SMAP) generated in the sketch map generation module 120 and the sketch map corresponding to each of the images included in the image database (20 in FIG. 1). there is. A sketch map corresponding to each image included in the image database (20 in FIG. 1) may be created using the same artificial intelligence model as the artificial intelligence model included in the sketch map creation module 120.

The sketch similarity calculation module 130 corresponds to the image calculated to have the highest similarity to the sketch map (SMAP) generated by the sketch map creation module 120 among the images included in the image database (20 in FIG. 1). The glass bottle may be determined as the type of glass bottle indicated by the first image data (IDAT1). As described above, since each type of recyclable product may have a unique borderline shape, the similarity between the borderlines can be a criterion for determining the type of recyclable product. The classification result for the first image data (IDAT1) may be provided to the color similarity calculation module 140 for additional classification operation.

The color similarity calculation module 140 calculates the color similarity between the normal recyclable product represented by the first image data (IDAT1) and the images included in the image database (20 in FIG. 1), and the damaged recyclable product represented by the second image data (IDAT2). The color similarity between images included in the and image database (20 in FIG. 1) can be calculated. Here, the color may include both the color of the recycling container and the color of the label attached to the recycling product. For example, the color similarity calculation module 140 may include an artificial intelligence model for performing calculations based on color information of an image.

In particular, when calculating color similarity to a normal recyclable product (e.g., when the received image is labeled as a normal recyclable product), the color similarity calculation module 140 of the present disclosure calculates the calculation result of the sketch similarity calculation module 130. For reference, among the images belonging to the image database (20 in FIG. 1), color similarity can be calculated only for images representing recycled products that are the same type of normal recycled products indicated by the first image data (IDAT1). By first performing the calculation on the border of the image in this way, there is no need to calculate color similarity for all images in the image database (20 in Figure 1), thus reducing the amount of calculation required for the entire classification process and enabling fast and accurate classification. It can happen.

The color similarity calculation module 140 calculates first image data (IDAT1) or second image data according to the color of the recycled product corresponding to the image with the highest color similarity calculated among the images included in the image database (20 in FIG. 1). Recyclables indicated by (IDAT2) can be classified. For example, in the case of glass bottles, color similarity calculations show that normal glass bottles are classified according to their color (e.g., a green bottle, a brown bottle) or according to the company indicated on the label (e.g., a beer bottle or a soju bottle). (by company) can be further classified, and broken glass bottles can be sorted according to color.

The color similarity calculation module 140 may generate control signals CTRL1 and CTRL2 for controlling the conveyor belts (14 and 15 in FIG. 1) according to the final classification result. Specifically, the first control signal CTRL1 may control the first conveyor belt (14 in FIG. 1) to transport the recyclables corresponding to the first image data IDAT1 to a sorting bin corresponding to the sorting result. For example, in the case of a glass bottle, the glass bottle represented by the first image data (IDAT1) is classified as a beer bottle through the sketch similarity calculation module 130, and is classified as a beer bottle of company A through the color similarity calculation module 140. In this case, the first conveyor belt (14 in FIG. 1) may transport glass bottles to a sorting bin corresponding to beer bottles of company A according to the first control signal (CTRL1). Likewise, the second control signal CTRL2 may control the second conveyor belt (15 in FIG. 1) to transport the broken glass bottle corresponding to the second image data IDAT2 to a sorting bin corresponding to the sorting result.

As described with reference to FIG. 1, the artificial intelligence model included in the recycling product sorting device 100 can be replaced depending on the type of recyclable product to be classified. For example, if you want to classify plastic bottles, the artificial intelligence model may be an artificial intelligence model trained to create a sketch map for plastic bottles and perform similarity calculations, and if you want to classify glass bottles, the artificial intelligence model may be an artificial intelligence model trained to create a sketch map for plastic bottles and perform similarity calculations. It may be an artificial intelligence model trained to generate a sketch map and perform similarity calculations. However, even if the type of recycled product is different, the overall operation of the sketch map creation module 120, sketch similarity calculation module 130, and color similarity calculation module 140 described above is the same.

FIG. 3 is a block diagram showing an example configuration of an artificial intelligence model included in the sketch map creation module 120 of FIG. 2. As explained with reference to FIG. 2, the artificial intelligence model 200 of FIG. 3 uses two encoders (a first encoder 210 and a second encoder 220) and one decoder 230, 1 It may be implemented to generate a sketch map (SMAP) corresponding to the image of a normal recyclable product represented by the image data (IDAT1). The first encoder 210 and the second encoder 220 may filter the first image data (IDAT1) using different algorithms, and divide the boundary of the recycled product image into the first feature (F1) and the second feature (F2). ), and the first feature (F1) and the second feature (F2) can be concatenated into one feature (F). For example, the first feature (F1) and the second feature (F2) extracted using different algorithms may have different sparsities. The decoder 230 may decode the combined features (F) to generate a sketch map (SMAP) of the recycled product image. As described below with reference to FIG. 2, an embodiment in which the first encoder 210 performs Gabor filtering and the second encoder 220 performs Gaussian Laplacian (LoG) filtering will be described.

The first encoder 210 may be configured using a Gabor convolution layer to perform Gabor filtering on the first image data IDAT1. When extracting the border of a recyclable image, information about the label attached to the recyclable or the background area where the recyclable is placed is not required. By performing Gabor filtering, local information related to the label or background area is minimized, and the recyclable is Global information related to its boundaries (i.e., related to its shape) can be extracted.

Specifically, the Gabor convolution layer constituting the first encoder 210 may be generated by applying a Gabor filter to an arbitrary learned filter. First, a learned filter with N random channels and size W is created. Then, the Gabor filter is That is, the learned filter with the Gabor filter applied is calculated as in Equation 1 below.

Here, u and v are parameters representing the orientation and scale of the Gabor filter, respectively, and the direction of the Gabor filter can be set to U and the size to V in advance. Convolutional Gabor Orientation Filter (GoF), which will perform Gabor filtering on the Gabor convolution layer. is defined as Equation 2 below.

By passing the input feature map F as shown in Equation 3 below through the convolutional Gabor direction filter created in this way, the Gabor convolution layer of the first encoder 210 produces an output feature map. Can be learned to generate . For example, the input feature map F could be an image of a random recyclable product, and the output feature map may be a sketch map corresponding to the image of the input feature map F. For simple notation below, The v notation is omitted.

Additionally, the output feature map Each channel of can be obtained through a convolution operation as shown in Equation 4 below.

where n is F and represents the channel of is the output feature map The response corresponding to the kth direction can be expressed. In the process of obtaining the output feature map through the above-described process, a filter is learned to accurately generate a sketch map corresponding to the input image. can be continuously updated. For example, a learned filter Can be updated using a back-propagation algorithm as shown in Equation 5 and Equation 6 below.

Here, L may represent a loss function related to the accuracy of the sketch map. Through this, the first encoder 210 can be trained to more accurately extract the first feature (F1) related to the boundary line of the recycled product image indicated by the first image data (IDAT1). However, as described above, the first feature (F1) extracted from the first encoder 210 composed of a Gabor convolution layer may be very sparse because it mainly includes only global information. That is, the accuracy of the sketch map generated using only the rare first feature (F1) may be somewhat low.

To compensate for this, the second encoder 220 may perform Gaussian Laplacian filtering on the first image data IDAT1. The Gaussian Laplacian filtered image of the recyclable product can be passed through convolutional layers and extracted as a second feature (F2). Unlike the Gabor convolution layer, which minimizes local information and mainly extracts only global information, a normal convolution layer extracts not only global information but also local information, so the second feature (F2) is different from the first feature (F2). It may be less rare than F1). For example, the convolution layer of the second encoder 220 may also be updated (eg, updated using a backpropagation algorithm) in a similar manner to the first encoder 210 described above.

In this way, by combining the first feature (F1) and the second feature (F2) with different scarcity into one feature (F), information about the border of recycled products can be enriched, and the combined feature (F) By being input to this decoder 230, the artificial intelligence model 200 can generate a more accurate sketch map (SMAP) than when using only one encoder. In addition, the artificial intelligence model 200 learned in this way can be equally used to generate a sketch map of images included in the image database (20 in FIG. 1), and as described with reference to FIG. 2, the first image data The similarity between the sketch map (SMAP) generated through (IDAT1) and the sketch map corresponding to the images included in the image database (20 in FIG. 1) can be calculated in the sketch similarity calculation module (130 in FIG. 2). And recyclables can be classified by type according to their unique boundary shape.

Although an embodiment of performing Gabor filtering and Gaussian Laplacian filtering has been described with reference to FIG. 3, the present disclosure is not limited thereto, and the artificial intelligence model 200 includes a first encoder 210 and a second encoder 220, respectively. It can be implemented to use any different filtering methods that can extract the first feature (F1) and the second feature (F2) with higher sparsity than the first feature (F1).

So far, with reference to FIGS. 2 and 3 , an embodiment of the classification operation of recyclables and, in particular, the classification operation by type according to the type of the recyclables has been described. Hereinafter, through FIGS. 4 to 8, an example of the sketch map (SMAP) described with reference to FIGS. 2 to 3 is shown.

Figures 4 to 8 show examples of sketch maps generated through the operation of the artificial intelligence model 200 of Figure 3 in an embodiment in which the recyclable product to be classified is a glass bottle. Figure 4 shows a beer bottle, Figures 5 and 6 show a soju bottle, and Figures 7 and 8 show a beverage bottle and a corresponding sketch map. Referring to Figures 4 to 8, the border of the glass bottle is clearly shown on the sketch maps, but the border of the label is blurred. Using the sketch maps created in this way, the sketch similarity calculation module (130 in FIG. 2) can calculate the similarity between the input sketch map and each sketch map of the images stored in the database, and classify glass bottles by type. there is.

Figure 9 conceptually shows the overall operation of the recycling classification system 10 in an embodiment in which the recycling product to be classified is a glass bottle. For clear explanation, Figure 9 assumes that the recyclable product subject to classification is a glass bottle, and is described below with reference to Figures 1 to 3 along with Figure 9. First, through the operation of the radar sensor 11, glass bottles can be classified into normal glass bottles and broken glass bottles. Normal glass bottles are primarily classified into types (e.g., beer bottles, soju bottles, beverage bottles) according to the unique shape of the boundary line through the operation of the sketch map generation module 120 and the sketch similarity calculation module 130. You can. Thereafter, the glass bottles separated by type can be further classified by color or company through the operation of the color similarity generation module 140. For example, beer bottles can be classified into beer bottle 1 and beer bottle 2, soju bottles can be classified into soju bottle 1 and soju bottle 2, and beverage bottles can be classified into beverage bottle 1 and beverage bottle 2, where beer bottle 1 and beer bottle 2, soju bottle 1 and Soju bottle 2, beverage bottle 1, and beverage bottle 2 may have different bottle colors (e.g., beer bottle 1 is a brown bottle, beer bottle 2 is a transparent bottle), and may be from different companies (e.g., beer bottle 1 is company A). , beer bottle 2 is from company B). As described above, according to the embodiment of the present disclosure, the sorting operation according to shape (shown by a dotted line) is performed prior to the sorting operation according to color, so the overall amount of computation can be reduced and the speed of sorting recycled glass bottles can be improved. there is.

The broken glass bottle travels on a conveyor belt (e.g., the second conveyor belt 15) that is different from the conveyor belt that carries the normal glass bottle (e.g., the first conveyor belt 14), and the color similarity calculation module Through the operation of (140), they can be classified by color. That is, the colors of broken glass bottle 1, broken glass bottle 2, and broken glass bottle 3 are different. Depending on the final classification result, the recycling glass bottle sorting device 100 may generate control signals (CTRL1, CTRL2), and conveyor belts 14 so that each glass bottle can be transported to the sorting box corresponding to the sorting result. 15) can be controlled. So far, the overall operation of the recycling classification system 10 has been shown focusing on the embodiment of classifying glass bottles, but the present disclosure is not limited to this, and even in the case of recycling products other than glass bottles, the overall operation is similar to that shown in FIG. 9. something to do.

Figure 10 is a flowchart showing a method for classifying recyclables according to an embodiment of the present disclosure. Hereinafter, the description will be made with reference to FIGS. 1 and 2 along with FIG. 10.

In step S110, the radar sensor 11 may classify the plurality of recyclables into normal recyclables and damaged recyclables. In step S120, the camera 12 captures first image data IDAT1 of normal recyclables on the first conveyor belt 14 and second image data IDAT2 of damaged recyclables on the second conveyor belt 15. Can be provided as a recycling classification device (13). Steps S130 to S170 below show a method of operating the recycling classification device 13 in the recycling classification system 10.

In step S130, the preprocessing module 110 may preprocess the first image data (IDAT1) and the second image data (IDAT2) to be suitable as input for an artificial intelligence model for classifying recycled products. In step S140, the sketch map creation module 120 and the sketch similarity calculation module 130 may classify the normal recyclables included in the first image data (IDAT1) by type according to shape, and in step S150, the color similarity calculation module (140) can further classify recycled products classified by type according to color. In step S160, the color similarity calculation module 140 may classify damaged recyclables included in the second image data IDAT2 according to color. Finally, in step S170, the color similarity calculation module 140 generates a first control signal (CTRL1) and a second control signal (CTRL2) according to the final classification result, so that the first conveyor belt 14 and the second conveyor Each belt 15 can be controlled.

FIG. 11 is a flowchart showing in more detail the step (S140) of classifying the normal recyclables of FIG. 10 into types according to shape. Hereinafter, it will be described with reference to FIGS. 1 to 3 along with FIG. 11 , and in particular, steps S141 to S143 correspond to the operation of the artificial intelligence model 200 of the sketch map creation module 120.

In step S141, the first encoder 210 may extract the first feature (F1) by performing Gabor filtering on the image of the normal recyclable product included in the first image data (IDAT1). And in step S142, the second encoder 220 may extract the second feature (F2) by performing Gaussian Laplacian filtering on the image of the normal recyclable product included in the first image data (IDAT1). In step S143, the decoder 230 may generate a sketch map (SMAP) using a feature (F) that is a combination of the first feature (F1) and the second feature (F2). Finally, in step S144, the sketch similarity calculation module 130 may complete step S140 of FIG. 10 by calculating the similarity between the generated sketch map (SMAP) and the sketch map of the image stored in the image database 20. For example, the operations described with reference to FIGS. 10 and 11 may be performed to sort recycled glass bottles. Furthermore, steps S141 and S142 are shown to use Gabor filtering and Gaussian Laplacian filtering, respectively, but the present disclosure is not limited thereto, and steps S141 and S142 use a first feature (F1) having different sparsities. ) and other types of filtering that can extract the second feature (F2) can be performed.

FIG. 12 is an exemplary hardware configuration diagram when the recycling classification device 100 of FIG. 2 is implemented as a computing device 500. As shown in FIG. 12, the computing device 500 loads one or more processors 510, a bus 550, a communication interface 570, and a computer program 591 performed by the processor 510. It may include a memory 530 that stores a computer program 591 and a storage 590 that stores a computer program 591. However, only components related to the embodiment of the present disclosure are shown in FIG. 12 . Accordingly, those skilled in the art to which this disclosure pertains can recognize that other general-purpose components may be included in addition to the components shown in FIG. 12 .

The processor 510 controls the overall operation of each component of the computing device 500. The processor 510 includes at least one of a Central Processing Unit (CPU), Micro Processor Unit (MPU), Micro Controller Unit (MCU), Graphic Processing Unit (GPU), or any type of processor well known in the art of the present disclosure. It can be configured to include. Additionally, the processor 510 may perform operations on at least one application or program to execute methods/operations according to various embodiments of the present disclosure. Computing device 500 may include one or more processors.

Bus 520 provides communication functionality between components of computing device 500. The bus 520 may be implemented as various types of buses, such as an address bus, a data bus, and a control bus.

The communication interface 530 supports wired and wireless Internet communication of the computing device 500. The communication interface 530 may support various communication methods other than Internet communication. To this end, the communication interface 530 may be configured to include a communication module well known in the technical field of the present disclosure.

Memory 540 stores various data, commands and/or information. The memory 540 may load one or more programs 591 from the storage 550 to execute methods/operations according to various embodiments of the present invention. For example, when the computer program 551 is loaded into the memory 540, modules as shown in FIG. 2 (e.g., 110, 120, 130, and 140 of FIG. 2) are loaded on the memory 540. It can be implemented. An example of the memory 530 may be RAM, but is not limited thereto.

Storage 550 may non-temporarily store one or more computer programs 551. The storage 550 is a non-volatile memory such as Read Only Memory (ROM), Erasable Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), flash memory, a hard disk, a removable disk, or a device well known in the technical field to which the present invention pertains. It may be configured to include any known type of computer-readable recording medium.

The computer program 551 may include one or more instructions implementing methods/operations according to various embodiments of the present disclosure. When the computer program 551 is loaded into the memory 540, the processor 510 can perform methods/operations according to various embodiments of the present disclosure by executing the one or more instructions.

Several embodiments of the present disclosure have been described so far with reference to FIGS. 1 to 12. In particular, according to the embodiment of the present disclosure, the process of sorting recycled glass bottles among recycled products can be efficiently performed. Hereinafter, the effects according to an embodiment of the present disclosure will be explained using the classification of recycled glass bottles as an example.

According to an embodiment of the present disclosure, the task of manually sorting recycled glass bottles can be performed automatically based on artificial intelligence. In addition, according to an embodiment of the present disclosure, even if a new type of glass bottle is added, the learned artificial intelligence model can be used to generate a sketch map without the need to learn a new artificial intelligence model, thereby saving time and cost.

Furthermore, according to the embodiment of the present disclosure, classification by type is first performed according to the shape of the border forming the glass bottle, and then detailed classification (classification by color or classification by company) is performed using color information, Compared to the case of trying to sort recycled glass bottles using color information from the beginning, the computational amount can be significantly reduced. In particular, in the case of artificial intelligence models that perform learning and inference using color information (for example, convolutional neural networks using RGB images), new models must be developed because operation performance is not smooth in embedded systems. By prioritizing glass bottle classification according to shape, the need to develop a new model that processes images using color information can be reduced.

Also, when extracting the boundary line of a glass bottle using an artificial intelligence model, a method that mainly considers only global information (e.g., Gabor filtering) and a method that considers both global information and local information (e.g., Gaussian Laplacian filtering) ) can be used together to increase the accuracy of sketch map creation. In this way, as the efficiency of artificial intelligence model learning improves and the amount of computation is reduced, the speed of sorting recycled glass bottles can be accelerated. The effects of the present disclosure are not limited to the effects mentioned above, and the type of recyclable product is not limited to glass bottles, and an artificial intelligence model trained to perform boundary-based calculations according to the type of recyclable product is downloaded. Since the above-mentioned effects can be applied to any type of recyclable product whose type can be determined by the boundary line. In other words, even if the recyclables to be classified change, the recyclables classification system 10 according to the embodiment of the present disclosure can be used as is, as long as the artificial intelligence model is replaced. In addition, other effects not mentioned will be clearly understood by those skilled in the art from the description below.

The concept of the present disclosure described so far with reference to FIGS. 1 to 12 may be implemented as computer-readable code on a computer-readable medium. The computer-readable recording medium may be, for example, a removable recording medium (CD, DVD, Blu-ray disk, USB storage device, removable hard disk) or a fixed recording medium (ROM, RAM, computer-equipped hard disk). You can. The computer program recorded on the computer-readable recording medium can be transmitted to another computing device through a network such as the Internet, installed on the other computing device, and thus used on the other computing device.

In the above, even though all the components constituting the embodiments of the present disclosure have been described as being combined or operated in combination, the present disclosure is not necessarily limited to these embodiments. That is, within the scope of the purpose of the present disclosure, all of the components may be operated by selectively combining one or more of them.

Although operations are shown in the drawings in a specific order, it should not be understood that the operations must be performed in the specific order shown or sequential order or that all illustrated operations must be performed to obtain the desired results. In certain situations, multitasking and parallel processing may be advantageous. Moreover, the separation of the various components in the embodiments described above should not be construed as necessarily requiring such separation, and the program components and systems described may generally be integrated together into a single software product or packaged into multiple software products. You must understand.

Although embodiments of the present disclosure have been described above with reference to the attached drawings, those skilled in the art will understand that the present disclosure can be implemented in other specific forms without changing its technical idea or essential features. can understand. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. The scope of protection of this disclosure should be interpreted in accordance with the claims below, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of rights of this disclosure.

Claims (10)

A radar sensor that radiates electromagnetic waves to a plurality of recyclables and classifies the plurality of recyclables into undamaged normal recyclables and broken recyclables;
a camera that generates first image data capturing the normal recyclable product and second image data capturing the damaged recyclable product;
The normal recyclables are classified by type according to the type of the normal recyclables indicated by the first image data, the recyclables classified by type are classified by color, and the damaged recyclables indicated by the second image data are classified by color. A sorting device for recyclables that sorts them according to;
A first conveyor belt on which the normal recyclables are transported; and
Including a second conveyor belt on which the damaged recyclables are transported,
The recycling classification device is,
Based on the classification result of the normal recyclable product and the classification result of the damaged recyclable product, outputting a first control signal for controlling the first conveyor belt and a second control signal for controlling the second conveyor belt,
Recyclables sorting system. According to claim 1,
The recycling classification device is,
a preprocessing module that preprocesses the first image data and the second image data;
a sketch map generation module that generates a sketch map by extracting a boundary line of the normal recyclable product indicated by the preprocessed first image data;
a sketch similarity calculation module that calculates similarity between the sketch map and each sketch map of recycled product images stored in an image database; and
A color similarity calculation module that calculates the similarity between the color of the normal recyclable product indicated by the pre-processed first image data and the damaged recyclable product indicated by the pre-processed second image data, and the colors of each of the recyclable product images stored in the image database. Including,
Recyclables sorting system. According to claim 2,
The sketch similarity calculation module,
Determining the type corresponding to the recycling product image with the highest similarity among the recycling product images stored in the image database as the type of the normal recycling product,
Recyclables sorting system. According to claim 2,
The sketch map creation module,
a first encoder that extracts first features by performing first filtering on the preprocessed first image data;
a first encoder that performs second filtering on the pre-processed first image data to extract second features with higher sparsity than the first features; and
Comprising a decoder that generates the sketch map based on a combination of the first feature and the second feature,
Recyclables sorting system. According to claim 4,
The first filtering is Gabor filtering, and the second filtering is Laplacian of Gaussian (LoG) filtering,
Recyclables sorting system. A method of sorting recycled glass bottles by a computing device, comprising:
Generating first image data of a normal glass bottle and second image data of a broken glass bottle;
Classifying the normal glass bottle into types according to the shape of the normal glass bottle indicated by the first image data;
Sorting the glass bottles classified by type according to the color of the normal glass bottles; and
Comprising the step of classifying the broken glass bottle represented by the second image data according to color,
How to sort recycled glass bottles. According to claim 6,
Further comprising preprocessing the first image data and the second image data,
The steps for classifying the normal glass bottles by type are:
extracting first features by performing first filtering on the first image data;
performing second filtering on the second image data to extract a second feature with higher sparsity than the first feature;
generating a sketch map using a combination of the first feature and the second feature; and
Comprising the step of calculating similarity between the sketch map and the sketch map of each glass bottle image stored in the image database,
How to sort recycled glass bottles. According to claim 7,
The step of classifying the normal glass bottles by type is,
Further comprising determining as the type of the normal glass bottle the type corresponding to the glass bottle image for which the similarity is calculated to be highest among the glass bottle images stored in the image database,
How to sort recycled glass bottles. According to claim 7,
The first filtering is Gabor filtering, and the second filtering is Laplacian of Gaussian (LoG) filtering,
How to sort recycled glass bottles. As a method of sorting recycled products,
Classifying a plurality of recyclables into normal recyclables and damaged recyclables using a radar sensor;
Generating first image data by photographing the normal recyclable product and second image data by photographing the damaged recyclable product;
Classifying the normal recyclable products by type according to the type of the normal recyclable product indicated by the first image data;
Sorting the recycled products classified by type according to color;
classifying the damaged recyclables indicated by the second image data according to color;
Controlling a first conveyor belt through which the normal recyclables are transported according to the classification results of the normal recyclables; and
Comprising the step of controlling a second conveyor belt on which the damaged recyclables are transported according to the classification result of the damaged recyclables,
How to sort recyclables.

Images