KR102613341B1 - Method and device for reading cargo improved through augmentation of training data based on deep learning - Google Patents

Abstract

According to an embodiment of the present invention, a deep learning-based cargo recognition method through cargo image augmentation performed by a server is: a) extracting some or all of the previously collected cargo images and setting them as reference images; Learning an image with a first deep learning model based on image conversion and generating a virtual cargo image by converting the reference image according to preset external factors; b) Set the previously collected cargo image and the virtual cargo image as learning data, learn the learning data through a second deep learning model based on object recognition, and when a random cargo image is input, the type of the preset cargo A step of building a cargo recognition model that classifies cargo into one of the following; and c) inputting a new cargo image for cargo captured in real time by a camera placed at a work site into the cargo recognition model to identify an object representing the cargo in the new cargo image and determining the type of the cargo. Includes.

Description

Cargo recognition method and device improved through deep learning-based training data augmentation {METHOD AND DEVICE FOR READING CARGO IMPROVED THROUGH AUGMENTATION OF TRAINING DATA BASED ON DEEP LEARNING}

The present invention is in the field of image deep learning technology used in the logistics industry, and relates to an improved cargo recognition method and device through augmentation of training data of deep learning by generating a virtual cargo image using an image conversion technique.

Recently, technology has emerged that recognizes objects in images and determines the type or state of the object by analyzing images through deep learning techniques. This image analysis technology is also expected to be useful in the logistics industry, such as cargo distribution and delivery.

For example, a technology is being developed to determine the type or damage of the cargo by installing a camera to photograph cargo at specific points and inputting the captured images into a deep learning model.

However, due to the nature of the logistics industry, the location and time of distribution for each cargo is very different, and the types are also very diverse, so if images for training are not secured to reflect this, there is a problem in ensuring an accurate recognition rate.

For example, the location where a specific cargo is loaded may be indoors, but the location where it is sorted for delivery may be outdoors, and time or weather may continuously change depending on the distribution process. In other words, even if an image is taken of the same cargo, the shading, brightness, and color tone may vary depending on external factors such as location, time, and weather.

Therefore, for precise learning, cargo images reflecting the above external factors must be collected, but this is realistically impossible because it requires excessive time, effort, and capital. For example, filming the same cargo multiple times at different times in the morning and afternoon, or placing cameras at each location where the distribution process takes place, actually causes delays in logistics processing time, unnecessary labor, and capital, and causes the original It may cause results that are contrary to the purpose.

Therefore, in order to utilize image deep learning in the logistics industry, in addition to collecting actually captured cargo images, a technology is needed to additionally secure cargo images whose style has been modified by reflecting external factors as training data.

Meanwhile, deep learning technology has recently progressed from object recognition to technology that converts or creates images. In general, object recognition in images mainly uses Convolutional Neural Networks (CNN) series models to learn patterns of given data, but image creation and transformation uses Generative Adversarial Networks (GAN) series models. The model is used.

An adversarial neural network learns a generative model (G) that approximates the distribution of images, and is a model that can generate virtual images that are likely to exist rather than images that actually exist.

Referring to Figure 1, an adversarial neural network consists of two networks: a generator and a discriminator. The generator creates a virtual image (Fake image) to deceive the discriminator, and the discriminator determines the authenticity of the incoming image. They learn as if competing with each other, and as a result, the virtual image created by the creator becomes very similar to the real image.

However, since the purpose of an adversarial neural network is to derive images similar to real images, the virtual image created by the generator may be similar to the real image, but the problem is that it may be created as an image that is completely unrelated to the original intended. This exists. In other words, if the actual images used for learning are the same, the same and similar images may be output regardless of the original input.

To compensate for these limitations, the pix2pix model, an improved adversarial neural network, was developed. pix2pix is a type derived from cGAN (conditional GAN) that inputs conditions that control the learning mode. It receives the image itself as a condition and is one of the Image to image translation techniques that receives pixels as input and predicts virtual pixels.

pix2pix performs learning by pairing two pieces of data that match each other. In other words, only the input and domain are different, but the corresponding correct answers are paired together to form a data set, and learning is performed until the generated virtual image is close to the correct answer.

In other words, pix2pix has a limitation that the learning data must consist of only matched pairs. Accordingly, tasks such as converting a hand drawing of the same object to a photo or converting a color image to a black and white image are possible, but learning is not possible for images that do not match each other and do not form a pair, or incorrect results are obtained. There are limits.

In this regard, the concept of obtaining a transformed virtual image by reflecting external factors in the previously collected cargo image using a GAN-type neural network can be introduced into the logistics industry.

However, as mentioned above, there are too many restrictions to photograph the same type of cargo in different locations, times, weather, etc., and even if it is done, it is inevitable that the shape of the image containing the cargo will change, so images paired with existing cargo images are used. It is virtually impossible to secure.

Therefore, unlike the conventional pix2pix, improved deep learning that does not require pairing is applied to secure virtual cargo images by preserving the form of the existing collected originals but only modifying the style according to external factors, thereby providing more precise images. There is a need for technology that promotes cargo recognition.

The present invention is intended to solve the problems of the prior art described above. Based on a deep learning model that converts the style of the image, the actually captured cargo image is modified to suit environmental changes to create a virtual cargo image, thereby generating robust training data. The purpose is to provide a cargo recognition method and device with improved recognition rates through this technology.

However, the technical challenges that this embodiment aims to achieve are not limited to the technical challenges described above, and other technical challenges may exist.

According to an embodiment of the present invention, a deep learning-based cargo recognition method through cargo image augmentation performed by a server is: a) extracting some or all of the previously collected cargo images and setting them as reference images; Learning an image with a first deep learning model based on image conversion and generating a virtual cargo image by converting the reference image according to preset external factors; b) Set the previously collected cargo image and the virtual cargo image as learning data, learn the learning data through a second deep learning model based on object recognition, and when a random cargo image is input, the type of the preset cargo A step of building a cargo recognition model that classifies cargo into one of the following; and c) inputting a new cargo image for cargo captured in real time by a camera placed at a work site into the cargo recognition model to identify an object representing the cargo in the new cargo image and determining the type of the cargo. Includes.

According to an embodiment of the present invention, the external factors include location, weather, and time as surrounding environments that can change during the distribution process of cargo.

According to an embodiment of the present invention, in step a), the configuration, position, and shape of the object included in the reference image by the first deep learning model are preserved and the reference image is changed according to the external factors. It includes the step of generating a virtual cargo image with a modified style.

According to one embodiment of the present invention, step a) is the difference between the first loss function corresponding to the image suitability for the virtual cargo image and the result of converting the virtual cargo image back to the original and the reference image. It includes repeating learning of the first deep learning model in the direction of minimizing the second loss function corresponding to .

According to an embodiment of the present invention, in step a), at least one conversion condition of the reference image for generating the virtual cargo image is set for each external factor, and the first deep learning is performed together with the reference image. It is applied as input data to the model.

According to one embodiment of the present invention, the conversion conditions are condition values for color temperature, brightness, shade, and illuminance divided into preset numerical ranges, or pre-collected random images are classified based on the condition values.

According to one embodiment of the present invention, when the external factor is a place, the conversion condition is set to a color temperature divided into two different numerical ranges corresponding to outdoor and indoor, and when the external factor is weather, The conversion conditions are set to color temperature, brightness, and shade divided into two or more different numerical ranges corresponding to two or more preset clear and cloudy degrees, and when the external factor is time, the conversion conditions are preset. It is set with color temperature, shading, and illuminance divided into two or more different numerical ranges corresponding to two or more time periods.

According to one embodiment of the present invention, in step a), the conditions of the cargo exterior, which are preset in response to packing or damage to the cargo, are learned with the first deep learning model along with the reference image to match the cargo exterior conditions. It includes generating a virtual cargo image by converting the reference image.

According to an embodiment of the present invention, step b) includes, in addition to the first object representing the cargo in the learning data, a second object representing damage, a third object representing an invoice, and a fourth object representing a barcode. It includes performing training of the second deep learning model to identify at least one or more.

According to an embodiment of the present invention, step c) determines whether the cargo is damaged and the type of damage depending on whether the second object in the new cargo image is identified by the cargo recognition model, and the cargo recognition model When a third or fourth object in the new cargo image is identified, it includes obtaining detailed data of the cargo based on the identified third or fourth object.

According to an embodiment of the present invention, a cargo recognition server includes: a memory storing a program for performing a deep learning-based cargo recognition method through cargo image augmentation; And a processor that executes the program, wherein the processor extracts some or all of the previously collected cargo images and sets them as a reference image, learns the reference image as a first deep learning model based on image conversion, and sets the preset image as a reference image. an image enhancement unit that generates a virtual cargo image by converting the reference image according to external factors; The previously collected cargo image and the virtual cargo image are set as learning data, and the learning data is learned through a second deep learning model based on object recognition. When a random cargo image is input, it is selected as one of the preset cargo types. A learning unit that builds a cargo recognition model to classify; And a cargo recognition unit that inputs a new cargo image for cargo captured in real time by a camera placed at the work site into the cargo recognition model, identifies objects representing the cargo in the new cargo image, and determines the type of the cargo. Includes.

According to one embodiment of the present invention, the image augmentation unit adjusts the style of the reference image according to the external factors while preserving the configuration, position, and shape of the object included in the reference image through the first deep learning model. This transformed virtual cargo image is created.

According to one embodiment of the present invention, the image augmentation unit determines the difference between a first loss function corresponding to the image suitability for the virtual cargo image, a result of converting the virtual cargo image back to the original, and the reference image. Training of the first deep learning model is repeated in the direction of minimizing the corresponding second loss function.

According to one embodiment of the present invention, the image augmentation unit sets at least one conversion condition of the reference image for generating the virtual cargo image for each external factor and generates the first deep learning model together with the reference image. Apply as input data.

According to one embodiment of the present invention, the image enhancement unit converts the condition values for color temperature, luminance, shade, and illuminance divided into preset numerical ranges, or the previously collected random images classified based on the condition values. Set as a condition.

According to one embodiment of the present invention, when the external factor is a location, the image enhancement unit sets the color temperature divided into two different numerical ranges corresponding to outdoor and indoor as the conversion condition, and the external factor is set to the conversion condition. In the case of weather, color temperature, brightness, and shading divided into two or more different numerical ranges corresponding to two or more preset degrees of clear and cloudy are set as the conversion conditions, and if the external factor is time, the preset Color temperature, shading, and illuminance divided into two or more different numerical ranges corresponding to two or more time periods are set as the conversion conditions.

According to one embodiment of the present invention, the image augmentation unit learns the conditions of the cargo exterior that are preset in response to packing or damage of the cargo with the first deep learning model along with the reference image, and according to the conditions of the cargo exterior. A virtual cargo image is created by converting the reference image.

According to one embodiment of the present invention, the learning unit, in addition to the first object representing the cargo in the learning data, at least one of a second object representing damage, a third object representing an invoice, and a fourth object representing a barcode. The second deep learning model is trained to identify abnormalities.

According to one embodiment of the present invention, the cargo recognition unit determines whether the cargo is damaged and the type of damage depending on whether the second object in the new cargo image is identified by the cargo recognition model, and determines whether the cargo is damaged and the type of damage according to the cargo recognition model. If a third or fourth object is identified in the new cargo image, detailed data of the cargo is obtained based on the identified third or fourth object.

According to an embodiment of the present invention, it is possible to automatically secure not only actually captured cargo images but also virtual cargo images to which various external factors due to environmental changes are applied.

In other words, since labeling only needs to be done as before without the need to create an artificial environment and photograph the cargo, time, effort, and capital are significantly reduced.

Accordingly, more extensive learning data is constructed and precise learning is performed, so a robust cargo recognition model can be constructed.

Therefore, even if the environment changes due to location, weather, time, etc. during the distribution of cargo, misidentification as another cargo is prevented, and the accuracy of damage detection and decoding of invoices or barcodes can also be improved.

Figure 1 is a conceptual diagram for explaining an adversarial neural network.
Figure 2 is a structural diagram of a cargo recognition system according to an embodiment of the present invention.
Figure 3 is a block diagram showing the configuration of a cargo recognition server according to an embodiment of the present invention.
Figure 4 is a block diagram showing the configuration of a processor according to an embodiment of the present invention.
Figure 5 is an example diagram for explaining a first deep learning model according to an embodiment of the present invention.
Figure 6 is an example diagram of a virtual cargo image created according to an embodiment of the present invention.
Figure 7 is a conceptual diagram for explaining a cargo recognition model according to an embodiment of the present invention.
Figure 8 is an operation flowchart of a method for determining cargo damage according to an embodiment of the present invention.
Figure 9 is a flowchart of a cargo recognition method according to an embodiment of the present invention.

Below, with reference to the attached drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily implement the present invention. However, the present invention may be implemented in many different forms and is not limited to the embodiments described herein. In order to clearly explain the present invention in the drawings, parts that are not related to the description are omitted, and similar parts are given similar reference numerals throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected," but also the case where it is "electrically connected" with another element in between. . Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

In this specification, 'part' includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Additionally, one unit may be realized using two or more pieces of hardware, and two or more units may be realized using one piece of hardware. Meanwhile, '~ part' is not limited to software or hardware, and '~ part' may be configured to reside in an addressable storage medium or may be configured to reproduce one or more processors. Therefore, as an example, '~ part' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuits, data, databases, data structures, tables, arrays, and variables. The functions provided within the components and 'parts' may be combined into a smaller number of components and 'parts' or may be further separated into additional components and 'parts'. Additionally, components and 'parts' may be implemented to regenerate one or more CPUs within a device or a secure multimedia card.

The “terminal” mentioned below may be implemented as a computer or portable terminal that can connect to a server or other terminal through a network. Here, the computer is, for example, a laptop equipped with a web browser, a desktop, a laptop, a VR HMD (e.g., HTC VIVE, Oculus Rift, GearVR, DayDream, PSVR, etc.), etc. may include. Here, VR HMD is for PC (e.g. HTC VIVE, Oculus Rift, FOVE, Deepon, etc.), mobile (e.g. GearVR, DayDream, Storm Magic, Google Cardboard, etc.), and console (PSVR). Includes independently implemented Stand Alone models (e.g. Deepon, PICO, etc.). Portable terminals are, for example, wireless communication devices that ensure portability and mobility, including smart phones, tablet PCs, and wearable devices, as well as Bluetooth (BLE, Bluetooth Low Energy), NFC, RFID, and ultrasonic devices. , may include various devices equipped with communication modules such as infrared, WiFi, and LiFi. In addition, “network” refers to a connection structure that allows information exchange between nodes such as terminals and servers, including a local area network (LAN), a wide area network (WAN), and the Internet. (WWW: World Wide Web), wired and wireless data communication network, telephone network, wired and wireless television communication network, etc. Examples of wireless data communication networks include 3G, 4G, 5G, 3GPP (3rd Generation Partnership Project), LTE (Long Term Evolution), WIMAX (World Interoperability for Microwave Access), Wi-Fi, Bluetooth communication, infrared communication, and ultrasound. This includes, but is not limited to, communication, Visible Light Communication (VLC), LiFi, etc.

Hereinafter, an embodiment of the present invention will be described in detail using the attached drawings.

Figure 2 is a structural diagram of a cargo recognition system according to an embodiment of the present invention.

Referring to FIG. 2, the cargo recognition system may include a server 100, a camera 200, and an administrator terminal 300.

The manager terminal 300 according to one embodiment may be a terminal used by a manager of a company operating a logistics center or a worker who sorts and processes cargo at the logistics center. The manager can remotely check the status of cargo flowing into the logistics center through the manager terminal 300 and perform overall logistics processing procedures such as classification, loading, and delivery accordingly.

According to one embodiment, the manager terminal 300 includes a memory, a processor, and a communication module, and the communication module performs data communication with the server 100 under the control of the processor.

A program (or application) for monitoring cargo status is stored in the memory. The processor executes a program stored in memory to process a series of operations to provide a cargo status monitoring service to the manager.

As an example of the overall operation, the processor receives the status of cargo flowing into the distribution center in real time from the server 100 and provides it to the manager. Specifically, the processor can receive cargo images of cargo flowing into the distribution center from the server 100 in real time. The processor may receive from the server 100 an image in which an area representing the cargo in the cargo image is specified and the type of the cargo. The processor may receive from the server 100 an image specifying an area indicating damage within the cargo image, whether the cargo is damaged, and the type of damage. The processor may receive from the server 100 an image in which the area representing the invoice and barcode in the cargo image is specified and detailed cargo data corresponding thereto.

The camera 200 according to one embodiment is placed in a distribution center and is designed to photograph cargo flowing into the distribution center. When cargo transported within a logistics center enters the shooting range, the camera 200 recognizes the cargo and activates the shooting function. The camera 200 transmits the cargo image obtained accordingly to the server 100 in real time. Meanwhile, the camera 200 is sufficient to perform recognition, photographing, and communication functions, and its type or form does not limit the present invention.

According to one embodiment, the camera 200 may be installed on or around a means of transporting cargo, such as a conveyor belt in a logistics center. Preferably, it may be connected to a conveyor belt or included as a component forming part of a cargo handling means, and may be installed in a position to automatically distribute cargo according to cargo recognition.

According to one embodiment, the system may further include cargo handling means (not shown). Cargo processing means may include, for example, transport means such as a conveyor belt, cargo distribution means such as sorters such as robot arms, loading means such as loading robots, and invoice recognition means including a barcode reader, but are limited thereto. This does not mean that various equipment needed during the cargo distribution process are further configured. In addition, in the case of the distribution means, under the control of the server 100, it can perform the role of distributing each cargo in the distribution center to a predetermined area according to the type or damage of the cargo and invoice data. In addition, in the case of the invoice recognition means, cargo data is acquired based on the invoice image detected by the server 100, or the barcode image detected by the server 100 is received, the barcode is decoded, and the corresponding cargo data is sent to the server. It can be sent to (100).

The server 100 according to one embodiment is an operating server of a company that operates a logistics center, and is involved in the overall process related to logistics processing and cargo distribution. In particular, it recognizes cargo flowing into the logistics center based on deep learning and responds accordingly. Carry out the process of collecting the following data. Accordingly, the server 100 may define the term as a cargo recognition server.

Figure 3 is a block diagram showing the configuration of a cargo recognition server according to an embodiment of the present invention.

According to one embodiment, the server 100 includes a communication module 110, a memory 120, a processor 130, and a database 140, and the communication module 110 includes a camera 200 and an administrator terminal. (300) and handles various cargo processing means and data communications.

The memory 120 stores a program (or application) for performing a deep learning-based cargo recognition method through cargo image augmentation, and the processor 130 executes the program stored in the memory 120 to recognize cargo. How to handle various processes.

An example of the overall process of the processor 130 is as follows. The processor 130 collects cargo images in which various cargoes are photographed using the camera 200 or other methods. The processor 130 learns some or all of the collected cargo images using a first deep learning model and generates a converted virtual cargo image by reflecting changes in the surrounding environment. The processor 130 learns the cargo image and the virtual cargo image with a second deep learning model to build a cargo recognition model that recognizes the cargo part in the image and derives the cargo type. The processor 130 uses a cargo recognition model to recognize cargo in the image of new cargo and determine its type.

The database 140 stores various cargo images collected by the processor 130 and virtual cargo images created using them. In addition, new cargo images captured by the camera 200 of cargo flowing into the distribution center in real time are stored, and the cargo and its type identified by the cargo recognition model are matched to the new cargo image and stored. In addition, if at least one area among damage, invoice, and barcode is identified in the new cargo image by the cargo recognition model, the damage, type of damage, or cargo detailed data derived accordingly are matched with the new cargo image and stored. do.

Meanwhile, the processor 130 may be divided into more detailed components according to the process of the cargo recognition method, and specific embodiments for each component will be described using FIGS. 4 to 7.

Figure 4 is a block diagram showing the configuration of a processor according to an embodiment of the present invention. Referring to FIG. 4, the processor 130 may include an image enhancement unit 131, a learning unit 132, and a cargo recognition unit 133.

The image enhancement unit 131 according to one embodiment first collects cargo images for various types of cargo. Here, the cargo type is preset to be classified according to the type of cargo or the shape of the cargo due to packing method such as vinyl or box, and is not particularly limited. The image augmentation unit 131 may internally receive cargo images captured from cameras 200 placed at logistics centers and distribution sites, or may collect them externally through methods such as Internet crawling. However, the collection method is limited to this. That is not the case. Cargo images collected in this way are later used as learning data for deep learning of cargo recognition.

The image augmentation unit 131 extracts some or all of the collected cargo images and sets them as reference images for augmenting learning data. The image augmentation unit 131 learns this reference image as a first deep learning model to generate a virtual cargo image. Here, the first deep learning model is designed as an algorithm for performing image transformation, and may be an adversarial neural network (GAN) series model, but is not limited to this. However, the cycleGAN model is preferable, and the algorithm of the first deep learning model to which cycleGAN is applied will be explained using Figure 4.

Figure 5 is an example diagram for explaining a first deep learning model according to an embodiment of the present invention.

The first deep learning model is designed as an algorithm that preserves the content composition of the input X domain image and transforms only the style set in the Y domain in the original. Here, the is not a problem. For example, in the case of the picture shown in Figure 5, when you want to convert the brown horse located in the center into a zebra while keeping other parts of the original Perform. In other words, the Y domain does not need to be composed of a picture of a zebra in a grassland; an image containing only zebra patterns is sufficient. Even if it is not an image, it can be included as data in the Y domain if it is a condition value set in response to the zebra pattern. In this way, the first deep learning model uses an image dataset to perform image to image translation, which maps the original to an image of the desired style. However, matching between input data is not an issue, so pairing is not necessary, so the dataset can be easily converted. It is configurable and effectiveness is guaranteed.

Referring to Figure 5, the first deep learning model is designed to use two generators (G, F), and each generator is learned so that the X domain and Y domain can be converted to each other's domains. Additionally, the first deep learning model can be trained to minimize at least two loss functions. For example, a first loss function may be applied that indicates how appropriately the virtual image (y) converted from the original (x) of the X domain according to the characteristics of the Y domain was created as an image. In addition, a second loss function that indicates how similar the image (x') restored through the original (x) to the original (x) through the G and Y generators, respectively, can be applied. In other words, the first deep learning model is designed with a learning principle that transforms the style of the original of the X domain according to the characteristics of the Y domain, but only to the extent that it can be restored to the original. Therefore, problems with the existing GAN model, such as the phenomenon of creating a virtual image that is completely unrelated to the originally intended original, can be solved.

Using the above-described learning principle, the image augmentation unit 131 can generate a virtual cargo image in which the reference image is converted according to preset external factors through the first deep learning model. Here, external factors are various surrounding environments that can change during the distribution process of cargo, and may include, but are not limited to, location, weather, and time. For example, in the case of location, even if the cargo is the same, different images are obtained when it is indoors and outdoors, in the case of weather, clear and cloudy are reflected in the image, and in the case of time, the brightness of the image is different depending on the day and night, so the exact cargo It is necessary to consider all such changes in the external environment in order to minimize recognition and errors. However, it is virtually impossible to take pictures every time the environment to which cargo is exposed changes, and even requires an inefficient process of artificially creating the environment, so conventionally, only cargo images in specific environments could be collected. On the other hand, in the case of the present invention, cargo images that satisfy all the diversity of the external environment can be automatically generated using the first deep learning model, so the above limitations can be actively improved.

According to one embodiment, the image enhancement unit 131 uses the reference image as a dataset and sets it as the first domain (X domain). In addition, the image enhancement unit 131 sets at least one conversion condition of the reference image for generating a virtual cargo image for each external factor and assigns a unique identification number to each conversion condition to classify it into a second domain (Y domain). can do. At this time, the conversion conditions can be set as condition values for color temperature, brightness, shading, and illuminance, which are divided into preset numerical ranges for each external factor. Alternatively, randomly collected random images can be classified according to the condition values set above and grouped into a dataset to set conversion conditions. For example, in the external factor of weather, the luminance in the range a or images classified accordingly may be set as a second domain and managed with a unique identification number.

Specific examples of conversion conditions for each external factor are as follows. When the external factor is a location, the image enhancement unit 131 sets the color temperature divided into two different numerical ranges corresponding to outdoor and indoor as the conversion condition. That is, outdoors, the first color temperature with a numerical range corresponding to natural light is applied as the image conversion condition, and indoors, the second color temperature with a numerical range corresponding to artificial light is applied as the image conversion condition. The conversion conditions classified in this way are each set with a unique identification number and are composed of two second domains.

When the external factor is weather, the image enhancement unit 131 sets color temperature, brightness, and shading divided into two or more different numerical ranges corresponding to two or more preset degrees of clear and cloudy as conversion conditions. For example, when clear and cloudy levels are divided into n levels, the image conversion conditions are applied from the first color temperature, first luminance, and first shade to the nth color temperature, nth luminance, and nth shade. A second domain is configured with a total of n identification numbers set.

When the external factor is time, the image enhancement unit 131 sets color temperature, shading, and illuminance divided into two or more different numerical ranges corresponding to two or more preset time periods as conversion conditions. For example, assuming that the time period is simply divided into day and night for explanation purposes, the first color temperature, first shade level, and first illuminance and the second color temperature and second shade level have numerical ranges corresponding to day and night. and Article 2 are applied as image conversion conditions, respectively, to form a second domain with two identification numbers set.

According to one embodiment, the image enhancement unit 131 may further configure a second domain according to changes in packaging or damage to cargo in addition to changes in the external environment. In other words, the image enhancement unit 131 can additionally secure a virtual cargo image that reflects the diversity of cargo appearance and uses the cargo appearance conditions as conversion conditions. For example, the image augmentation unit 131 randomly collects images containing a preset number of different patterns, groups images with common patterns to build one dataset, and configures each of them into a separate second domain. can do. Alternatively, a second domain can be constructed that uses the previously matched identification value for each pattern as a conversion condition.

The image augmentation unit 131 learns a first deep learning model using the reference image and conversion conditions as input data, thereby generating and outputting a virtual cargo image to which each conversion condition is applied. That is, when each reference image of the first domain is input to the first deep learning model along with the conversion conditions constituting each second domain, a virtual cargo image is generated in which each reference image is converted according to each conversion condition. Specifically, taking as an example the process of converting a reference image ( It is input into the learning model and output Y, a virtual cargo image. The image enhancement unit 131 repeats the first deep learning until the first loss function indicating whether Y has been appropriately generated as an image (image suitability) is minimized. At the same time, the first deep learning model learns a generator that converts Y back to the original, and accordingly, X', in which Y is restored like the original, is output and comparison with X is performed. The image enhancement unit 131 repeats the first deep learning in the direction of minimizing the second loss function corresponding to the difference in the comparison results. As a result, a virtual cargo image converted to color temperature, shading, and illuminance corresponding to the night is created while preserving the composition, location, and shape of the objects included in X. In this way, the image enhancement unit 131 generates virtual cargo images to which all conversion conditions of the second domain are applied to each of the reference images of the first domain, and an example of this embodiment is shown in FIG. 6. there is.

Referring again to FIG. 4, the learning unit 132 according to one embodiment uses the previously collected cargo images and the virtual cargo image generated by the image augmentation unit 131 as learning data for the second deep learning model. . Here, the second deep learning model is designed as an algorithm that identifies specific objects in the image and classifies them according to preset standards. It may be a neural network of the CNN (Convolutional Neural Network) series, and YOLO (You Only Look Once) is preferable. However, it is not necessarily limited to this.

According to one embodiment, the learning unit 132 transmits the collected cargo images and virtual cargo images to the manager terminal 300 and receives labeling values for each image from the manager terminal 300. For example, the manager terminal 300 may mark an area representing cargo within the image as a box, assign an identification code preset to a specific type to the area, and transmit it to the server 100. That is, the labeling value may include an area representing the cargo and an identification code of the cargo type.

The learning unit 132 may generate text data for the image based on the received labeling value. Specifically, the learning unit 132 can check boxes representing cargo and convert them into coordinate values. Here, the coordinate value can be composed of a total of four codes, which are the x and y components for the center of the box, the horizontal length value (width), and the vertical length value (height) of the box. The learning unit 132 matches the identification code of the cargo type with the coordinate value of the box representing the cargo, and can generate text data in which the box representing the cargo in the image is defined by a total of five codes. For example, if the identification code of the box type is set to '1', text data may be generated in the form of '[1, a, b, c, d]'. Here, a and b are the x and y components of the center of the box representing the cargo, c is the width of the box, and d is the height of the box. In this way, the learning unit 132 can configure one cargo image or virtual cargo image and the text data of the image into a set and set it as one learning data.

Referring to FIG. 7, the learning unit 132 repeats learning of the learning data through the second deep learning model to optimize the process of recognizing the first object corresponding to the cargo in the image. For example, in the case of a CNN or YOLO-based neural network model, the image of the training data passes through a filter in the order of Convolutional Layer, Pooling layer, Activation layer, and Fully connected-layer, and feature values in tensor units are extracted. The first object in the image is identified through probability calculation on the extracted feature values, and the identified first object is processed as a bounding box. The learning unit 132 performs a process of optimizing the parameters of the second deep learning model by repeating learning until all of the learning data is bounded and extracted as the first object.

When the object recognition process is optimized, the learning unit 132 repeats learning to optimize the classification process. In other words, depending on whether cargo is recognized in the image of the learning data and the first object is extracted, the process of classifying and outputting the first object into a preset type is optimized. For example, the learning unit 132 compares the type of output cargo with the text data of the corresponding learning data, calculates the difference, and repeats the learning of the classification process in a way that minimizes the corresponding loss.

According to one embodiment, in addition to the first object representing the cargo, the learning unit 132 selects at least one of a second object representing damage, a third object representing an invoice, and a fourth object representing a barcode in the learning data. The second deep learning model is trained to identify. For example, when receiving the labeling value from the manager terminal 300, the learning unit 132 may further receive a box mark for at least one of damage, invoice, and barcode in the image. In addition, the manager terminal 300 can assign an identification code preset to the damage type, invoice type (carrier), and barcode pattern to the damage, invoice, and box marked with a barcode and transmit it to the server 100. The learning unit 132 can use these labeling values to check boxes representing damage, invoices, and barcodes, convert them into coordinate values, and then generate text data consisting of an identification code and coordinate values for each of them. The learning unit 132 can configure one cargo image or virtual cargo image and all text data generated for the image into a set and set it as one learning data. For example, if it is an image of damaged cargo and all labeling values for the invoice and barcode are received, a total of four text data generated for the cargo image, cargo, damage, invoice, and barcode can constitute one learning data. You can. Afterwards, the learning unit 132 repeats learning of the learning data through the second deep learning model, and selects a second object corresponding to the damaged part in the image, a third object corresponding to the invoice, and a fourth object corresponding to the barcode. The process of recognizing at least one object and the process of classifying the type of each recognized object are optimized, and specific details are replaced with tactics for the first object.

The cargo recognition unit 133 according to one embodiment receives a new cargo image for cargo captured in real time by the camera 200 and inputs it into the cargo recognition model built by the learning unit 132 to create a new cargo image. It is responsible for identifying objects representing cargo in cargo images and determining the type of cargo. The cargo recognition unit 133 can generate control values for the path and work operation of the cargo according to the type of cargo determined and transmit it to the cargo processing means. Additionally, the cargo recognition unit 133 can create a work schedule based on the determined cargo type and transmit it to the manager terminal 300.

According to one embodiment, the cargo recognition unit 133 further determines whether the cargo is damaged and the type of damage depending on whether the second object is identified in the new cargo image by the cargo recognition model, and distributes the cargo based on this. can be performed, and a specific example thereof is shown in FIG. 8.

Referring to FIG. 8, the cargo recognition unit 133 inputs a new cargo image into the cargo recognition model. Accordingly, the recognition process of the first object proceeds, and the part recognized as cargo in the image is processed as a bounding box and extracted as the first object. If there is no part recognized as cargo, no object is extracted, and accordingly, the presence or absence of cargo in the image is determined.

Once the existence of the first object is determined, the recognition process for the second object proceeds. In other words, the part recognized as damaged in the image is processed as a bounding box and extracted as a second object. On the other hand, if no part is recognized as damaged, no object is extracted, and accordingly, damage is determined within the image.

Meanwhile, the classifier according to one embodiment adopts the bounding box method in object recognition, but is not limited to this, and various object recognition technologies, including the polygon method or keypoint method, may be applied.

Next, the classification process of new cargo represented by the image is performed by the cargo recognition model. That is, according to the result of the determination of whether the first object and the second object exist, the new cargo photographed is classified into one of four types: normal cargo, damaged cargo, total loss cargo, and learning cargo.

Specifically, when it is determined that the first object exists and the second object does not exist, the new cargo is classified as a normal cargo type. In other words, if cargo is recognized through the object recognition process but no damaged parts are recognized, a normal cargo type is output through the classification process.

If it is determined that both the first object and the second object exist, the new cargo is classified as a damaged cargo type. In other words, if both cargo and damage are recognized through the object recognition process, the type of damaged cargo is output through the classification process.

If the first object does not exist and the second object exists, the new cargo is classified as a total loss cargo type. In other words, if no cargo is recognized through the object recognition process, but damage is recognized, the main form of the cargo is considered damaged due to the damage, and the total loss cargo type is output through the classification process.

If it is determined that both the first object and the second object do not exist, the new cargo is classified as a learning cargo type. In other words, if no cargo or damaged parts are recognized through the object recognition process, it is considered a new type of cargo or an object other than cargo, and is output as learning cargo through the classification process, and the cargo recognition model is updated based on this. It goes on.

According to one embodiment, when a third or fourth object in a new cargo image is identified by the cargo recognition model, the cargo recognition unit 133 may obtain detailed data of the cargo based on this. Here, the detailed data may include the type, nature, weight, size, and delivery address of the cargo, but this is representatively selected for explanation purposes and may include all information to identify and specify each cargo, and is specifically limited. It doesn't work. For example, when a third object is identified by the cargo recognition model and a specific transport company according to the invoice type is output, the cargo recognition unit 133 can request detailed data from the server of the transport company and receive it. In addition, when the fourth object is identified by the cargo recognition model and the location and pattern of the barcode are output, the cargo recognition unit 133 transmits it to the barcode reader of the cargo processing means and obtains detailed data through it.

Hereinafter, an embodiment of the cargo recognition method performed by the server 100 will be described using FIG. 9. Figure 9 is a flowchart of a cargo recognition method according to an embodiment of the present invention, and overlapping content will be replaced with the above.

In step S910, the server 100 extracts some or all of the previously collected cargo images, sets them as a reference image, learns the reference image as a first deep learning model based on image conversion, and selects the reference image according to preset external factors. Create a converted virtual cargo image. Here, external factors are the surrounding environment that can change during the cargo distribution process and include location, weather, and time.

According to one embodiment, a virtual cargo image is created in which only the style is transformed according to transformation conditions set for external factors while preserving the composition, position, and shape of objects included in the reference image by the first deep learning model. . Here, the external factor is the surrounding environment that can change during the cargo distribution process, and is a virtual cargo image in which at least one of color temperature, brightness, shading, and illuminance is transformed from the reference image according to changes in location, weather, and time. can be created.

According to one embodiment, in response to packing or damage to cargo, preset cargo appearance conditions are learned with a reference image using a first deep learning model to generate a virtual cargo image by converting the reference image according to the cargo appearance conditions. can do. In other words, in order to prepare for situations where damage occurs during the cargo distribution process or the external pattern changes depending on the packaging, the first deep learning model is used on a virtual cargo image reflecting the above situation to improve the accuracy of cargo recognition. It can be created automatically through .

In step S920, the server 100 sets the previously collected cargo images and virtual cargo images as learning data, learns the learning data through a second deep learning model based on object recognition, and trains the training data when a random cargo image is input. Build a cargo recognition model that classifies cargo into one of the set cargo types.

According to one embodiment, in addition to the first object representing the cargo within the learning data, a second deep learning is used to identify at least one of a second object representing damage, a third object representing an invoice, and a fourth object representing a barcode. Learning of the model may be performed.

In step S930, the server 100 inputs a new cargo image for cargo captured in real time by the camera 200 placed at the work site into a cargo recognition model to identify objects representing the cargo in the new cargo image and identify the cargo. Determine the type.

In this way, according to the present invention, a robust cargo recognition model is built through augmentation of learning data, so even if the surrounding environment changes due to location, weather, time, etc. during the cargo distribution process, the phenomenon of being mistaken for another cargo is prevented, and the barcode Decoding also has the effect of improving accuracy.

According to one embodiment, the server 100 determines whether the cargo is damaged and the type of damage depending on whether the second object in the new cargo image is identified by the cargo recognition model, and the third object in the new cargo image is identified by the cargo recognition model. If an object or a fourth object is identified, detailed data of the cargo can be obtained based on the identified third or fourth object.

Although the methods and systems of the present invention have been described with respect to specific embodiments, some or all of their components or operations may be implemented using a computer system having a general-purpose hardware architecture.

The description of the present invention described above is for illustrative purposes, and those skilled in the art will understand that the present invention can be easily modified into other specific forms without changing the technical idea or essential features of the present invention. will be. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive. For example, each component described as unitary may be implemented in a distributed manner, and similarly, components described as distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims described below rather than the detailed description above, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present invention. do.

100: server
200: Camera
300: Administrator terminal
130: processor

Claims (18)

In a deep learning-based cargo recognition method through cargo image augmentation performed by a server,
a) Virtual cargo that extracts some or all of the previously collected cargo images, sets them as a reference image, learns the reference image with a first deep learning model based on image conversion, and converts the reference image according to preset external factors. generating an image;
b) Set the previously collected cargo image and the virtual cargo image as learning data, learn the learning data through a second deep learning model based on object recognition, and when a random cargo image is input, the type of the preset cargo A step of building a cargo recognition model that classifies cargo into one of the following; and
c) inputting a new cargo image for cargo captured in real time by a camera placed at a work site into the cargo recognition model to identify an object representing the cargo in the new cargo image and determining the type of the cargo;
Including,
The external factors include location, weather, and time as the surrounding environment that can change during the cargo distribution process.
In step a), the conditions of the cargo appearance preset in response to packing or damage to the cargo are learned with the first deep learning model together with the reference image, and the reference image is converted according to the conditions of the cargo appearance. A cargo recognition method comprising: generating a cargo image. According to clause 1,
In step a), a virtual cargo image is created in which the style of the reference image is modified according to the external factors while the configuration, position, and shape of the object included in the reference image are preserved by the first deep learning model. A cargo recognition method including a generating step. According to clause 1,
Step a) minimizes a first loss function corresponding to the image suitability for the virtual cargo image and a second loss function corresponding to the difference between the result of converting the virtual cargo image back to the original and the reference image. A cargo recognition method comprising: repeating learning of the first deep learning model in the direction of: According to clause 1,
In step a), at least one conversion condition of the reference image for generating the virtual cargo image is set for each external factor and applied as input data of the first deep learning model together with the reference image. , Cargo recognition method. According to clause 4,
The conversion condition is a condition value for color temperature, brightness, shade, and illuminance divided into a preset numerical range, or pre-collected random images are classified based on the condition value. According to clause 4,
When the external factor is a location, the conversion condition is set to a color temperature divided into two different numerical ranges corresponding to outdoor and indoor,
When the external factor is weather, the conversion condition is set to color temperature, brightness, and shading divided into two or more different numerical ranges corresponding to two or more preset degrees of clear and cloudy,
When the external factor is time, the conversion condition is set to color temperature, shading, and illuminance divided into two or more different numerical ranges corresponding to two or more preset time periods. delete According to clause 1,
Step b) is to identify at least one of a second object representing damage, a third object representing an invoice, and a fourth object representing a barcode in the learning data in addition to the first object representing the cargo. Cargo recognition method comprising: performing learning of a deep learning model. According to clause 8,
In step c), the damage and damage type of the cargo are determined depending on whether the second object in the new cargo image is identified by the cargo recognition model, and the third object in the new cargo image is determined by the cargo recognition model. When an object or a fourth object is identified, obtaining detailed data of the cargo based on the identified third or fourth object. In the cargo recognition server,
A memory storing a program for performing a deep learning-based cargo recognition method through cargo image augmentation; and
Including a processor executing the program,
The processor,
Extract some or all of the previously collected cargo images and set them as reference images, learn the reference images with the first deep learning model based on image conversion, and create a virtual cargo image by converting the reference images according to preset external factors. an image enhancement unit that generates;
The previously collected cargo image and the virtual cargo image are set as learning data, and the learning data is learned through a second deep learning model based on object recognition. When a random cargo image is input, it is selected as one of the preset cargo types. A learning unit that builds a cargo recognition model to classify; and
A cargo recognition unit that inputs a new cargo image for cargo captured in real time by a camera placed at a work site into the cargo recognition model, identifies an object representing the cargo in the new cargo image, and determines the type of the cargo;
Including,
The external factors include location, weather, and time as the surrounding environment that can change during the cargo distribution process.
The image augmentation unit learns the conditions of the cargo appearance preset in response to packing or damage of the cargo with the first deep learning model together with the reference image, and converts the reference image according to the conditions of the cargo appearance to create a virtual cargo. Cargo recognition server that generates images. According to clause 10,
The image augmentation unit generates a virtual cargo image in which the style of the reference image is modified according to the external factors while preserving the configuration, position, and shape of the object included in the reference image through the first deep learning model. Cargo recognition server. According to clause 10,
The image augmentation unit minimizes a first loss function corresponding to the image suitability for the virtual cargo image and a second loss function corresponding to the difference between the result of converting the virtual cargo image back to the original and the reference image. A cargo recognition server that repeats learning of the first deep learning model in each direction. According to clause 10,
The image augmentation unit sets at least one conversion condition of the reference image for generating the virtual cargo image for each external factor and applies it as input data of the first deep learning model along with the reference image. server. According to clause 13,
The image augmentation unit sets condition values for color temperature, luminance, shading, and illuminance divided into preset numerical ranges, or pre-collected random images classified based on the condition values, as the conversion conditions. A cargo recognition server. According to clause 13,
The image enhancement unit,
When the external factor is a location, set the color temperature divided into two different numerical ranges corresponding to outdoor and indoor as the conversion condition,
When the external factor is weather, color temperature, brightness, and shading divided into two or more different numerical ranges corresponding to two or more preset clear and cloudy degrees are set as the conversion conditions,
When the external factor is time, the cargo recognition server sets color temperature, shading, and illuminance divided into two or more different numerical ranges corresponding to two or more preset time intervals as the conversion conditions. delete According to clause 10,
The learning unit uses the second deep learning to identify at least one of a second object representing damage, a third object representing an invoice, and a fourth object representing a barcode in the learning data, in addition to the first object representing the cargo. Cargo recognition server that performs model learning. According to clause 17,
The cargo recognition unit determines whether the cargo is damaged and the type of damage depending on whether the second object in the new cargo image is identified by the cargo recognition model, and determines the third object in the new cargo image according to the cargo recognition model. Or, when a fourth object is identified, a cargo recognition server that obtains detailed data of the cargo based on the identified third or fourth object.

Images