KR102159472B1 - System for managing materials based on deep learning - Google Patents

Abstract

The present invention relates to a deep learning-based material management system. According to the present invention, the deep learning-based material management system comprises: a real image camera for obtaining a real image and a thermal image camera for obtaining a thermal image; a wireless network device; a gateway for transmitting second material management data to a solution server; the solution server for transmitting third material management data to a mapping server; the mapping server; and an operation server. According to the present invention, material management and monitoring can be accurately performed.

Description

Deep learning-based material management system {SYSTEM FOR MANAGING MATERIALS BASED ON DEEP LEARNING}

The present invention relates to a deep learning-based material management system, and more particularly, based on image images of materials obtained from a real image camera and a thermal image camera provided in a warehouse, through a wireless network transmission and reception through a server. Material management data is generated by performing imaging processing for material management, and real-time material management and monitoring are performed by performing artificial intelligence learning based on CNN (Convolutional Neural Network) and RNN (Recurrent Neural Network) through the generated material management data. It relates to a material management system based on deep learning that allows it to be achieved.

In general, warehouses in manufacturing factories are managed in units of storage racks and boxes or areas. The materials stored in the warehouse are mainly dispensed and input whenever necessary for the operation of the production process. In the case of warehouses with a large amount of materials, it is difficult to accurately grasp the location and quantity of materials stored in the case of material dispensing.

In addition, since materials are delivered and stored mainly by workers, various problems arise due to the lack of accurate inventory management such as loss of materials, difficulty in identifying storage locations, and incorrect input of materials in the production process. There are also problems such as material loss in the production process due to incorrect input and an increase in product defect rates.

Therefore, there is a growing demand for a method or system that enables problems such as incorrect material input during the production process to be solved through accurate material management and ultimately improves the defect rate in the production process, and solves the above problems. There is an urgent need for a plan to do so.

The present invention is to solve the problems of the prior art described above, based on image images of materials acquired through a real image camera and a thermal image camera provided in a warehouse, and for material management through a wireless network transmission/reception and server It aims to provide a deep learning-based material management system that enables accurate real-time material management and monitoring by performing imaging processing to generate material management data and performing artificial intelligence learning based on the generated material management data. do.

The objects of the present invention are not limited to the objects mentioned above, and other objects not mentioned will be clearly understood from the following description.

According to an embodiment of the present invention for achieving the above object, there is provided a deep learning-based material management system, comprising: a real image camera installed in a warehouse to obtain a real image and a thermal image camera to obtain a thermal image; The real image and the thermal image are received from the real image camera and the thermal image camera, the real image and the thermal image are grouped and matched with a first identification code, and the first identification code and the real image are A wireless network device that transmits information and first material management data including information on the thermal image to a gateway; Receiving the first material management data from the wireless network device, a second identification code updated the first identification code, second material management data including information on the real image and information on the thermal image A gateway to transmit to the solution server; Generate a plurality of image data by performing an image correction process and an image processing process based on the real image information and the thermal image information included in the second material management data, and update the second identification code A solution server that transmits a third identification code and third material management data including the plurality of image data to a mapping server; Based on the third material management data received from the solution server, material identification information corresponding to at least one of letters, numbers, barcodes, and QR codes is derived, and the material identification information and the third material management data are mapped. A mapping server for transmitting to the operation server; And an operation server that manages material information in a warehouse by performing deep learning based on the material identification information and the third material management data, wherein the image correction process includes: i) contrast for the real image , A real image corrected image by correcting at least one of saturation, hue, shadow, and illuminance, or ii) a process of generating a thermal image corrected image by correcting at least one of contrast, saturation, hue, shade, and illuminance of the thermal image A deep learning-based material management system, which is meant to mean, is provided.

In the image processing process, the solution server generates first image data by i) performing a first image processing for grasping stock of materials in the warehouse based on the information on the real image and the thermal image. Process, ii) The process of generating second image data by performing second image processing to determine the storage status of materials in the warehouse, iii) Performing third image processing to derive the material identification information to obtain third image data. It may be characterized in that at least one of the generating processes is performed.

When it is assumed that the material included in the real image correction image consists of an outline area and a material state area included in the outline area, the solution server, in the first image processing, is included in the real image correction image. The thermal image may be synthesized as a correction image in the material state region to generate first image data.

When it is assumed that the material included in the thermal image correction image is composed of an outline area and a material state area included in the outline area, the solution server, in the second image processing, is included in the thermal image correction image. It may be characterized in that the second image data is generated by synthesizing the real image as a correction image in the outline region.

The solution server, in the third image processing, i) performs black and white shading processing on the real image, ii) recognizes the material identification information derived part from the black and white shaded image, and iii) the recognized part It may be characterized in that the third image data is generated by cutting the image of.

The mapping server may be characterized in that it derives material identification information by performing any one of character recognition, number recognition, barcode recognition, and QR code recognition from the third image data.

The operation server, i) by performing deep learning using a convolutional neural network (CNN) based on the material identification information and the mapped first image data, real-time identification of material inventory in the warehouse, A process for managing and monitoring and ii) deep learning using a convolutional neural network (CNN) based on the material identification information and the mapped second image data, It may be characterized by generating and storing material monitoring data by performing at least one of the processes of identifying the storage state of the material in the warehouse and further determining whether the process input is possible for the material stored in the warehouse.

The operation server performs deep learning using a recurrent neural network (RNN) on the basis of the material identification information, the mapped third material management data and the material monitoring data, thereby , It may be characterized in that it derives and stores material management history information including dispensing details and movement information according to process input.

On the other hand, according to another embodiment of the present invention, in a method for managing materials based on deep learning, a real image camera installed in a warehouse acquires a real image, and a thermal image camera installed in the warehouse acquires a thermal image. step; The wireless network device receives the real image and the thermal image from the real image camera and the thermal image camera, groups the real image and the thermal image to match a first identification code, and the first identification code, Transmitting first material management data including information on the real image and information on the thermal image to a gateway; The gateway receives the first material management data from the wireless network device, a second identification code that updates the first identification code, a second material including information on the real image and information on the thermal image Transmitting management data to a solution server; The solution server generates a plurality of image data by performing an image correction process and an image processing process based on the information on the real image and the information on the thermal image included in the second material management data, and the second Transmitting a third identification code updated with the identification code and third material management data including the plurality of image data to a mapping server; The mapping server derives material identification information corresponding to at least one of letters, numbers, barcodes, and QR codes based on the third material management data received from the solution server, and the material identification information and the third material Mapping the management data and transmitting it to the operation server; And performing, by the operation server, deep learning based on the material identification information and the third material management data to manage material information in the warehouse, wherein the image correction process includes: i) the real image For a real image corrected image by correcting at least one of contrast, saturation, hue, shade, and illuminance for the thermal image, or ii) a thermal image corrected image by correcting at least one of contrast, saturation, hue, shadow, and illuminance for the thermal image. A deep learning-based material management method, which means the process of creation, is provided.

According to an embodiment of the present invention, when material is disbursed, by acquiring images of materials from a real image camera and thermal image camera provided in a warehouse, based on the photographed image, the server determines and stocks the material delivery Since the management of the material is performed, there is an effect that the convenience of the worker (worker) is increased as the work such as recording the material dispensing of the worker (worker) for material management is not required.

In addition, according to an embodiment of the present invention, it is possible to determine the status of the material such as whether or not the material is deteriorated through deep learning after imaging based on the image of the material acquired from the warehouse, so not only inventory management of the material but also storage management It also has the advantage of being possible.

The effects of the present invention are not limited to the above effects, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description or claims of the present invention.

1 is a diagram schematically showing the configuration of a material management system based on deep learning according to an embodiment of the present invention.
2 is a diagram showing an example of a deep learning-based material management system implemented in a warehouse.
3 is a flowchart illustrating a process of implementing a material management system based on deep learning according to an embodiment of the present invention.
4 is a diagram illustrating an example in which image data is generated through an image correction process and an image processing process based on information about a real image and information about a thermal image.
5 is a diagram showing an example of an identification code according to an embodiment of the present invention.

Hereinafter, exemplary embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the rights of the patent application is not limited or limited by these embodiments. It should be understood that all changes, equivalents, or substitutes to the embodiments are included in the scope of the rights.

Specific structural or functional descriptions of the embodiments are disclosed for illustrative purposes only, and may be changed in various forms and implemented. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from other components. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

When a component is referred to as being "connected" to another component, it is to be understood that it may be directly connected or connected to the other component, but other components may exist in the middle.

The terms used in the examples are used for illustrative purposes only and should not be interpreted as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in this application. Does not.

In addition, in the description with reference to the accompanying drawings, the same reference numerals are assigned to the same components regardless of the reference numerals, and redundant descriptions thereof will be omitted. In describing the embodiments, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the embodiments, the detailed description thereof will be omitted.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram schematically showing the configuration of a material management system based on deep learning according to an embodiment of the present invention.

Referring to Figure 1, a deep learning-based material management system according to an embodiment of the present invention, a real image camera 100, a thermal image camera 200, a wireless network device 300, a gateway 400, a solution It includes a server 500, a mapping server 600, and an operation server 700.

The real image camera 100, the thermal image camera 200, the wireless network device 300, the gateway 400, the solution server 500, the mapping server 600, and the operation server 700 are each connected to a communication network. In this case, the communication network may be configured regardless of its communication mode, such as wired or wireless. It may be composed of various communication networks such as a local area network (LAN), a metropolitan area network (MAN), and a wide area network (WAN).

The real image camera 100 may be installed in a warehouse to take pictures of a predetermined area in the warehouse, acquire a real image of a material stacked in the corresponding area, and transmit it to the wireless network device 300.

Similarly, the thermal imaging camera 200 is installed in the warehouse to take pictures of a predetermined area in the warehouse, acquires a thermal image of the material stacked in the area, and transmits it to the wireless network device 300. have.

When a real image and a thermal image are received from each of the visual camera 100 and the thermal imager 200, the wireless network device 300 may group the received real image and thermal image to match the first identification code. .

The first identification code is generated and assigned to facilitate identification of data when transmitting data between servers, and specifically, is assigned to group the real and thermal images and transmit them to the gateway 400 Can be. According to an embodiment of the present invention, the first identification code is information including a warehouse identification number, area code information, and camera identification number, and a detailed description will be described later through the embodiment of FIG. 5.

The wireless network device 300 may transmit the first material management data including the first identification code, information on the real image, and information on the thermal image to the gateway 400.

The information on the real image may mean that the real image acquired through the real image camera 100 includes time information at which the corresponding real image was acquired, and as deep learning is performed in the operation server 700 It may be information that is referenced when deriving material management history information listed in time series.

Similarly, the information on the thermal image may mean that the thermal image acquired through the thermal imaging camera 200 includes time information at which the corresponding thermal image was acquired, and deep learning is performed in the operation server 700. As implemented, it may be information that is referenced when deriving material management history information listed in time series.

When the first material management data is received from the wireless network device 300, the gateway 400 may update the first identification code included in the first material management data to a second identification code, and the updated second material management data Second material management data including an identification code, information on the real image, and information on the thermal image may be transmitted to the solution server 500.

The second identification code, as described above, is an update of the first identification code, specifically, may be information further including the identification code of the gateway 400 in the first identification code, and the first identification Like the code, a detailed description will be described later through the embodiment of FIG. 5.

When the second material management data is received from the gateway 400, the solution server 500 performs an image correction process and an image based on the information on the real image and the information on the thermal image included in the second material management data. A plurality of image data may be generated by performing a processing process.

The image correction process may refer to a process of generating a real image corrected image by correcting at least one of contrast, saturation, hue, shadow, and illuminance for the real image, or contrast, saturation, and color tone for a thermal image. , It may mean a process of generating a thermal image correction image by correcting at least one of shade and illuminance.

The image processing process may mean a process of performing at least one of a first image processing process, a second image processing process, and a third image processing process.

The first image processing process may be a process for identifying the stock of materials in the warehouse, and when it is assumed that the material included in the real image correction image consists of an outline area and a material state area included in the outline area, the actual image It may refer to a process of generating first image data by synthesizing a thermal image as a corrected image in the material state region included in the image corrected image. The first image processing process may correspond to a process of generating first image data using only a real image corrected image and a thermal image without using a thermal image corrected image.

In this case, the first image data may be data for identifying the stock of materials in the warehouse, and may mean that the image has been processed to highlight the outline of the material.

The second image processing process may be a process for identifying the storage status of materials in the warehouse, and when it is assumed that the material included in the thermal image correction image consists of an outline area and a material status area included in the outline area, It may mean a process of generating second image data by synthesizing a real image as a corrected image on a contour area included in the thermal image correction image. The second image processing process may correspond to a process of generating second image data using only a thermal image correction image and a real image without using a real image correction image.

In this case, the second image data may be data for determining the storage state of the material in the warehouse, and may mean that the image is processed so that whether the material is deteriorated or the degree of deterioration is displayed as thermal image data.

Meanwhile, the third image processing process is an image processing process for deriving material identification information, performing black and white shading processing on a real image, and recognizing the material identification information derivation part from the black and white shaded image, It may refer to a process of generating third image data by cutting the image of. For reference, when recognizing a part of material identification information derived from a black-and-white shaded image, a difference in color values (ex, grayscale, etc.) included in the image may be used.

In addition, if necessary, the solution server 500 may perform black and white shading on the acquired real image, and then enlarge the size of the real image to facilitate recognition of material identification information.

Accordingly, the plurality of image data generated through the solution server 500 includes first image data generated through first image processing, second image data generated through second image processing, and third image processing. It may include at least one of the third image data generated through.

In addition, the solution server 500 may update the second identification code included in the second material management data to the third identification code, and includes the updated third identification code and the generated plurality of image data. 3 Material management data can be transmitted to the mapping server 600.

The third identification code, as described above, is an update of the second identification code, and specifically, may be information further including the identification code of the solution server 500 in the second identification code, and the first As with the identification code and the second identification code, a detailed description will be described later through the embodiment of FIG. 5.

When the third material management data is received from the solution server 500, the mapping server 600 may derive material identification information based on the received third material management data, and the derived material identification information and the third material Management data may be mapped and transmitted to the operation server 700.

The material identification information may correspond to at least one of letters, numbers, barcodes, and QR codes. Accordingly, the derivation of material identification information performed by the mapping server 600 may mean performing any one of character recognition, number recognition, barcode recognition, and QR code recognition from the third image data.

The operation server 700 may perform deep learning based on the third material management data received from the mapping server 600 to manage material information in the warehouse.

According to an embodiment of the present invention, the operation server 700 synthesizes the material identification information received from the mapping server 600 and the first image data among the third material management data mapped with the material identification information. By conducting deep learning using a convolutional neural network (CNN), real-time identification, management, and monitoring of material inventory in warehouses can be performed, and material monitoring data including corresponding management information and monitoring information can be created and stored. have.

According to another embodiment of the present invention, the operation server 700 is synthesized based on the material identification information received from the mapping server 600 and the second image data of the third material management data mapped with the material identification information. By performing deep learning using a convolutional neural network (CNN), it is possible to understand the storage status of materials stacked in the warehouse, and to further determine whether the process can be input for the materials stacked in the warehouse. Material monitoring data can be created and saved, including information on checking the storage status of materials and information on determining whether material processing is possible.

According to another embodiment of the present invention, the operation server 700 includes material identification information received from the mapping server 600, the third material management data mapped with the material identification information, and the convolutional neural network (CNN, Based on the material monitoring data generated through deep learning using Convolutional Neural Network), deep learning using a recurrent neural network (RNN) is additionally performed, so that the details of material in/out and dispensing according to process input and movement Material management history information including information can be derived and stored.

That is, the operation server 700 generates material monitoring data by performing primary deep learning using a convolutional neural network (CNN) based on material identification information and first image data or second image data, and generates material identification information. , Based on the third material management data and the material monitoring data generated through the first deep learning, the second deep learning using a recurrent neural network (RNN) is performed, so that not only the inventory of the material, but also due to movement, dispensing, and corruption By deriving and storing historical information such as disposal, it is possible to perform overall information management and monitoring on materials stored in the warehouse.

2 is a diagram showing an example of a deep learning-based material management system implemented in a warehouse.

As shown in FIG. 2, the real image camera 100, the thermal image camera 200, the wireless network device 300, the gateway 400, and the solution server 500 may be configured in a plurality of units.

The number of visual cameras 100, thermal imaging cameras 200, wireless network devices 300, gateways 400, and solution servers 500 constituted in a deep learning-based material management system is the number of materials based on deep learning. The number of warehouses set to be included in the management system, the number of areas, the number of wireless network devices 300 or gateways 400 installed according to the wireless data transmission/reception network environment, the real image camera 100 installed by the administrator, and It may be changed differently depending on a factor due to at least one of the number of thermal imaging cameras 200, which may be set by an administrator.

Visual cameras (101, 106), thermal cameras (201, 206) and wireless network devices (301, 305) are installed at the entrances and exits of the warehouse to acquire images of material input/output or dispensing, and the gateway 401 Can be sent to.

In particular, the real image camera 100, the thermal image camera 200, and the wireless network device 300 may be installed in units of one or more according to a predetermined area, and, like the "C zone" of "Warehouse 1," The network device 304 may receive and collect real and thermal images from the plurality of visual cameras 104 and 105 and the thermal imagers 204 and 205 and transmit them to the gateway 401.

That is, at least one visual camera (101-110) and thermal imaging camera (201-210) are installed for each area in the warehouse (ex A area, B area, C area, etc.), and a wireless network for each of the areas The devices 301-309 may be installed one by one to transmit information about the acquired real image or thermal image to the gateway 401-402. The gateways 401-402 may be installed for each warehouse to receive information on each warehouse, and may transmit this to the solution servers 501-502 corresponding to each warehouse. The solution server 501-502 may transmit the received information related to each warehouse to the mapping server 600 again, and the mapping server 600 may manage information obtained from a plurality of warehouses.

3 is a flowchart illustrating a process of implementing a material management system based on deep learning according to an embodiment of the present invention.

First, when a real image and a thermal image of a warehouse or material are acquired through the real image camera 100 and the thermal image camera 200 installed in the warehouse (S301), the real image and the thermal image are wireless network device 300 ) Can be transmitted (S302).

The wireless network device 300 may group the received real and thermal images (S303) and match them with a first identification code (S304), and information about the real image matched with the first identification code and the column Information on the image may be transmitted to the gateway 400 as first material management data (S305).

The gateway 400 may add the gateway identification code to the first identification code included in the received first material management data and update the second identification code (S306), and the updated second identification code and the real image The information on and the thermal image may be transmitted to the solution server 500 as second material management data (S307).

The solution server 500 generates a plurality of image data by performing an image correction process and an image processing process based on the information on the real image and the information on the thermal image included in the received second material management data (S308). ), the solution server identification code may be added to the second identification code included in the second material management data to be updated with the third identification code (S309).

Accordingly, the solution server 500 may transmit the third identification code and third material management data including the generated plurality of image data to the mapping server 600 (S310).

The mapping server 600 derives material identification information corresponding to at least one of letters, numbers, barcodes, and QR codes based on the third material management data (S311), and the derived material identification information and the third material management data By mapping (S312), it is possible to transmit the material identification information and the third material management data mapped to the operation server 700 (S313).

The operation server 700 can generate and store material monitoring information by performing deep learning using a convolutional neural network (CNN) based on the material identification information and the third material management data received from the mapping server 600, By performing deep learning using a recurrent neural network (RNN) based on the generated material monitoring information, the material identification information, and the third material management data, material management history information, material input/output details, dispensing details and movement information according to process input Material management history information including a can be derived and stored to perform management for the material in the warehouse (S314).

4 is a diagram illustrating an example in which image data is generated through an image correction process and an image processing process based on information on a real image and information on a thermal image.

As shown in (a) of FIG. 4, the real image correction image or the thermal image correction image 401 generated through the image correction process includes an outline area 403 and a material state area 402 included in the outline area. ) May include the shape of the material.

Meanwhile, FIG. 4B shows an example of a process in which first image data is generated according to the first image processing.

As shown in (b) of FIG. 4, a real image correction image may be generated through an image correction process of correcting contrast based on a real image acquired through the real image camera 100. Thereafter, as the thermal image is synthesized as a corrected image in a predetermined region included in the real image corrected image, first image data in which the outline of a specific shape in the real image is clear may be generated.

In this way, after correcting the image of the real image whose shape was difficult to understand due to the faint outline area, the thermal image was synthesized as a corrected image, thereby obtaining image data in which the outline area became clear, that is, the image shape classification was improved, In the present invention, it is possible to obtain image data processed so that the outline of the material is highlighted with respect to an image obtained in a state where the shape of the material is not clear, and accordingly, it is possible to easily identify the quantity and stock of materials in the warehouse.

4C shows an example of a process in which second image data is generated according to second image processing.

As shown in (c) of FIG. 4, a thermal image correction image may be generated through an image correction process of correcting saturation based on a thermal image acquired through the thermal imager 200. Thereafter, as the real image is synthesized as a corrected image in the contour region included in the thermal image correction image, second image data in which information on the thermal image is displayed in a specific shape may be generated.

In this way, after correcting the image of the thermal image for which it was difficult to know which object the thermal image state information was displayed because the shape was not displayed, the real image was synthesized as a corrected image, thereby obtaining image data displaying thermal image information in a specific shape. By doing so, in the present invention, it is possible to obtain image data displaying thermal image data that enables identification of the material's deterioration in the shape of the material or information on the deterioration of the thermal image obtained in a state where the shape of the material is not clear, and According to this, it is possible to understand the storage status of materials in the warehouse, and furthermore, determine whether the process can be inputted according to the material status.

According to an embodiment of the present invention, the first image processing or the second image processing may be performed using at least one of a Gaussian filter or a Laplacian filter.

4D shows an example of a process of generating third image data according to third image processing.

As shown in (d) of FIG. 4, the real image may include an image of a material and material identification information, and the material identification information corresponds to the material identification information 404 and QR code including letters and numbers. The material identification information 405 and the material identification information 406 corresponding to the barcode may be implemented in any one form and attached to the material.

The solution server 500 processes information on the real image in black and white to facilitate the recognition of the material identification information derived part, and then recognizes the material identification information derived part, and cuts the image of the recognized part to process the third image. Data may be generated, and through this, the mapping server 600 may facilitate material identification information.

5 is a diagram showing an example of an identification code according to an embodiment of the present invention.

As shown in Figure 5, the identification code generated in the deep learning-based material management system is composed of a warehouse identification number, area code information, camera identification number, gateway identification code, and solution server identification code sequentially listed. Can be.

Specifically, the first identification code may be an identification code in the form of sequentially listing the warehouse identification number, area code information, and camera identification number, and the second identification code is updated by adding a gateway identification code to the first identification code. It may be a type of identification code. That is, the second identification code may be configured in a form in which the warehouse identification number, area code information, camera identification number, and gateway identification code are sequentially arranged.

The third identification code may be an updated identification code by listing the solution server identification code in the second identification code. In other words, the warehouse identification number, area code information, camera identification number, gateway identification code and solution server The identification code may be configured in a sequential order.

In addition, the warehouse identification number is composed of two digits, the area code information is a three digit code composed of numbers and letters, the camera identification information is a two digit number, the gateway identification code is a three digit number, and the solution server identification code is As it is composed of 3 digits, the first identification code is a total of 7 digits including numbers and letters, the second identification code is a total of 10 digits including numbers and letters, and the third identification code is a number and It may be composed of a total of 13 digits of code including characters.

As described above, according to an embodiment of the present invention, when material is dispensed, the server determines the material dispensing based on the captured image through image acquisition of the material of the real image camera and the thermal image camera provided in the warehouse. And since the management of inventory, etc. is performed, there is an effect that the convenience of the worker (worker) is increased because the work such as recording of material dispensing of the worker (worker) for material management is not required.

In addition, according to an embodiment of the present invention, it is possible to determine the status of the material such as whether or not the material is deteriorated through deep learning after imaging based on the image of the material acquired from the warehouse, so not only inventory management of the material but also storage management It also has the advantage of being possible.

The effects of the present invention are not limited to the above effects, and should be understood to include all effects that can be inferred from the configuration of the invention described in the detailed description or claims of the present invention.

The embodiments described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices, methods, and components described in the embodiments include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions, such as one or more general purpose computers or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. In addition, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of the program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

As described above, although the embodiments have been described by the limited drawings, a person of ordinary skill in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order from the described method, and/or components such as a system, structure, device, circuit, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments and claims and equivalents fall within the scope of the following claims.

100: real image camera
200: thermal imaging camera
300: wireless network device
400: gateway
500: solution server
600: mapping server
700: production server

Claims (9)

In a deep learning-based material management system,
A real image camera installed in the warehouse to obtain a real image and a thermal image camera to obtain a thermal image;
The real image and the thermal image are received from the real image camera and the thermal image camera, the real image and the thermal image are grouped and matched with a first identification code, and the first identification code and the real image are A wireless network device that transmits information and first material management data including information on the thermal image to a gateway;
Receiving the first material management data from the wireless network device, a second identification code updated the first identification code, second material management data including information on the real image and information on the thermal image A gateway to transmit to the solution server;
Generate a plurality of image data by performing an image correction process and an image processing process based on the real image information and the thermal image information included in the second material management data, and update the second identification code A solution server that transmits a third identification code and third material management data including the plurality of image data to a mapping server;
Based on the third material management data received from the solution server, material identification information corresponding to at least one of letters, numbers, barcodes, and QR codes is derived, and the material identification information and the third material management data are mapped. A mapping server for transmitting to the operation server; And
It includes an operation server for managing material information in the warehouse by performing deep learning based on the material identification information and the third material management data,
The image correction process includes: i) a real image corrected image by correcting at least one of contrast, saturation, hue, shade, and illuminance for the real image, or ii) contrast, saturation, hue, shadow, and illuminance for the thermal image. A deep learning-based material management system that refers to a process of generating a thermal image correction image by correcting at least one of them.
The method of claim 1,
The solution server,
In the image processing process, based on the information on the real image and the information on the thermal image, i) a process of generating first image data by performing a first image processing to identify material stock in a warehouse, ii) a warehouse At least one of the processes of generating second image data by performing second image processing to determine the storage status of my material, iii) performing third image processing for deriving the material identification information to generate third image data A material management system based on deep learning, characterized by performing one.
The method of claim 2,
Assuming that the material included in the real image correction image consists of an outline area and a material state area included in the outline area,
The solution server,
In the first image processing, the material management system based on deep learning, characterized in that the thermal image is synthesized as a corrected image in the material state area included in the real image corrected image to generate first image data.
The method of claim 2,
Assuming that the material included in the thermal image correction image consists of an outline area and a material state area included in the outline area,
The solution server,
In the second image processing, the real image is synthesized as a corrected image in the contour area included in the thermal image correction image to generate second image data.
The method of claim 2,
The solution server,
In the third image processing, i) black-and-white shading is performed on the real image, ii) the material identification information derived portion is recognized in the black-and-white shaded image, and iii) the image of the recognized portion is cropped. Material management system based on deep learning, characterized in that to generate the third image data.
The method of claim 5,
The mapping server,
A material management system based on deep learning, characterized in that the material identification information is derived by performing any one of character recognition, number recognition, barcode recognition, and QR code recognition from the third image data.
The method of claim 2,
The operation server,
i) Real-time identification, management and monitoring of material inventory in warehouses by performing deep learning using a convolutional neural network (CNN) based on the material identification information and the mapped first image data. Process and ii) deep learning using a convolutional neural network (CNN) based on the material identification information and the mapped second image data, and the storage state of the materials stacked in the warehouse A deep learning-based material management system, characterized in that by performing at least one of the processes of grasping and further determining whether a process input is possible for the material stored in the warehouse, and generating and storing material monitoring data.
The method of claim 7,
The operation server,
Based on the material identification information, the mapped third material management data, and the material monitoring data, deep learning is performed using a recurrent neural network (RNN), thereby A deep learning-based material management system, characterized in that it derives and stores material management history information including dispensing details and movement information.
In the method of managing materials based on deep learning,
A step of obtaining a real image by a real image camera installed in the warehouse, and obtaining a thermal image by a thermal image camera installed in the warehouse;
The wireless network device receives the real image and the thermal image from the real image camera and the thermal image camera, groups the real image and the thermal image to match a first identification code, and the first identification code, Transmitting first material management data including information on the real image and information on the thermal image to a gateway;
The gateway receives the first material management data from the wireless network device, a second identification code that updates the first identification code, a second material including information on the real image and information on the thermal image Transmitting management data to a solution server;
The solution server generates a plurality of image data by performing an image correction process and an image processing process based on the information on the real image and the information on the thermal image included in the second material management data, and the second Transmitting a third identification code updated with the identification code and third material management data including the plurality of image data to a mapping server;
The mapping server derives material identification information corresponding to at least one of letters, numbers, barcodes, and QR codes based on the third material management data received from the solution server, and the material identification information and the third material Mapping the management data and transmitting it to the operation server; And
The operation server includes the step of performing deep learning based on the material identification information and the third material management data to manage material information in the warehouse,
The image correction process includes: i) a real image corrected image by correcting at least one of contrast, saturation, hue, shade, and illuminance for the real image, or ii) contrast, saturation, hue, shadow, and illuminance for the thermal image. A deep learning-based material management method that refers to a process of generating a thermal image correction image by correcting at least one of them.

Images