KR102447253B1 - Method and system for detecting unauthorized storage item - Google Patents

Abstract

The present disclosure provides a method for detecting an unauthorized stored item performed by at least one processor. The method comprises: a step of obtaining an image for an item stored in an unmanned storage; a step of obtaining a storage risk index for the item from a first machine learning model by inputting the obtained image into the first machine learning model; and a step of determining whether the storage is allowed for the item based on the obtained storage risk index, wherein the first machine learning model may be learned by using the data for learning, which includes an image for learning and a reference storage risk index. Therefore, the present invention is capable of enabling a storage state of the item to be checked.

Description

METHOD AND SYSTEM FOR DETECTING UNAUTHORIZED STORAGE ITEM

The present disclosure relates to a method and system for detecting unauthorized storage items, and more particularly, to a method and system for detecting unauthorized items stored in unattended storage.

A service for renting small warehouses for individuals or small businesses has been launched. These small warehouses are located in the city center and are leased to users for a relatively short period of time, and the users exclusively use the small warehouses during the rental period.

These small warehouses are often operated unmanned. For example, the user accesses the server, pays the rent for the small warehouse, obtains a password or a key that can be used in the small warehouse, and then visits the small warehouse to store his or her goods.

However, in the small warehouse, flammable substances may be stored, and perishable food, etc. may be stored. If flammable substances are stored in small warehouses, the flammable substances can cause a fire. In addition, when food is stored in a small warehouse, an odor may occur due to spoilage of the food.

Accordingly, there is a need to detect unauthorized items in a small warehouse operated unattended.

The present disclosure provides a method for detecting unauthorized storage items, a computer program stored in a recording medium, and an apparatus (system) for solving the above problems.

The present disclosure may be implemented in various ways, including a method, an apparatus (system) and/or a computer program stored in a computer-readable storage medium, and a computer-readable storage medium in which the computer program is stored.

According to an embodiment of the present disclosure, a method for detecting unauthorized storage items performed by at least one processor includes obtaining an image of an item stored in an unattended storage, inputting the obtained image to a first machine learning model, , obtaining a storage risk index for the article from the first machine learning model, and based on the obtained storage risk index, determining whether to allow storage of the article, wherein the first machine learning model includes a learning image and It can be learned using training data that includes a reference storage risk index.

In addition, the unauthorized storage article detection method includes the steps of acquiring an image of the article in a first period, acquiring an image of the article in a second period, after storage permission is determined and the article is stored, the step of acquiring an image of the article in the first period The method may further include comparing the image with the image acquired in the second period to determine whether the item has changed and, in response to determining that the item has changed, providing a notification message to at least one of an administrator or a user. .

Also, the providing of the notification message may include providing a notification message including the image acquired in the first period and the image acquired in the second period to at least one of an administrator and a user.

In addition, the first period and the second period may be included in a period in which the locked state of the unattended storage is maintained.

Further, determining whether the article has changed may include: identifying a first region in which the article appears from the image acquired in the first period; identifying a second region in which the article appears from the image acquired in the second period; based on the matching rate of the identified first region and the second region, calculating a fluctuation index and determining that the article has changed when the fluctuation index exceeds a threshold value.

In addition, the method for detecting unauthorized storage includes receiving measurement data from a gas sensor after storage permission is determined and the product is stored, inputting the received measurement data into a second machine learning model, and the article from the second machine learning model and determining whether the quality of the article is deteriorated based on the obtained deterioration index and obtaining the deterioration index.

In addition, the second machine learning model may be trained using sample measurement data including a plurality of types of gas concentrations and training data including a reference deterioration index.

In addition, the step of determining whether the quality of the article is, when the obtained deterioration index exceeds a threshold, the step of controlling the door of the unattended storage to be opened.

In addition, the unauthorized storage article detection method includes the steps of detecting a fire in the unmanned storage, identifying the location where the fire occurred in the unmanned storage, and the spray hole of the fire extinguishing device included in the unmanned storage pointing to the identified location. It may further include the step of adjusting the angle of the fire extinguishing device and spraying the extinguishing liquid to the identified position by controlling the fire extinguishing device.

In addition, the step of identifying the location where the fire was detected includes identifying the location where the fire is detected based on at least one of smoke or flame included in an image obtained from a camera, and adjusting the angle of the injection hole may include, based on the location of the fire extinguishing device and the location where the fire is identified from the image, determining the angle of the injection hole and adjusting the angle of the injection hole to the determined angle.

Also, when the communication disconnection is sensed, the fire extinguishing device may spray the fire extinguishing liquid while rotating the injection port.

In addition, the fire extinguishing device may be installed on the inner surface of the ceiling of the unmanned storage.

In addition, the acquired image may be an image photographed using a fisheye camera, and the training image may be an image that is distorted and pre-processed based on a photographing method of the fisheye camera.

In addition, a computer program stored in a computer-readable recording medium may be provided to execute the above-described method for detecting unauthorized storage items in a computer.

An information processing system according to an embodiment of the present disclosure includes a memory and at least one processor connected to the memory and configured to execute at least one computer-readable program included in the memory, the at least one program being unattended Acquire an image of an article stored in storage, input the obtained image to a first machine learning model, obtain a storage risk index for the article from the first machine learning model, and based on the obtained storage risk index, Including instructions for determining whether to allow storage of the article, the first machine learning model may be trained using training data including a training image and a reference storage risk index.

According to some embodiments of the present disclosure, it may be determined whether the article included in the unattended storage is an unauthorized article, and storage of the article may be permitted based on the determination result. Accordingly, a situation in which unauthorized items such as flammable materials, animals, fresh food, etc. are stored in an unmanned storage box can be minimized.

According to some embodiments of the present disclosure, an article image is input to the machine learning model, and a storage risk index for the article may be obtained from the machine learning model in response thereto. Based on the storage risk index, whether the article is an unauthorized storage article can be objectively and accurately determined.

According to some embodiments of the present disclosure, an image of an article stored in the unattended storage may be analyzed, and a storage deformation state of the article may be determined. If deformation of the storage of the article is detected, such as expansion, collapse, reduction, movement, etc., a notification message may be provided to at least one of a user or an administrator. Accordingly, the user or the manager may visit the unmanned storage box and check the storage state of the article.

According to some embodiments of the present disclosure, based on data measured from the gas sensor, deterioration of the article stored in the unmanned storage may be detected. When the quality of the product is detected, a subsequent procedure such as opening the door of the unmanned storage may be performed, thereby minimizing the odor caused by the decomposition of the product.

According to some embodiments of the present disclosure, by detecting the deterioration of the article using a machine learning model, it is possible to accurately detect the deterioration of the article at an initial point in time before the article is completely decomposed.

According to some embodiments of the present disclosure, when a fire occurs inside the unmanned storage, the angle of the injection port may be adjusted in the direction of the point where the fire occurred, so that the extinguishing liquid may be sprayed to the point where the fire occurred. Accordingly, not only can the fire be extinguished at an early stage, but the area where the extinguishing liquid is sprayed can be minimized.

The effect of the present disclosure is not limited to the above-mentioned effects, and other effects not mentioned are clear to those of ordinary skill in the art (referred to as 'person of ordinary skill') from the description of the claims. will be able to be understood

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present disclosure will be described with reference to the accompanying drawings described below, wherein like reference numerals denote like elements, but are not limited thereto.
1 is a schematic diagram illustrating a configuration in which an information processing system according to an embodiment of the present disclosure is connected to communicate with a plurality of unmanned storage devices.
2 is a block diagram illustrating an internal configuration of an information processing system according to an embodiment of the present disclosure.
3 is a block diagram illustrating an internal configuration of a processor included in an information processing system according to an embodiment of the present disclosure.
4 is a diagram illustrating data output through a first machine learning model according to an embodiment of the present disclosure.
5 is a diagram illustrating a process in which a first machine learning model is trained, according to an embodiment of the present disclosure.
6 is a diagram illustrating data output through a second machine learning model, according to an embodiment of the present disclosure.
7 is a diagram illustrating a process in which a second machine learning model is trained, according to an embodiment of the present disclosure.
8 is a diagram illustrating an example of a machine learning model including an artificial neural network, according to an embodiment of the present disclosure.
9 is a diagram illustrating an unattended storage with an open door, according to an embodiment of the present disclosure.
10 is a view exemplarily showing the inside of an unattended storage into which a digestive fluid is sprayed, according to an embodiment of the present disclosure.
11 is a flowchart illustrating a method of detecting an unauthorized storage article stored in an unattended storage according to an embodiment of the present disclosure.
12 is a flowchart illustrating a method of detecting a change state of an article stored in an unattended storage according to an embodiment of the present disclosure.
13 is a flowchart illustrating a method of detecting a deterioration state of an article stored in an unattended storage according to an embodiment of the present disclosure.
14 is a flowchart illustrating a method of responding to a fire occurring in an unattended storage according to an embodiment of the present disclosure.

Hereinafter, specific contents for carrying out the present disclosure will be described in detail with reference to the accompanying drawings. However, in the following description, if there is a risk of unnecessarily obscuring the gist of the present disclosure, detailed descriptions of well-known functions or configurations will be omitted.

In the accompanying drawings, identical or corresponding components are assigned the same reference numerals. In addition, in the description of the embodiments below, overlapping description of the same or corresponding components may be omitted. However, even if description regarding components is omitted, it is not intended that such components are not included in any embodiment.

Advantages and features of the disclosed embodiments, and methods of achieving them, will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the present disclosure to be complete, and the present disclosure provides those skilled in the art with the scope of the invention. It is provided for complete information only.

Terms used in this specification will be briefly described, and the disclosed embodiments will be described in detail. The terms used in the present specification have been selected as currently widely used general terms as possible while considering the functions in the present disclosure, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the terms used in the present disclosure should be defined based on the meaning of the term and the contents of the present disclosure, rather than the simple name of the term.

Expressions in the singular herein include plural expressions unless the context clearly dictates the singular. Also, the plural expression includes the singular expression unless the context clearly dictates the plural. In the entire specification, when a part includes a certain component, this means that other components may be further included, rather than excluding other components, unless otherwise stated.

In addition, the term 'module' or 'unit' used in the specification means a software or hardware component, and 'module' or 'unit' performs certain roles. However, 'module' or 'unit' is not meant to be limited to software or hardware. A 'module' or 'unit' may be configured to reside on an addressable storage medium or may be configured to reproduce one or more processors. Thus, as an example, a 'module' or 'unit' refers to components such as software components, object-oriented software components, class components, and task components, and processes, functions, and properties. , procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, database, data structures, tables, arrays or at least one of variables. Components and 'modules' or 'units' are the functions provided within are combined into a smaller number of components and 'modules' or 'units' or additional components and 'modules' or 'units' can be further separated.

According to an embodiment of the present disclosure, a 'module' or a 'unit' may be implemented with a processor and a memory. 'Processor' should be construed broadly to include general purpose processors, central processing units (CPUs), microprocessors, digital signal processors (DSPs), controllers, microcontrollers, state machines, and the like. In some contexts, a 'processor' may refer to an application specific semiconductor (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), or the like. 'Processor' refers to a combination of processing devices, such as, for example, a combination of a DSP and a microprocessor, a combination of a plurality of microprocessors, a combination of one or more microprocessors in combination with a DSP core, or any other such configurations. You may. Also, 'memory' should be construed broadly to include any electronic component capable of storing electronic information. 'Memory' means random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), programmable read-only memory (PROM), erase-programmable read-only memory (EPROM); may refer to various types of processor-readable media, such as electrically erasable PROM (EEPROM), flash memory, magnetic or optical data storage, registers, and the like. A memory is said to be in electronic communication with the processor if the processor is capable of reading information from and/or writing information to the memory. A memory integrated in the processor is in electronic communication with the processor.

In the present disclosure, a 'system' may include at least one of a server device and a cloud device, but is not limited thereto. For example, a system may consist of one or more server devices. As another example, a system may consist of one or more cloud devices. As another example, the system may be operated with a server device and a cloud device configured together.

In addition, terms such as first, second, A, B, (a), (b) used in the following embodiments are only used to distinguish a certain component from other components, and the term The essence, order, or order of the components is not limited by the

In addition, in the following embodiments, when it is described that a component is 'connected', 'coupled' or 'connected' to another component, the component may be directly connected or connected to the other component, but , it should be understood that another element may be 'connected', 'coupled' or 'connected' between each element.

In addition, as used in the following embodiments, 'comprises' and/or 'comprising' refers to a referenced component, step, operation and/or element being one or more other components, steps, operations. and/or the presence or addition of elements.

Hereinafter, various embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

1 is a schematic diagram illustrating a configuration in which the information processing system 110 according to an embodiment of the present disclosure is connected to communicate with a plurality of unattended storages 120_1 , 120_2 , and 120_3 . Here, the unmanned storage 120_1 , 120_2 , and 120_3 may be an article storage facility operated unattended. A plurality of unattended storage may be located in a specific place.

The unattended storage 120_1 , 120_2 , and 120_3 may include a plurality of Internet of Things (IoT) devices. IoT devices may communicate with the information processing system 110 that may provide an unattended storage management service through the network 130 . According to an embodiment, as an IoT device, at least one of a fire extinguishing device, a camera, an electronic lock, or a gas sensor may be included in each of the unattended storages 120_1 , 120_2 , and 120_3 .

The network 130 may be configured to enable communication between the IoT devices included in the plurality of unattended storages 120_1 , 120_2 , and 120_3 and the information processing system 110 . Network 130 according to the installation environment, for example, Ethernet (Ethernet), wired home network (Power Line Communication), telephone line communication device and wired networks such as RS-serial communication, mobile communication network, WLAN (Wireless LAN), It may consist of a wireless network such as Wi-Fi, Bluetooth and ZigBee, or a combination thereof. The communication method is not limited, and short-distance wireless communication between IoT devices as well as a communication method using a communication network (eg, mobile communication network, wired Internet, wireless Internet, broadcasting network, satellite network, etc.) that the network 130 may include is also included. can

In one embodiment, information processing system 110 includes one or more server devices and/or databases capable of storing, providing and executing computer executable programs (eg, downloadable applications) and data associated with unattended storage management services and the like; Alternatively, it may include one or more distributed computing devices and/or distributed databases based on cloud computing services.

The information processing system 110 may communicate with the IoT devices included or installed in the unmanned storage via the network 130 , and may also control the IoT devices. For example, the information processing system 110 controls the electronic lock included in the unattended storage (120_1, 120_2, 120_3) for storing the article when the deterioration of the article is detected to control the unattended storage (120_1, 120_2, 120_3) can open the door of As another example, when a fire is detected, the information processing system 110 controls the fire extinguishing device so that the injection port is directed to the location where the fire occurred, and then opens the valve of the fire extinguishing device to control the fire extinguishing liquid to be sprayed to the fire point. can

The information processing system 110 may use a first machine learning model for determining whether an article stored in an unattended storage is an unauthorized article, as will be described later. Additionally or alternatively, the information processing system 110 may use a second machine learning model to determine whether the archived item has deteriorated.

2 is a block diagram illustrating an internal configuration of the information processing system 110 according to an embodiment of the present disclosure. The information processing system 110 may include a memory 210 , a processor 220 , a communication module 230 , and an input/output interface 240 . As shown in FIG. 2 , the information processing system 110 may be configured to communicate information and/or data through a network using the communication module 230 .

The memory 210 may include any non-transitory computer-readable recording medium. According to one embodiment, the memory 210 may include a non-volatile mass storage device such as read only memory (ROM), a disk drive, a solid state drive (SSD), a flash memory, or the like. can As another example, a non-volatile mass storage device such as a ROM, an SSD, a flash memory, a disk drive, etc. may be included in the information processing system 110 as a separate permanent storage device distinct from the memory. Also, an operating system and at least one program code (eg, a code for an unattended storage service, etc.) may be stored in the memory 210 . In FIG. 2 , the memory 210 is illustrated as a single memory, but this is only for convenience of description, and the memory 210 may include a plurality of memories.

The software components may be loaded from a computer-readable recording medium separate from the memory 210 . Such a separate computer-readable recording medium may include a recording medium directly connectable to the information processing system 110, for example, a floppy drive, a disk, a tape, a DVD/CD-ROM drive, a memory card, etc. It may include a computer-readable recording medium. As another example, the software components may be loaded into the memory 210 through the communication module 230 rather than a computer-readable recording medium. For example, the at least one program is a computer program (eg, for unmanned storage service program, etc.) may be loaded into the memory 210 .

The communication module 230 may provide a configuration or function for a user terminal and/or an external device and the information processing system 110 to communicate with each other through a network, and for the information processing system 110 to communicate with an external system. It may provide a configuration or function.

In addition, the input/output interface 240 of the information processing system 110 is connected to the information processing system 110 or means for interfacing with a device (not shown) for input or output that the information processing system 110 may include. can be For example, the input/output interface 240 may include at least one of a PCI express interface and an Ethernet interface. Although the input/output interface 240 is illustrated as an element configured separately from the processor 220 in FIG. 2 , the present invention is not limited thereto, and the input/output interface 240 may be configured to be included in the processor 220 . The information processing system 110 may include more components than those of FIG. 2 .

The processor 220 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The instructions may be provided to the processor 220 by the memory 210 or the communication module 230 . For example, the processor 220 may be configured to execute a received instruction according to program code stored in a recording device such as the memory 210 . In an embodiment, at least one program including instructions for executing methods according to various embodiments to be described later may be stored in the memory 210 . Also, the processor 220 may be configured to execute at least one program.

According to an embodiment, the program provides commands for determining whether an item stored in the unattended storage is an unauthorized item, commands for determining whether the state of the item stored in the unattended storage has changed, and a fire inside the unattended storage It may include at least one of commands for detecting whether .

3 is a block diagram illustrating an internal configuration of a processor 220 included in the information processing system 110 according to an embodiment of the present disclosure. As shown in FIG. 3 , the processor 220 includes a model learning unit 310 , a dangerous goods determination unit 320 , a state change detection unit 330 , a fire detection unit 340 , and an article deterioration detection unit 350 . may include

The model learning unit 310 may perform learning on the first machine learning model by using a plurality of first learning data. The first training data may include a pre-processed training image and a reference storage risk index. For example, the first learning data may include a learning image including an object for unauthorized items, such as flammable substances, animals, and food with short-term storage properties, and additionally include a reference storage risk index related to the object of unauthorized items. As another example, the first learning data may include a learning image including an object for an authorized article, such as a bicycle, clothing, electronic product, tool, etc., and additionally include a reference storage risk index related to the object of the corresponding authorized article. .

According to an embodiment, the pre-processed learning image may be an image that is intentionally distorted from an original image based on a photographing method of a camera installed in an unmanned storage. For example, the camera installed in the unmanned storage may be a fisheye camera, and in this case, a partial area (eg, an area near a corner) among the entire photographing area may be distorted according to the fisheye effect according to a photographing method of the fisheye camera. In some embodiments of the present disclosure, the original image may be intentionally distorted and pre-processed by applying the fisheye effect to the undistorted original image.

Meanwhile, a camera other than the fisheye camera may be installed in the unmanned storage. In this case, pre-processing of the image for training may be omitted. That is, when a camera that does not generate distortion is disposed in the unattended storage, an original image that is not distorted may be used as an image for learning.

The model learning unit 310 may learn the second machine learning model using a plurality of second learning data. The second learning data may include measurement data for learning and a reference article deterioration index. Here, the measurement data for learning may include a plurality of types of gas concentrations. For example, the measurement data for training may include a first type of gas concentration, a second type of gas concentration, and a third type of gas concentration. In addition, the reference article deterioration index may be associated with a plurality of types of gas concentrations. A method for learning the first machine learning model by the model learning unit 310 will be described later with reference to FIG. 5 , and a method for learning the second machine learning model will be described with reference to FIG. 7 .

The dangerous goods determination unit 320 may determine whether an article (hereinafter, referred to as a 'target article') stored in the unmanned storage is an unauthorized article. According to an embodiment, the dangerous goods determining unit 320 acquires an image of the target article, and after inputting the image of the target article into the first machine learning model, the storage risk of the article output from the first machine learning model index can be obtained. In this case, the dangerous article determination unit 320 may acquire an image of the target article by using a camera installed in the unmanned storage.

If the storage risk index is less than or equal to a preset first threshold, the dangerous goods determination unit 320 may determine that the target article is not an unauthorized article and allow storage of the target article. If the storage of the stored items is allowed, the electronic locking device of the unmanned storage box is activated (that is, made operable), and the dangerous goods determination unit 320 can provide the user with an authentication key that can open the unattended storage. have. In this case, the dangerous goods determination unit 320 may transmit the authentication key to the user's terminal. On the other hand, when the storage risk index for the stored goods exceeds the first threshold, the dangerous goods determination unit 320 may not allow storage of the stored goods by determining that the stored goods are unauthorized goods. In this case, the inactive state of the electronic lock included in the unattended storage is maintained, and the door lock using the electronic lock may not proceed.

The state change detection unit 330 may monitor whether the state of the item being stored has changed, such as expansion, contraction, or collapse. According to an embodiment, the state change detection unit 330 may store the item in the unattended storage and maintain the locked state of the electronic lock, while the item is stored at a plurality of different time points (eg, the first time point and the second time point). An image may be acquired and the acquired image may be compared to determine whether the state of the article has changed. The state change detection unit 330 identifies a first region in which an article (ie, an article object) appears in the image acquired at the first time point, and identifies a second region in which the article appears in the image acquired at the second time point. , a matching rate between the first region and the second region may be calculated. Also, the processor may calculate a variation index of the article based on a matching rate between the first region and the second region. For example, the processor may calculate the variation index of the article by applying a negative weight to the matching rate of the first area and the second area. That is, the magnitude of the variation index may be inversely proportional to the coincidence rate between the first region and the second region. Here, the weight may be dynamically determined based on at least one of the size of the article or the storage period of the article, or may be predetermined as a static value.

The state change detection unit 330 may determine that the product has not changed if the calculated variation index is less than or equal to the second threshold, and may determine that the product has changed if the variation index exceeds the second threshold. When it is determined that the item has changed, the state change detection unit 330 may generate an item change notification message including an image photographed at a first time point and an image photographed at a second time point and provide it to at least one of a user or an administrator. have.

The fire detection unit 340 may determine whether a fire has occurred inside the unattended storage. For example, the fire detection unit 340 may determine whether a fire has occurred by using a gas sensor included in the unmanned storage. Also, the fire detection unit 340 may determine whether a fire has occurred based on an image inside the unattended storage received from the camera. In this case, the fire detection unit 340 may detect a fire by recognizing a flame or smoke from an image captured by the camera. In addition, the fire detection unit 340 may identify a point where a flame or smoke is generated in the image captured by the camera, and control the fire extinguishing device to adjust the angle of the spray hole by controlling the fire extinguishing device so that the spray port of the fire extinguishing device is directed to the identified point. In addition, when the angle adjustment is completed, the fire detection unit 340 may open the valve of the fire extinguishing device to spray the extinguishing liquid to the fire point.

The article deterioration detection unit 350 may determine whether the article stored in the unmanned storage has deteriorated. According to an embodiment, the article quality detection unit 350 receives measurement data including a plurality of different types of gas concentrations by using a gas sensor, and inputs the measured data to the second machine learning model for the article. You can get a quality index.

When the deterioration index is less than or equal to a preset third threshold, the article deterioration detection unit 350 may determine that the stored article is not deteriorated. On the other hand, when the deterioration index exceeds the third threshold, the article quality detection unit 350 may determine that the stored article is deteriorated, and transmit a message notifying the deterioration of the article to the terminal of the manager or the terminal of the user. In this case, the article deterioration detection unit 350 may control the unmanned storage to be opened by transmitting a door opening command to the electronic locking device of the unmanned storage in which the article is stored. In addition, the article quality detection unit 350 may control the fan to maximize the output of the fan installed in the space in which the unmanned storage box is located. Here, the fan may be an IoT device, and may be controlled by an information processing system. As the door of the unmanned storage is opened and the output of the fan is maximized, the odor generated from the deteriorated article may be minimized.

4 is a diagram illustrating data output through the first machine learning model 410 according to an embodiment of the present disclosure. As illustrated in FIG. 4 , the first machine learning model 410 may be an artificial intelligence-based model configured to output a storage risk index 440 for items stored in unmanned storage. An image 430 of the stored article photographed by the camera 420 may be input to the first machine learning model 410 . For example, the camera 420 may be a fisheye camera, and the image 430 of the stored article may be an image distorted by the fisheye effect. As another example, the camera 420 may be a camera that does not generate distortion. The first machine learning model 410 may output a storage risk index 440 for the article by performing an operation based on the image 430 of the stored article.

5 is a diagram illustrating a process in which the first machine learning model 410 is learned, according to an embodiment of the present disclosure. Learning for the first machine learning model 410 may be performed by at least one processor (eg, the model learning unit of FIG. 3 ) included in the information processing system.

Referring to FIG. 5 , the processor may extract the first training data including the pre-processed training image 510 and the reference storage risk index 530 . Here, the preprocessed learning image 510 may be an image in which the original image is intentionally distorted by reflecting the fisheye effect. The reference storage risk index 520 may be an index set in association with the article object included in the pre-processed learning image 510 , and may be used as a ground truth in the learning process.

The first machine learning model 410 may analyze the training image 510 , identify an object related to an article from the training image, and output a storage risk index 520 for the identified article object. A loss value 540 that is a difference between the output storage risk index 520 and the reference storage risk index 530 may be calculated and fed back to the first machine learning model 410 . The fed back loss value 540 may be reflected in the first machine learning model 410 , and weights for at least one node included in the first machine learning model 410 may be adjusted. Here, the node may be a node included in the artificial neural network.

When a plurality of different first learning data are repeatedly input to the first machine learning model 410 and repeated learning is performed, the weights of the nodes included in the first machine learning model 410 converge to an optimal value. can

Meanwhile, according to another embodiment, the first machine learning model 410 may be learned by using the training data including the original training image (ie, the undistorted image) and the reference storage risk index 530 . . Specifically, the training image is input to the first machine learning model 410 in an undistorted state, and then between the storage risk index 520 output from the first machine learning model 410 and the reference storage risk index 530 . The loss value 540 may be fed back to the first machine learning model 410 so that the first machine learning model 410 may be trained.

In other words, when a camera that generates a distortion effect, such as a fisheye camera, is disposed in the unmanned storage, the preprocessed image 510 for learning may be used to learn the first machine learning model 410 . On the other hand, when a camera that does not generate a distortion effect is disposed in the unattended storage, an original image that is not distorted may be used to learn the first machine learning model 410 .

6 is a diagram illustrating data output through the second machine learning model 610 according to an embodiment of the present disclosure. As illustrated in FIG. 6 , the second machine learning model 610 may be an artificial intelligence-based model configured to output a deterioration index 640 for articles stored in unmanned storage. The gas measurement data 630 collected by the gas sensor 620 may be input to the second machine learning model 610 . Here, the gas measurement data may include a plurality of different types of gas concentrations. The second machine learning model 610 may perform an operation based on a plurality of gas concentrations included in the measurement data to output a deterioration index 640 for the article.

7 is a diagram illustrating a process in which the second machine learning model 610 is trained, according to an embodiment of the present disclosure. Learning of the second machine learning model 610 may be performed by at least one processor (eg, the model learning unit of FIG. 3 ) included in the information processing system.

Referring to FIG. 7 , the processor may extract the second learning data including the learning measurement data 710 and the reference article deterioration index 730 . Here, the learning measurement data 710 may include a plurality of types of gas concentrations that may be measured through a gas sensor. In addition, the reference article deterioration index may include an index related to measurement data for learning, and may be used as a ground truth in the learning process.

The second machine learning model 610 may perform an operation based on the concentration of a plurality of types of gases included in the input measurement data 710 for learning, and may output the deterioration index 720 for the article. A loss value 740 that is a difference between the output product deterioration index 720 and the reference product deterioration risk index 730 may be calculated and fed back to the second machine learning model 610 . The fed back loss value 740 may be reflected in the second machine learning model 610 , and weights for at least one node included in the second machine learning model 610 may be adjusted. Here, the node may be a node included in the artificial neural network.

When a plurality of different second learning data is repeatedly input to the second machine learning model 610 and repeated learning is performed, the weights of the nodes included in the second machine learning model 610 converge to an optimal value. can

8 is a diagram illustrating an example of a machine learning model 800 including an artificial neural network, according to an embodiment of the present disclosure. The machine learning model 800 may refer to a statistical learning algorithm implemented based on the structure of a biological neural network or a structure for executing the algorithm in machine learning technology and cognitive science. That is, the machine learning model 800 repeatedly adjusts the weights of synapses, which are artificial neurons that form a network by combining synapses, as in a biological neural network, so that the correct output corresponding to a specific input and the inferred output are obtained. It represents a machine learning model with problem-solving ability by learning to reduce the error between them.

According to an embodiment, the machine learning model 800 may be implemented as a multilayer perceptron (MLP) consisting of nodes of multiple layers and connections therebetween. The machine learning model 800 according to the present embodiment may be implemented using one of various artificial neural network structures including MLP. The machine learning model 800 is located between an input layer that receives an input signal or data from the outside, an output layer that outputs an output signal or data corresponding to the input data, and an input layer and an output layer, and receives a signal from the input layer to extract characteristics It is composed of n hidden layers that are transmitted to the output layer. Here, the output layer receives a signal from the hidden layer and outputs it to the outside. In the learning process, a weight associated with at least one node may be adjusted.

According to an embodiment, at least one of the first machine learning model and the second machine learning model may be configured to include the artificial neural network illustrated in FIG. 8 . The input data illustrated in FIG. 8 may be measurement data including an image or gas concentration for an article, and the output data illustrated in FIG. 8 may be a storage risk index or an article deterioration index.

9 is a diagram illustrating an unattended storage 900 in which the door 910 is opened, according to an embodiment of the present disclosure. As illustrated in FIG. 9 , the unmanned storage 900 may be manufactured in the form of a cabinet, and an openable and openable door 910 , a camera 930 capable of capturing an internal image, an electronic lock 920 , and a fire extinguishing device ( 940 , and a gas sensor 950 .

The electronic locking device 920 may lock the door 910 or open the door 910 under the control of the information processing system or the user terminal. The electronic lock 920 is a device that supports IoT communication, and may communicate with the information processing system through a network.

A camera 930 capable of photographing the inner space of the unattended storage 900 may be installed at the door 910 of the unattended storage 900 . Meanwhile, the camera 930 may be installed at another point (eg, a ceiling) that can photograph the interior of the unattended storage 900 other than the door 910 . The camera 930 may continuously photograph an image inside the unattended storage 900 , or may photograph it only during a specific period. The camera 930 may be a camera capable of network communication, and in this case, may communicate with the information processing system.

The fire extinguishing device 940 is a device capable of injecting a fire extinguishing liquid, and may be an IoT device. According to an embodiment, in the fire extinguishing device 940, the injection port through which the fire extinguishing liquid is sprayed may rotate up/down/left/right within a predetermined angle range. Accordingly, the injection hole may be rotated at a three-dimensional angle. The angle of the fire extinguishing device 940 may be adjusted under the control of the information processing system. Also, the fire extinguishing device 940 may include a pipe forming a flow path between the storage tank for storing the fire extinguishing liquid and the injection port. The inside of the pipe may include an electromagnetic valve that determines whether to open or close. The opening and closing of the electromagnetic valve may be controlled by an information processing system. That is, according to the control of the information processing system, the valve may be opened and the extinguishing liquid may be injected through the injection port. The extinguishing liquid sprayed through the fire extinguishing device 940 has a directional characteristic, and thus, when the extinguishing liquid is sprayed, splashing of the extinguishing liquid to an area separated from the fire occurrence point can be minimized.

Meanwhile, when a fire spreads inside the unattended storage 900 , communication between the fire extinguishing device 940 and the information processing system may be cut off. In this case, the fire extinguishing device 940 may not be controlled according to the control of the information processing system. In preparation for such a situation, the fire extinguishing device 940 moves the injection port in a certain direction so that the fire extinguishing liquid is sprayed throughout the storage if communication with the information processing system continues beyond the threshold time while the extinguishing liquid is sprayed to the fire point. By rotating at speed, you can spray fire extinguishing fluid. That is, the fire extinguishing device 940 may spray the fire extinguishing liquid to the entire area inside the unattended storage.

The gas sensor 950 may measure concentrations of a plurality of types of gases generated inside the unattended storage. For example, the gas sensor 950 may measure the concentration of hydrogen sulfide (H ₂ S) and measure the concentration of ammonia (NH ₃ ). Additionally, the gas sensor 950 may measure a concentration for nitrogen dioxide, and may measure a concentration for various types of gases according to embodiments. The gas sensor 950 may provide measurement data including each gas concentration to the information processing system.

Meanwhile, the unattended storage may be manufactured in various sizes and shapes other than the size illustrated in FIG. 9 . In addition, the unattended storage may be equipped with a communication device (eg, an access point) capable of communicating with a network. In this case, the unattended storage may communicate with the information processing system through the communication device. In addition, at least one of the electronic locking device 920 , the camera 930 , the gas sensor 950 , and the fire extinguishing device 940 is connected to a communication device by wire or wirelessly, and communicates with the information processing system using the communication device. can

10 is a view exemplarily showing the inside of the unattended storage 1000 to which the extinguishing liquid is sprayed, according to an embodiment of the present disclosure. As illustrated in FIG. 10 , when a fire occurs inside the unmanned storage, the injection port 1040 included in the fire extinguishing device is directed to the fire occurrence point 1010 under the control of the information processing system. angle can be adjusted.

According to an embodiment, the information processing system may periodically or in real time photograph an internal image of the unmanned storage from a camera included in the unmanned storage, and identify a point where smoke or flame is generated from the captured image as a fire occurrence point. The information processing system may adjust the angle of the injection hole 1040 so that the injection hole 1040 of the fire extinguishing device is directed to a point where a fire occurs in the entire area of the unmanned storage. When the angle adjustment of the injection hole is completed, the information processing system may open the valve of the fire extinguishing device to allow the extinguishing liquid to be sprayed to the fire generating point 1010 .

11 is a flowchart illustrating a method 1100 of detecting an unauthorized storage article stored in an unattended storage according to an embodiment of the present disclosure. The method shown in FIGS. 11 to 14 is only one embodiment for achieving the object of the present disclosure, and it goes without saying that some steps may be added or deleted as necessary. Also, the method illustrated in FIGS. 11 to 14 may be performed by at least one processor included in the information processing system illustrated in FIG. 2 . For convenience of explanation, each step shown in FIGS. 11 to 14 will be described as being performed by a processor included in the information processing system shown in FIG. 2 .

Referring to FIG. 11 , the processor may acquire an image of an article stored in the unattended storage (S1110). An image of the stored article may be taken through a camera included in the unmanned storage. For example, after the user pays for the unmanned storage service to rent the unattended storage, the item may be stored in the unattended storage. In this case, the information processing system may acquire an image of the article by using a camera included in the unmanned storage.

Thereafter, the processor may input the obtained image to the first machine learning model to obtain a storage risk index for the article from the first machine learning model ( S1120 ). As described above, the first machine learning model may be a machine learning model learned using the training data including the training image and the reference storage risk index. In some embodiments, the camera disposed in the unattended storage may be a fisheye camera, and in this case, the training image may be an image pre-processed to be distorted by applying a photographing method of the fisheye camera.

Next, the processor may determine whether to allow storage of the article based on the storage risk index (S1130). In an embodiment, if the storage risk index is equal to or less than a first threshold value set in advance, the processor may determine that the stored item is not an authorized item and permit storage of the stored item. If the storage of the stored item is allowed, the processor may provide an authentication key for opening the unattended storage to the user, and may also allow locking of the electronic lock included in the unattended storage. Accordingly, the article is stored in the unattended storage, and the user can open the unattended storage by using the authentication key.

On the other hand, if the storage risk index of the stored article exceeds the first threshold, the processor may determine that the stored article is an unauthorized article and disallow storage of the stored article. In this case, the processor may control not to set the locked state of the electronic lock, and may transmit a warning message indicating that the unauthorized item cannot be stored to at least one of the user's terminal and the manager's terminal. Accordingly, even if the unauthorized item is stored in the unattended storage, the door is not locked, and thus a normal unattended storage service may not be provided to the user.

12 is a flowchart illustrating a method 1200 of detecting a change state of an article stored in an unattended storage according to an embodiment of the present disclosure. The method illustrated in FIG. 12 may be performed after the method illustrated in FIG. 11 .

Referring to FIG. 12 , the processor may detect storage of an article in the unattended storage ( S1210 ). For example, when the electronic lock is in a locked state after the article is stored in the unattended storage, the processor may determine that the article is stored in the unattended storage.

Thereafter, the processor may acquire an article image in a preset first period using the camera ( S1220 ). Subsequently, the processor may acquire an article image in a second period, which is a viewpoint after the first period, by using the camera ( S1230 ). Here, the first cycle and the second cycle may be included in a period in which the locked state of the electronic lock is continuously maintained. That is, while the locked state of the electronic locking device is maintained, the processor may acquire the article image in the first period and acquire the article image in the second period.

Next, the processor may compare the image acquired in the first period with the image acquired in the second period to calculate a variation index of the article ( S1240 ). In an embodiment, the processor may identify a first area occupied by an article (ie, an article object) in the image acquired at the first time point, and identify a second area occupied by the article in the image acquired at the second time point . Subsequently, the processor may calculate a matching rate between the identified first area and the second area, and then calculate a variation index of the article based on the calculated matching rate. For example, the processor may calculate the variation index of the article by applying a negative weight to the matching rate of the first area and the second area.

Next, the processor may determine whether the calculated variation index exceeds a second threshold (S1250). In response to determining that the calculated variation index exceeds the second threshold, the processor generates an article change notification message including an image taken in the first cycle and an image taken in the second cycle, and provides it to at least one of a user or an administrator It can be done (S1260). The product change notification message may be transmitted to at least one of a user's terminal and a manager's terminal.

On the other hand, the processor illustrated in FIG. 12 is related to one cycle for determining the item change state, and while the item is stored in the unattended storage, the process from step S1220 of FIG. 12 may be repeatedly performed.

According to the present disclosure, when the state of the item in the unmanned storage is changed, such as collapsed, expanded, or contracted, the user or manager may check this through a notification message. Accordingly, the user can quickly recognize a situation in which the goods are unintentionally changed. In addition, unauthorized items (eg, pets in a box, etc.) that are not detected based on the storage risk index may be detected through the item change index and reported to the manager.

13 is a flowchart illustrating a method 1300 of detecting a deterioration state of an article stored in an unattended storage according to an embodiment of the present disclosure. The method illustrated in FIG. 13 may be performed after the method illustrated in FIG. 11 .

Referring to FIG. 13 , the processor may detect storage of an article in the unattended storage ( S1310 ). For example, when the electronic lock is in a locked state after the article is stored in the unattended storage, the processor may detect that the article is stored in the unattended storage.

Thereafter, the processor may determine whether a data measurement period arrives ( S1320 ). When a data measurement period arrives, the processor may receive gas measurement data including a plurality of types of gas concentrations from the gas sensor by using the gas sensor included in the unmanned storage ( S1330 ).

Subsequently, the processor may input the received gas measurement data into the second machine learning model to obtain a deterioration index for the article (S1430). As described above, the second machine learning model may be a machine learning model learned using sample measurement data including a plurality of types of gas concentrations and learning data including a reference change index.

Thereafter, the processor may determine whether the deterioration index exceeds a third threshold (S1350). When the deterioration index exceeds the third threshold, the processor may determine that the article stored in the unattended storage is corrupted, and may proceed with a process corresponding to the deterioration of the article ( S1360 ). For example, the processor may transmit a message notifying the deterioration of the article to the terminal of the manager or the terminal of the user. As another example, the processor may control the unattended storage to be opened by transmitting a door opening command to the electronic lock of the unmanned storage in which the article is stored. Additionally, the processor may control the fan to maximize the output of the fan installed in the space in which the unmanned storage box is located.

According to the present embodiment, deterioration of the goods stored in the unmanned storage is detected at an early stage, so that odors and the like generated due to the decay of the goods can be prevented.

14 is a flowchart illustrating a method 1400 for responding to a fire occurring in an unattended storage according to an embodiment of the present disclosure. The processor may use at least one of a gas sensor and a camera included in the unmanned storage to detect a fire that has occurred inside the unmanned storage (S1410). According to an embodiment, the processor may periodically or in real time take an image inside the unattended storage from a camera included in the unattended storage, and when smoke or flame is identified from the captured image, it may be determined that a fire has occurred inside the unattended storage. . An object image related to fire or flame may be stored in the information processing system, and the processor may determine that a fire has occurred inside the unattended storage when an object related to fire or flame is identified in the captured image.

When a fire is detected, the processor may identify a point where a fire occurred in the unmanned storage based on the captured image (S1420). Thereafter, the processor may adjust the angle of the injection port so that the injection port of the fire extinguishing device is directed to the point where the fire occurred ( S1430 ). In one embodiment, the processor may control the fire extinguishing device included in the unmanned storage to adjust the angle of the injection hole. In this case, the processor may control the fire extinguishing device so that the angle of the injection hole becomes the determined angle after determining the angle of the injection hole based on the relative position between the installation position of the injection hole and the location where the fire is identified in the image.

Then, when the angle adjustment of the injection hole is completed, the processor may open the valve of the fire extinguishing device to control the fire extinguishing liquid to be sprayed to the fire generating point (S1440).

According to the present embodiment, a fire may be detected early in the unmanned storage, and the extinguishing liquid may be sprayed to the point where the fire occurred. Accordingly, the splashing of the extinguishing liquid other than the point where the fire occurred can be minimized.

The above flowchart and the above description are only examples, and may be implemented differently in some embodiments. For example, in some embodiments, the order of each step may be changed, some steps may be repeatedly performed, some steps may be omitted, or some steps may be added.

The above-described method may be provided as a computer program stored in a computer-readable recording medium for execution by a computer. The medium may be to continuously store a program executable by a computer, or to temporarily store it for execution or download. In addition, the medium may be various recording means or storage means in the form of a single or several hardware combined, it is not limited to a medium directly connected to any computer system, and may exist distributed on a network. Examples of the medium include hard disks, magnetic media such as floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto optical media such as floppy disks, and There may be ones configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media include recording media or storage media managed by an app store that distributes applications, sites that supply or distribute various other software, and servers.

The method, operation, or techniques of this disclosure may be implemented by various means. For example, these techniques may be implemented in hardware, firmware, software, or a combination thereof. Those of ordinary skill in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design requirements imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementations should not be interpreted as causing a departure from the scope of the present disclosure.

In a hardware implementation, the processing units used to perform the techniques include one or more ASICs, DSPs, digital signal processing devices (DSPDs), programmable logic devices (PLDs). ), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, electronic devices, other electronic units designed to perform the functions described in this disclosure. , a computer, or a combination thereof.

Accordingly, the various illustrative logic blocks, modules, and circuits described in connection with this disclosure are suitable for use in general purpose processors, DSPs, ASICs, FPGAs or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or the present disclosure. It may be implemented or performed in any combination of those designed to perform the functions described in A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, eg, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other configuration.

In firmware and/or software implementations, the techniques include random access memory (RAM), read-only memory (ROM), non-volatile random access memory (NVRAM), PROM ( on computer-readable media such as programmable read-only memory), erasable programmable read-only memory (EPROM), electrically erasable PROM (EEPROM), flash memory, compact disc (CD), magnetic or optical data storage devices, etc. It may be implemented as stored instructions. The instructions may be executable by one or more processors, and may cause the processor(s) to perform certain aspects of the functionality described in this disclosure.

delete

delete

A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium may be coupled to the processor such that the processor can read information from, or write information to, the storage medium. Alternatively, the storage medium may be integrated into the processor. The processor and storage medium may reside within the ASIC. The ASIC may exist in the user terminal. Alternatively, the processor and the storage medium may exist as separate components in the user terminal.

Although the embodiments described above have been described utilizing aspects of the presently disclosed subject matter in one or more standalone computer systems, the present disclosure is not limited thereto and may be implemented in connection with any computing environment, such as a network or distributed computing environment. . Still further, aspects of the subject matter in this disclosure may be implemented in a plurality of processing chips or devices, and storage may be similarly affected across the plurality of devices. Such devices may include PCs, network servers, and portable devices.

Although the present disclosure has been described in connection with some embodiments herein, various modifications and changes can be made without departing from the scope of the present disclosure that can be understood by those skilled in the art to which the present disclosure pertains. Further, such modifications and variations are intended to fall within the scope of the claims appended hereto.

110: information processing system
120: unattended storage
130: network
210: memory
220: processor
230: communication module
240: input/output interface

Claims (15)

A method for detecting unauthorized storage items, performed by at least one processor, the method comprising:
acquiring an image of an article stored in an unattended storage;
Inputting the obtained image to a first machine learning model, obtaining a storage risk index for an article from the first machine learning model - The first machine learning model includes a learning image and a reference storage risk index It is a model trained using training data - ;
determining whether to allow storage of the article based on the obtained storage risk index;
after storage permission is determined and the article is stored, acquiring an image of the article in a first period and acquiring an image of the article in a second period;
identifying a first region in which the article appears from the image acquired in the first period, and identifying a second region in which the article appears from the image acquired in the second period;
calculating a variation index based on the identified matching rates of the first and second regions;
determining that the article has changed when the change index exceeds a threshold, and providing a notification message to at least one of an administrator or a user;
receiving measurement data from the gas sensor;
inputting the received measurement data into a second machine learning model to obtain a deterioration index of the article from the second machine learning model, wherein the second machine learning model is sample measurement data including a plurality of types of gas concentrations It is a model trained using training data including - and reference change index;
controlling the door of the unattended storage to be opened when the obtained deterioration index exceeds a threshold value;
determining whether a fire has occurred in the unattended storage based on at least one of smoke or flame included in an image obtained from a camera, and when it is determined that a fire has occurred, identifying a location where the fire occurred;
adjusting the angle of the injection port so that the injection port of the fire extinguishing device included in the unmanned storage faces the identified position; and
Controlling the fire extinguishing device to spray the fire extinguishing liquid to the identified location
including,
The fire extinguishing device sprays the fire extinguishing liquid while rotating the injection port when a communication disconnection is detected while injecting the fire extinguishing liquid at an angle corresponding to the identified position;
The image for the article is an image taken using a fisheye camera,
The training image is a pre-processed image that is distorted based on the shooting method of the fisheye camera,
How to detect unauthorized storage items.
delete According to claim 1,
The step of providing the notification message includes:
providing the notification message including the image acquired in the first period and the image acquired in the second period to at least one of the administrator and the user
A method for detecting unauthorized storage items, comprising:
According to claim 1,
The first period and the second period are,
Included in the period in which the locked state of the unattended storage is maintained,
How to detect unauthorized storage items.
delete delete delete delete delete delete delete According to claim 1,
The fire extinguishing device is
installed on the inner surface of the ceiling of the unmanned storage,
How to detect unauthorized storage items.
delete A computer program stored in a computer-readable recording medium for executing the method according to any one of claims 1, 3, 4 and 12 on a computer.
delete

Images