KR102234555B1 - Method, apparatus and system for managing crime prevention and security for unmanned store operation - Google Patents

Abstract

Disclosed is a system for managing crime prevention and security for unmanned store operation. According to an embodiment of the present invention, the system comprises: a 360-degree camera installed around a main facility of an unmanned store in an overlapping manner, and photographing the unmanned store in real time; an integrated server for identifying and tracking a user, and detecting, tracking, and responding to an event occurring in the unmanned store, on the basis of real-time photographing data obtained through the 360-degree camera; a database for storing the real-time photographing data obtained from a server, and storing user identification information on the user of the unmanned store; an operator portable terminal for receiving an event occurrence signal when the event occurs, from the server; and security control devices including a door opening/closing facility, a disaster prevention facility and a theft bell controlled by the server on the basis of the event occurrence signal when the event occurs. The integrated server transceives signals with the 360-degree camera, the database, the operator portable terminal, and the security control devices, and includes artificial neural networks for identifying and tracking the user, or performing calculation for detecting, tracking, and responding to the event occurring in the unmanned store. The 360-degree camera is photographing equipment capable of obtaining an image at all 360-degree positions of three-axial directions without movement or rotation from one position, and is installed in an overlapping manner not to have a blind spot excluding an exceptional space in which the image photographed by each camera corresponds to a vector predetermined in the unmanned store, and the event includes theft, damage, fire, and flood. Accordingly, the preference of a user is analyzed based on artificial intelligence, such that a recommendation order can be generated therefrom.

Description

Security and security management method, device and system for unmanned store operation {METHOD, APPARATUS AND SYSTEM FOR MANAGING CRIME PREVENTION AND SECURITY FOR UNMANNED STORE OPERATION}

The following embodiments relate to a technology for managing crime prevention and security for the operation of an unmanned store.

Crime prevention and security are very important factors in the operation of unmanned stores. The current crime prevention and security system may include observation through CCTV during the open time, analysis of movements after the deadline, and reporting through it. However, it is difficult to receive orders optimized for users, which may lead to inefficiency, and detailed response to theft and damage is difficult. In addition, in the event of an event accompanied by fire or flood, there is no separate manual corresponding to it, and the means for storing information is also insufficient. Therefore, it is required to develop a technology that can respond to each by providing an optimized use environment for users, providing a means to cope with theft and damage, and classifying fire and flood damage.

Korean Patent Publication No. KR10-2132086 Korean Patent Application Publication KR10-2020-0056498 Republic of Korea Patent Publication KR10-1935470 Republic of Korea Patent Publication KR10-2040574

The embodiments attempt to analyze a user's preferences based on artificial intelligence and generate a recommendation order from this.

The embodiments seek to analyze whether or not theft or damage is stolen by analyzing photographing data of an identification tag based on artificial intelligence, and respond to this.

The actual examples are to classify and classify fires and floods based on artificial intelligence, and respond to fires and floods based on classification.

A security and security management system for operating an unmanned store according to an embodiment, comprising: a 360-degree camera installed to be superimposed around major facilities of an unmanned store and photographing the unmanned store in real time; An integrated server that identifies and tracks a user based on real-time photographing data acquired through the 360-degree camera, and detects, tracks, and responds to events occurring in the unmanned store; A database for storing the real-time photographing data obtained from the server and storing user identification information for the user of the unmanned store; An operator portable terminal receiving an event occurrence signal when the event occurs from the server; And security control devices including a door opening/closing facility, a disaster prevention facility, and a burglar bell controlled by the server, based on the event occurrence signal when the event occurs, and the integrated server includes the 360-degree camera, the database, and the An operator portable terminal and artificial neural networks that transmit and receive signals with the security control devices, identify and track the user, or perform an operation to detect, track, and respond to the event occurring in the unmanned store, The 360-degree camera is a photographing equipment capable of acquiring images of all 360-degree positions in the three-axis direction without moving or rotating from one location, and the image captured by each camera corresponds to a predefined vector in the unmanned store. Except for the exception space, it is overlapped and installed so that there is no blind spot, and the event may include theft, damage, fire, and flood damage.

According to an embodiment, when the user enters the unmanned store, the integrated server acquires photographing data including the user's face image by controlling the 360-degree camera, and preprocesses the photographing data to input the first And, by inputting the first input into a first artificial neural network, obtaining a first output, and obtaining the user identification information from the database based on the first output, and the user identification information in the database If is not present, new user identification information is generated and stored in the database, based on the user identification information, a pre-stored recommended order is obtained, and an order signal is generated based on the user's response to the recommended order. Generate, based on the order signal, generate a second input, input the second input to a second artificial neural network to obtain a second output, and based on the second output, the content of the recommended order And, the order signal and the modified recommendation order may be stored in a database.

According to an embodiment, when the user moves out of the angle of view of the 360-degree camera together with an item on which an identification tag is mounted from the unmanned store, the integrated server transfers the photographing data taken through the 360-degree camera during the event. Classify as the theft photographing data corresponding to the theft, store theft photographing data including images up to 24 hours before the time of the classification as theft in the database, pre-process the theft photographing data to generate a third input, and , By inputting the third input into a third artificial neural network, obtaining a third output, and generating edited theft photographing data based on the third output, and the edited theft photographing data and identification information of the user As a device that transmits the event occurrence signal including a to the operator's portable terminal, and the identification tag continuously emits a special wavelength identifiable through the infrared vision of the 360-degree camera, when the special wavelength is damaged or removed, It may be considered to correspond to the hand among the events through the third artificial neural network.

According to an embodiment, the event occurrence signal is classified into a total of four grades: A grade, B grade, C grade, and D grade, and the A grade, the B grade and the C grade are the third It is determined by an artificial neural network, a fourth artificial neural network, and a fifth artificial neural network, and the D grade is determined through an automatically generated signal from the operator portable terminal according to a communication failure of the integrated server, and the A grade is the theft and the It corresponds to a case classified as damage, and the B class corresponds to a case where the fire and the flood disaster occurred, but it is determined that it can be extinguished through the disaster prevention facility among the security control devices, and the C class is the fire and This corresponds to the case where the flood has occurred and it is determined that it is impossible to extinguish through the disaster prevention facility among the security control devices, and the D grade is a signal received by the operator's portable terminal due to a failure and communication failure of the integrated server. This may be the case when there is no response.

According to an embodiment, when the event occurrence signal is Class A, the integrated server transmits a question to confirm the fact of the damage and the theft to the device used by the user, and the user's response to this When absent or inadequate, the door is closed by controlling the door opening/closing facility among the security control devices, and the voice signal is output by controlling the theft bell. Among the security control devices, the disaster prevention facility is controlled to perform an appropriate disaster prevention operation, the door opening and closing facility is controlled to open the door, the theft bell is controlled to output a voice signal, and the event occurrence signal is C grade. In this case, the integrated server controls the disaster prevention facility among the security control devices to perform the maximum disaster prevention operation, controls the door opening/closing facility to open the door, controls the burglar bell to output a voice signal, and the When the event occurrence signal corresponds to the class D, the operator portable terminal automatically generates the event occurrence signal, and the operator portable terminal transmits an automatic operation execution signal, and the door opening/closing facility among the security control devices The door opening operation and the voice signal output of the burglar bell may be performed, and the integrated server may automatically transmit the event occurrence signal to the operator's portable terminal in the A, B, and C grades of the event generation signal.

The apparatus according to an embodiment may be controlled by a computer program stored in a medium in order to execute the method of any one of the above-described methods in combination with hardware.

Embodiments may analyze a user's preferences based on artificial intelligence, and generate a recommendation order from this.

The embodiments may analyze the theft or damage by analyzing the photographing data of the identification tag based on artificial intelligence, and respond to this.

Real-life examples can be classified by subdividing fire and flood damage based on artificial intelligence, and responding to fire and flood damage based on the classification.

1 is a view for explaining a security and security management system for unmanned store operation according to an embodiment.
2 is a diagram illustrating a process of identifying a user and generating a recommended order according to an exemplary embodiment.
3 is a view for explaining a process of determining theft and damage using an identification tag according to an embodiment.
4 is a diagram for describing a process of classifying an event occurrence signal according to an exemplary embodiment.
5 is a view for explaining the operation of security control equipment according to the level of the event occurrence signal according to an embodiment.
6 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, since various changes may be made to the embodiments, the scope of the patent application is not limited or limited by these embodiments. It is to be understood that all changes, equivalents, or substitutes to the embodiments are included in the scope of the rights.

Specific structural or functional descriptions of the embodiments are disclosed for the purpose of illustration only, and may be changed and implemented in various forms. Accordingly, the embodiments are not limited to a specific disclosure form, and the scope of the present specification includes changes, equivalents, or substitutes included in the technical idea.

Although terms such as first or second may be used to describe various components, these terms should be interpreted only for the purpose of distinguishing one component from other components. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component.

When a component is referred to as being "connected" to another component, it is to be understood that it may be directly connected or connected to the other component, but other components may exist in the middle.

The terms used in the examples are used for illustrative purposes only and should not be construed as limiting. Singular expressions include plural expressions unless the context clearly indicates otherwise. In the present specification, terms such as "comprise" or "have" are intended to designate the presence of features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, but one or more other features. It is to be understood that the presence or addition of elements or numbers, steps, actions, components, parts, or combinations thereof does not preclude in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiment belongs. Terms as defined in a commonly used dictionary should be interpreted as having a meaning consistent with the meaning in the context of the related technology, and should not be interpreted as an ideal or excessively formal meaning unless explicitly defined in the present application. Does not.

In addition, in the description with reference to the accompanying drawings, the same reference numerals are assigned to the same components regardless of the reference numerals, and redundant descriptions thereof will be omitted. In describing the embodiments, when it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the embodiments, the detailed description thereof will be omitted.

The embodiments may be implemented in various types of products such as a personal computer, a laptop computer, a tablet computer, a smart phone, a television, a smart home appliance, an intelligent vehicle, a kiosk, and a wearable device.

1 is a view for explaining a security and security management system for unmanned store operation according to an embodiment.

The security and security management system for unmanned store operation according to an embodiment includes a 360-degree camera 101 that is superimposed around the main facilities of the unmanned store and photographs the unmanned store in real time; An integrated server 100 that identifies and tracks a user based on real-time photographing data acquired through the 360-degree camera 101 and detects, tracks, and responds to events occurring in unmanned stores; A database 102 that stores real-time photographing data acquired from the server and stores user identification information for users of unmanned stores; An operator portable terminal 103 that receives an event occurrence signal when an event occurs from the server; And security control devices 104 including door opening/closing facilities, disaster prevention facilities, and burglar bells controlled by the server based on the event generation signal when an event occurs.

The security and security management system for unmanned store operation according to an embodiment may include a server and a terminal. The server and the terminal are connected to the network and can exchange data through wired or wireless communication. The server for the security and security management system for unmanned store operation may be the integrated server 100. The server may collect photographing data photographed through the 360-degree camera 101 and analyze it using artificial intelligence therefrom. The server may analyze the photographed data captured from the 360-degree camera 101 through artificial intelligence, and control the security control devices 104 from the result of querying the database 102.

The server may be its own server owned by a person or organization that provides services using the server; May be a cloud server; It may be a peer-to-peer (p2p) set of distributed nodes. The server has a computing function of a typical computer; Save/refer to function; Input/output function; And it may be configured to perform all or part of the control function. The server may include at least one artificial neural network that performs an inference function. The server may be configured to communicate with the operator's portable terminal 103 by wired or wirelessly.

The server communicates with the operator's portable terminal 103 and performs operations to generate control signals of the security control devices 104 to respond to events inferred from the analysis of the photographing data of the 360-degree camera 101. can do. Therefore, the service provided through the server may be suitable for individuals and organizations who want to automatically perform crime prevention and security while operating an unmanned store.

The terminal according to an embodiment may be an operator portable terminal 103. The terminal may be a desktop computer, a laptop computer, a tablet, a smartphone, or the like. The terminal includes an operation function of a general computer; Save/refer to function; Input/output function; And it may be configured to perform all or part of the control function. The terminal may be configured to communicate with the server by wired or wirelessly.

The terminal may be installed with an application linked to the server or connected to a web page linked to the server. The terminal may receive various signals transmitted by the server through calculations through a web page or an application, and transmit a remote signal for this to manually manipulate operations that can be controlled by the server. The user of the operator's portable terminal 103 may be a company or an individual who wants to operate an unmanned store, but there is no particular restriction on the job. As long as the processing capacity of the server allows, the number of terminals is not particularly limited.

Hereinafter, embodiments will be described centering on the operation of the server, but the embodiments are not limited by the subject or aspect of communication, and various application examples may be employed.

The 360-degree camera 101 according to an embodiment is a photographing equipment capable of acquiring images of all 360-degree positions in the three-axis direction without moving or rotating from one location. It can be installed so that there is no blind spot except for the exception space corresponding to the vector specified in advance. The integrated server 100 may form a panoramic vision of the entire unmanned store through an overlapped range in the image captured from the 360-degree camera 101, and track the location of the event from this. Exception spaces may include spaces infringing on privacy, including toilets, and may include all spaces designated by the operator.

The integrated server 100 according to an embodiment transmits and receives signals with the 360-degree camera 101, the database 102, the operator's portable terminal 103, and the security control devices 104, and identifies and tracks a user, or It may include artificial neural networks that perform operations to detect, track, and respond to events occurring in unmanned stores. The operation of the integrated server 100 according to an example is to track and identify a user of an unmanned store, retrieve identification information on the identified user, obtain a recommended order based on the user's existing order history, and recommend a suggested recommendation. When a user places an order in response to an order, learning for a future recommended order may be performed from an order signal for this. In addition, when an event in an unmanned store occurs, the integrated server 100 may use an artificial neural network to respond to the event, and may perform a response to the event through analysis of the artificial neural network. A detailed description of the user's identification and order process and response to events will be described later with reference to FIGS. 2 to 5.

The database 102 according to an embodiment may store all user identification information for a user, an order signal including an order history of a corresponding user, and recommended orders. In addition, the database 102 may store the user's usage status, including the grade of the corresponding user, as a result of analysis by the artificial neural network of the integrated server 100. The database 102 may include all information on an event, and may store a response manual for the event. In addition, when an event occurs, the database 102 automatically transmits all the information in the database 102 to the cloud shared with the operator's portable terminal 103 to help track the cause of the event problem in the future. I can. In general, events may include theft, damage, fire, and flood damage, but are not limited thereto, and the types and numbers may increase according to the operator's request.

The security control devices 104 according to an embodiment may include a door opening and closing facility, a disaster prevention facility, and a burglar bell. The door opening/closing facility is a facility for controlling the main door closer to the outside of the two types of doors installed to enter the unmanned store.When an event occurs, the main door is opened or closed to help appropriate response to the event situation. Can give. Disaster prevention facilities are facilities that respond to events that may cause human damage such as fire or flood, and may include sprinklers, fire hydrants, fire extinguishers, fire extinguishing gas, emergency exit guides, etc. in case of fire. A device for discharging internal water to the outside based on a device including a pump may be included, but is not limited thereto. Theft bell is a device capable of outputting a high-decibel voice signal according to the occurrence of an event, and can be activated when most events occur. All security control devices 104, including the theft bell, can be manually controlled by the operator through the operator's portable terminal 103.

According to an embodiment, the operator may directly transmit a signal for controlling the security adjustment devices 104 through the operator's portable terminal 103 to the integrated server 100 or the security adjustment devices 104. Even when the event occurrence signal is not received from the integrated server 100, the operator can check the photographing data taken through the 360-degree camera 101 through the operator portable terminal 103 in real time, and can directly check the operator portable terminal ( 103) can be selected in advance to generate an event occurrence signal. The event generation signal generated through this may have the same operation control effect as the event generation signal generated through the integrated server 100, and all may be stored in the database 102. The operator can directly control the security control devices 104 through the operator's portable terminal 103, even if it is not through the integrated server 100, which is not the case that corresponds to the class D of the event occurrence signal, respectively. This may be possible by manually inputting a signal for the control operation to the security control devices 104 of the. Even if the operator does not correspond to the class D of the event occurrence signal in the operator's portable terminal 103, the operator can store and use the settings for the security control devices 104 that have been set in advance.

2 is a diagram illustrating a process of identifying a user and generating a recommended order according to an exemplary embodiment.

According to an embodiment, when a user enters an unmanned store, the integrated server may control a 360-degree camera to obtain photographing data 202 including the user's face image 201. The integrated server may extract a feature point inside the face image 201 by specifying the face image 201 from the acquired photographing data 202, and perform pre-processing including the relationship and coordinates of the corresponding feature points. The integrated server may generate the first input including information processed through pre-processing. The integrated server may obtain a first output by inputting the first input to the first artificial neural network 203. The first input may correspond to an input layer node of the first artificial neural network 203, and the first output may correspond to an output layer node of the first artificial neural network 203. The first artificial neural network 203 may be a convolutional neural network. The first artificial neural network 203 is composed of a feature extraction neural network and a classification neural network, and the feature extraction neural network processes an input signal by sequentially stacking a convolutional layer and a pooling layer. The convolution layer includes convolution operations, convolution filters, and activation functions. The calculation of the convolution filter is adjusted according to the size of the matrix of the target input, but generally a 9X9 matrix is used. The activation function generally uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The pooling layer is a layer that reduces the size of an input matrix, and uses a method of extracting a representative value by grouping pixels in a specific area. In general, an average value or a maximum value is often used for the operation of the pooling layer, but is not limited thereto. This operation is performed using a square matrix, usually a 9X9 matrix. The convolutional layer and the pooling layer are alternately iterated until the corresponding input is small enough while maintaining the difference. The first artificial neural network 203 may have a hidden layer and an output layer. In general, the first artificial neural network 203 has three or more hidden layers, and the number of nodes of each hidden layer is designated as 100, but may be set to more or less depending on the case. The activation function of the hidden layer uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The number of output layer nodes of the first artificial neural network 203 may be a total of 14, but is not limited thereto. The output layer activation function of the first artificial neural network 203 may use a softmax function, but is not limited thereto. The softmax function is a representative function of one-hot encoding, which makes the sum of all output nodes total 1, the output of the output node with the largest value as 1, and the output of the remaining output nodes as 0. . The integrated server can generate 14 outputs as a binary code with a value of 0 or 1, respectively, and accordingly, can indicate a total of 16,384 values with 14 outputs. The bar indicated by each code may be used as the identification information 205 of the user of the unmanned store.

The learning device may learn the first artificial neural network 203 through the first labels. The learning device may be the same as the integrated server, but is not limited thereto. The first labels may include photographing data 202 including 100,000 face images 201 and identification information 205 indicating that the person is the same person or different person from the corresponding photographing data 202. The first artificial neural network 203 may be formed by calculating a loss function by comparing first labeled training outputs obtained by inputting first labeled training inputs with first labels. As the loss function, a known mean squared error (MSE), cross entropy error (CEE), or the like may be used. However, the present invention is not limited thereto, and loss functions used in various artificial neural network models may be used as long as the deviation, error, or difference between the output of the first artificial neural network 203 and the label can be measured.

Next, the learning device may optimize the first artificial neural network 203 based on the comparison value. By updating the weight of the nodes of the artificial neural network so that the comparison value of the learning device becomes smaller, the output of the artificial neural network corresponding to inference and the label corresponding to the correct answer can be gradually matched. It can be optimized to output an inference close to the correct answer. Specifically, the learning apparatus may optimize the artificial neural network by repeating the process of resetting the weight of the artificial neural network so that the loss function corresponding to the comparison value approaches the estimated value of the minimum value. For the optimization of artificial neural networks, known backpropagation algorithms, stochastic gradient descent, and the like can be used. However, the present invention is not limited thereto, and a weight optimization algorithm used in various neural network models may be used.

According to an embodiment, the integrated server may obtain the user identification information 205 from the database 204 based on the first output. If the user identification information 205 does not exist, the integrated server may generate new user identification information 205 and store it in the database 204. The integrated server may obtain various types of information about a user by inquiring the database 204 from the user identification information 205. For example, the integrated server may query the database 204 to inquire the class classified according to the user's usage condition from the user identification information 205. Classified grades may include A, B, C, D, or F, and each may be a value randomly designated to facilitate the use of the corresponding unmanned store or manage the corresponding unmanned store. For example, A is a user who has cleanly managed the use of an unmanned store, B is a user who has used the unmanned store without an event, but shows an unclean management state, and C is a user who has generated an event corresponding to damage less than two times. , D may be a user who has generated an event corresponding to damage three or more times, and F may be a user who has generated an event excluding damage one or more times. The integrated server may have an additional artificial neural network for classifying the user's usage status. In general, the tenth artificial neural network may correspond to this, but is not limited thereto.

According to an embodiment, the integrated server may obtain a recommendation order 206 from the user identification information 205 by querying the database 204. The recommended order 206 is pre-calculated data through the second artificial neural network 208 of the integrated server, and may be performed when the user ends using the unmanned store on the same day. The integrated server provides the obtained recommended order 206 to the user, so that the user can save the trouble of recording the order one by one. As a response to the recommended order 206, the user may perform an initial response including yes and no, and a process of modifying a detailed order when selecting no. The revised order may be generated as an order signal 207 via the integration server. The integration server may generate a second input based on the order signal 207. The integrated server may input the second input to the second artificial neural network 208 to obtain a second output. The second input may correspond to an input layer node of the second artificial neural network 208, and the second output may correspond to an output layer node of the second artificial neural network 208. The second artificial neural network 208 may have a hidden layer and an output layer. In general, the second artificial neural network 208 has five or more hidden layers, and 50 nodes of each hidden layer are designated, but more or less may be determined depending on the case. The activation function of the hidden layer uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The number of output layer nodes of the second artificial neural network 208 may be a total of 10, but is not limited thereto. Each of the 10 output layer nodes may correspond to a type of selection options for an order, and the type and number of options for one selection option may be designated according to an operator's request.

The learning device may learn the second artificial neural network 208 through the second labels. The learning device may be the same as the integrated server, but is not limited thereto. The second labels may include 100,000 pieces of identification information 205, a combination of order details corresponding thereto, and recommended orders 206 extracted therefrom. The second artificial neural network 208 may be formed by comparing the second labeled training outputs obtained by inputting the second labeled training inputs with the second labels and calculating a loss function. As the loss function, a known mean squared error (MSE), cross entropy error (CEE), or the like may be used. However, the present invention is not limited thereto, and loss functions used in various artificial neural network models may be used as long as the deviation, error, or difference between the output of the second artificial neural network 208 and the label can be measured.

Next, the learning device may optimize the second artificial neural network 208 based on the comparison value. By updating the weight of the nodes of the artificial neural network so that the comparison value of the learning device becomes smaller, the output of the artificial neural network corresponding to inference and the label corresponding to the correct answer can be gradually matched. It can be optimized to output an inference close to the correct answer. Specifically, the learning apparatus may optimize the artificial neural network by repeating the process of resetting the weight of the artificial neural network so that the loss function corresponding to the comparison value approaches the estimated value of the minimum value. For the optimization of artificial neural networks, known backpropagation algorithms, stochastic gradient descent, and the like can be used. However, the present invention is not limited thereto, and a weight optimization algorithm used in various neural network models may be used.

According to an embodiment, the integrated server may modify the contents of the recommended order 206 based on the second output. The integrated server may store the result of modifying the existing recommendation order 206 and the corresponding order signal 207 in the database 204. The stored data may be used for learning of an artificial neural network in the future or for managing user identification information 205.

The integrated server according to an embodiment may analyze a user's preferences based on artificial intelligence and generate a recommendation order from this.

3 is a view for explaining a process of determining theft and damage using an identification tag according to an embodiment.

The integrated server according to an embodiment, when the user moves out of the angle of view 302 of the 360 degree camera together with the item on which the identification tag 301 is mounted from an unmanned store, the photographed data captured through the 360 degree camera is prevented from being stolen during an event. It can be classified as the corresponding theft photographing data 303. The identification tag 301 may be a device that continuously emits a special wavelength that can be identified through infrared vision of a 360 degree camera. A detailed determination method for the size, shape, shape and application method of the identification tag 301 may be determined through selection of an operator. The identification tag 301 may also include a method of applying a liquid type material to the surface, so that the entire range of the corresponding article may be included. Accordingly, when the special wavelength of the identification tag 301 is damaged or removed from the photographing data, the integrated server may classify it as damaged photographing data corresponding to the damage 307 during the event. The theft photographing data 303 and the damaged photographing data may include all photographing data including an image 24 hours before the time when the theft and damage 307 is classified as occurring. Both the theft photographing data 303 and the damaged photographing data may be automatically stored in the database.

The integrated server may generate a third input by pre-processing the stolen photographing data 303 and the damaged photographing data. The integrated server may input the third input to the third artificial neural network 304 to obtain a third output. The third input may correspond to an input layer node of the third artificial neural network 304, and the third output may correspond to an output layer node of the third artificial neural network 304. The third artificial neural network 304 may be a convolutional neural network. The third artificial neural network 304 is composed of a feature extraction neural network and a classification neural network, and the feature extraction neural network processes an input signal by sequentially stacking a convolutional layer and a pooling layer. The convolution layer includes convolution operations, convolution filters, and activation functions. The calculation of the convolution filter is adjusted according to the size of the matrix of the target input, but generally a 9X9 matrix is used. The activation function generally uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The pooling layer is a layer that reduces the size of an input matrix, and uses a method of extracting a representative value by grouping pixels in a specific area. In general, an average value or a maximum value is often used for the operation of the pooling layer, but is not limited thereto. This operation is performed using a square matrix, usually a 9X9 matrix. The convolutional layer and the pooling layer are alternately iterated until the corresponding input is small enough while maintaining the difference. The third artificial neural network 304 may have a hidden layer and an output layer. In general, three or more hidden layers are present in the third artificial neural network 304, and the number of nodes of each hidden layer is designated as 100, but may be set to more or less depending on the case. The activation function of the hidden layer uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The number of output layer nodes of the third artificial neural network 304 may be 100, but is not limited thereto. The 100 outputs can include information on the editing point and editing method for the video, and the integrated server can use the third output generated therefrom to edit the theft photographing data 303 and the damaged photographing data. . Among the 100 outputs, the first node may include a value indicating whether the corresponding photographing data is stolen or damaged 307.

The learning device may learn the third artificial neural network 304 through third labels. The learning device may be the same as the integrated server, but is not limited thereto. The third labels may include photographing data including situations of theft and damage 307 and editing points and editing methods designated for editing only images deemed to be meaningful by experts therefrom. The third artificial neural network 304 may be formed by calculating a loss function by comparing third labeled training outputs obtained by inputting third labeled training inputs with third labels. As the loss function, a known mean squared error (MSE), cross entropy error (CEE), or the like may be used. However, the present invention is not limited thereto, and loss functions used in various artificial neural network models may be used as long as the deviation, error, or difference between the output of the third artificial neural network 304 and the label can be measured.

Next, the learning device may optimize the third artificial neural network 304 based on the comparison value. By updating the weight of the nodes of the artificial neural network so that the comparison value of the learning device becomes smaller, the output of the artificial neural network corresponding to inference and the label corresponding to the correct answer can be gradually matched. It can be optimized to output an inference close to the correct answer. Specifically, the learning apparatus may optimize the artificial neural network by repeating the process of resetting the weight of the artificial neural network so that the loss function corresponding to the comparison value approaches the estimated value of the minimum value. For the optimization of artificial neural networks, known backpropagation algorithms, stochastic gradient descent, and the like can be used. However, the present invention is not limited thereto, and a weight optimization algorithm used in various neural network models may be used.

According to an embodiment, the integrated server may edit the stolen photographing data 303 and the damaged photographing data based on the third output. The edited theft photographing data 305 and the edited damaged photographing data may be included in the event occurrence signal 306 by the integrated server. The integrated server may include the edited theft photographing data 305 and the edited damaged photographing data together with the user's identification information in the event occurrence signal 306, and transmit the event occurrence signal 306 to the operator's portable terminal. Can be transmitted.

According to an embodiment, the integrated server may analyze photographing data of an identification tag based on artificial intelligence to analyze whether it is stolen or damaged, and respond to it.

4 and 5 are diagrams for explaining a process of classifying a class of an event occurrence signal and an operation of security control devices according to the class according to an exemplary embodiment.

The event occurrence signal according to an embodiment can be classified into a total of four grades: A grade 401, B grade 402, C grade 403, and D grade 404, and A grade 401, B The grade 402 and the C grade 403 are determined by the third artificial neural network 410, the fourth artificial neural network 420 and the fifth artificial neural network 430 of the integrated server, and the D grade 404 is the integrated server. It can be determined through an automatically generated signal of the operator's portable terminal according to the communication failure.

The integrated server according to an embodiment may analyze photographing data on an event that has occurred through the third artificial neural network 410 and classify whether the event corresponds to damage or theft. Accordingly, photographing data that is not classified as damaged or stolen may be graded through the fourth artificial neural network 420 and the fifth artificial neural network 430. The description of the third artificial neural network 410 is the same as described above.

According to an embodiment, the integrated server may generate a fourth input by pre-processing photographing data that is not classified as damaged or stolen. The integrated server may input the fourth input to the fourth artificial neural network 420 to obtain a fourth output. The fourth input may correspond to the input layer node of the fourth artificial neural network 420, and the fourth output may correspond to the output layer node of the fourth artificial neural network 420. The fourth artificial neural network 420 may be a convolutional neural network. The fourth artificial neural network 420 is composed of a feature extraction neural network and a classification neural network, and the feature extraction neural network processes an input signal by sequentially stacking a convolutional layer and a pooling layer. The convolution layer includes convolution operations, convolution filters, and activation functions. The calculation of the convolution filter is adjusted according to the size of the matrix of the target input, but generally a 9X9 matrix is used. The activation function generally uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The pooling layer is a layer that reduces the size of an input matrix, and uses a method of extracting a representative value by grouping pixels in a specific area. In general, an average value or a maximum value is often used for the operation of the pooling layer, but is not limited thereto. This operation is performed using a square matrix, usually a 9X9 matrix. The convolutional layer and the pooling layer are alternately iterated until the corresponding input is small enough while maintaining the difference. The fourth artificial neural network 420 may have a hidden layer and an output layer. In general, the fourth artificial neural network 420 has three or more hidden layers, and the number of nodes of each hidden layer is designated as 100, but may be set to more or less depending on the case. The activation function of the hidden layer uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The number of output layer nodes of the fourth artificial neural network 420 may be 100, but is not limited thereto. The 100 outputs may include information on the editing point and editing method for the corresponding image, and the integrated server may use the fourth output generated therefrom to edit the corresponding photographing data. The first node among the 100 outputs may include a value indicating whether the corresponding photographing data corresponds to fire or flood.

The learning device may learn the fourth artificial neural network 420 through the fourth labels. The learning device may be the same as the integrated server, but is not limited thereto. The fourth labels may include event-occurring photographing data excluding situations of theft and damage, and editing points and editing methods designated for editing only images deemed to be meaningful by experts therefrom. The fourth artificial neural network 420 may be formed by calculating a loss function by comparing fourth labeled training outputs obtained by inputting fourth labeled training inputs with fourth labels. As the loss function, a known mean squared error (MSE), cross entropy error (CEE), or the like may be used. However, the present invention is not limited thereto, and loss functions used in various artificial neural network models may be used as long as the deviation, error, or difference between the output of the fourth artificial neural network 420 and the label can be measured.

Next, the learning device may optimize the fourth artificial neural network 420 based on the comparison value. By updating the weight of the nodes of the artificial neural network so that the comparison value of the learning device becomes smaller, the output of the artificial neural network corresponding to inference and the label corresponding to the correct answer can be gradually matched. It can be optimized to output an inference close to the correct answer. Specifically, the learning apparatus may optimize the artificial neural network by repeating the process of resetting the weight of the artificial neural network so that the loss function corresponding to the comparison value approaches the estimated value of the minimum value. For the optimization of artificial neural networks, known backpropagation algorithms, stochastic gradient descent, and the like can be used. However, the present invention is not limited thereto, and a weight optimization algorithm used in various neural network models may be used.

The integrated server can determine whether the corresponding shooting data corresponds to fire or flood, based on the fourth output, and classifies the shooting data as either fire shooting data or flood shooting data, and edited fire shooting by editing the data. Data and edited handwriting data can be obtained. All fire shooting data and flood shooting data may be automatically stored in a database through an integrated server, and the edited fire shooting data and the edited flood shooting data may be transmitted to the operator's portable terminal.

The integrated server may generate the fifth input by pre-processing the edited fire photographing data and the edited flood photographing data obtained from the fourth output. The integrated server may input the fifth input to the fifth artificial neural network 430 to obtain the fifth output. The fifth input may correspond to an input layer node of the fifth artificial neural network 430, and the fifth output may correspond to an output layer node of the fifth artificial neural network 430. The fifth artificial neural network 430 may be a convolutional neural network. The fifth artificial neural network 430 is composed of a feature extraction neural network and a classification neural network, and the feature extraction neural network processes an input signal by sequentially stacking a convolutional layer and a pooling layer. The convolution layer includes convolution operations, convolution filters, and activation functions. The calculation of the convolution filter is adjusted according to the size of the matrix of the target input, but generally a 9X9 matrix is used. The activation function generally uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The pooling layer is a layer that reduces the size of an input matrix, and uses a method of extracting a representative value by grouping pixels in a specific area. In general, an average value or a maximum value is often used for the operation of the pooling layer, but is not limited thereto. This operation is performed using a square matrix, usually a 9X9 matrix. The convolutional layer and the pooling layer are alternately iterated until the corresponding input is small enough while maintaining the difference. The fifth artificial neural network 430 may have a hidden layer and an output layer. In general, the fifth artificial neural network 430 has three or more hidden layers, and the number of nodes of each hidden layer is designated as 100, but may be set to more or less depending on the case. The activation function of the hidden layer uses a ReLU function, a sigmoid function, and a tanh function, but is not limited thereto. The number of output layer nodes of the fifth artificial neural network 430 may be a total of 4, but is not limited thereto. The output layer activation function of the fifth artificial neural network 430 may use a softmax function, but is not limited thereto. The softmax function is a representative function of one-hot encoding, which makes the sum of all output nodes total 1, the output of the output node with the largest value as 1, and the output of the remaining output nodes as 0. . The integrated server may obtain information for determining whether or not the corresponding fire and flood prevention is at a level that can be suppressed through the disaster prevention facility 503 from the four outputs.

The learning device may learn the fifth artificial neural network 430 through fifth labels. The learning device may be the same as the integrated server, but is not limited thereto. The fifth labels may include photographing data on fire and flood damage, and whether or not disaster prevention is possible through the fire and flood prevention facility 503 determined by an expert. The fifth artificial neural network 430 may be formed by calculating a loss function by comparing fifth labeled training outputs obtained by inputting fifth labeled training inputs with fifth labels. As the loss function, a known mean squared error (MSE), cross entropy error (CEE), or the like may be used. However, the present invention is not limited thereto, and loss functions used in various artificial neural network models may be used as long as the deviation, error, or difference between the output of the fifth artificial neural network 430 and the label can be measured.

Next, the learning device may optimize the fifth artificial neural network 430 based on the comparison value. By updating the weight of the nodes of the artificial neural network so that the comparison value of the learning device becomes smaller, the output of the artificial neural network corresponding to inference and the label corresponding to the correct answer can be gradually matched. It can be optimized to output an inference close to the correct answer. Specifically, the learning apparatus may optimize the artificial neural network by repeating the process of resetting the weight of the artificial neural network so that the loss function corresponding to the comparison value approaches the estimated value of the minimum value. For the optimization of artificial neural networks, known backpropagation algorithms, stochastic gradient descent, and the like can be used. However, the present invention is not limited thereto, and a weight optimization algorithm used in various neural network models may be used.

A grade 401 according to an embodiment corresponds to a case classified as theft and damage. Theft and damage may be classified through the third artificial neural network 410, and theft and damage are related to the removal and damage of the identification tag. This allows operators to install identification tags on items that are likely to be stolen or damaged.

According to an embodiment, when the event occurrence signal is Class A (401), the integrated server transmits a question to confirm the fact of damage or theft to the device being used by the user, and the user's response is absent or insufficient. In this case, among the security control devices, the door opening/closing facility 501 may be controlled to close the door, and the theft bell 502 may be controlled to output a voice signal. Users are obligated to faithfully respond to questions to confirm theft or damage, and if the response is inappropriate, a report can be automatically received.

Class B 402 according to an embodiment may correspond to a case where fire and flooding have occurred, but it is determined that it is possible to extinguish through the disaster prevention facility 503 among security control devices. Whether the fire or flood can be extinguished may be determined through the fifth artificial neural network 430.

According to an embodiment, when the event occurrence signal is a B class 402, the integrated server controls the disaster prevention facility 503 among the security control devices to perform an appropriate disaster prevention operation, and controls the door opening/closing facility 501. By opening the door and controlling the anti-theft bell 502, a voice signal can be output.

The C grade 403 according to an embodiment may correspond to a case where fire and flooding have occurred, and it is determined that it is impossible to extinguish through the disaster prevention facility 503 among security control devices. Whether the fire or flood can be extinguished may be determined through the fifth artificial neural network 430.

According to an embodiment, when the event occurrence signal is a C class 403, the integrated server controls the disaster prevention facility 503 among the security control devices to perform the maximum disaster prevention operation, and controls the door opening/closing facility 501. By opening the door and controlling the anti-theft bell 502, a voice signal can be output.

According to an embodiment, the operator may directly transmit the event occurrence signal to the integrated server through the operator's portable terminal. In general, this may be the case when the operator determines that an event such as theft or damage has occurred additionally through the photographing data, but is not limited thereto. The operator may arbitrarily transmit an event occurrence signal corresponding to the A grade 401, B grade 402 and C grade 403 among the event occurrence signals through the operator's portable terminal possessed by the operator to the integrated server. On the other hand, the integrated server can control the security control devices, even if it is transmitted through the operator's portable terminal, as if it went through an artificial neural network. In addition, all the event occurrence signals and photographing data are stored in the database, and can be used as data for learning the artificial neural network in the integrated server in the future.

D-grade 404 according to an embodiment may correspond to a case in which there is no signal response received by the operator's portable terminal due to a failure of the integrated server and communication failure. Determination of the D-grade 404 may be determined through the operator's portable terminal, for example, if there is no periodic signal exchange of the integrated server five or more times, it may be determined as the D-grade 404.

According to an embodiment, when the event occurrence signal corresponds to the D-class 404, the operator portable terminal automatically generates an event occurrence signal, and the operator portable terminal transmits an automatic operation execution signal, among the security control devices. The door opening operation of the door opening/closing facility 501 and the voice signal output of the burglar bell 502 may be performed. In addition, in other cases, the operator can control the security control devices at any time through his operator's portable terminal. The integrated server may automatically transmit the event occurrence signal to the operator's portable terminal in all cases of the A grade 401, B grade 402 and C grade 403 of the event occurrence signal.

According to an embodiment, the integrated server may classify and classify fires and floods based on artificial intelligence, and respond to fires and floods based on the classification.

6 is an exemplary diagram of a configuration of an apparatus according to an embodiment.

The device 601 according to one embodiment includes a processor 602 and a memory 603. The device 601 according to an embodiment may be the above-described server or terminal. The processor may include at least one of the devices described above with reference to FIGS. 1 to 5, or may perform at least one of the methods described above with reference to FIGS. 1 to 5. The memory 603 may store information related to the above-described method or a program in which the above-described method is implemented. The memory 603 may be a volatile memory or a nonvolatile memory.

The processor 602 can execute programs and control the device 601. The code of a program executed by the processor 602 may be stored in the memory 603. The device 601 is connected to an external device (for example, a personal computer or a network) through an input/output device (not shown), and can exchange data.

The embodiments described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices, methods, and components described in the embodiments are, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate (FPGA). array), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. -A hardware device specially configured to store and execute program instructions such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those produced by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware device described above may be configured to operate as one or more software modules to perform the operation of the embodiment, and vice versa.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be interpreted by a processing device or, to provide instructions or data to a processing device, of any type of machine, component, physical device, virtual equipment, computer storage medium or device. , Or may be permanently or temporarily embodyed in a transmitted signal wave. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

As described above, although the embodiments have been described by the limited drawings, a person of ordinary skill in the art can apply various technical modifications and variations based on the above. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and claims and equivalents fall within the scope of the following claims.

Claims (3)

In the crime prevention and security management system for unmanned store operation,
360-degree cameras installed to be superimposed around the main facilities of the unmanned store to photograph the unmanned store in real time;
An integrated server that identifies and tracks a user based on real-time photographing data acquired through the 360-degree camera, and detects, tracks, and responds to events occurring in the unmanned store;
A database for storing the real-time photographing data obtained from the server and storing user identification information for the user of the unmanned store;
An operator portable terminal receiving an event occurrence signal when the event occurs from the server; And
When the event occurs, based on the event occurrence signal, security control devices including door opening/closing facilities, disaster prevention facilities, and burglar bells controlled by the server
Including,
The integrated server
Transmitting and receiving signals with the 360-degree camera, the database, the operator's portable terminal, and the security control devices,
Including artificial neural networks that identify and track the user or perform an operation to detect, track, and respond to the event occurring in the unmanned store,
The 360 degree camera
It is a photographing equipment that can acquire images of all 360-degree positions in the three-axis direction without moving or rotating from one position.
The images captured by each camera are superimposed and installed so that there is no blind spot except for an exception space corresponding to a pre-designated vector in the unmanned store,
The above event is
Including theft, damage, fire and flood,
The integrated server
When the user enters the unmanned store, by controlling the 360-degree camera to obtain photographing data including the user's face image,
Preprocessing the photographing data to generate a first input,
Inputting the first input to a first artificial neural network to obtain a first output,
Based on the first output, obtain the user identification information from the database,
If the user identification information does not exist in the database, new user identification information is created and stored in the database,
Obtaining a pre-stored recommended order based on the user identification information,
Based on the user's response to the recommended order, an order signal is generated,
Based on the order signal, generate a second input,
Inputting the second input to a second artificial neural network to obtain a second output,
Based on the second output, correct the content of the recommended order,
To store the order signal and the modified recommendation order in a database
Security and security management system for unmanned store operation.
delete In the crime prevention and security management system for unmanned store operation,
360-degree cameras installed to be superimposed around the main facilities of the unmanned store to photograph the unmanned store in real time;
An integrated server that identifies and tracks a user based on real-time photographing data acquired through the 360-degree camera, and detects, tracks, and responds to events occurring in the unmanned store;
A database for storing the real-time photographing data obtained from the server and storing user identification information for the user of the unmanned store;
An operator portable terminal receiving an event occurrence signal when the event occurs from the server; And
When the event occurs, based on the event occurrence signal, security control devices including door opening/closing facilities, disaster prevention facilities, and burglar bells controlled by the server
Including,
The integrated server
Transmitting and receiving signals with the 360-degree camera, the database, the operator's portable terminal, and the security control devices,
Including artificial neural networks that identify and track the user or perform an operation to detect, track, and respond to the event occurring in the unmanned store,
The 360 degree camera
It is a photographing equipment that can acquire images of all 360-degree positions in the three-axis direction without moving or rotating from one position.
The images captured by each camera are superimposed and installed so that there is no blind spot except for an exception space corresponding to a pre-designated vector in the unmanned store,
The above event is
Including theft, damage, fire and flood,
The integrated server
When the user moves out of the angle of view of the 360-degree camera together with an item equipped with an identification tag from the unmanned store,
Classifying the photographing data photographed through the 360-degree camera as theft photographing data corresponding to the theft among the events,
Store theft photographing data including images up to 24 hours before the time classified as theft in the database,
Preprocessing the theft photographing data to generate a third input,
Inputting the third input to a third artificial neural network to obtain a third output,
Based on the third output, edited theft photographing data is generated,
Transmitting the event generation signal including the edited theft photographing data and the user's identification information to the operator's portable terminal,
The identification tag is
As a device that continuously emits a special wavelength that can be identified through the infrared vision of the 360-degree camera,
When the special wavelength is damaged or removed, the third artificial neural network is considered to correspond to the damage among the events.
Security and security management system for unmanned store operation.

Images