KR20230022595A - Efficient method for transmitting and receiving imformation between apparatuses in unmanned shop - Google Patents

Abstract

According to one aspect of the present invention, a method for transmitting and receiving information by a device in an unmanned store is disclosed. The method comprises the steps of: providing, in response to sensing first sensing information related to customer entering and leaving to a store from a first sensor, the first sensing information to a first camera, wherein the first sensor is a sensor which detects the customer entering and leaving; photographing, in response to the first camera receiving the first sensing information, the customer entering and leaving; transmitting an image captured by the first camera to a server; providing, in response to a second sensor sensing a customer into a product warehouse, second sensing information to the second camera; photographing, in response to the second camera receiving the second sensing information, the customer located around the product warehouse; transmitting n image captured by the second camera to the server; providing, in response to initiation of payment at a payment terminal, payment initiation information to a third camera; photographing, in response to the third camera receiving the payment initiation information, a customer performing payment for a product; and transmitting an image captured by the third camera to the server. Accordingly, security of the unmanned store is strengthened, and store management becomes more efficient.

Description

Efficient information transmission/reception method of device in unmanned store, and unmanned store system for efficient information transmission/reception

The present invention relates to a method for transmitting and receiving information, and more particularly, to a method for efficiently transmitting and receiving information between equipment in an unmanned store.

In the operation of an unmanned store, crime prevention and security are very important factors. The current crime prevention and security system may include observation through CCTV during open hours, analysis of movements after closing hours, and a reporting process through them. However, it is difficult to receive orders optimized for users, which can lead to inefficiency, and it is difficult to respond quickly and accurately to theft and damage. In particular, when theft or damage occurs while the unattended state lasts for a long time, the manager must carefully examine the behavior of all customers who have entered and exited during that time while replaying the entire time section of multiple CCTVs one by one. It is very inefficient in detecting the culprit, and in practice, it is almost impossible to find the cause of theft or damage.

Moreover, in the case of unmanned stores, even though disasters caused by smoking and drinking occur very frequently, there is no separate manual that can respond to them, and a means for storing related information is insufficient.

Therefore, development of technology that provides store managers with an optimized use environment for store management as well as method and security aspects, provides means to respond to theft and damage, and responds immediately to accident events caused by smoking and drinking. this is required

An object according to an aspect of the present invention for solving the above-described problems is to analyze sensors and a plurality of camera images provided in an unmanned store in a server, between in-bound equipment in an unmanned store, or inbound It is to provide a method for transmitting and receiving information of a device in an unmanned store that enables efficient communication between equipment and external equipment.

According to an aspect of the present invention for achieving the above object, a method for transmitting and receiving information of a device in an unmanned store is provided by a first camera in response to sensing first sensing information related to a customer entering and exiting a store from a first sensor. Providing first sensing information (the first sensor is a sensor for detecting entry/exit of a customer), photographing a customer entering or exiting in response to receiving the first sensing information by the first camera, Transmitting the image taken by the camera to the server, providing second sensing information to the second camera in response to the second sensor sensing the customer going to the product storage, the second camera receiving the second sensing information Corresponding to, photographing a customer located near the product storage, transmitting an image captured by the second camera to the server, providing payment start information to a third camera in response to payment being initiated at the payment terminal The third camera may include a step of photographing a customer performing product payment in response to receiving payment start information, and a step of transmitting an image captured by the third camera to the server.

At least one of the first sensor and the second sensor may include at least one of a door open/close detection sensor, an approach sensor, a motion recognition sensor, and an infrared sensor.

The product storage may include at least one of a refrigerator for refrigerating products, a freezer for storing products in a frozen state, and a product display stand for displaying products in the form of a shelf.

The first camera, the second camera, and the third camera may start photographing by being triggered by receiving signals from the first sensor, the second sensor, and the payment terminal.

The first camera, the second camera, and the third camera are triggered when signals from the first sensor, the second sensor, and the payment terminal are received, and the reception time point is a reference point index in the captured video. (index).

The first sensor, the second sensor, the payment terminal, and the first camera, the second camera, and the third camera have IP addresses allocated from an access point (AP) in the store, and , The first sensor transmits the first sensing information to the first camera via the IP of the first camera having a mapping relationship therewith, and the second sensor transmits the IP of the second camera having a mapping relationship therewith. The second sensing information is transmitted to the second camera through the medium, and the payment terminal can transmit the payment initiation information to the third camera through the IP of the third camera having a mapping relationship therewith.

A plurality of cameras may have a mapping relationship with one of the first sensor, the second sensor, and the payment terminal.

The first sensor, the second sensor, and the payment terminal may transmit the first sensing information, the second sensing information, and the payment start information to the server via the access point.

The first sensor, the second sensor, and the payment terminal, and the first camera, the second camera, and the third camera are Wi-Fi, Zigbee, and Bluetooth At least one of them may be used to communicate with each other.

The first sensor, the second sensor, and the payment terminal, and the first camera, the second camera, and the third camera are NarrowBand IoT (NB-IoT), LoRa, and oneM2M .

In response to the smoke detection sensor detecting smoke, the smoke detection signal may be transmitted to a fourth camera having a mapping relationship with the smoke detection sensor, and the fourth camera may activate photographing in response to receiving the smoke detection signal. there is.

The plurality of smoke detection sensors are arranged to sense the unmanned store by zone, and the plurality of smoke detection sensors individually operate with the IP address of the corresponding fourth camera. Shooting corresponding to the above fourth camera may be activated.

The payment terminal may include at least one of a kiosk and a POS terminal.

The mapping relationship between the first sensor and the first camera, the mapping relationship between the second sensor and the second camera, and the mapping relationship between the third sensor and the third camera are set by the server or manager terminal It can be.

According to another aspect of the present invention for achieving the above object, an unmanned store system for transmitting and receiving information sends first sensing information to a first camera in response to sensing first sensing information related to a customer entering and exiting a store. A first sensor (the first sensor is a sensor that detects the customer's entry/exit), a first camera that photographs and transmits a photograph of a customer entering and exiting to a server in response to receiving the first sensing information, and a A second sensor for providing second sensing information to a second camera in response to sensing a customer, a second camera for photographing a customer located near a product storage and transmitting the photograph to the server in response to receiving the second sensing information, It may include a payment terminal for providing payment initiation information to a third camera in response to initiation of payment, and a third camera for photographing a customer making a product payment and transmitting the photograph to the server in response to receiving the payment initiation information. there is.

According to another aspect of the present invention for achieving the above object, an unmanned store system for transmitting and receiving information provides the first sensing information to the server in response to sensing the first sensing information related to the customer entering and exiting the store A first sensor (the first sensor is a sensor that detects the entry and exit of a customer), a first camera for photographing and transmitting a customer entering and exiting to the server in response to receiving the first sensing information from the server, and a product A second sensor providing second sensing information to the server in response to sensing a customer entering the storage space, and a photograph of a customer located near the product storage space and transmitting the second sensing information to the server in response to reception of the second sensing information from the server. A second camera for photographing a payment terminal providing payment start information to the server in response to payment initiation and a customer making a product payment in response to receiving the payment start information from the server and transmitting the photograph to the server A third camera may be included.

According to the information transmission and reception method of the device in the unmanned store of the present invention, the security of the unmanned store is strengthened and the store management is enhanced by enabling efficient communication between in-bound equipment or inbound equipment and external equipment in the unmanned store has the effect of helping to make it more efficient.

1 is a conceptual diagram showing an unmanned store system for performing customer behavior pattern analysis according to an embodiment of the present invention;
2 is a view showing an unmanned store system for performing customer behavior pattern analysis according to another embodiment of the present invention;
3 is a flow chart showing an information exchange process between equipment in an unmanned store of FIG. 1 or 2;
4 is a conceptual diagram for explaining a method of collectively managing customers by labeling image data obtained through a plurality of camera equipment in an unmanned store;
5 is a table expressing results of analyzing the labeled image data of FIG. 4;
6 is a flowchart showing a method of visualizing the video data analysis result so that the user can intuitively understand when the inventory quantity of the unmanned store does not match;
7 is a conceptual diagram for explaining a method of efficiently reproducing image data corresponding to a user input when there is a user input during visualization according to the method of FIG. 6;
8 is a conceptual diagram for explaining an operation of displaying in-store customer behavior patterns in real time through the in-store display equipment and speakers of FIG. 1;
9 is a flowchart for explaining a method of warning a smoker through an in-store display device and a speaker when smoke is detected in the store;
10 is a conceptual diagram for explaining a method of learning an artificial intelligence model that infers an optimal product display method by using data obtained through the system of FIG. 1 for unmanned store management;
11 is a detailed block diagram showing the configuration of a device constituting an unmanned store system for performing customer behavior pattern analysis according to an embodiment of the present invention.

Since the present invention can make various changes and have various embodiments, specific embodiments are illustrated in the drawings and described in detail.

However, this is not intended to limit the present invention to specific embodiments, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and scope of the present invention.

Terms such as first and second may be used to describe various components, but the components should not be limited by the terms. These terms are only used for the purpose of distinguishing one component from another. For example, a first element may be termed a second element, and similarly, a second element may be termed a first element, without departing from the scope of the present invention. The terms and/or include any combination of a plurality of related recited items or any of a plurality of related recited items.

It is understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, but other elements may exist in the middle. It should be. On the other hand, when an element is referred to as “directly connected” or “directly connected” to another element, it should be understood that no other element exists in the middle.

Terms used in this application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly dictates otherwise. In this application, the terms "include" or "have" are intended to designate that there is a feature, number, step, operation, component, part, or combination thereof described in the specification, but one or more other features It should be understood that the presence or addition of numbers, steps, operations, components, parts, or combinations thereof is not precluded.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which the present invention belongs. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and unless explicitly defined in the present application, they should not be interpreted in an ideal or excessively formal meaning. don't

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention will be described in more detail. In order to facilitate overall understanding in the description of the present invention, the same reference numerals are used for the same components in the drawings, and redundant descriptions of the same components are omitted.

1 is a conceptual diagram illustrating an unmanned store system for performing customer behavior pattern analysis according to an embodiment of the present invention. As shown in FIG. 1, the unmanned store system according to an embodiment of the present invention includes a door sensor 112, a camera 120, a product storage sensor 132, a camera 134, a payment terminal 140, It may include a camera 142 , a display device 150 , a speaker 160 , an access point 170 (Access Point (AP)), a proxy server 180 and an integrated server 190 .

Referring to FIG. 1 , the door sensor 112 is a sensor that detects customers entering and exiting the unmanned store. This may include at least one of a door open/close sensor, an approach sensor, a motion recognition sensor, and an infrared sensor. When the door sensor 112 detects a customer entering or exiting, the camera 120 operates to photograph the shape of the customer entering the door. At this time, it is preferable to photograph the entire shape of the customer, and in particular, it is preferable to photograph in the direction in which the shape of the customer's face appears in order to use technology such as face tracking later. The door sensor 112 performs sensing not only when a customer enters but also when exiting the store, and sends the sensed signal to the camera 120 so that the appearance of the customer leaving the store can be captured as well.

A plurality of product storage 130 may exist in an unmanned store. Although the term product storage 130 is used, it may be implemented as a display stand that simply displays products, rather than a place in the form of a box for storing products therein. That is, it means a unit space in which a product is placed. In one example, merchandise storage 130 may include a refrigerator, freezer, display stand, and the like. In addition, a sensor 132 is disposed in or around the product storage 120 to detect an action of selecting a product (an action of picking up the product) by the customer to purchase the product. The sensor 132 includes at least one of a door open/close sensor, an approach sensor, a motion recognition sensor, and an infrared sensor. In particular, in a situation where products are stored with the door closed, such as in a refrigerator or freezer, it is preferable to detect a customer's operation of selecting a specific product to purchase through a door open/close detection sensor. In an environment that does not require a special door opening/closing operation, such as a shelf, a customer's selection of a product can be detected through an approach sensor, a motion recognition sensor, an infrared sensor, and the like.

Meanwhile, when an act of selecting a product related to a specific product storage 130 of a customer is detected through the sensor 132 , a camera 134 corresponding thereto takes a picture of the customer during product selection. One or more sensors 132 and cameras 134 may be provided for each individual product storage 130 . Therefore, when the sensor 132 of the first product storage 130 detects the customer's product selection behavior, the corresponding camera 134 takes a picture, and the customer's product selection behavior is detected by the sensor of the other product storage. When it is, the shooting is performed by another camera corresponding to it. Here, also, photographing may be performed in a direction of looking at the front, back, or top of the customer.

After the customer makes a product selection, payment is requested. Accordingly, payment is performed by accessing the payment terminal 140 and inputting a payment-related user input to the payment terminal 140 . At this time, the customer's product payment behavior may be detected through the execution of the payment terminal 140 and/or an access sensor (not shown) separately provided around the payment terminal 140 . In response to the customer's payment (initiation) behavior detected in this way, the camera 142 starts shooting. The payment terminal 140 includes a kiosk and a POS terminal. Alternatively, it may be a kiosk device including a POS terminal function.

According to an embodiment of the present invention, in-store equipment may communicate with each other using short-range communication. In this case, the short-range communication may include Wi-Fi, Zigbee, Bluetooth, and the like. When performing Wi-Fi communication, information is transmitted and received to each other using the internally distributed IP through the access point 170. That is, the access point 170 allocates individual IPs to the sensors 112 and 132, the payment terminal 140, and the cameras 120, 134, and 142, and stores each other's IPs among inbound devices to obtain information. can send and receive. For example, in an embodiment in which inbound devices communicate directly without going through the proxy server 180, the first sensor 112 transmits sensing information to the camera 120 through the IP of the camera 120, and The sensor 132 transmits sensing information to the camera 134 via the IP of the camera 134, and the payment terminal 140 transmits payment initiation information to the camera 142 via the IP of the camera 142. Thus, it is possible to control each of the cameras 120, 134, and 142 to activate shooting. The manager uses the manager terminal 105 to transmit the information sensed by the manager to the individual sensors 112 and 132 and/or the payment terminal 140 via the integration server 190 or the proxy server 180. IP information can be determined. And, you can change it whenever you want. That is, a mapping relationship between a sensor/terminal and a camera may be arbitrarily set. In another example, inbound devices in the store do not necessarily use only short-range communication, but NarrowBand IoT (NB-IoT), LoRa, oneM2M, Open Connectivity Foundation (OCF), Lightweight M2M (LwM2M) Information may be transmitted and received between each other or to an external device according to at least one of the standard communication protocols established by the global IoT standard.

In addition, sensing information from at least two sensors 112 and 132, image data captured by at least three cameras 120, 134 and 142 (including video), and payment terminal 140 Payment initiation and progress information and payment detail information in may be provided to the proxy server 180 . In addition, the proxy server 180 may provide corresponding data to the integrated server 190 using a broadband communication network.

The proxy server 180 is a server deployed at an unmanned store site. This may exist in the form of a daemon tool emulated in a computing device disposed in an unmanned store. The corresponding server works with the integrated server 190. This may be a server or program installed in member stores using unmanned store security and business management services provided by the integrated server 190 .

However, according to an embodiment of the present invention, the proxy server 180 may not necessarily be required. The sensors 112 and 132 , the cameras 120 , 134 and 142 , and the payment terminal 140 may directly send related information to the integration server 190 without going through the proxy server 180 .

The integration server 190 analyzes the customer's behavioral pattern according to labeling of the customer in the video data based on the received sensing information, video data, and payment information, and sends the analysis result to the manager terminal 105, in-store Control to output through the display device 150 and the speaker 160. Output to the in-store display equipment 150 and the speaker 160 may be made through the proxy server 180 . The display equipment 150 and the speaker 160 may be referred to as an output device, which is an information acquisition unit (not shown) that obtains a customer behavior pattern analysis result (and an expression method for the analysis result) and an output unit that outputs it ( not shown) may be included. In addition, the integration server 190 (or proxy server 180) may be called an output control device, a visualization device, a visual expression device, an auditory expression device, an expression control device, and the like.

On the other hand, the customer's behavior pattern analyzes the store entry and exit behavior based on the image captured by the camera 120, analyzes the customer's product selection behavior based on the image captured by the camera 134, and analyzes the customer's product selection behavior based on the image captured by the camera 142. It is derived by comprehensively analyzing the results of analyzing the customer's product payment behavior based on the captured video. Accordingly, some customers only entered and left the store, some customers only entered and selected products and left the store without paying, and some customers left the store after proceeding from entry to product selection and payment. After confirming whether or not the customer has gone, customers can be classified by type, and the classified result can be provided to the manager terminal 105 or the in-store output devices 150 and 160 .

In addition, the integration server 190 classifies and stores the sensing information, image data, and payment information in the database 192 by customer, by product, and/or by time. Then, a plan for increasing sales can be derived through an artificial intelligence-based machine learning model using the stored information. An exemplary one of the methods derived may be a product display method suitable for a customer's product selection propensity.

Meanwhile, the functions of the integration server 190 may be performed by the proxy server 180 disposed in the store. That is, the analysis function provided by the integrated server 190 is installed in the proxy server 180 in the form of an agent, so that the proxy server 180 itself can analyze and output analysis results. Since the integrated server 190 or the proxy server 180 analyzes sensing information and/or image information, it may be referred to as an analysis device or an image analysis device.

The sensors 112 and 132 may be implemented as various types of IoT sensors other than those listed as examples above. This may include a temperature sensor, a humidity sensor, a motion detection sensor, and the like. In addition, one or more other sensors may be present in the store or at other locations around the store in addition to the sensors 112 and 132 and the sensor at the location of the payment terminal 140 . In addition, at least one of the cameras 120 , 134 , and 142 may take an image by interlocking with sensors disposed elsewhere as above, and transmit the image to the integrated server 190 .

The manager terminal 105 is a user terminal possessed by a manager who performs store management, and may be implemented as a smart phone, a desktop, a laptop, or a personal digital assistant (PDA). The manager terminal 105 can be linked with the integration server 190 by installing an application (Application: hereinafter referred to as "app") provided by the integration server 190, and the information provided by the integration server 190 is stored in the app. It can be displayed through the provided user interface (UX/UI). In a special case, the manager terminal 105 may be implemented as a terminal in a store, and the proxy server 180 and the manager terminal 105 may be implemented as one device.

2 is a diagram illustrating an unmanned store system for performing customer behavior pattern analysis according to another embodiment of the present invention.

Referring to FIG. 2 , the relationship between a sensor attached to a product storage and detecting a customer's product selection operation and a camera does not necessarily have to be a 1:1 relationship. It can have a 1:N, N:1, or N:M relationship.

As in the embodiment of FIG. 2 , 6 sensors may correspond to 4 cameras. In this case, a matching relationship between an individual sensor and a corresponding camera may be predetermined. For example, the first sensor and the second sensor are connected to the first camera, the third sensor and the fourth sensor are connected to the second camera and the third camera, the fifth sensor is connected to the third camera and the fourth camera, and the sixth sensor may be matched with the fourth camera. Such a scenario can be set in the integration server, or can be set through the integration server and proxy server using an administrator's terminal.

According to an embodiment of the present invention, at least some of the cameras are installed to be rotatable, and when sensing information is received from the sensor, they rotate in a direction toward the sensed product storage area and look at the product storage area in a rotated state. It can be controlled to start shooting in the viewing direction. Rotation of such a camera includes PTZ (Pan Tilt Zoom) control. In addition, such a rotation and PTZ control mechanism can be applied to a camera that photographs a customer's action of entering a store, a camera that photographs a product payment action, and the like, in addition to photographing a customer's action of selecting a product.

3 is a flowchart illustrating an information exchange process between equipment in an unmanned store of FIG. 1 or 2 .

Referring to FIG. 3 , when the entrance sensor senses the customer's position (S310), related information is provided to the first camera (120 in FIG. 1) (S312). The first camera starts shooting based on the sensing information (S314). Shooting is performed for a predetermined time. For example, it may be taken for one minute. However, it does not necessarily have to be set to 1 minute, and it may be set to 2 minutes, 3 minutes, 4 minutes, 5 minutes, 10 minutes, 15 minutes, and 20 minutes. When the first camera captures an image, the captured image data is transmitted to the proxy server (S316). The proxy server transmits the received image data to the integration server in real time (S318).

When the door opening/closing sensor of the warehouse senses the opening and closing of the door (S320), it is regarded as the customer's act of selecting a product, and the sensing information is provided to the second camera (134 in FIG. 1) (S322). The second camera starts capturing images by triggering the opening/closing sensing information (S324) and provides the image to the proxy server (S326). The proxy server transmits it to the integration server (S328).

Next, in relation to product payment, when the payment process proceeds in the payment terminal (S330), the payment terminal provides information related to the payment process to the third camera (142 in FIG. 1) (S332). The third camera uses this as a trigger to start capturing an image (S334) and provides it to the proxy server (S336). The proxy server transmits it to the integration server (S338).

In the embodiment of FIG. 3 , it is described that individual sensors directly provide sensing information to a corresponding camera, but information may be transmitted and received through a proxy server without directly transmitting and receiving sensing information. For example, when an access sensor senses a customer's entry, it transfers the sensing information to a proxy server, the proxy server identifies which sensor the sensing information is from, and then takes a picture with a corresponding camera (ie, a first camera). It can operate by sending a request message. That is, the sensor transmits all sensing information to the proxy server, the proxy server identifies the sensing information from the sensor, extracts a corresponding camera, and transmits a photographing request message to the extracted camera. In this embodiment, the proxy server may have a mapping table representing a mapping relationship between sensors and cameras. In addition, the administrator can change or newly set it arbitrarily.

Further, according to another embodiment of the present invention, when sensing is performed, photographing may be started, but other operations may be performed while the camera continues photographing. The camera may be controlled to set a reference viewpoint for identifying a customer's behavioral pattern or behavioral stage based on the sensing information. For example, the first camera may continue to photograph the direction of the door, and recognize the receiving time of the sensing information as the time of entering the store by the customer in response to receiving sensing information from the entrance sensor. Accordingly, it is possible to set the customer's access point to the store based on the sensing information reception point and use it for image analysis. For example, an image of a predetermined predetermined time before and after a reference point may be classified as a store access video related to a corresponding customer, and when a store access video is needed, the video of the predetermined time may be called and used for analysis. That is, the camera operates in a form of assigning a view-related reference index to an image based on sensing information.

4 is a conceptual diagram illustrating a method of collectively managing image data obtained through a plurality of camera equipment in an unmanned store by labeling them.

Referring to FIG. 4, the integrated server collects first image data obtained from an access detection camera, second image data obtained from a camera related to product selection, and third image data obtained from a camera related to product payment, and individually Labeling is performed for each customer present in the image. When more than one customer is present in a single image, each is labeled differently. There are two customers in the first image data, one labeled #1 and the other #2. The detection of customers in the video is performed through an artificial intelligence (AI)-based machine learning model (hereinafter referred to as an 'artificial intelligence model'). That is, a person in an image may be recognized and a plurality of recognized persons may be distinguished by using face tracking or an image analysis/object detection model that tracks a person's impression.

After labeling is performed in the first image data through the artificial intelligence model as described above, the labeled image of the customer is detected from among the second image data (product selection action shooting data). Then, the same labeling is performed on the corresponding image. That is, a customer labeled as #1 is detected in the plurality of second image data, and the customer in the detected image is labeled as #1. In the same way, a customer labeled as #1 is detected in a plurality of third image data (product payment action photographing data), and the customer in the detected image is labeled as #1.

Then, the image containing the customer labeled #1 is stored in a specific physical location (folder labeled #1) in the database. In the same way, images related to the customer labeled #2 are stored in another physical location (folder labeled #2). When there are multiple customers in an image, one image data can be simultaneously stored in multiple folders. Such labeling may be performed for a specific product, and image data for each product may be stored in a predetermined physical location.

On the other hand, the integration server can classify the customer's motion using an artificial intelligence-based motion recognition model. For example, a customer's entrance and exit will not deviate much from the basic entry and exit operations, but various motions may exist for product selection operations. For example, there may be an act of picking up a product, an act of picking up and putting it back, an act of opening and closing only a refrigerator door, an act of testing a product in various ways, and the like.

These various motions may be generated as various learning data through a method such as crowd sourcing and classified into more subdivided motions. In other words, it is possible to secure a vast amount of learning data set by collecting relevant images from all over the world through crowdsourcing. In addition, self-learning can be performed by creating a learning data set again with image data captured while executing an actual artificial intelligence model and analysis results thereof.

At this time, in relation to the subdivided motion classification, the second image data is not an image related to product selection, but a video of picking up a product as the 2-1 image and a video related to the act of putting the product into the shopping cart as the 2-2 image. It can be divided into images indicative of more subdivided actions, such as images related to an act of taking products out of a shopping cart, such as images and second and third images, and can be applied as customer behavior patterns or behavior stages.

Regarding the payment action, an action of removing a product from a shopping cart, an action of putting a product on a checkout counter, an action of reading a barcode in a product with a barcode reader, and an action of inserting a payment card can be distinguished. Furthermore, the number of products placed on the checkout counter can be inferred based on the act of placing an object on the counter, and the number of paid products can be inferred based on the act of reading the barcode of the product and the payment history, and the difference between the two By comparing , it is possible to infer the number of products that are missing at the time of payment.

In addition, in the second image data obtained through the camera 134, the information on the type and number of products selected by the labeled customer is compared with the type and number of paid products in the payment details obtained through the payment terminal Thus, it is possible to calculate whether there are missing products and the number of missing products.

In particular, through artificial intelligence-based object detection and motion recognition technology, the more subdivided steps and/or patterns, the more useful it is for data for theft and security-related analysis and management data for increasing store sales.

Basically, the integration server determines whether a specific customer is a customer who has only entered the store, has selected a product, and a product through analysis of the existence of the labeled customer in the first to third video data and the customer's behavior. You can distinguish whether a customer has made a payment. That is, based on the fact that the customer #1 also exists in the first video data, the second video data, and the third video data, it can be determined that the customer has entered the store, selected a product, and paid for the product. . Based on the fact that the customer #2 exists in the first image data but does not exist in the second and third image data, it can be determined as a customer who only entered the store and left without selecting a product or paying for a product. In addition, a customer who exists in the first image data and the second image data but does not exist in the third image data may be determined as a customer who entered the store and even selected a product, but did not pay for the product. At this time, it is preferable to subdivide the second image data and check whether there is data related to the act of placing the product in the road product storage. Furthermore, by using the customer's exit video (fourth video data (not shown)), the customer's behavior pattern can be analyzed more effectively. This is because through the exit video analysis, it is possible to clearly confirm whether the customer carried the product or not.

In other words, the integration server checks whether an individual customer has entered or exited using the first image data, checks whether or not the customer has selected a product using the second image data, and checks whether the customer has selected a product using the third image data and/or payment Using the information, it is possible to check whether the customer has paid for the product. In addition, it is possible to finally check whether the customer went out with the product or went out in the same or similar way as when the customer came in without taking the product by using the exit video data. At this time, if the bag that the customer carries out the product is a white bag with the inside visible, such as a plastic bag, the type and number of products taken can be more clearly identified.

In particular, since time information is included in all image data captured by the camera, the customer's behavior over time can be inferred more clearly. That is, after entering the store, he went to the first refrigerator to select a product, then to the third refrigerator, then to the second refrigerator to select a product, and after paying for the product, he left at 14:05:12. It provides an almost perfect picture of the in-store behavioral flow of a series of customers. Since there is sensing information and payment information for each of the customer's important actions as described above, and there is a camera operating based on this, it is possible to combine the customer's behavioral flow based on a specific behavioral index in the video.

FIG. 5 is a table expressing a result of analyzing the labeled image data of FIG. 4 .

Referring to FIG. 5 , the integrated server may classify a behavior pattern into a plurality of preset types through the video analysis method of FIG. 4 . In the embodiment of FIG. 4, three types are classified from A to C, but it will be obvious to those skilled in the art that more types can be subdivided as behavior pattern scenarios are set differently.

First, the integrated server can classify a customer like #1 who entered the store, selected a product, and paid as type A. In addition, customers such as #2 who entered the store and selected products but did not pay may be classified as type B. In addition, customers such as #3 who only entered the store, but did not select products and pay, can be classified as type C. That is, by using such a classification method, behavior patterns of all customers from the first time point to the second time point set by the user can be analyzed and classified into one of three types.

According to an embodiment of the present invention, the integration server can control display means to display different visual expressions for the above three types. Alternatively, the auditory expression may be different. For example, when displaying the customer behavior pattern analysis results classified by type as described above in the administrator terminal, the server displays Type A customers in blue, Type C customers in yellow, and Type B customers in red. You can control it. Relatively, in terms of theft, type A customers can be seen as the customers with the most normal purchasing behavior, followed by type C customers, and type B customers with the most unusual behavior patterns. am. In other words, even though I entered the store and selected a product, I did not pay because I cannot rule out the possibility that I just took the product and left without paying.

On the other hand, as previously mentioned in FIG. 4, the artificial intelligence model can recognize more diverse motions, and accordingly, it is possible to set more subdivided action types than three based on subdivided actions. For example, by analyzing a store exit video, a type may be classified according to whether a product was taken out upon exit. In particular, among B-type customers, customers who did not take out the product based on the video at the time of exiting can be further subdivided into B-1 types, and customers who took out products into B-2 types. At this time, the B-2 type customer may be highlighted.

6 is a flowchart illustrating a method of visualizing an image data analysis result so that a user can intuitively understand when the inventory quantity of an unmanned store does not match.

Referring to FIG. 6 , the unmanned store manager may periodically check the stock quantity (S610). The manager may set period information from the first point in time when the inventory quantity has already been confirmed to the second time point for checking the next inventory quantity, and may utilize the customer behavior pattern analysis result. That is, by comparing the quantity in stock within the period set by the manager with the quantity of products paid, it is checked whether there is any abnormality (S620). The inventory at the first point in time must match the value added to the inventory at the second point in time so that all products have been paid for during the period without being stolen. This can be considered normal. By the way, if the above two values do not match, it is desirable to look at the results of image analysis of customers in relation to theft.

At this time, in displaying the result of image analysis, customers of a type most likely to be related to theft are displayed first. That is, B-type customers are displayed first (S630), then C-type customers (S640), and A-type customers are displayed at the end (S650). When displaying, as shown in FIG. 5, customers of the corresponding type are displayed in a table form. You can have the customer's label in the row and the customer's action stage in the column. The opposite is also fine.

The manager can sort and view the B-type customers first, then the C-type customers. Then, while looking at the A-type customer, it finds out the customer who has committed an act related to theft. Hereinafter, this process will be described in detail with reference to FIG. 7 .

FIG. 7 is a conceptual diagram for explaining a method of efficiently reproducing image data corresponding to a user input when there is a user input during visualization according to the method of FIG. 6 .

Referring to FIG. 7 , the integration server controls a customer's behavior pattern to be visualized and displayed, whether it is a manager terminal, a display means connected to a proxy server, or a display device in a store. At this time, as in the method of FIG. 6, it is preferable to display the customers in a format in which each individual type is organized. For example, when only B-type customers are listed and displayed, as in the embodiment of FIG. 7 , only customers #3, #7, and #8 may be displayed within the set period. These are all customers who entered the store and selected products, but did not pay for the products.

At this time, the integration server may further display the staying time from the customer's entrance to the exit in consideration of the customer's exit video. In another example, as described above, it may be analyzed and displayed whether the customer has taken out the product when leaving.

In general, when such a theft accident occurs, the administrator finds the screen at the time of the accident by playing backwards the CCTV screen installed in the store. That is, even if images for a set period are quickly forwarded, it is not efficient because images for all time periods must be checked one by one. It is also difficult to find the actual theft accident screen.

However, according to an embodiment of the present invention, the unmanned store system selects a customer with a specific label on a user interface displaying customers by type in a table, and quickly loads and reproduces the customer's image data to prevent theft and prevent accidents. You can quickly check related videos.

In the embodiment of FIG. 7 , the manager who recognizes that the staying time of the customer #8 is abnormally long clicks the customer #8 in the table, loads the entire image data related to the customer, and then plays it. At this time, image data may be prepared for all images related to customer #8 (eg, the first image to the nth image). In addition, since the individual image data is extracted and reproduced for a time of ±1 minute before and after the specific action, it does not take a long time to check it. Since this method is used, the administrator does not need to check the video for the entire set period one by one. Here, the time of ±n minutes before and after does not necessarily need to be 1 minute, and may be set to other periods such as 2 minutes, 3 minutes, 4 minutes, 5 minutes, 10 minutes, and 20 minutes. For example, when reproducing a video of customer #8 entering the store, the video from 1 minute before entering the store to 1 minute after entering the store (photographed through the camera 120 video) to play. And, in relation to product selection, when the second image data related to customer #8 exists, based on the product selection time point, a video from 1 minute before product selection to 1 minute after product selection is prepared and reproduced. Regarding product selection, since selection can be made while searching through a number of refrigerators (Refrigerator 1, Refrigerator 3, Refrigerator 4, ...), in the multiple product selection video data, Multiple images can be played. Here, in the case of a refrigerator, the product selection time may be the time when the refrigerator door is opened and closed. In addition, if the customer #8 is displayed on the video data that illuminates other places, such as the door, refrigerator, and payment terminal, data for one minute before and after the display is prepared based on the customer's display time. can be reproduced. In this case, if the user wants to view images for more time sections before or after the reference point, content for a section one minute before or one minute after the corresponding video can be reproduced by inputting a command from the user.

As mentioned earlier in the embodiment of FIG. 5, the operation can be divided into more subdivided operations, it is possible to recognize the product selection cancellation operation along with the product selection on the table of FIG. 7, and the presence or absence of the product selection cancellation operation is displayed on the table. can do it That is, customers #3 and #8 both have a product selection action and a product selection cancellation action, while #7 customers have only a product selection action and no product selection cancellation action, and the analysis result may be displayed to the manager. Then, the manager will want to view the detailed video of customer #7, and the manager can check where the theft occurred by looking at the video related to customer #7.

According to an embodiment of the present invention, by specifically clicking the customer label part 710 and the action parts 720-1, 720-2, ... in the table, only the screen corresponding to the corresponding action of the corresponding customer is displayed. You can control it. For example, if a customer's label portion 710 is clicked, all images related to the customer may be prepared. However, if a specific customer's store entrance part 720-1 is clicked, only the first image data related to the store entrance may be reproduced. If you want to see only the product selection part, click the row part 720-2 indicating the product selection operation within the column of the corresponding customer, and only the second image data related to the corresponding operation can be reproduced.

FIG. 8 is a conceptual diagram illustrating an operation of displaying a behavioral pattern of a customer in a store in real time through the in-store display equipment and speaker of FIG. 1 .

Referring to FIG. 8 , the integrated server (or proxy server) analyzes the customer's behavior pattern in real time and outputs the result to a display device and/or a speaker in the store. For example, divide the screen of the display device to display a plurality of customers, and for currently entered customers, to output an image of the customer with a customer label, and at this time, take action ahead of the customer's action step. It can be identified and output in the same or similar form as the pattern analysis. For example, if the customer #1 has entered, completed product selection, and is currently making a payment at the payment terminal, when the customer's entry is detected, the first camera (120 in FIG. 1) image is recorded (at the same time, labeling When there is product selection sensing information, the second camera (134 in FIG. 1) image is displayed while displaying the operation step "product selection", and payment is made at the payment terminal. When progress is sensed, it can be controlled to display an operation step of “paying” while displaying an image of a third camera (142 in FIG. 1). Through this, when the customer is in the store, it is possible to have the effect of preventing theft and vandalism by making the customer feel that his/her behavior is continuously monitored. The expression of the action step is not output only as a video, but through the speaker, you can control the customer's label and action step information to be output through the speaker, such as "Customer No. 2 has entered" or "Customer No. 3 is selecting a product." there is.

Meanwhile, the image of a specific customer in the display device may be deleted 5 minutes after the customer leaves. At this time, it is not necessarily 5 minutes, and it may be set to 1 minute, 2 minutes, etc.

In addition, as the number of customers in the store increases, it is desirable to further subdivide the screen division to secure a plurality of smaller screen divisions. This can be done adaptively according to the number of customers entering.

9 is a flowchart illustrating a method of warning a smoker through an in-store display device and a speaker when detecting smoke in the store.

Referring to FIG. 9 , an unmanned store system according to an embodiment of the present invention includes a system for detecting and warning smoking. As described above, the unmanned store system according to an embodiment of the present invention can interwork with various IoT sensors. For example, smoking smoke in the entire store may be detected by arranging smoke detection sensors for each predetermined area. In fact, although smoking in unmanned stores has become a big problem, it is not possible to arrest it because there is no person in the store.

Accordingly, the unmanned store system according to an embodiment of the present invention includes a smoking detection sensor and displays a warning message through an in-store display device when smoke is detected, as in the embodiment of FIG. 8 . At this time, an image of a camera corresponding to the current location of the labeled customer is displayed together. In the case of zone B, it may have a mapping relationship with camera 1, and in response to detection of smoke in zone B, an image of the customer is taken by camera 1, and the captured image is displayed on the display device. At this time, a warning message is displayed by changing the background screen to red along with the phrase "smoking prohibited". At this time, it is preferable to also output a voice warning through a speaker. For example, a comment such as "This is a no-smoking area for customer No. 9. Please refrain from smoking" may be output. In spite of this primary warning, if smoking is continuously detected through the smoke detection sensor (for more than a critical period of time), the server may take measures such as dispatching security personnel in association with a preset security system. Alternatively, a smoking detection message is transmitted to the administrator's terminal so that the administrator can take appropriate action.

As with smoking, in-store consumption of alcoholic beverages is a major social problem. An unmanned store system according to an embodiment of the present invention may detect that a liquor product is selected by a customer by arranging sensors (a door open/close sensor, an proximity sensor, an infrared sensor, etc.) at a place where liquor products are arranged. When alcohol selection is detected, an image of the customer is taken through a camera corresponding to the corresponding sensor. At this time, when the act of drinking alcohol (drinking) is detected through an artificial intelligence-based motion detection model, the system detects smoking and smoking Similarly, a liquor consumption image and a warning message are displayed together through a display device in a store. In addition, a warning can be made through a speaker. If the consumption of alcoholic beverages continues despite the above primary warning, the server takes measures to have security personnel dispatched in connection with the preset security system or sends a message to detect drinking alcohol to the administrator terminal so that appropriate measures can be taken. can be made to be taken.

FIG. 10 is a conceptual diagram illustrating a method of learning an artificial intelligence model that infers an optimal product display method by using data acquired through the system of FIG. 1 for unmanned store management.

Referring to FIG. 10, the integrated server of the unmanned store system according to an embodiment of the present invention analyzes the number of store entries, the number of product selections, the number of paid products, store sales, etc. together with customer behavior patterns to provide data related to store management. can be derived.

For example, by using at least some of the number of store entries, the number of product selections, the number of paid products, and store sales indicators by time slot as a learning data set, the product display is changed (A-B-C-D → B-C-A-D → ...), and the learning data set When input into an artificial intelligence-based machine learning model, the optimal product display method can be inferred in the direction of maximizing store sales. At this time, parameters related to display may also be entered as inputs, and when inferring the relationship between store sales index and/or number of product selections while changing display parameters, the direction in which store sales index and/or number of product selections increase It learns an artificial intelligence model in the form of inferring the optimal product arrangement method. The server infers an optimal product arrangement method using the model learned in this way.

In particular, optimal product display can be inferred from the relationship between the number of product choices and the number of payment products. The integration server analyzes the number of selections and payments for each product, detects products that have been selected a lot, but has not been paid for, and can display them to a manager. For example, there may be many cases in which product A of ice cream is selected, but is placed in the refrigerator again after being selected. This can be confirmed by recognizing the customer's motion pattern for each product through the motion recognition artificial intelligence model described above. The server may determine that the current display form is not appropriate for these products, may change it, and may input the changed product display format as learning data. In addition, the product display format change and the resultant value, such as whether sales increase or payment is made without returning after product selection according to the product display format change, is created as learning data to derive the optimal product display format. can create, train, and run artificial intelligence models that

In another example, the server extracts the corresponding time zone based on the sales index and customer behavior pattern analysis result for each time zone, in response to the fact that there are not many product selection and product payment compared to the large number of store admissions during a specific time period. , it is possible to help the manager in the form of recommending events such as product discounts to customers who entered the extracted time zone.

11 is a detailed block diagram showing the configuration of a device constituting an unmanned store system for performing customer behavior pattern analysis according to an embodiment of the present invention. As shown in FIG. 11 , the device of the unmanned store system according to an embodiment of the present invention may include a communication unit 1110, a processor 1120, a memory 1130, and an input/output module 1140. Here, the device may be a proxy server or an integrated server.

Referring to FIG. 11 , a communication unit 1110 is a component for exchanging data with another terminal using a wired or wireless network. The communication unit 1210 may include an antenna. The communication unit 1110 receives image data from cameras as well as sensing information from sensors and payment information from payment terminals.

The processor 1120 is a component that processes information received through the communication unit 1110 . This may be implemented as a microprocessor in terms of hardware. The processor 1120 identifies sensing information from the sensor and transmits a photographing command to a corresponding camera. In addition, the collected image data is labeled by customer by executing an artificial intelligence model such as face recognition, object detection, and motion recognition, and the customer's behavior pattern and stage are analyzed.

The processor 1120 is connected to the memory 1130 through a signal line, and executes a program stored in the memory 1130. As described above, the processor 1120 not only analyzes customer behavior related to theft and security, but also processes the analyzed customer behavior pattern into data useful for store management by linking it with a sales index. At this time, an artificial intelligence-based learning model can also be executed.

The memory 1130 is a storage device that stores instructions related to programs to be executed by the processor 1120 and various data requested by the processor 1120 . The memory 1130 stores various sensing data obtained from sensors in the store and image data obtained from a camera. In addition, not only the mapping relationship between the sensor and the camera, but also a scenario such as a certain operation to be performed as a next operation when sensing information is obtained may be stored in the form of a program language. The memory 1130 may be implemented as a local memory inside the device or as an external large-capacity database.

The input/output module 1140 includes information input means such as a keyboard and mouse, and information output means such as a monitor, TV, and touch screen. The input/output module 1140 may be used to change various setting values such as the setting of a target time period for customer behavior pattern analysis and to display analysis results.

The system or apparatus described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, systems, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA) ), programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. , or may be permanently or temporarily embodied in a transmitted signal wave. Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

Claims (16)

In the information transmission and reception method of the device in the unmanned store,
providing first sensing information to a first camera in response to sensing of first sensing information related to a customer entering and exiting a store from a first sensor, wherein the first sensor is a sensor for detecting a customer entering and exiting the store;
In response to receiving the first sensing information by the first camera, photographing a customer entering or exiting;
Transmitting an image captured by the first camera to a server;
providing second sensing information to a second camera in response to the second sensor sensing a customer entering the product storage;
In response to the second sensing information being received by the second camera, photographing a customer located around the product storage;
transmitting the image captured by the second camera to the server;
providing payment start information to a third camera in response to payment being initiated in the payment terminal;
photographing, by the third camera, a customer who pays for a product in response to receiving payment start information; and
A method for transmitting and receiving information of a device in an unmanned store comprising transmitting an image captured by the third camera to the server. According to claim 1,
At least one of the first sensor and the second sensor includes at least one of a door open/close detection sensor, an approach sensor, a motion recognition sensor, and an infrared sensor. According to claim 1,
The method of transmitting and receiving information of a device in an unmanned store, wherein the product storage includes at least one of a refrigerator for refrigerating products, a freezer for storing products frozen, and a product display stand for displaying products in the form of a shelf. According to claim 1,
The first camera, the second camera, and the third camera are triggered by receiving signals from the first sensor, the second sensor, and the payment terminal to start shooting. How to send and receive information. According to claim 1,
The first camera, the second camera, and the third camera are triggered when signals from the first sensor, the second sensor, and the payment terminal are received, and the reception time point is a reference point index in the captured video. (index), a method of transmitting and receiving information of a device in an unmanned store. According to claim 1,
The first sensor, the second sensor, the payment terminal, and the first camera, the second camera, and the third camera have IP addresses allocated from an access point (AP) in the store, and ,
The first sensor transmits the first sensing information to the first camera via the IP of the first camera having a mapping relationship with the first sensor;
The second sensor transmits the second sensing information to the second camera via the IP of the second camera having a mapping relationship therewith,
wherein the payment terminal transmits the payment start information to the third camera via the IP of the third camera having a mapping relationship therewith. According to claim 6,
A method for transmitting and receiving information of a device in an unmanned store, wherein a plurality of cameras have a mapping relationship with one of the first sensor, the second sensor, and the payment terminal. According to claim 6,
The first sensor, the second sensor, and the payment terminal transmit the first sensing information, the second sensing information, and the payment initiation information to the server via the access point. How to send and receive. According to claim 6,
The first sensor, the second sensor, and the payment terminal, and the first camera, the second camera, and the third camera are Wi-Fi, Zigbee, and Bluetooth A method for transmitting and receiving information between devices in an unmanned store, which communicate with each other using at least one of the following. According to claim 6,
The first sensor, the second sensor, and the payment terminal, and the first camera, the second camera, and the third camera are NarrowBand IoT (NB-IoT), LoRa, and oneM2M , OCF (Open Connectivity Foundation), LwM2M (Lightweight M2M) using at least one related communication protocol to communicate with each other, a method for transmitting and receiving information between devices in an unmanned store. According to claim 1,
In response to the smoke detection sensor detecting smoke, a smoke detection signal is transmitted to a fourth camera having a mapping relationship with the smoke detection sensor;
wherein the fourth camera activates photographing in response to receiving a smoke detection signal. According to claim 11,
A plurality of smoke detection sensors are arranged to sense the unmanned store by zone,
The plurality of smoke detection sensors operate individually with the IP address of the corresponding fourth camera,
One smoke sensor corresponds to one or more fourth cameras. According to claim 1,
The payment terminal includes at least one of a kiosk and a POS terminal. According to claim 1,
The mapping relationship between the first sensor and the first camera, the mapping relationship between the second sensor and the second camera, and the mapping relationship between the third sensor and the third camera are set by the server or manager terminal A method for transmitting and receiving information of a device in an unmanned store. In the unmanned store system for transmitting and receiving information,
A first sensor that provides first sensing information to a first camera in response to sensing of first sensing information related to a customer's entering and exiting the store, wherein the first sensor is a sensor that detects a customer's entering and exiting;
In response to receiving the first sensing information, a first camera for capturing and transmitting to a server a customer entering and exiting;
a second sensor providing second sensing information to a second camera in response to sensing a customer entering the product storage;
a second camera for taking a photograph of a customer located near a product storage and transmitting the photograph to the server in response to receiving the second sensing information;
a payment terminal providing payment start information to a third camera in response to payment being initiated; and
In response to receiving the payment initiation information, a third camera for photographing a customer performing a product payment and transmitting the photograph to the server, unmanned store system for transmitting and receiving information. In the unmanned store system for transmitting and receiving information,
A first sensor that provides the first sensing information to a server in response to sensing of first sensing information related to a customer entering and exiting the store, wherein the first sensor is a sensor that detects a customer entering and exiting the store;
a first camera for photographing a customer entering and exiting and transmitting the photograph to the server in response to receiving the first sensing information from the server;
a second sensor providing second sensing information to the server in response to sensing a customer entering the product storage;
a second camera for taking a picture of a customer located near a product storage and transmitting the photograph to the server in response to receiving the second sensing information from the server;
a payment terminal providing payment start information to the server in response to payment being initiated; and
In response to receiving the payment initiation information from the server, the unmanned store system for transmitting and receiving information including a third camera for photographing a customer performing product payment and transmitting the photograph to the server.

Images