KR102451378B1 - Unmanned store management system - Google Patents

Abstract

The present invention relates to an unmanned store management system. The unmanned store management system comprises: an IoT environmental complex sensor for detecting environmental information in an unmanned store; a kiosk provided in the unmanned store and controlling an operation of an IoT control sensor provided in the unmanned store in consideration of the environmental information acquired from the IoT environmental complex sensor; and an unmanned store management device for interworking with each of the IoT environmental complex sensor, the IoT control sensor, and the kiosk through a network and performing the management of the unmanned store at a remote location. The unmanned store management device may control device operations of each of the IoT environmental complex sensor, the IoT control sensor, and the kiosk in relation to the management of the unmanned store. Accordingly, a manager can remotely manage an unmanned store using IoT technology.

Description

Unmanned Store Management System {UNMANNED STORE MANAGEMENT SYSTEM}

The present application relates to an unmanned store management system.

With the development of Internet of Things (IoT) technology, interest in unmanned stores is increasing as the demand for untact or non-face-to-face has recently increased.

Unmanned stores can be applied to numerous industries. For example, the unmanned store may be variously applied not only to industries that provide tangible products such as restaurants and retail stores, but also to industries that provide spaces or intangible services such as reading rooms and study rooms.

Accordingly, there is an increasing demand for an unmanned store operation method that can provide convenience and efficiency to all users who operate an unmanned store or use an unmanned store.

The background technology of the present application is disclosed in Korean Patent Application Laid-Open No. 10-2019-0104078.

An object of the present application is to solve the problems of the prior art described above, and an object of the present application is to provide an unmanned store management system capable of remotely managing an unmanned store using IoT technology.

However, the technical problems to be achieved by the embodiments of the present application are not limited to the technical problems as described above, and other technical problems may exist.

As a technical means for achieving the above technical problem, the unmanned store management system according to an embodiment of the present application includes an IOT environmental complex sensor for detecting environmental information in the unmanned store; a kiosk provided in the unmanned store, which controls the operation of the IOT control sensor provided in the unmanned store in consideration of the environmental information obtained from the IOT environmental complex sensor; and an unmanned store management device that is interlocked with each of the IOT environment complex sensor, the IOT control sensor, and the kiosk through a network, and manages the unmanned store in a remote location, wherein the unmanned store management device is In relation to the management of the store, it is possible to control the operation of each device of the IOT environment complex sensor, the IOT control sensor, and the kiosk.

In addition, the unmanned store management system is installed at the store entrance of the unmanned store, acquires user authentication information of a user who wants to enter the unmanned store, transmits it to the unmanned store management device, and obtains from the unmanned store management device Further comprising an access authentication device for controlling the opening and closing of the door of the unmanned store based on the access authentication result information for the user authentication information, the unmanned store management device, the access authentication in relation to the management of the unmanned store You can further control the operation of the device.

In addition, the unmanned store management device, the IOT environment complex sensor, the IOT control sensor, the kiosk and the device control management unit for controlling and managing each device operation of the access authentication device; Based on the user authentication information received from the access authentication device, user authentication is performed to determine whether the user is a pre-registered customer registered in the customer DB, and an access authentication result generated in consideration of the result of the user authentication an authentication unit for transmitting information to the access authentication device; a product management unit that manages products sold in the unmanned store; and a screen controller for providing a response screen corresponding to the input signal to the display unit of the manager terminal based on an input signal of a manager who manages the unmanned store management apparatus.

In addition, the device control and management unit may further perform operation control and management of a plurality of electronic price indicators provided to be 1:1 matched for each type of product with respect to the plurality of types of products displayed in the unmanned store. .

The above-described problem solving means are merely exemplary, and should not be construed as limiting the present application. In addition to the exemplary embodiments described above, additional embodiments may exist in the drawings and detailed description.

According to the above-described problem solving means of the present application, by providing an unmanned store management system, an administrator can remotely manage an unmanned store using IoT technology. That is, the present application may provide a manager to operate and manage the unmanned store more efficiently.

However, the effects obtainable herein are not limited to the above-described effects, and other effects may exist.

1 and 2 are diagrams showing a schematic configuration of an unmanned store management system according to an embodiment of the present application.
3 is a diagram schematically illustrating a process in which an authentication unit of an unmanned store management apparatus according to an embodiment of the present application performs user authentication.
4 is a view for explaining a payment process of a product through a kiosk in the unmanned store management apparatus according to an embodiment of the present application.
5 is a schematic diagram for explaining a method of operating various devices in an unmanned store management system according to an embodiment of the present application.
6 is a diagram illustrating an example of a screen display of a kiosk in an unmanned store management system according to an embodiment of the present application.
7 to 11 are diagrams for explaining an example in which the screen of the manager terminal is controlled by the screen controller of the unmanned store management apparatus according to an embodiment of the present application.
12 is a diagram illustrating an example of an access authentication device applied to an unmanned store management system according to an embodiment of the present application.
13 is a diagram for explaining a Harr Features calculation method that is considered when the authentication unit in the unmanned store management apparatus according to an embodiment of the present application performs user authentication on a face image.
14 and 15 are diagrams for explaining the integral image that the authentication unit in the unmanned store management apparatus according to an embodiment of the present application considers when performing face recognition.
16 is a view for explaining a meaningful feature that is considered when the authentication unit in the unmanned store management apparatus according to an embodiment of the present application performs face recognition.
17 is a diagram for explaining the MODEBUS protocol considered in the unmanned store management system according to an embodiment of the present application.
18 is a view for explaining a function code in relation to the MODEBUS-TCP technology considered in the unmanned store management system according to an embodiment of the present application.
19 is a view for explaining a case in which the device control management unit in the unmanned store management apparatus according to an embodiment of the present application controls a plurality of cctvs in the IOT control sensor.
20 is an operation flowchart of an unmanned store management method according to an embodiment of the present application.

Hereinafter, embodiments of the present application will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art to which the present application pertains can easily implement them. However, the present application may be implemented in several different forms and is not limited to the embodiments described herein. And in order to clearly explain the present application in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout this specification, when a part is said to be “connected” to another part, it is not only “directly connected” but also “electrically connected” or “indirectly connected” with another element interposed therebetween. "Including cases where

Throughout this specification, when it is said that a member is positioned "on", "on", "on", "under", "under", or "under" another member, this means that a member is located on the other member. It includes not only the case where they are in contact, but also the case where another member exists between two members.

Throughout this specification, when a part "includes" a component, it means that other components may be further included, rather than excluding other components, unless otherwise stated.

1 and 2 are diagrams showing a schematic configuration of an unmanned store management system 1000 according to an embodiment of the present application. Hereinafter, the unmanned store management system 1000 according to an embodiment of the present application will be referred to as the present system 1000 for convenience of description.

1 and 2 , the present system 1000 may include the unmanned store management apparatus 100 and various devices 3a provided in the unmanned store 3 .

The unmanned store 3 considered herein is a store operated unmanned, and may be, for example, a convenience store, a supermarket, a retail store, and the like, but is not limited thereto. The unmanned store 3 may be otherwise referred to as an unmanned distribution store or the like.

The unmanned store management apparatus 100 is an unmanned store management apparatus 100 according to an embodiment of the present application, and hereinafter, it will be referred to as the present apparatus 100 for convenience of description. The unmanned store management apparatus 100 may be a device managed by the manager 1 , that is, a device owned and managed by the manager 1 .

For example, the device 100 may be provided in the form of a device included in the manager terminal 1a, which is a terminal possessed by the manager 1 . In this case, the device 100 may be a device that can be implemented in the form of a program or application (application, app) installed in the manager terminal 1a. In other words, the unmanned store management method provided through the device 100 may be implemented in the form of a program or application, for example, and provided to the manager through the manager terminal 1a. However, the present invention is not limited thereto, and as another example, the device 100 may be provided in the form of a server capable of transmitting and receiving data with the manager terminal 1a. The device 100 provided in the form of a server controls the operation of the manager terminal 1a, which is a terminal of the manager 1 connected to the application provided by the device 100 (eg, the manager terminal 1a). can control the screen display of In the description with reference to FIG. 1 , the present device 100 will be described as an example provided in the form of a server.

In addition, in the present application (application, app) provided by the present device 100 will be referred to as this app for convenience of description below. Accordingly, the device 100 may control the terminal operation (eg, screen display operation, etc.) of a user (ie, an administrator) who has installed the present application.

The manager terminal 1a may be referred to as a user terminal or the like. The manager terminal 1a is, for example, PCS (Personal Communication System), GSM (Global System for Mobile communication), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication) )-2000, CDMA(Code Division Multiple Access)-2000, W-CDMA(WCode Division Multiple Access), Wibro(Wireless Broadband Internet) terminal, Smartphone, SmartPad, Tablet PC, Laptop, Wearable It may include all types of wired/wireless communication devices such as devices and desktop PCs, but is not limited thereto.

The device 100 may transmit and receive data by interworking with each of the various devices 3a through the network 4 .

The network 4 is, for example, a 3rd Generation Partnership Project (3GPP) network, a Long Term Evolution (LTE) network, a World Interoperability for Microwave Access (WIMAX) network, the Internet, a Local Area Network (LAN), and a Wireless LAN (Wireless LAN) network. Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), Bluetooth (Bluetooth) network, NFC (Near Field Communication) network, satellite broadcasting network, analog broadcasting network, DMB (Digital Multimedia Broadcasting) network, etc. It may include, but is not limited thereto, and may include various wired/wireless communication networks.

The various devices 3a provided in the unmanned store 3 may include an IOT environmental complex sensor 200, an IOT control sensor 300, a kiosk 400, an access authentication device 500, and an electronic price display 600. can

Herein, the IOT environment complex sensor 200 and the IOT control sensor 300 may be referred to as IOT equipment (devices, devices) and the like.

The IOT environmental complex sensor 200 may be provided in one area in the unmanned store 3 to detect environmental information in the unmanned store 3 . The IOT environmental complex sensor 200 may include, for example, a temperature sensor, a humidity sensor, a harmful gas detection sensor, a fine dust detection sensor, etc., but is not limited thereto, and various sensors for detection of environmental information (environmental data). may be included. The type and number of IOT environmental complex sensors 200 applied to the present system 1000 may be set/applied in various ways.

The IOT control sensor 300 may be a device (sensor device) that is provided in the unmanned store 3 and can electronically control the operation by the device control management unit 110 in the kiosk 400 or the device 100 . The IOT control sensor 300 may include, as an example, an air conditioner, an air purifier, a light, etc., but is not limited thereto, and all electronically controllable devices (eg, TV, speaker, fan, cctv, etc.) may be applied. have. The type and number of IOT control sensors 300 applied to the present system 1000 may be set/applied in various ways.

The kiosk 400 is provided in the unmanned store 3, and in consideration of the environmental information obtained from the IOT environmental complex sensor 200, the operation of the IOT control sensor 300 provided in the unmanned store 3 can be controlled. . The kiosk 400 may be otherwise referred to herein as a self-checkout kiosk or the like.

The access authentication device 500 is installed at the store entrance of the unmanned store 3, acquires the user authentication information of the user 2 who wants to enter the unmanned store 3, and through the network 4, the unmanned store management device ( 100) (in particular, the authentication unit in the device), and based on the access authentication result information for the user authentication information obtained from the unmanned store management device 100, the opening and closing of the door of the unmanned store 3 can be controlled. have. The access authentication device 500 may be otherwise referred to as an access authentication system or the like herein. The user 2 who wants to enter the unmanned store 3 may be referred to as a customer.

The electronic price display 600 is provided in the unmanned store 3 , and in particular, with respect to a plurality of types of products displayed in the unmanned store 3 , a plurality of electronic price indicators 600 may be provided so as to match 1:1 for each type of product. That is, the present system 1000 is an electronic price indicator 600 , and may include a plurality of electronic price indicators 600 .

As an example, the electronic price indicator 600 refers to a device that displays specific information about a product, such as a product name, price, and barcode, on a plastic tag, and may be expressed as an ESL (Electric Shelf Label). The description of the structure and operation principle of the electronic price indicator 600 may be applied to various electronic price indicators that have been previously developed or developed in the future, and detailed descriptions thereof will be omitted below.

The device control management unit 110 in the device 100, which will be described later, may control and manage operations for a plurality of electronic price displays 600 provided in the unmanned store 3 . Since the device 100 can control and manage the operation of a plurality of electronic price indicators 600, there is no need to replace paper labels with product-related information as in the prior art, so the number of manpower is reduced compared to the prior art. It is possible to reduce the wastage of wasted paper, and when information such as the price of a product is changed, the changed information can be easily changed in real time.

The device control management unit 110 may conveniently change the price list of the product, which is changed in real time, online through the control of the electronic price display 600 . A new product (new product)/discounted price may be displayed through the electronic price display 600 , and may be applied to each product. The device 100 can operate an unmanned store more easily by simply changing the price tag through the control of the electronic price display 600 without having to manually change the price tag.

According to the electronic price display 600 , the present system 1000 can improve customer satisfaction by providing convenient product information guidance and improve store productivity by replacing paper price tags. The operating principle of the electronic price indicator 600 is, for example, linking the network 4 (eg, cloud, etc.) with the kiosk 400 and ERP, and then transmitting data from the cloud to the store, and thereafter, the electronic price through the Ble router. It may be operated by displaying the price on the indicator 600 , but is not limited thereto. The unmanned store 3 to which the electronic price indicator 600 is applied may include, for example, a convenience store, an unmanned fruit store, a logistics warehouse, and the like.

By accessing this app, the manager 1 can make the electronic price display 600 conveniently change the product price with the app and perform inventory check in real time.

The device 100 is interlocked with each of the IOT environmental complex sensor 200 , the IOT control sensor 300 and the kiosk 400 through the network 4 , and can manage the unmanned store 3 at a remote location. have.

In this case, the device 100 may control the operation of each of the IOT environment complex sensor 200 , the IOT control sensor 300 , and the kiosk 400 in relation to the management of the unmanned store 3 . Also, the device 100 may control the operation of the access authentication device 500 in relation to the management of the unmanned store 3 . In addition, the device 100 may control the operation of the plurality of electronic price indicators (600). Accordingly, the device 100 may control and manage operations of various devices 3a in the unmanned store 3 in a remote location. The manager 1 can remotely control and manage various devices 3a in the unmanned store 3 by accessing the app through the manager terminal 1a. A more detailed description of the apparatus 100 is as follows.

The apparatus 100 may include a device control management unit 110 , an authentication unit 120 , a customer DB 130 , a product management unit 140 , and a screen control unit 150 .

The device control management unit 110 may control and manage the operation of each of the IOT environment complex sensor 200 , the IOT control sensor 300 , the kiosk 400 , and the access authentication device 500 . In addition, the device control management unit 110 may control and manage the operation of the plurality of electronic price display 600 . That is, the device control manager 110 may control and manage the operation of at least one of the various devices 3a.

The authentication unit 120 performs user authentication to determine whether the user 2 is a pre-registered customer registered in the customer DB 130 based on the user authentication information of the user 2 received from the access authentication device 500 . and transmits the generated access authentication result information to the access authentication device 500 in consideration of the result of user authentication.

Here, the user authentication information is information (identification information) for authenticating the user 2 himself, and may include, for example, at least one of a face image (face image information), card information, and QR recognition information.

The face image is information obtained to perform user authentication based on face recognition (facial recognition), and may be an image obtained by photographing the face of the user 2 . The face image may be acquired by being photographed, for example, by a camera for authentication provided in one area of the access authentication device 500 . That is, the user 2 wishing to enter the unmanned store 3 may have his or her face image taken through an authentication camera, for example, in order to perform user authentication, and through this, the access authentication device 500 may A face image (face image information) may be acquired as user authentication information.

The card information may be information obtained to perform user authentication based on a card (eg, a credit card, a check card, etc.) possessed by the user. The card information may be obtained, for example, through an IC card reader provided in one area of the access authentication device 500 . That is, the user 2 who wants to enter the unmanned store 3 can insert or contact the card he has with the IC card reader to perform user authentication, and through this, the access authentication device 500 is Card information may be acquired as user authentication information.

The QR recognition information may be information obtained to perform user authentication based on a terminal (user terminal) possessed by the user. The QR recognition information may be obtained, for example, through a QR recognition button provided in one area of the access authentication device 500 . That is, the user 2 who wants to enter the unmanned store 3 clicks a QR recognition button provided in one area of the screen of the access authentication device 500 for example to perform user authentication, and responds to the click on the screen The QR code displayed in can be recognized as a terminal (ie, user terminal) possessed by the user. Through this, the access authentication device 500 may acquire QR recognition information as user authentication information, and according to the obtained QR recognition information, as an example, identification information of a terminal (user terminal) possessed by the user (eg, mobile phone number, etc.) ) can be identified (recognized).

The customer DB 130 may register (store) user information of users (customers) who use the unmanned store 3 . In this case, the customer DB 130 may be registered as user information, in which the user's name and user authentication information are linked to each other. The user authentication information may include at least one of a face image (face image information), card information, and QR recognition information. In addition, the user information may further include, for example, the user's gender and age group information, in addition to the user's name and user authentication information.

The access authentication device 500 obtains user authentication information from the user 2 who enters and exits the unmanned store 3 , and uses the acquired user authentication information through the network 4 to the authentication unit 120 of the device 100 . can be sent to The authentication unit 120 performs user authentication to determine whether the user 2 corresponding to the acquired user authentication information is a pre-registered customer registered in the customer DB 130 based on the obtained user authentication information. can In this case, the result of user authentication may be 'a registered customer' or 'not a registered customer (ie, an unregistered customer').

Thereafter, the authenticator 120 may generate access authentication result information in consideration of the result of user authentication and transmit it to the access authentication device 500 . Here, the access authentication result information may be 'authentication success' or 'authentication failure'.

The access authentication device 500 may control opening and closing of the door of the unmanned store 3 based on the access authentication result information received from the authentication unit 120 . When the access authentication result information is 'authentication success', the access authentication device 500 opens the door of the unmanned store 3 so that the user 2 can access the unmanned store 3 in an open (open, open) state. can be controlled On the other hand, when the access authentication result information is 'authentication failure', the access authentication device 500 may control the door of the unmanned store 3 to be closed to prevent the user 2 from entering the unmanned store 3 . .

In the present specification, as an example, the authentication unit 120, which is a means for performing user authentication, is illustrated as being provided in the device 100, but this is only an example for helping understanding of the present application, and is not limited thereto. As another example, the means for performing user authentication in the system 1000 may be provided in the form of an authentication server (not shown) that is provided as a separate configuration from the device 100 and capable of transmitting and receiving data with the device 100 . may be

The user authentication process by the authenticator 120 may be more easily understood with reference to FIG. 3 .

3 is a diagram schematically illustrating a process in which the authentication unit 120 of the unmanned store management apparatus 100 according to an embodiment of the present application performs user authentication.

Referring to FIG. 3 , when the user 2 wants to enter the unmanned store 3 (when he wants to enter the unmanned store), the kiosk 400 obtains user authentication information from the user 2 and uses the authentication unit ( 120) can be transmitted. Accordingly, the authenticator 120 may receive the user authentication information transmitted from the kiosk 400 and perform user authentication on the received user authentication information (S1).

At this time, if the user authentication information received in step S1 is face image information (S2-Y), the authenticator 120 compares the received face image with the face images of a plurality of customers previously registered in the customer DB 130 . By doing so, user authentication can be performed, and it can be checked whether an image matching the received face image is in the customer DB 130 . That is, the authenticator 120 may check whether the user corresponding to the received face image is a pre-registered customer already registered in the customer DB 130 by performing user authentication.

At this time, if the user corresponding to the face image is a pre-registered customer (S3-Y), the authentication unit 120 generates information 'authentication success' as access authentication result information (S4) and transmits it to the access authentication device 500 can Accordingly, the access authentication device 500 controls the door so that the door of the unmanned store 3 is opened (S5) based on the access authentication result information corresponding to 'authentication success' received from the authenticator 120 . can Accordingly, the user 2 can enter and exit the unmanned store 3 .

On the other hand, if the user corresponding to the face image is not a registered customer (S3-N), the authentication unit 120 sends a message requesting new customer registration to the kiosk 400 in order to induce the user to register a new customer. You can control what is displayed on the screen. At this time, the message requesting new customer registration is, for example, 'You are not a registered customer (member) in the unmanned store. Please enter user authentication information to proceed with new customer registration'.

At this time, if the user wants to perform the new customer registration procedure in response to the message, the user allows his or her face image to be taken through the kiosk 400 or inserts (or touches) the card he/she owns into the IC card reader card to be recognized, or a terminal (mobile phone, etc.) possessed by the user can be recognized with a QR code.

At this time, when the user performs the new customer registration procedure in response to the message (S6-Y), that is, for the new customer registration procedure, the user takes a face image as user authentication information through the kiosk 400, recognizes card information, and performs QR When at least one of recognition is performed (input), the authentication unit 120 may register the user authentication information obtained from the user in the course of the new customer registration procedure as the new customer information in the customer DB 130 (S7) . After the registration of new customer information is made, the authentication unit 120 may perform the process of step S4.

If the user does not perform the new customer registration procedure in response to the message (S6-N), the authentication unit 120 generates the information 'authentication failure' as the access authentication result information (S8), and the access authentication device ( 500) can be sent. Accordingly, the access authentication device 500, based on the access authentication result information corresponding to 'authentication failure' received from the authentication unit 120, allows the door of the unmanned store 3 to remain closed (S9). can be controlled. Accordingly, the user 2 may not be able to enter the unmanned store 3 .

On the other hand, if the user authentication information received in step S1 is not face image information (S2-N), the authenticator 120 may check whether the user authentication information received in step S1 is card information. At this time, if the received user authentication information is card information (S10-Y), the authentication unit 120 performs user authentication by comparing the received card information with card information of a plurality of customers previously registered in the customer DB 130 . This can be performed, and through this, it can be checked whether card information matching the received card is in the customer DB 130 . That is, by performing user authentication, the authenticator 120 may confirm ( S11 ) whether the user corresponding to the received card information is a pre-registered customer in the customer DB 130 .

At this time, when it is confirmed that the user corresponding to the received card information is a registered customer (S11-Y), the authentication unit 120 does not directly generate 'authentication success' as the access authentication result information, but user authentication information As a result, a verification process can be performed to check whether the user who has entered card information is actually the owner of the card (ie, whether he is not using a card secretly by picking up a card lying on the road). In order to perform this verification process, the authenticator 120 acquires a face image of the user through the authentication camera provided in the kiosk 400 in step S11-Y, and the obtained face image is previously obtained from the customer DB 130 . It is possible to check (S12) whether it matches the stored face image in connection with the card information registered in the . have.

At this time, if the acquired face image and the face image corresponding to the pre-registered card information match each other (S12-Y), the authenticator 120 determines that the user who entered the card information as user authentication information in step S10 is actually using the corresponding card. The process of S4 may be performed by determining that it matches the name (owner) of (ie, determining that it is the person's card).

If, in step S12-N, the authentication unit 120 determines that the user who entered the card information as user authentication information in step S10 does not actually match the name (owner) of the corresponding card (that is, not the card of the user). can be considered). Accordingly, the authentication unit 120 is linked with card information previously registered in the customer DB 130 (in particular, card information that matches the card information received in step S10 among a plurality of card information registered in the customer DB) A message for checking whether the card is used without permission may be transmitted to the stored mobile phone number (S13).

At this time, if 'Card unauthorized use is correct' is obtained in response to the message from the terminal corresponding to the mobile phone number that received the message transmitted in step S13 (S14-Y), the authentication unit 120 goes to the unmanned store 3 The process of step S8 may be performed by recognizing that the user who attempted to enter the store tried to enter the unmanned store 3 to use someone else's card, not his or her own card, without permission.

If 'not using the card without permission' is obtained in response to the message from the terminal corresponding to the mobile phone number that received the message transmitted in step S13 (S14-N), the authentication unit 120 goes to the unmanned store 3 Recognizing that the user who attempted to enter the unattended store 3 to purchase a product with the card of the parent or spouse, for example, may be recognized and the process of step S4 may be performed.

As such, when the card information is input as user authentication information in step S1, the authentication unit 120 does not simply generate access authentication result information based on whether or not it matches the customer DB, but the process of steps S12 to S14 (that is, In addition, through the process of determining whether the face image is matched after acquiring the face image, sending a message to check whether the card is used without permission by the mobile phone number and checking the response) By creating , it is possible to prevent the act of the user stealing (using another person's card) to purchase a product in advance before entering the unmanned store 3 .

If, in step S11-N, the authentication unit 120 allows, for example, other user authentication information (eg, face image, QR recognition information) other than card information to be acquired (ie, performs step S12), or other For example, the process of step S6 may be performed.

In addition, when the user authentication information received in step S1 is neither face image information nor card information (S10-N), the authentication unit 120 may check whether the user authentication information received in step S1 is QR recognition information.

At this time, if the received user authentication information is QR recognition information (S15-Y), the authentication unit 120 is a mobile phone number corresponding to the received QR recognition information and the mobile phone information of a plurality of customers pre-registered in the customer DB 130 It is possible to perform user authentication by comparing with those, and by performing user authentication, it is possible to confirm (S3) whether the user corresponding to the received QR recognition information is a pre-registered customer already registered in the customer DB 130.

If the received user authentication information is not QR recognition information (S15-N), the authentication unit 120 may generate a signal requesting the user to input the user authentication information and transmit it to the kiosk 400, and in response Accordingly, the kiosk 400 may display information requesting input of user authentication information on its own screen (S16). Here, the information requesting input of user authentication information may be, for example, 'Please input user authentication information', 'Please perform one of face image recognition, card information recognition, and QR code recognition'.

In the above description, steps S1 to S16 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted if necessary, and the order between the steps may be changed.

4 is a view for explaining a payment process of a product through the kiosk 400 in the unmanned store management apparatus 100 according to an embodiment of the present application.

Referring to FIG. 4 , when the access authentication device 500 receives 'authentication success' as the access authentication result information from the authentication unit 120, it can control the door of the unmanned store 3 in the open (S20) state. . Through this, the user 2 who wants to visit the unmanned store 3 can enter (enter) the unmanned store 3 . In this case, step S20 may be the same step as step S5 in FIG. 3 described above.

The user 2 may select a product to purchase from among products provided in the unmanned store 3 , and may use the kiosk 400 to purchase the selected product.

For example, when a user input is made to the 'Payment' button displayed on one area of the screen of the kiosk 400, the kiosk 400 detects that a payment request has been made from the user for the purchase (purchase) of the product selected by the user ( S21) can be done.

At this time, the kiosk 400 determines whether the user 2 has succeeded in user authentication only with a face image when the user 2 performs user authentication for entry into the unmanned store 3 (ie, when performing user authentication through the authentication unit 120 ). can judge

At this time, if the user authentication is successful only with the face image (S22-Y) (that is, when the process of S2 → S3 → S4 in FIG. 3 is passed), the kiosk 400 displays payment method information on its own screen (screen display unit) may display information requesting the input of (S23).

If it is S22-N, the kiosk 400 may determine whether the user 2 has successfully authenticated the user with the card information when performing user authentication for access to the unmanned store 3 . At this time, if the user authentication is successful with the card information (S24-Y) (that is, when the process of S10 → S11 → S12 → S4 in FIG. 3 is passed, or when the process of S12 → S13 → S14 → S4 is passed), the kiosk 400 may display information for selecting whether to agree to the automatic payment process on the screen display unit (S25).

At this time, when the user agrees to proceed with the automatic payment (S25-Y), the kiosk 400 may proceed with the payment so that automatic payment is made with the card information received when performing user authentication for access (S26) . If the user does not agree to proceed with the automatic payment (S25-N), the kiosk 400 may perform step S23.

If, in step S24-N, the kiosk 400 may determine whether the user 2 has succeeded in user authentication through QR recognition when performing user authentication for entry into the unmanned store 3 .

At this time, if the user authentication is successful by QR recognition (S27-Y) (that is, when the process of S15 → S3 → S4 in FIG. 3 is passed), the kiosk 400 may perform step S25. If it is step S27-N, the kiosk 400 may perform step S23.

Through the above-described process, if the user authenticates the user through card information or QR recognition to enter the unmanned store 3 , the device 100 then transfers the product from the unmanned store 3 to the kiosk 400 . If the user agrees to proceed with the automatic payment when attempting to pay, the payment can be made automatically with the information authenticated at the time of entry (ie, card information or a mobile phone that supports QR recognition). Through this, the device 100 may help the user to perform payment more conveniently.

In the above description, steps S21 to S27 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted if necessary, and the order between the steps may be changed.

The product management unit 140 may perform product management for products sold in the unmanned store 3 . Here, the product management includes the first management of product registration and stocking processing performed in order to register the product stocked in the unmanned store 3 to the kiosk 400, the second management of the product inventory, and the product stocking history. It may include a third management related to product orders and a fourth management related to product orders.

On the basis of the input signal of the manager 1 who manages the unmanned store management apparatus 100, the screen control unit 150 controls the manager terminal 1a, which is a terminal in which the manager 1 possesses a response screen corresponding to the input signal. It may be provided on the display unit. Here, the display unit of the manager terminal 1a may mean a liquid crystal display (LCD), a liquid crystal panel, or the like.

As described above, the system 1000 is a kiosk to which artificial intelligence (AI) and Internet of Things (IOT) are applied in unmanned stores 3 such as convenience stores, supermarkets, and retail stores operated unmanned. (400) and relates to a system capable of managing various devices (3a) in the kiosk (400) as well as the unmanned store (3). That is, the present system 1000 may be referred to as a kiosk management system in which artificial intelligence and the Internet of Things are applied to the unmanned store 3 (unmanned distribution store).

The present system 1000 provides facial recognition (ie, face recognition using a face image), card authentication (for example, credit Card authentication, etc.), or after authenticating the user (customer, 2) with QR recognition (ie, QR code recognition) (especially after performing user authentication in conjunction with the authentication unit of the device), the user (2) ) may include an access authentication device 500 for controlling access. The access authentication device 500 may be otherwise referred to as an access authentication kiosk system, an access authentication system, or the like.

In addition, the present system 1000 may include a kiosk 400 that enables a user (2, customer) to directly scan a product barcode or RF tag in the unmanned store 3 to order and calculate. Such a kiosk 400 may be referred to differently as a self-payment kiosk, a self-payment kiosk system, or the like.

In addition, the IOT environmental complex sensor 200 in the present system 1000 detects environmental information such as temperature, humidity, harmful gas, fine dust, etc. in the unmanned store 3 , and the kiosk 400 and/or the device 100 can be passed to Thereafter, the kiosk 400 and/or the device 100 controls the IOT of air conditioners, air purifiers, lights, cctv, etc. provided in the unmanned store 3 in consideration of the environmental information received from the IOT environmental complex sensor 200 . The operation of the sensor 300 may be automatically controlled remotely.

The device 100 can remotely control various devices 3a, which are all systems to which artificial intelligence and the Internet of Things are applied, in the unmanned store 3 to provide the manager 1 for convenient store operation. The device 100 may provide a store manager app (ie, this app) to the manager terminal 1a so that the manager 1 can more conveniently manage and operate the unmanned store 3 in a remote location. have. The present device 100 and the present system 1000 may enable the implementation of a smart unmanned store 3 (unmanned distribution store).

In general, a kiosk (KIOSK) is a marketing device using a touch screen information delivery system mainly installed in public places. By providing a payment system, it is widely used as a simple order payment agency. The present application applies artificial intelligence and the Internet of Things to such a conventional simple order payment agency device (ie, a conventional kiosk), thereby increasing operational productivity in the unmanned store 3 and users (2, customers) using the unmanned store 3 ), we propose a technology (ie, this system and this device) to maximize satisfaction.

When the user (2, customer) wants to enter the unmanned store 3, the user 2 enters the user authentication information (for example, face/face at least one of face image information, card information, and QR recognition information for recognition) may be provided. The access authentication device 500 performs user authentication through the authentication unit 120 or an authentication server (not shown) in the device 100 based on user authentication information obtained from the user 2 , and performs user authentication Based on the access authentication result information generated based on the result, the door can be opened or kept in the closed state.

At this time, user authentication information (customer personal information) may be obtained by obtaining the consent of the user 2 and data processing, for example, may be set to be discarded after being stored for one month. In other words, the user information registered (stored) in the customer DB 130 (that is, the user's name and user authentication information) is stored in the customer DB 130 only for a preset period (eg, one month), for example, and Thereafter, when a preset period is exceeded, it may be set to be automatically discarded.

The device 100 may set an access authentication time zone in, for example, a store (ie, an unmanned store) operated unattended at night. In other words, the device control management unit 110 of the device 100 automatically performs access authentication for a user in a preset time zone (eg, 10 pm to 7 am) for the access authentication device 500 , for example. It is possible to set the access authentication time zone so that the access authentication time period is set in this way, the access authentication for the user (ie, user authentication for access to the unmanned store) can be automatically performed.

The manager 1 can check and monitor the real-time access status of users (customers) to the unmanned store 3 by accessing the app through the manager terminal 1a. In addition, the manager 1 can remotely monitor the unmanned store 3 through this app and control various devices 3a in the unmanned store 3 . This app, which is an app provided by the device 100 , may be referred to as a store manager app or the like.

The kiosk 400 may provide a function (service) that enables the user 2 to directly order and calculate by scanning the barcode information or RF tag of the product selected by the user (2, customer) in the unmanned store 3 . .

In this case, an AI camera capable of recognizing and tracking the user 2 may be provided in one area (eg, the upper part) of the kiosk 400 . Utilizing such an AI camera, the kiosk 400 is located in a portion of the screen (eg, the upper screen on the screen) of the screen display unit (not shown, for example, a liquid crystal panel, etc.) provided in the kiosk 400 through the AI camera. The acquired user's image (customer image, that is, the user's face image) can be displayed, and through this, attention can be called upon payment.

In addition, the kiosk 400 may transmit, for example, a face image, which is facial information of the user 2 recognized through the AI camera, to the device 100 through the network 4, through which the Gender and age group information of the user 2 identified in real time using the customer DB 130 may be transmitted from the device 100 . Thereafter, the kiosk 400 may provide a customized recommendation event to the user 2 based on the received gender and age group information of the user 2 , for example. That is, the kiosk 400 may provide information of the generated customized recommendation event to the user by displaying the information on the screen display unit of the kiosk 400 . By accessing this app (store manager app) through the manager terminal 1a, the manager 1 can check and monitor the real-time sales status for the unmanned store 3 based on the payment information made through the kiosk 400. have.

In addition, the present system 1000 may include an IOT environment complex sensor 200 and an IOT control sensor 300 to provide a system for automatically operating a comfortable environment in the unmanned store 3 . Environmental information (ie, temperature, humidity, harmful gas, fine dust, etc.) in the unmanned store detected (measured) through the IOT environmental complex sensor 200 in the unmanned store 3 may be transmitted to the kiosk 400 . Thereafter, the kiosk 400 may automatically control the IOT control sensors (eg, air conditioners, air purifiers, lights, cctv, etc.) in the unmanned store 3 in consideration of the received environmental information.

5 is a schematic diagram for explaining an operating method of various devices 3a in the unmanned store management system 1000 according to an embodiment of the present application.

Referring to FIG. 5 , the operation/control/operation method of the access authentication system may be simply as follows. Here, the access authentication system may include an access authentication device 500 (an access authentication kiosk) and an entrance door (smart door lock) of the unmanned store 3 .

The access authentication device (500, access authentication kiosk) obtains user authentication information (at least one of card authentication information, face recognition information, and QR authentication information) from the user and verifies it with an authentication server through a network (eg, the Internet) You can request information. Here, the authentication server may mean a case in which the authentication unit 120 is separated from the device 100 and provided in the form of a server. Thereafter, the authentication server may provide authentication confirmation information to the access authentication device 500 in response to the authentication confirmation information request. The authentication confirmation information may refer to the above-described access authentication result information (authentication success or authentication failure). The access authentication device 500 may control opening and closing of an entrance door (for example, a smart door lock) in the unmanned store 3 based on the authentication confirmation information.

The explanation of the kiosk operation method applied with AI and IOT is as follows. The kiosk 400 may acquire a face image of a user who wants to make a payment with the kiosk through a camera (AI camera) provided in one area of the kiosk 400 . The kiosk 400 may request facial recognition information from the AI server by transmitting the acquired user's face image to the AI server through the network 4 (eg, the Internet, etc.). In this case, the AI server may mean, for example, a case in which the customer DB 130 is separated from the present device 100 and provided in the form of a server. In response to the request for face recognition information, the AI server checks whether a face image matching the user's face image obtained from the kiosk 400 exists in the AI server, and if there is, the user's information corresponding to the matching face image It can be transmitted to the kiosk 400 . Here, the user information corresponding to the matching face image may include, for example, information such as gender and age. Thereafter, the kiosk 400 may generate a customized recommendation event customized to the user based on the obtained user information (gender, age, etc.) and display it on the screen of the kiosk 400 . Through this, the kiosk 400 may provide a customized recommendation event to the user.

The description of the operation method of the store environment sensor and control system is as follows. This may be referred to differently as an operation/operation operation/control/operation method of the IOT environment complex sensor 200 and the IOT control sensor 300 . In brief, the kiosk 400 receives environmental information from the IOT environmental complex sensor 200 such as a temperature sensor, and according to the received environmental information, a comfortable environment in the unmanned store 3 (environment such as appropriate temperature and humidity) of For maintenance, it is possible to automatically control the IOT control sensors 300 such as air conditioners, air purifiers, and lights. In this case, the kiosk 400 transmits a request for environment information setting details to the device 100 when the environment information is received, for example, and may receive environment information setting details information from the device 100 in response thereto. Here, the environmental information setting details information may mean the set values of appropriate environmental information (temperature, humidity, etc.) set for the unmanned store 3, which is set to various values by the manager 1, for example. can be set.

6 is a view showing an example of a screen display of the kiosk 400 in the unmanned store management system 1000 according to an embodiment of the present application.

Referring to FIG. 6 , the kiosk 400 may display (expose) the screen shown in FIG. 6 as an example on the screen of the screen display unit provided therein. The user 2 may press a start button on the screen of the kiosk 400 in order to pay for a product that he/she wants to purchase among products sold in the unmanned store 3 . Thereafter, the user 2 clicks on a product to purchase from the product list displayed on the screen of the kiosk 400 , or applies a product barcode to the product selected by the user from among the products displayed in the unmanned store 3 to the barcode (of the kiosk). A barcode scanning unit provided in one area) can be scanned. Thereafter, the user 2 may check the scanned list of products through the screen of the kiosk 400 , and at this time, the captured image of the user performing payment may be displayed on the screen of the kiosk 400 . After checking the product list, the user 2 may perform payment by pressing a payment button and selecting a payment method.

Meanwhile, the product management unit 140 may perform product management for products sold in the unmanned store 3 as described above. The manager 1 may remotely perform product management for products sold in the unmanned store 3 by accessing the app through the manager terminal 1a. Here, as the type of product management, the first to fourth management and the like may be included as described above.

7 to 11 are diagrams for explaining an example in which the screen of the manager terminal 1a is controlled by the screen controller 150 of the unmanned store management apparatus 100 according to an embodiment of the present application. That is, the screens shown in FIGS. 7 to 11 may be displayed (expressed, exposed) on the display unit of the manager terminal 1a by the device 100 .

7 to 11 , the screen control unit 150 displays a screen as shown in FIGS. 7 to 11 on the display unit so that the manager 1 can efficiently manage the unmanned store 3 remotely. You can control the display.

Herein, the manager 1 is, for example, a manager who manages the unmanned store 3 , and may mean a store owner of the unmanned store 3 , but is not limited thereto. As another example, the system 1000 may include a store owner separately from the manager 1 , and in this case, the apparatus 100 is interlocked with the store owner's terminal (storekeeper terminal) through a network as a store owner terminal in FIG. 7 . 11 to 11 may be displayed, and in this case, the manager 1 may mean a manager who collectively manages the present system 1000 separately from the store owner.

As shown in FIG. 7 as an example, in relation to the management (product management) of the product in the unmanned store 3 by the manager 1 accessing the app through the manager terminal 1a, the device 100 is It can be provided to perform product registration and warehousing related management, product inventory management, product stocking history management, product order management, and the like. Also, the device 100 may allow the manager 1 to set the operating hours (operation start time, operation end time) of the unmanned store 3 . Through this, the user's access to the unmanned store 3 may be enabled only for a set operating time. In addition, the device 100 may provide the manager 1 to check the history (detail) of users (heads) of entering and leaving the unmanned store 3 .

In addition, the screen control unit 150 may display the manager menu as shown in FIG. 8 on the manager terminal 1a, for example. The manager menu may include items for checking/setting closing, card sales, cash sales, sales screen, and the like. The manager 1 may check the settlement result and work deadline, etc. through the manager terminal 1a. In addition, the manager 1 may check the settlement result for the kiosk 400 through the manager terminal 1a, and set a work deadline.

In addition, as shown in FIG. 9 as an example, the manager 1 registers (product registration) products sold in the unmanned store 3 through the manager terminal 1a with respect to the kiosk 400 and performs the stocking process. can do.

Also, as shown in FIG. 10 , the manager 1 may perform product management, sales management, etc. in relation to the unmanned store 3 through the manager terminal 1a and check (inquire). In addition, as shown in FIG. 11 , the manager 1 may perform product stock management and product inventory harmonization management in relation to store inventory management through the manager terminal 1a . In this case, FIGS. 10 and 11 may be expressed differently as examples of a screen display provided on the screen of the store owner terminal.

As described above, the present application (ie, the present system 1000 and the present device 100) proposes smart solution technology for unmanned distribution stores, kiosk unmanned distribution technology, and the like. Through this system 1000, we provide AWS cloud system (eg, high availability and secure cloud service) technology, customer mobile ordering (eg, mobile ordering service in cafes and restaurants) function, store employee mobile operation management (eg. (Raw, mobile app sales, menu, inventory management, etc.) technology, various functions for customers (e.g., customer points, easy payment, employee discount, etc.), convenience store unmanned self-checkout kiosk technology, screen and voice guidance service in 4 languages It is possible to provide a provision function, major simple payment (pay) application technology, and the like.

This application can increase sales (increase in unit price) compared to the operation of the incentives when operating the kiosk 400 . In addition, the present hospital can increase the selection ratio of additional options by exposing the options (photos) at the bottom when selecting the menu, and reduce the psychological burden compared to ordering in person with the staff. In addition, the present application can improve the sales turnover by increasing the cooking time by reducing the customer response time through the provision of the present system 1000 and the present apparatus 100 .

In addition, the present application can enhance hygiene (preferably non-face-to-face) and solve the hygiene problem by completely separating cooking and ordering so that you can focus on cooking only. In addition, the present application may provide for non-face-to-face purchase (purchase) of goods, food, etc. in all fields.

In addition, the present application can provide a labor cost reduction effect, and can be made smart store management. In addition, the hospital can provide functions such as improved response to foreigners with foreign language services, simple and smart store operation management and integrated sales management through the store manager App, and smart ordering using mobile ordering.

In addition, the present application can provide an unmanned retail store solution, an access authentication system, an IOT solution, a restaurant/service store order solution, KDS, factory DID, text guide solution, and the like through the system 1000 . Also, the present application may provide a customer marketing point system, a mobile menu board, and a mobile order payment function through the system 1000 . The present application can manage the present system 1000 so that rapid failure handling support and stable operation support of the system are guaranteed.

In addition, the present system 1000 may provide, for example, a convenient point system that the user earns with a phone number and uses. In addition, the present application may provide an order / payment / point integrated operation technology at the kiosk affiliate. For example, we can provide points that can be conveniently accumulated/used at the kiosk with a phone number, and can provide SMS/LMS sending services, and secure customer DB 130 management and various services through encryption of AWS customer information. Statistical analysis function can be provided, and merchant point operation management can be provided with low operating fee.

Herein, the user 2 through this app provided through the device 100, for example, customer point history inquiry, customer points available, information guidance for each nearby store, real-time sales menu information for each store, visit store event You can receive services such as /discount information guide.

In addition, in the present application, the manager (1) (for example, back office / store owner), through this app provided through the device 100, franchise / store service accrual rate, point accrual rate for each event period, point accumulation for each store / You can receive a service to check usage status, point accumulation/use status by customer, SMS/push marketing by target customer, etc.

In addition, in the present application, the kiosk 400 may provide services (functions), such as service subscription, point accumulation, point inquiry, point use, and output of receipt details related to product purchase, to the user 2 as an example. The user 2 may accumulate or use points by, for example, pressing his/her phone number (mobile phone number) in the kiosk 400 . For example, a new member can easily join directly at the kiosk 400 . The user 2 can use the points after entering the amount to be used among the points possessed by the user.

The manager 1 can remotely operate the unmanned store 3 simply by accessing the app through the manager terminal 1a. For example, the manager 1 through the device 100, real-time product registration and stocking management function, real-time inventory confirmation function, kiosk 400 restart function with the app, product arrival history and change history confirmation function, sales confirmation app You can receive direct connection function and CCTV direct connection function used in the app.

12 is a diagram illustrating an example of an access authentication device 500 applied to the unmanned store management system 1000 according to an embodiment of the present application.

Referring to FIG. 12 , the access authentication device 500 applied to the present system 1000 has, for example, the same specifications as those shown in FIG. 12 , and includes a night motion sensor light, an IC card reader, and the like as main components. , and a more detailed description thereof will be omitted.

On the other hand, with respect to the face image of the user 2 acquired through the AI camera provided in one area of the kiosk 400 and/or the authentication camera provided in one area of the access authentication device 500, face recognition (facial recognition) In order to perform user authentication as an example, the following technology may be applied. That is, when a face image is received as user authentication information, for example, the authenticator 120 may perform user authentication on the received face image by using the following technique.

The authenticator 120 may use open source computer vision (OpenCv) to perform face recognition-based user authentication on the face image of the user received from the user 2 , for example. OpenCv refers to an open source library that is free for academic and commercial use as it is distributed under the Berkeley Software Distribution (BSD) license, and can be used for various image and video processing.

The authenticator 120 generates a library suitable for Qt by compiling OpenCv using Qt5.8.0, which is the development language of the device 100, for example, and uses a face recognition algorithm Harr Cascades to create a face (that is, a plurality of By performing machine learning on the face images of users), an object (face object) in the received face image may be detected and used.

The authenticator 120 may extract (detect) the user's face object in the face image through the learned machine learning by learning a plurality of (eg, 686) sampled face images without a mask as an example through machine learning. have. In addition, the authenticator 120 extracts (detects) a user's face object in the face image through machine learning by learning a plurality of (eg, 690, for example) sampling mask face images wearing a mask through machine learning. can do.

In this case, the machine learning considered may mean a deep learning model, an artificial intelligence (AI) algorithm model, a machine learning model, a neural network model (an artificial neural network model), a neurofuzzy model, and the like. In addition, machine learning is exemplarily a convolutional neural network (CNN, convolutional neural network), a recurrent neural network (RNN), a deep neural network, etc. A neural network model can be applied.

Also, when performing face recognition-based user authentication on the received user's face image, the authenticator 120 may use the Haar Cascades algorithm as an example of a face recognition algorithm. Haar Cascades refers to an object detection algorithm based on machine learning, which can be understood more easily by referring to the paper “Rapid Object Detection using a Boosted Cascade of Simple Features” submitted in 2001 by Paul Viola and Michael Jones. A more detailed description of the algorithm will be omitted below.

Briefly, this paper presented a method for detecting an object in a video or image based on a feature. There are advantages.

The face recognition process using the Harr Cascade face recognition algorithm can be largely comprised of four steps as follows. Here, among the four steps, step 1 may be the same as Haar Feature Selection, step 2 may be Integral Images, step 3 may be Adaboost Training, and step 4 may be the same as Cascade Classifier.

In step 1 (Haar Feature Selection), Harr Features can be calculated from the image. At this time, one image or one frame of a video can be imaged and the Harr Features can be calculated by scanning the whole. For example, when using a 24x24 image, more than 160,000 Harr Features can be generated. Harr Features can use the difference in the sum of pixels within the area of adjacent rectangles to move the position while scanning the image. Three methods for calculating the difference of the sum of pixels using a rectangle may be as shown in FIG. 13 .

13 is a diagram for explaining a method of calculating Harr Features that is considered when the authentication unit 120 in the unmanned store management apparatus 100 according to an embodiment of the present application performs user authentication on a face image.

Referring to FIG. 13 , the authenticator 120 may obtain the difference of the sum of pixels using a quadrangle when calculating the HAR feature using three methods as (a) to (c).

At this time, (a) in FIG. 13 is an edge features method, which means a method obtained by summing pixels inside two rectangular areas and subtracting the sum of the white areas from the sum of the black areas. 13(b) shows the Line Features method, which means a method obtained by subtracting the sum of pixels inside two white rectangular areas outside from the sum of pixels inside the central black rectangular area inside three rectangular areas. 13( c ) is a Four-rectangle features method, which means a method for obtaining a difference between regions located on diagonal lines in four rectangular regions.

Meanwhile, in the second step (Integral Images), integral images may be used to quickly obtain the sum of the square difference (black/white) pixels obtained in the first step.

14 and 15 are diagrams for explaining an integral image that the authentication unit 120 in the unmanned store management apparatus 100 according to an embodiment of the present application considers when performing face recognition.

Referring to FIG. 14 , in order to use the integral image, the authenticator 120 may add 1 to the width and height as shown in FIG. 14 and fill the left end and the upper end with 0 as shown in FIG. 14 . In addition, after obtaining the sum of internal values by designating a region in the existing image, the sum may be input at a position corresponding to the lower right pixel of the region specified in the original image in the integral image.

Also, the integral image may be used as shown in FIG. 15 as an example. Specifically, when an internal value is obtained by designating a region in the existing image shown on the left in FIG. 15 , the sum of the regions may be calculated using pixel values of four corresponding regions in the integral image. Also, in the integral image, the values of the upper and lower pixels may be subtracted from the values of the lower-right pixel of the corresponding region, and the pixels in the diagonal direction may be added.

In step 3 (Adaboost Training), for example, an image is extracted by connecting a device of a webcam to OpenCV, and can be calculated using a Harr Feature selected from the extracted image. In this case, as an example, more than 160,000 features may be detected, and it is necessary to select a meaningful feature that helps to detect a face among them. In this case, the authenticator 120 may use Adaboost to improve performance. For example, most of the features obtained above may be meaningless features.

16 is a view for explaining meaningful features that are considered when the authentication unit 120 in the unmanned store management apparatus 100 according to an embodiment of the present application performs face recognition.

Referring to FIG. 16 , for example, the two Harr features of the figure shown in FIG. 16 may be said to be meaningful features only around the eyes. That is, two Harr features can be calculated as meaningful values only near the eyes. In this case, in the case of a feature having a black rectangular area and a white rectangular area in the horizontal direction, the characteristic that the eyes are darker than the nose and cheeks may be used. In the case of a feature in which there is a white rectangular area in the vertical direction and a black rectangular area on the left and right, the characteristic that the eyes on both sides are darker than the nose in the center can be used.

Adaboost can be made through the following process. Specifically, a feature can be applied to all training images to select the optimal feature. Thereafter, for each feature, an optimal threshold for classifying an image including a face and an image without a face may be found. Thereafter, if the difference between the white area and the black area is equal to or greater than a certain threshold, it may be an algorithm that determines that it is a feature for a face, and determines that there is little difference between the two areas in a non-face area. In this case, since there is a possibility of misclassification, it is necessary to select a feature with a low error rate. Through Adaboost, for example, 160,000 features can be reduced to 6,000 features.

In step 4 (Cascade Classifier), the user's face in the received face image can be detected by applying 6,000 Harr features to the prepared image (24x24). However, if the Har feature is applied to the entire received face image, the amount of calculation may increase, which may be inefficient. Therefore, the authenticator 120 checks step by step whether the area with the current window is a face area, considering that most of the space in the received face image is an area without a face (ie, a background area other than a face). can be used At this time, it is determined whether it is a face region in a short time in a lower stage, and an operation that takes a longer time can be performed as it goes to an upper stage, and this method may be referred to as a cascade classifier.

Accordingly, whenever the window moves over the image, the authenticator 120 may group and use the 6,000 features as a group of several steps without applying all of the features. When it is determined that the feature is not a face region in the first step, the window is moved to the next position. If it is determined that the feature is a face region in the feature in the first step, the feature of the next step can be applied to the location where the current window is located. The authenticator 120 may perform face recognition of the user's face in the received face image by performing the processes of steps 1 to 4 described above.

Meanwhile, the device control management unit 110 of the device 100 may use the following technology, for example, when controlling the IOT (ie, controlling the operation of various devices in an unmanned store).

The device control management unit 110 may use MODEBUS-TCP technology for IOT control. MODBUS-TCP is a communication protocol frequently used in the field of industrial fieldbus, and refers to a communication method designed for the interface between upper and lower devices on a TCP/IP network.

The MODBUS-TCP communication standard divides the roles of the master (Client) and the slave (Server), and the slave (Server) can respond to the data requested by the master (Client). Mainly, an industrial touch HMI device or a higher level device such as a PC may be located in the master (Client). In addition, TCPPORT or PLC may be located in the slave (Server).

The slave (Server) is in a passive position that only performs the operations requested by the upper device, whereas on the master (Client) side, it is necessary to actively handle the slave device, such as reading the desired data or writing the desired data. there is

In the present system 1000 as an example, the kiosk 400 may correspond to a master (client), and IOT equipment (ie, IOT control sensor, IOT environment complex sensor, etc.) may correspond to a slave (Server). The kiosk 400 may be a MODBUS TCP Client, and the IOT equipment may be a PLC MODBUS module (motor, sensor, etc.) as a MODBUS TCP Server. When the kiosk makes a request to the IOT device, the IOT device may respond to the request. In this case, communication (data transmission/reception) may be performed between the kiosk and the IOT equipment based on the MODBUS-TCP protocol.

The description of the MODEBUS protocol is as follows. For normal data communication between the master (Client) and the slave (Server), it is necessary to transmit a data packet conforming to the MODBUS-TCP frame structure. The MODBUS-TCP frame structure may be similar to the general-purpose MODBUS (RTU, ASCII) frame structure. This can be more easily understood with reference to FIG. 17 .

17 is a view for explaining the MODEBUS protocol considered in the unmanned store management system 1000 according to an embodiment of the present application.

Referring to FIG. 17, MODBUS-TCP can access the memory of the slave server by using the frame-type protocol as shown in FIG. 17 to read the data value of the corresponding address or write it as a desired value.

The description of MBAP Header is as follows. MODBUS-TCP consists of MBAP (Modbus Application Protocol), followed by Function code and Data in order. MBAP is 7 bytes in total and represents the following byte values.

Specifically, in the case of Transaction ID [2Bytes], the master (Client) increments by 1 at the start of communication from the initial 0x0000 value, and the slave (Server) can copy and use the value as it is, and the corresponding byte is stored in the request/response. It may mean a part that confirms whether work has been done.

In the case of Protocol ID [2Bytes], it indicates the ID of the protocol, and a fixed value of 0x0000 can be used for MODBUS-TCP.

In the case of Length [2Bytes], it indicates the length from the position of the Length field to the end of the frame, that is, it indicates the number of bytes from Unit ID to the end of Data.

In the case of Unit ID [1 Byte], it indicates information that identifies a connected slave of a communication line other than TPC/IP, and Tcpport may be fixed as 0x01.

18 is a view for explaining a function code in relation to the MODEBUS-TCP technology considered in the unmanned store management system 1000 according to an embodiment of the present application.

Referring to FIG. 18 , a description of the function code is as follows. Function Code means the command set code provided by the Modbus protocol. This is a service that can read or write values to the slave memory (Coil, Register) using the function code. In fact, a value between 1 and 127 of the function code code is used, but in TCPPORT, 1, 2, 4, 5, 6, 15 , can support 16 values.

MODBUS data model can be divided into a total of 4 types based on input and output, bit-wise access and word-unit access. Data models may vary from manufacturer to manufacturer. For example, as shown in the upper figure of FIG. 18 , each data block can be designated for each data type and used as four data blocks, or two data blocks can be used by dividing a bit area and a word area.

Referring to the middle figure of FIG. 18, TCPPORT is divided into two data memories into a 16-bit word area (Resisters) and a bit area (Coils). The memory referred to here refers to the memory of the Slave (server) device, and the Master (Client) can read the memory of the Slave (server) device using the above function code or change it to a desired value. In addition, as shown in the lower figure of FIG. 18 , it is divided into which memory to access and which work to perform (Read, Write) according to the corresponding function code.

The description of data in relation to MODEBUS-TCP technology is as follows. The structure of data may vary slightly depending on the function code. Data can basically have the form of Start Address, Length, Byte Count, and Data.

Start Address [2Bytes] indicates the start address of the memory to be accessed (eg, 0x40 0x01 when accessing address 0x4001). Length [2Bytes] indicates the length of reading or writing values from the start address. Byte Count [1Bytes] indicates the number of bytes of memory data according to Request and Response, in other words, the number of bytes of memory data to be read or written. DATA [N Byte] indicates the value of memory data according to Request and Response, in other words, the memory value to be read or written.

As described above, the device 100 may control various IOT equipment (devices) included in the system 1000 using MODEBUS-TCP technology, and may transmit/receive data to/from them.

On the other hand, the IOT control sensor 300 may include a plurality of cctv for taking images in the unmanned store (3). The device control manager 110 may control the operation of a plurality of cctvs. In addition, the screen control unit 150, for example, when a request to provide a cctv image is made from the manager 1, by displaying at least some of the images obtained from a plurality of cctv in response to the screen of the manager terminal 1a, the manager ( 1) can be monitored (confirmed).

The device control management unit 110, if there is a second cctv that shoots a second image having an overlapping area at least partially overlapping with the first image of the first cctv among the plurality of cctvs, but in a situation in which acquisition of the first image is impossible , an image acquisition angle of the second cctv may be differently controlled in order to receive an alternative image of the first image from the second cctv.

Here, the situation in which the acquisition of the first image is impossible may mean a case in which the apparatus 100 cannot acquire the first image, for example, because the first cctv is broken or the like.

As such, when it is difficult to obtain an image through the first cctv, the device control manager 110 may receive an alternative image of the first image from the second cctv by differently controlling the angle (image acquisition angle) of the second cctv.

In this case, when controlling the image acquisition angle of the second cctv, the device control management unit 110 determines the size of the overlapping area between the first image and the second image before controlling the angle of the second cctv (that is, the second cctv). Based on the size of the overlapping area before controlling the angle of , the angle (image acquisition angle) of the second cctv may be controlled in a direction in which the size of the overlapping area between the first image and the second image becomes larger.

According to this, for example, when the size of the overlapping area in the second image acquired through the second cctv before the control of the image acquisition angle of the second cctv is performed is referred to as the first size, the image acquisition angle of the second cctv The size of the overlapping area in the second image acquired through the second cctv after the control of is made may be a second size larger than the first size. At this time, the second image acquired through the second cctv after the control of the image acquisition angle of the second cctv is performed is the angle of image acquisition (that is, compared to before the control of the image acquisition angle of the second cctv is performed) As an image in which the photographing angle) is changed, it may be referred to as an angle-changed image (particularly, an angle-changed second image). That is, the image acquired through the second cctv before the image acquisition angle is controlled differently is referred to as a second image, whereas the image acquired through the second cctv after the image acquisition angle is controlled differently is the angle change second It may be referred to as an image.

In other words, when it is impossible to obtain the first image from the first cctv, the device control management unit 110 controls the image acquisition angle of the second cctv differently based on the size of the overlapping area between the first image and the second image. , the angle (image acquisition angle, shooting angle) of the second cctv may be controlled in a direction in which the size of the overlapping region between the first image and the second image becomes larger.

According to this, the size of the overlapping area in the angle change image (angle change second image) acquired through the second CCTV after the control (change control) of the image acquisition angle of the second cctv is made, that is, the overlap with the first image The size of the area may be larger than the size of the overlapping area in the second image before the control (change control) of the image acquisition angle of the second CCTV is performed.

The device control management unit 110 receives the angle change image (particularly, the angle change second image) obtained from the second cctv in which the control (change control) of the image acquisition angle is made as an alternative image of the first image, and the manager terminal 1a ) can be provided.

This device 100, for example, even in a situation in which the first image of the first cctv requested by the manager 1 cannot be provided to the manager terminal 1a, the overlapping area at least partially overlapping with the corresponding image (the first image) By controlling the angle of the cctv (that is, the second cctv) shooting the The cctv (ie, the second cctv) may obtain an alternative image for the first image and provide it to the manager 1 .

That is, the apparatus 100 may change the image acquisition angle of the second cctv disposed adjacent to the first cctv even in a situation in which the administrator 1 cannot provide the administrator with the first image of the first cctv requested by the administrator 1, for example. By controlling it, an image similar to the first image (alternative image) can be provided to the user instead, thereby increasing the satisfaction of the image providing service for the manager 1 .

In addition, the device control management unit 110, when there are a plurality of overlapping cctv for photographing an image having an overlapping area at least partially overlapping with the first image in the plurality of cctvs, a plurality of overlapping images to obtain an alternate image of the first image It is possible to selectively control the angle of at least some of the CCTVs being photographed. In this case, the device control management unit 110 determines that the number of combinations (number of combinations) of at least some selectively controllable CCTVs among a plurality of overlapping CCTVs is a plurality in order to obtain an alternative image, and the number of controlled CCTVs is the minimum. The angle can be controlled by selecting at least some cctv corresponding to the combination.

That is, the device control management unit 110, when there are a plurality of overlapping cctvs for photographing an image having an overlapping area at least partially overlapping with the first image in the plurality of cctvs, in order to receive a replacement image of the first image, a plurality of An image acquisition angle of at least some of the overlapping CCTVs corresponding to a combination in which the number of controlled CCTVs among the overlapping CCTVs is minimized may be controlled differently. A description thereof may be more easily understood with reference to FIG. 19 .

19 is a view for explaining a case in which the device control management unit 110 in the unmanned store management apparatus 100 according to an embodiment of the present application controls a plurality of cctvs in the IOT control sensor 300 . In particular, FIG. 19 is a view for explaining a case in which the device control management unit 110 controls the image acquisition angle of at least some overlapping CCTVs corresponding to a combination in which the number of controlled CCTVs among a plurality of overlapping CCTVs is minimized. .

Referring to FIG. 19, for example, as shown in FIG. 19 (a), among a plurality of cctvs, an image is acquired so that at least a partial region of the first image I1 acquired through the first cctv overlaps. Assume that three cctvs exist (ie, second to fourth cctvs exist).

Here, the second image I2 is acquired such that the second cctv includes an overlapping region r1 overlapping with at least a partial region of the first image I1, and the third cctv is at least a part of the other at least part of the first image I1 A third image I3 is acquired to include an overlapping region r2 overlapping the region, and a fourth image so that the fourth cctv includes an overlapping region r3 overlapping with another at least partial region of the first image I1 Suppose we obtain (I4).

Assume that the manager 1 has requested the device 100 to provide the first image I1 of the first cctv. At this time, when the device 100 is in a state in which the acquisition of the first image is possible, in response to the request for provision of the manager (in response to the request for provision of the first image), the first image obtained from the first cctv is transferred to the manager terminal ( 1a) can be provided.

However, if the device 100 is in a state in which it is impossible to acquire the first image (for example, the first cctv is broken and the first image cannot be acquired, and accordingly, the first image cannot be provided to the administrator ), the device control management unit 110 obtains an image including at least a partial area of the first image I1 as an overlapping area within a plurality of CCTVs in order to provide an alternative image of the first image. existence can be determined.

In this case, it is assumed that there are overlapping CCTVs in the plurality of CCTVs, but a plurality of such overlapping CCTVs exist. Here, according to the example shown in (a) of FIG. 19 , the plurality of overlapping cctvs includes a second cctv acquiring a second image I2 , a third cctv acquiring a third image I3 , and a fourth image It may mean the fourth cctv to obtain (I4).

As such, in providing an alternative image of the first image I1, for example, when a plurality of overlapping cctvs (second cctv to fourth cctv) exist in a plurality of cctvs, the device control management unit 110 controls the first In order to obtain an alternative image of the image I1 , angles of at least some of the plurality of overlapping CCTVs (the second to fourth CCTVs) may be selectively controlled.

In this case, the device control manager 110 may control the angles of the second cctv and the third cctv to obtain an alternative image of the first image I1 as shown in FIG. 19(b) , for example. . In this case, the device control management unit 110 transmits the angle change image (angle change second image, I2') through the second cctv with angle control and the angle change image (angle change third) through the third cctv with angle control. image, I3') can be acquired (received).

Thereafter, the editing unit (not shown) in the device 100 acquires, based on the first image I1 , at least a partial region of the acquired second cctv angle change image (angle change second image, I2′) An alternative image of the first image I1 may be acquired (generated) by combining at least some regions of the angle change image (the angle change third image, I3') of the third cctv. That is, although not shown in the drawings, the apparatus 100 may include an editing unit (not shown) for generating (acquiring) an alternative image.

Specifically, the editing unit (not shown) is a first partial image corresponding to the overlap region r1'' overlapping the first image I1 within the angle change image (angle change second image, I2') of the second cctv can be extracted, and a second partial image corresponding to the overlap region r2'' overlapping the first image I1 within the angle change image (angle change third image, I3') of the third CCTV can be extracted. . Thereafter, the editing unit (not shown) may acquire (generate) an alternative image of the first image I1 by connecting (combining) the extracted first partial image and the second partial image.

In addition, as another example, as shown in FIG. 19( c ), the device control management unit 110 , the second cctv, the third cctv, and the fourth cctv image to obtain an alternative image of the first image I1 . You can control the acquisition angle. In this case, the device control management unit 1100 performs an angle change image (angle change second image, I2'') through a second cctv with angle control and an angle change image (angle change third through a third cctv with angle control). It is possible to acquire (receive) an angle-changed image (angle-changed fourth image, I4'') through the fourth cctv in which the image, I3'') and angle control are performed.

Then, the editing unit (not shown), based on the first image I1, at least a partial area of the acquired second cctv angle change image (angle changed second image, I2'') and the acquired third cctv Replacing the first image by combining at least a partial area of the angle change image (third angle change image, I3'') with at least some area of the angle change image (angle change fourth image, I4'') of the fourth cctv with each other Images can be acquired (generated).

Specifically, the editing unit (not shown) configures the first image corresponding to the overlapping area r1''' overlapping the first image I1 within the angle change image (angle change second image, I2'') of the second cctv. A partial image is extracted, and a second partial image corresponding to the overlap region (r2''') overlapping the first image I1 within the angle change image (angle change third image, I3'') of the third cctv Extract and extract the third partial image corresponding to the overlap region (r3''') overlapping the first image I1 within the angle change image (angle change fourth image, I4'') of the fourth cctv have.

Thereafter, the editing unit (not shown) may acquire (generate) an alternative image of the first image by connecting (combining) the extracted first partial image, second partial image, and third partial image.

In this case, as in the example of FIG. 19B , an example of obtaining an alternative image through angle control of two CCTVs (ie, the second CCTV and the third CCTV) may be referred to as an example of the first case. In addition, as in the example of FIG. 19C , an example of obtaining an alternative image through angle control of three cctvs (ie, the second cctv to the fourth cctv) may be referred to as an example of the second case.

According to this, when a plurality of overlapping cctvs (second cctv to fourth cctv) exist in the plurality of cctvs, the device control management unit 110 selectively selects one of the plurality of overlapping cctvs as in the first case example. Alternative images may be acquired by controlling the image acquisition angles of the second cctv and the th cctv, respectively, or as another example, all of the second to fourth cctvs selectively among a plurality of overlapping cctvs as in the second case example An alternative image can be obtained by controlling the image acquisition angle of .

According to this example of the present application, in the present device 100, the number of combinations (the number of combinations) of at least some of the plurality of overlapping CCTVs selectively controllable among the plurality of overlapping CCTVs is two (the first case) in order to obtain an alternative image. two cases including an example of and an example of the second case). That is, according to an example of the present application, it can be said that the number of selectively controllable combinations of at least some cctvs is a total of two including the example of the first case and the example of the second case.

In this way, when the number of combinations (the number of combinations) of at least some cctvs that are selectively controllable among a plurality of overlapping cctvs to obtain an alternative image is plural (including the first case and the second case) , the device control manager 1100 may select at least some cctvs (overlapping cctvs) corresponding to a combination in which the number of controlled cctvs is minimized to differently control the image acquisition angle.

In the example of the first case, the number of cctvs controlled to acquire the alternate image may be two, whereas in the example of the second case, the number of cctvs controlled to acquire the alternate image may be three. In this case, since the number of cctvs controlled to obtain an alternate image is smaller in the first case than in the second case, the device control management unit 110 controls as an example as in the first case among the two cases. Alternative image of the first image I1 by selecting the second cctv and the third cctv as at least some cctvs corresponding to the combination in which the number of cctvs is minimized, and controlling the image acquisition angles of the second cctv and the third cctv differently can be obtained.

This device 100, for example, in providing an alternative image of the first image in a situation in which the administrator 1 cannot provide the first image of the first cctv requested by the administrator 1 to the administrator 1, the generation of the alternate image It is possible to provide an image similar to or corresponding to the first image (alternative image) while minimizing the number of cctvs (overlapping cctvs) that are controlled for the purpose.

That is, the apparatus 100 generates an alternative image based on the angle change image obtained through the overlapping cctv controlled with a minimum number, thereby reducing the image editing time for generating the replacement image (faster of the replacement image). can be created), and the quality of the image (replacement image of the first image) provided to the manager 1 can be improved by combining images using a small number of images.

In addition, the screen control unit 150 displays the locations of a plurality of devices managed by the device control management unit 110 on the modeling unmanned store modeled in three dimensions in the unmanned store 3 as a mark to display the screen of the manager terminal 1a. can be expressed in

In this case, the plurality of devices managed by the device control management unit 110 may mean various devices 3a provided in the unmanned store 3 . Here, the various devices 3a include a plurality of IOT environmental complex sensors 200 , a plurality of IOT control sensors 300 , a plurality of kiosks 400 , an access authentication device 500 , a plurality of electronic price indicators 600 , etc. this may be included.

When displaying the positions of a plurality of devices as a mark in the modeling unmanned store (on the modeling unmanned store data), the screen control unit 150 recognizes or recognizes the mark of the abnormal device in which the abnormal signal is detected among the plurality of devices compared to the mark of the normal device. It can be displayed in a form that is easy to identify.

At this time, the screen control unit 150 determines a device that the device 100 makes a data request from among the plurality of devices but does not respond to the request as an abnormal device (that is, an abnormal signal is detected due to a malfunction or the like). device) can be identified.

In addition, displaying the mark in a form that is easy to recognize or identify may mean displaying at least one of a visually emphasized form, an expanded form, a form accompanying an auditory effect, and a dynamic animation form.

In this way, the device 100 displays the locations of a plurality of devices managed by the device control management unit 110 as a mark on the modeling unattended store 3 dimensionally modeling the unmanned store 3 managed by the manager 1, but , in particular, it is possible to provide a mark of an abnormal device among a plurality of devices in a form that is easier to recognize or identify than a mark of a normal device. This device 100 allows the manager to recognize the existence of a problematic device (such as a broken device) among a plurality of devices existing in the unmanned store 3 operated by the manager as well as the problematic device. It can be made to recognize the position of the position more intuitively and quickly.

Also, the screen controller 150 may determine whether the detected (identified) abnormal device among the plurality of devices is a replacement target abnormal device by satisfying a preset replacement condition. At this time, when it is determined that the detected abnormal device is the abnormal device to be replaced, the screen controller 150 controls to display replacement abnormal device recommendation information for recommending replacement of the replacement target abnormal device on the screen of the manager terminal 1a. can

Here, the preset replacement condition may mean a condition in which an abnormal device is detected (identified) as a preset number of times (eg, 5 times) within a preset period (eg, one month). That is, for example, when the first device (for example, a temperature sensor, which is one of a plurality of IOT environmental complex sensors) among a plurality of devices is detected as an abnormal device having a problem more than 5 times within one month, the screen control unit 150 It is determined that the first device is an abnormal device to be replaced, generates replacement abnormal device recommendation information, and controls it to be displayed on the screen of the manager terminal 1a.

In this case, the generated replacement faulty device recommendation information may include identical device recommendation information that is the same device as the replacement target faulty device and similar device recommendation information that is another device having similar performance to the replacement target faulty device.

Exemplarily, when the replacement target abnormal device is the first device (eg, a temperature sensor) as described above, the screen control unit 150 displays replacement abnormal device recommendation information in relation to the temperature sensor on the screen of the manager terminal 1a. can be displayed

In this case, the replacement abnormal device recommendation information may include recommended information (same device recommendation information) of the same temperature sensor as the first device temperature sensor, as well as a temperature sensor of a different type from the first device temperature sensor (for example, , temperature sensors of different manufacturers/shapes/structures) but recommending information (recommended information of similar devices) of a similar temperature sensor having similar performance to that of the temperature sensor, which is the first device, may be included.

In addition, as similar device recommendation information included in the replacement faulty device recommendation information, for example, the review (evaluation) is better or the price is higher than the first device (temperature sensor), which is the replacement target faulty device that is determined to need replacement. Information on lower or higher quality devices may be included.

According to this, when it is determined that replacement of a specific device among a plurality of devices managed by the device control management unit 110 is required (ie, when it is determined that the device is an abnormal device to be replaced), It is possible to provide (recommend) not only a device of the same type as the replacement target defective device, but also a different type but good cost-performance ratio as recommended information (recommended device recommendation information). This device 100 allows the manager to not only recognize the existence of a device requiring replacement among various devices existing in the unmanned store 3 managed by him/herself, but also replace the device determined to need replacement. It can be provided to make the process easier and more efficient.

This device 100 may enable efficient management of a plurality of devices (various devices 3a) in the unmanned store 3 interlocked with the device 100 . The apparatus 100 may reduce the operating cost of the various devices 3a, increase the management efficiency of the various devices 3a, and increase the convenience of the manager 1 who is in charge of operation. In other words, the device 100 may facilitate the identification and management of the status of the various devices 3a in the unmanned store 3 . In addition, the apparatus 100 may improve management convenience and user convenience of an administrator.

In addition, although not shown in the drawings, the present system 1000 may include an unmanned shelf (not shown) provided in the unmanned store 3 . The various devices 3a may include an unmanned stand (not shown). The unmanned shelf may mean, for example, a refrigerated shelf on which frozen food, beverages, and the like are displayed.

The unmanned shelf may have an accommodating space where products can be displayed. When some of the displayed products displayed in the accommodating space are withdrawn from the unmanned shelf by the user 2 , the unmanned kiosk stores information related to withdrawal products drawn out by the user 2 (ie, withdrawal product related information). It can be generated and transmitted to the kiosk 400 through the network 4 .

Here, the withdrawal product-related information may include information on the type and number of withdrawal products, and information on the face image of the user who withdraws the product (withdrawal product) from the unmanned shelf (ie, the product withdrawal user face image).

Here, an internal camera may be provided in one area inside the unmanned shelf to photograph the displayed products displayed in the accommodation space. In addition, the unmanned shelf may include a detection sensor for detecting the door opening and closing of the unmanned shelf. The detection sensor may be, for example, a magnetic field, a pressure sensor, a touch detection sensor, and the like, but is not limited thereto.

When the door is closed after a certain period of time after the door opening is detected through the detection sensor, the user (2) can recognize the act of opening and closing the door of the unmanned shelf to withdraw the products displayed on the unmanned shelf. can In this case, the unmanned shelf acquires the image of the first displayed product through the internal camera before a preset time (for example, 5 seconds before) based on the time when the door is opened, and a certain period of time after the door open is detected. The second displayed product image may be acquired through the internal camera after a preset time (for example, after 5 seconds) based on the time when the door closing is sensed. In this case, the unmanned shelf may identify the type and number of withdrawal products withdrawn by the user by comparing the first displayed product image and the second displayed product image.

In this case, the unmanned shelf may perform comparative analysis based on various image analysis algorithms that are known or developed in the future when performing analysis (image analysis) for comparison between the first displayed product image and the second displayed product image. .

In addition, in one area of the outer surface of the unmanned shelf, a shelf camera for acquiring a face image of a user withdrawing a product (withdrawal product) from the unmanned shelf may be installed (that is, a face image of the user withdrawing product). Through such a stand camera, a face image of a user withdrawing a product of a user withdrawing a product from an unmanned stand may be obtained.

The unmanned kiosk may generate withdrawal product related information including information on the type and number of withdrawal products and a face image of the user withdrawing the product, and transmit it to the kiosk 400 through the network 4 .

Thereafter, when a user input is made to the 'Make Payment' button displayed on one area of the screen of the kiosk 400, for example, the kiosk 400 may display the product selected by the user (the product selected by the user among products sold in the unmanned store). It may be detected that a payment request has been made from the user for purchase (purchase).

At this time, before the user input for the 'Payment' button is made, the user selects a product to be purchased from among the products sold in the unmanned store, and then uses the barcode of the product at the kiosk 400 by itself. ) can be scanned with the barcode scanning unit provided in Through this, a list of products (products to be purchased) scanned by the user 2 may be displayed on the screen of the kiosk 400 . Thereafter, the user may press the 'Payment' button if there is no problem after checking the list of scanned products. Here, the kiosk 400 may provide, for example, a payment method selection screen on the screen of the kiosk in response to a user input for a 'make payment' button, and provide the user to make a payment based thereon.

At this time, when a user input for the payment button is made (that is, when a payment request is made), the kiosk 400 does not provide a payment method selection screen directly on the screen of the kiosk, but provides a payment method selection screen It is possible to detect the existence of a stolen product by the user before the following. To this end, the kiosk 400 may include a stolen goods detection unit (not shown).

The stolen product detection unit (not shown) receives the user's face image obtained through the AI camera of the kiosk 400 and the withdrawal product related information obtained from the unmanned shelf when the user input for the payment button is made. By comparing the images, it is possible to determine whether they are the same person.

At this time, if it is determined that it is the same person, the stolen product detection unit (not shown) scans all of the withdrawal product information corresponding to the type and number information of the withdrawal product included in the withdrawal product related information is displayed on the screen of the kiosk 400 . It can be determined whether it is included in the list of products.

For example, when a user withdraws a display product corresponding to '5 carbonated beverages of brand A' from the unmanned shelf, withdrawal product information corresponding to information on the type and number of withdrawal products is 'brand A carbonated beverages' It can mean 'five'.

In this case, the stolen product detection unit (not shown) may compare the list information of the scanned product with the withdrawal product information, and check whether all (same) information on the withdrawal product is included in the list of the scanned product.

Here, not all of the withdrawal product information is included in the list of scanned products, but only some product information (that is, as product information related to withdrawal product information in the scanned product list, for example, 'Three carbonated drinks of brand A' ', the stolen product detection unit (not shown) provides product information related to withdrawal product information included in the scanned product list (ie, '3 carbonated drinks of brand A'). The product corresponding to the difference between the product information (ie, 'A brand's 5 carbonated drinks') and the withdrawal product information (ie, 'A brand's two carbonated drinks') is a stolen product that the user secretly tried to take without payment. can recognize

As such, when it is detected that a stolen product exists by the stolen product detection unit (not shown), the kiosk 400 may generate a warning sound through a speaker provided in one area of the kiosk 400 .

At this time, if the product recognized as a stolen product ('A brand 2 carbonated drinks') is not scanned through the barcode scanning unit of the kiosk 400 within a preset time period (for example, 10 minutes) after the warning sound is generated. (that is, when the user does not scan the product recognized as the stolen product in the kiosk within a preset time period), the kiosk 400 recognizes that the user intends to take the product recognized as the stolen product without payment (confirmed) ), a stolen product generation notification message notifying that a stolen product has occurred may be generated and transmitted to the present device 100 . The device 100 provides the stolen product occurrence notification message delivered from the kiosk 400 to the manager terminal 1a, so that the manager can recognize that the stolen product has occurred.

In general, since an unmanned store does not have a manager who calculates or manages products in the store at all times due to the nature of the store, users often take advantage of this to secretly take the products in the unmanned store. In order to solve this problem, the system 1000 may automatically determine whether a stolen product exists in consideration of the withdrawal product related information obtained by the kiosk 400 from the unmanned shelf. This system 1000 can prevent and reduce the occurrence of a product theft event in the unmanned store 3 .

Hereinafter, based on the details described above, the operation flow of the present application will be briefly reviewed.

20 is an operation flowchart of an unmanned store management method according to an embodiment of the present application.

The unmanned store management method illustrated in FIG. 20 may be performed by the apparatus 100 described above. Accordingly, even if omitted below, the description of the apparatus 100 may be equally applied to the description of the unmanned store management method.

Referring to FIG. 20 , in step S31, the device control management unit may control and manage device operations of the IOT environment complex sensor, the IOT control sensor, the kiosk, and the access authentication device.

Next, in step S32, the authentication unit performs user authentication to determine whether the user is a pre-registered customer registered in the customer DB based on the user authentication information received from the access authentication device, and considers the result of user authentication. The generated access authentication result information can be transmitted to the access authentication device.

Next, in step S33, the product management unit may perform product management for products sold in the unmanned store.

Next, in step S34, the screen controller may provide a response screen corresponding to the input signal to the display unit of the manager terminal based on the input signal of the manager who manages the unmanned store management apparatus.

In the above description, steps S31 to S34 may be further divided into additional steps or combined into fewer steps, according to an embodiment of the present application. In addition, some steps may be omitted if necessary, and the order between the steps may be changed.

The unmanned store management method according to an embodiment of the present application may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the present invention, and vice versa.

In addition, the above-described unmanned store management method may be implemented in the form of a computer program or application executed by a computer stored in a recording medium.

The foregoing description of the present application is for illustration, and those of ordinary skill in the art to which the present application pertains will understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present application. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may also be implemented in a combined form.

The scope of the present application is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be construed as being included in the scope of the present application.

1000: unmanned store management system
100: unmanned store management device
110: device control management unit 120: authentication unit
130: customer DB 140: product management department
150: screen control
1: Manager 1a: Manager terminal
2: User 4: Network
3: Unmanned Store
200: IOT environmental complex sensor 300: IOT control sensor
400: kiosk 500: access authentication device
600: electronic price indicator

Claims (4)

An unmanned store management system, comprising:
IOT environmental complex sensor that detects environmental information in unmanned stores;
a kiosk provided in the unmanned store, which controls the operation of the IOT control sensor provided in the unmanned store in consideration of the environmental information obtained from the IOT environmental complex sensor;
It is installed at the store entrance of the unmanned store, acquires user authentication information of a user who wants to enter the unmanned store, transmits it to the unmanned store management device, and access authentication for the user authentication information obtained from the unmanned store management device an access authentication device for controlling opening and closing of the door of the unmanned store based on the result information; and
An unmanned store management device that interworks with each of the IOT environmental complex sensor, the IOT control sensor, and the kiosk through a network, and manages the unmanned store in a remote location,
The unmanned store management device controls the operation of each device of the IOT environment complex sensor, the access authentication device, the IOT control sensor, and the kiosk in relation to the management of the unmanned store, and the user received from the access authentication device Performs user authentication to determine whether the user is a pre-registered customer registered in the customer DB based on authentication information, and transmits the access authentication result information generated in consideration of the result of the user authentication to the access authentication device includes an authentication unit;
The authentication unit,
If the received user authentication information is card information, compare the received card information with card information of a plurality of customers previously registered in the customer DB to determine whether the user corresponding to the received card information is a registered customer, In order to verify whether the user who entered card information as user authentication information is the actual card owner of the card information, the user's face image acquired through the authentication camera provided in the kiosk and the Check whether the two face images including the pre-stored face image in connection with the received card information match each other,
If the two face images match, the access authentication result information is generated as authentication success so that the door is opened and transmitted to the access authentication device,
If the two face images do not match, a message for checking whether the card has been used without permission is sent to a mobile phone number previously stored in connection with the received card information in the customer DB, and a response to the transmitted message is the unauthorized use of the card If the answer is yes, the access authentication result information is generated as an authentication failure so that the door remains closed and transmitted to the access authentication device. Generates and transmits the result information as authentication success,
In the kiosk, when the user who entered the unmanned store scans the barcode of the selected product among the products sold in the unmanned store by itself to the barcode recognition unit provided in the kiosk and presses the pay button, the pay button A payment method selection screen is provided on the screen of the kiosk in response to a user input for payment so that payment is performed, but before providing the payment method selection screen on the screen of the kiosk, the stolen product by the user through the stolen product detection unit in the kiosk performing detection of the presence of
The unmanned shelf provided in the unmanned store generates and transmits withdrawal product related information, which is information related to withdrawal products withdrawn from the unmanned shelf by a user among the displayed products displayed on the unmanned shelf, to the kiosk,
The withdrawal product-related information includes information on the type and number of withdrawal products identified through comparative analysis of displayed product images obtained through an internal camera provided in one area inside the unmanned shelf to photograph the displayed product, and the outer surface of the unmanned shelf. Including the face image of the user who withdraws the product obtained through the camera stand provided in one area of the
The stolen goods detection unit,
By comparing the face image of the user obtained through the kiosk and the face image of the withdrawing user in the withdrawal product related information received from the unmanned kiosk, it is determined whether it is the same person, and if it is the same person, the withdrawal product included in the withdrawal product related information Determining whether all of the withdrawal product information corresponding to the type and number information is included in the list of scanned products displayed on the screen of the kiosk through scanning of the barcode recognition unit,
If the list of scanned products does not include all of the withdrawal product information but only a part of the product information, It recognizes the product as a stolen product and generates a warning sound through a speaker provided in one area of the kiosk,
The kiosk is stolen so that a stolen product occurrence notification message notifying that the stolen product has occurred is provided to the manager terminal when the product recognized as a stolen product within a preset time after the warning sound is generated is not scanned through the barcode recognition unit. An unmanned store management system that generates and transmits a product occurrence notification message. delete The method of claim 1,
The unmanned store management device,
a device control management unit for controlling and managing device operations of the IOT environment complex sensor, the IOT control sensor, the kiosk, and the access authentication device;
a product management unit that manages products sold in the unmanned store; and
The unmanned store management system comprising a screen controller for providing a response screen corresponding to the input signal to the display unit of the manager terminal based on an input signal of a manager who manages the unmanned store management device. 4. The method of claim 3,
The device control management unit,
With respect to the plurality of types of products displayed in the unmanned store, the operation control and management of a plurality of electronic price indicators provided to be 1:1 for each type of product are further performed, the unmanned store management system.

Images