KR102525381B1 - System based on user behavior analysis and situational awareness using artificial intelligence - Google Patents

Abstract

In an unmanned vending system, at least one computing device for analyzing images acquired by a plurality of cameras provided inside and outside an unmanned vending machine in which a plurality of products can be displayed for sale; The computing device of recognizes the user's payment method, unlocks the lock of the unmanned vending machine, and acquires an external image obtained from at least one camera that acquires an external image of the unmanned vending machine when the payment method is approved. It is characterized in that the operation of the user including purchase, return, and exchange operations of goods is determined primarily by using .

Description

User behavior analysis and situational awareness based system using artificial intelligence

The present invention relates to a system based on user behavior analysis and context awareness using artificial intelligence.

A conventional vending machine or unmanned convenience store is an unmanned sales system in which consumers directly purchase and pay for goods without a store clerk. This system reduces labor costs in preparation for an increase in the minimum wage and enhances consumer convenience. In addition, there is an advantage in that waiting time is shortened as consumers directly pay with a credit card, smartphone app, or the like.

In the conventional unmanned sales system, since it is used non-face-to-face without a store clerk, it is necessary to directly select a discount and point accumulation method, select a payment method, insert a card, and recognize the barcode of the product.

However, there remains a problem that vulnerable consumers such as the elderly who do not use IT technology well are alienated.

In order to solve the above problems, Korean Patent Registration No. 10-1887685 discloses an unmanned vending machine including an operating system. By the way, Korean Patent Registration No. 10-1887685 requires a separate weight sensor, so there is a possibility of increasing the unit cost of manufacturing a shelf, increasing the cost of mass production process, and increasing the mechanical failure rate.

The present invention is an invention aimed at solving the technical problems described above, and based on user behavior analysis and situational awareness using artificial intelligence, not only does not require a separate weight sensor, but also purchases such as exchange or return Its purpose is to provide an unmanned sales system capable of discriminating various consumer behaviors that may follow afterward.

The unmanned vending system of the present invention includes at least one computing device for analyzing images obtained by a plurality of cameras provided inside and outside of an unmanned vending machine in which a plurality of items can be displayed for sale, wherein the At least one computing device may recognize the user's payment method and unlock the unmanned vending machine when the payment method is approved.

In addition, the at least one computing device may use an external image obtained from at least one camera that obtains an external image of the unmanned vending machine to perform user operations including purchase, return, and exchange of goods as a primary function. It is characterized by judging by Specifically, it is preferable that the at least one computing device primarily determines the user's motion by detecting the user's hand region using the external image and tracking the hand.

In addition, the at least one computing device determines the state of the displayed goods by using an internal image obtained from at least one camera that acquires an internal image of the unmanned vending machine.

Preferably, the at least one computing device is characterized in that the final user's motion is determined by comparing a result of determining the primarily determined user's motion with a result of determining the state of the items on display. .

In addition, the at least one computing device preferably displays the product on a display provided in the unmanned vending machine or transmits the damaged product to the user terminal as a result of determining the final user's operation.

In addition, the at least one computing device, when it is determined as a return or exchange operation of the product by the corresponding user as a result of determining the final user's operation, the corresponding return or exchange operation of the corresponding product by the corresponding user by browsing the database. search for the purchase history up to a pre-specified time for the corresponding return or exchange operation, and if there is a purchase history as a result of the search, the corresponding return or exchange operation is approved, and if there is no purchase history as a result of the search, the corresponding return Alternatively, the exchange operation is disapproved, and approval or disapproval information is displayed on a display provided in the unmanned vending machine or transmitted to a user terminal.

Preferably, the at least one computing device searches for event information related to the corresponding product from the database when it is determined that the final user's operation is a purchasing operation of the corresponding user, and the search result event If there is information, the corresponding event information is displayed on a display provided in the unmanned vending machine or transmitted to a user terminal.

In addition, the at least one computing device, when a separate gift certificate is authenticated in addition to the payment method, information on the type of product that can be purchased with the gift certificate or the product that can be purchased based on the price of the product, or an unmanned vending machine for the product At least one of location information in is displayed on a display provided in the unmanned vending machine or transmitted to a user terminal.

In addition, when the payment means is approved, the at least one computing device receives information about a product desired to be purchased from a display or user terminal provided in the unmanned vending machine, and information about the location of the product in the unmanned vending machine. is displayed on a display provided in the unmanned vending machine or transmitted to a user terminal.

In addition, the at least one computing device determines the current stock quantity and sales quantity for each item displayed in the unmanned vending machine and adjusts the number of items displayed in the unmanned vending machine according to the corresponding quantity sold. Accordingly, when the quantity in stock is less than or equal to a certain ratio or a certain number of items on display, it is preferable to transmit recommendation information for additionally displaying the corresponding item in the unmanned vending machine to the manager terminal.

According to the unmanned sales system of the present invention, based on user behavior analysis and situational awareness using artificial intelligence, a separate weight sensor is not required, and various consumer behaviors that may accompany purchase such as exchange or return can be identified.

1 is a configuration diagram of an unmanned sales system based on user behavior analysis and context awareness using artificial intelligence according to a preferred embodiment of the present invention.
2 is a flow diagram of a purchase of an item by at least one computing device.
3 is a flowchart illustrating a primary determination of a user's motion;

Hereinafter, an unmanned sales system based on user behavior analysis and context awareness using artificial intelligence according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

Of course, the following examples of the present invention are only intended to embody the present invention and are not intended to limit or limit the scope of the present invention. What can be easily inferred by an expert in the technical field to which the present invention pertains from the detailed description and examples of the present invention is interpreted as belonging to the scope of the present invention.

First, FIG. 1 shows a configuration diagram of an unmanned sales system 100 based on user behavior analysis and context awareness using artificial intelligence according to a preferred embodiment of the present invention.

As can be seen from FIG. 1, the unmanned vending system 100 according to a preferred embodiment of the present invention includes an unmanned vending machine 10, an image acquisition unit 20, a payment unit 30, and at least one computing device. It is composed of (40).

The unmanned vending machine 10 preferably has an inner space in which a number of items can be displayed for sale and a door locking device. In addition, the unmanned vending machine 10 further includes a touch display, so that the display serves as an input/output device.

The image acquisition unit 20 serves to acquire images of the inside and outside of the unmanned vending machine 10 by including a plurality of cameras.

That is, it is preferable that at least one camera installed inside the unmanned vending machine 10 mainly acquires images of products displayed inside the unmanned vending machine 10 . In addition, at least one camera installed outside the unmanned vending machine 10 is characterized in acquiring an image of a user located outside the unmanned vending machine 10 .

The payment unit 30 may be a card reader provided in the unmanned vending machine 10 as an example, and card information read by the card reader may be approved by at least one computing device 40 .

At least one computing device 40 serves to analyze the user's behavior by analyzing the image obtained by the image acquiring unit 20 . At least one computing device 40 may preferably analyze a user's action by artificial intelligence.

At least one computing device 40 may be included in the unmanned vending machine 10 or located outside the unmanned vending machine 10 . In addition, the at least one computing device 40 includes one computing device provided in the unmanned vending machine 10 and a computing device such as a server provided outside the unmanned vending machine 10, so that a plurality of computing devices can collaborate. may be operated by

2 is a flow diagram of purchase of an item by at least one computing device 40 .

When a user appears, the at least one computing device 40 tracks the user's image (S10). In addition, when a user's face image is input by a camera provided outside the unmanned vending machine 10, the user is recognized by the corresponding face image, and if the user is registered in the database, the user is authenticated. Specifically, when a user who is a purchaser of the unmanned vending machine 10 makes a purchase, face detection and face recognition are performed. A commonly used face detection algorithm may be used here, and it may be confirmed whether the buyer is a customer through a database having a previous purchase history.

However, user recognition may be performed by various methods other than face recognition, such as logging in from a user terminal to at least one computing device 40 . That is, the user may log in through a pre-installed application program of the user terminal.

When the user authentication is completed (S20), the card information as a payment method read by the card reader is recognized and approved if there is no problem (S30). This approval process may be performed through access to the card company server. However, since the user authentication task is not an essential task, the task of recognizing and approving the user's payment method may be performed first.

Alternatively, at least one computing device 40 may receive payment method information from the user terminal and receive corresponding payment method information. That is, the user may request the at least one computing device 40 to approve various payment means available in the application program through a pre-installed application program of the user terminal.

When the payment method is approved, the at least one computing device 40 unlocks the unmanned vending machine 10 (S40).

When the locking device is released, the user can access the items provided inside the shelf. Accordingly, when the locking device is unlocked, the at least one computing device 40 starts analyzing the user's behavior and whether or not the item is changed (S50).

When the door of the unmanned vending machine 10 is closed and locking by the locking device is resumed (S60), at least one computing device 40 determines the user's operation (S70), and pays or exchanges according to the purchase accordingly. Operations such as settlement and refund according to return are performed (S80). In addition, when determining the operation of the corresponding user (S70), it is also necessary to determine the type of product that is the target of the corresponding purchase, return, or exchange operation and to determine whether or not it is damaged.

Specifically, the at least one computing device 40 uses an external image obtained from at least one camera that obtains an external image of the unmanned vending machine 10, and the user performs purchase, return, and exchange operations of the product. It is characterized in that the operation of is primarily determined (S71). That is, the at least one computing device 40 is characterized in that the user's motion is primarily determined by detecting the user's hand region using an external image and tracking the hand. When the user's operation is primarily determined (S71), the type of product subject to the purchase, return, or exchange operation is determined and whether or not it is damaged is also determined.

However, it is preferable that at least one computing device 40 determines the state of the items on display using an internal image obtained from at least one camera that obtains an internal image of the unmanned vending machine 10 (S72). do. Determining the state of the items on display is intended to more accurately determine the user's motion by supplementing the primary judgment result of the user's motion.

That is, it can be judged as purchase if there is a disappeared item among the displayed items, return if there is a new item, and exchange if there are both a disappeared item and a new item appearing at the same time. In addition, when determining the state of the items on display (S72), it is preferable to determine the type of the item to be purchased, returned, or exchanged, and determine whether or not it is damaged.

The at least one computing device 40 finally determines the user's motion by comparing the primarily determined user's motion determination result with the judgment result of the state of the items on display (S73). It is preferable that the final determination of the user's operation is made after the locking of the unmanned vending machine 10 is resumed.

However, when one of the primarily determined judgment results of the user's motion or the state of the items on display is ambiguous, another clear judgment result may be used. That is, when a contradiction occurs between the primarily determined user's motion determination and the displayed state of goods determination, a clearer determination result may be used.

3 is a flowchart of a primary determination of a user's motion.

As can be seen from FIG. 3, the primary determination of the user's motion (S71) is to detect and track the hand (S711), the hand enters the unmanned vending machine 10 (S712), and the hand moves into the unmanned vending machine 10 again (S712). (10) When going outside (S713), the user's motion is primarily determined (S714), the hand enters the unmanned vending machine 10 (S712), and the hand goes outside the unmanned vending machine 10 again (S712). If it does not go out, it is preferable to continuously determine whether to go outside (S713).

Specifically, in the primary determination of the user's motion, the user's hand and objects in the hand are detected using an image-based behavior analysis algorithm. Based on this, it is checked whether the user took out the product (purchase operation), put the product in (return operation), or exchanged the product (exchange operation). In addition, the presence or absence of abnormalities of the goods is also checked. In the case of the user's motion, the user's motion may continue to be judged differently while checking the product based on the first item and taking it out and putting it in. After that, when the buyer closes the door, that is, when the lock of the unmanned vending machine 10 is resumed, the operation is finally determined. Here, in the case of a purchase operation, payment is made, and in the case of a returning customer, additional procedures are performed according to returns and exchanges. In the case of a return operation, a refund is performed, and in the case of an exchange operation, an exchange confirmation is performed for the same product, and a settlement process such as difference and additional payment or refund is performed in the case of a different product.

The detection of the user's hand and the detection of the object on the hand are actually performed only once for the first time. After that, it is preferable to use an algorithm that continuously detects and tracks the hand. As for the hand detection algorithm, SSD (Single Shot Multi-box Detector), which is widely used, can be used, and as tracking algorithms, algorithms such as IOU tracker and SORT (Simple Online and Realtime Tracking) are used to continuously find the location of the hand. let it be In the case of this tracking algorithm, the position of the hand is not missed even in frames where the middle is not detected, and even if there are several hands, the same hand can be compared.

In this way, if the hand is continuously followed and the hand enters the unmanned vending machine 10, hand detection continues only in the area in front of the unmanned vending machine 10 until the hand comes out of the unmanned vending machine 10, and when the hand comes out, the first 1 Use the hand detection and item classification algorithms implemented in the circuit. In the case of the classification algorithm used at this time, mobileNet, effientNet, VggNet, or an improved type of such a network is used, and at this time, the classification algorithm determines what kind of product is and whether the product exists. Compared to the result detected at the first time, if the item is not in hand, then it is determined as the current purchase action, if the item is in hand and then not, return action, if the item is in the same state as it is, exchange action, In the case of maintaining a state without an item, it is determined as an undetermined standby operation. Here, the operation is determined by including one of four states, and hand detection is performed again. If the hand is continuously detected and then enters the unmanned vending machine 10 again, it is determined that the operation is changed, and when the hand comes out of the unmanned vending machine 10, the operation is determined again using the detection and product classification algorithm. Here, the comparison is made with the first detected state, and this process is repeated until the door is closed.

In addition, the at least one computing device 40 may, as a result of determining the final user's operation, display on the display provided in the unmanned vending machine 10 or transmit it to the user terminal, if there is a damaged item by the user. to be characterized In this case, payment is made for the damaged item by at least one computing device 40 .

In addition, the at least one computing device 40, as a result of the determination of the final user's operation, when it is determined that the user's operation of returning or exchanging the goods, browses the database and returns or exchanges the corresponding goods by the corresponding user. It is preferable to search a purchase history from an exchange operation to a time prior to a predetermined time for a corresponding return or exchange operation. Recognition of the corresponding user is possible not only by a user authentication procedure such as face recognition, but also by the corresponding payment means.

The return and exchange period may be separately set according to the nature of the product or may be set collectively.

Preferably, the at least one computing device 40 approves the corresponding return or exchange operation when there is a purchase history as a result of the search, and disapproves the corresponding return or exchange operation when there is no purchase history as a result of the search. In addition, at least one computing device 40 is characterized in that it displays approval or disapproval information on a display provided in the unmanned vending machine 10 or transmits it to a user terminal.

The at least one computing device 40 searches for event information related to the corresponding product from the database when it is determined that the final user's action is a purchasing action of the corresponding user, and as a result of the search, the event information is found. In this case, the corresponding event information is displayed on a display provided in the unmanned vending machine 10 or transmitted to a user terminal. For example, it is determined that the user purchases 'A drink', but if an event is being held in which one is provided as a service when two 'A drinks' are purchased, the information is notified to the user and an additional 'A drink' is provided. You can encourage the purchase of 'beverages'. Examples of the event information include discount, provision of an additional product identical to a purchased product, addition of a service product different from a purchased product, and the like.

At least one computing device 40, when a separate gift certificate is authenticated in addition to the payment method, information on the type of item that can be purchased with the gift certificate or the item that can be purchased by the price of the item, or an unmanned vending machine for the item ( It is characterized in that at least one of the location information in 10) is displayed on a display provided in the unmanned vending machine 10 or transmitted to a user terminal.

For example, if a coupon is authenticated after the card is approved, items that can be purchased by the corresponding coupon are notified to the user. Authentication of the gift certificate may use an image by a camera of the image acquiring unit 20 or image information transmitted from the user terminal to at least one computing device 40 .

The location information of the corresponding product in the unmanned vending machine 10 is 'B' on the arrangement map of the entire product so that the user can easily know the location of 'B beverage' in the unmanned vending machine 10 where a number of items are placed. It can be specified by identifiablely displaying the position of 'beverage'.

In addition, after at least one computing device 40 authenticates the user of the corresponding user terminal, recognizes and approves the payment method of the authenticated user, and when the payment method is approved, the display provided in the unmanned vending machine 10 Alternatively, information on a product desired to be purchased is received from the user terminal, and location information of the product in the unmanned vending machine 10 is displayed on a display provided in the unmanned vending machine 10 or transmitted to the user terminal. .

In addition, at least one computing device 40 determines the current inventory quantity and sales quantity for a certain period of time for each item displayed in the unmanned vending machine 10, and the unmanned vending machine 10 of the corresponding item according to the corresponding sales quantity When the number of items in stock is less than or equal to a certain percentage or less than a certain number of items on display, recommendation information for additionally displaying the corresponding item in the unmanned vending machine 10 is transmitted to the manager's terminal.

Based on the recommended information about the display of such products, the manager can properly replenish the products to the unmanned vending machine 10 . In addition, the at least one computing device 40 adjusts the number of items to be sold so that many items can be displayed.

As described above, according to the unmanned sales system 100 of the present invention, based on user behavior analysis and situational awareness using artificial intelligence, a separate weight sensor is not required, and subsequent purchase such as exchange or return It can be seen that various consumer behaviors can be identified.

That is, according to the unmanned sales system 100 of the present invention, by recognizing the buyer based on the image and analyzing the buyer's behavior, it is possible to purchase, exchange, or return the product, so the intervention of the clerk can be further reduced, and the existing purchase Through data, it is possible to exchange or return without bringing a receipt. In addition, according to the unmanned sales system 100 of the present invention, exchange or return is simple without time constraints, and payment settlement or refund can be made simultaneously with exchange or return.

100: unmanned sales system
10 : Unmanned vending machine
20: image acquisition unit
30: payment unit
40: at least one computing device

Claims (11)

In the unmanned sales system,
At least one computing device for analyzing images obtained by a plurality of cameras provided inside and outside of an unmanned vending machine in which a plurality of items can be displayed for sale;
The at least one computing device recognizes the user's payment method and unlocks the unmanned vending machine when the payment method is approved;
The at least one computing device primarily determines a user's operation including purchasing, returning, and exchanging an item by using an external image acquired from at least one camera that obtains an external image of the unmanned vending machine. do,
The at least one computing device determines a state of the displayed goods by using an internal image obtained from at least one camera that acquires an internal image of the unmanned vending machine;
The at least one computing device determines a final user's motion by comparing a result of determining the user's motion, which is determined primarily, with a result of determining the state of the displayed item;
The at least one computing device searches for event information related to the corresponding product from the database when it is determined that the final user's action is a purchasing action of the corresponding user, and event information is found as a result of the search. displaying corresponding event information on a display provided in the unmanned vending machine or transmitting it to a user terminal;
When a separate gift certificate is authenticated in addition to the payment means, the at least one computing device displays location information of the unmanned vending machine of an item that can be purchased with the corresponding gift certificate on the display provided in the unmanned vending machine or the user send to terminal
When the payment method is approved, the at least one computing device receives information about a product desired to be purchased from a display or user terminal provided in the unmanned vending machine, and displays location information of the corresponding product in the unmanned vending machine as an input. displayed on the display provided in the unmanned vending machine or transmitted to the user terminal;
The at least one computing device determines the current stock quantity and sales quantity for each item displayed in the unmanned vending machine, and adjusts the number of items displayed in the unmanned vending machine according to the corresponding quantity sold, The unmanned vending system, characterized in that for transmitting recommendation information for displaying the corresponding item in the unmanned vending machine to a manager terminal when the inventory quantity is less than or equal to a predetermined ratio or a predetermined number relative to the displayed quantity. delete According to claim 1,
The at least one computing device,
The unmanned sales system characterized in that the user's motion is primarily determined by detecting the user's hand region using the external image and tracking the hand. delete delete According to claim 1,
The at least one computing device,
An unmanned vending system characterized in that, when a product damaged by the user exists as a result of the final user's operation, it is displayed on a display provided in the unmanned vending machine or transmitted to a user terminal. According to claim 1,
The at least one computing device,
As a result of the determination of the final user's operation, if it is determined that the return or exchange operation of the product by the user is determined, the database is browsed to determine the return or exchange operation of the product in advance from the corresponding user's return or exchange operation. Search purchase history up to the specified time, and if there is a purchase history as a result of the search, the corresponding return or exchange operation is approved. An unmanned vending system characterized in that the information is displayed on a display provided in the unmanned vending machine or transmitted to a user terminal.
delete delete delete delete

Images