KR20230167681A - Apparatus, method and program to determine whether goods are normally placed or not based on artificial intelligence - Google Patents

Abstract

The present disclosure relates to an artificial intelligence-based product position calculation device in a product showcase utilizing a load cell arrangement structure, including a plurality of load cells for sensing the positions of products placed on the shelves of a display stand divided into a plurality of columns, and and a first processor that calculates location information of the product placed on the shelf based on the sensing values sensed by the plurality of load cells and the location information of each of the plurality of load cells that sense the sensing values.

Description

Apparatus, method and program to determine whether goods are normally placed or not based on artificial intelligence}

This disclosure relates to a device for determining whether a product is properly placed, and more specifically, to a device for determining whether a product is properly placed based on artificial intelligence.

Recently, demand for unmanned refrigerators and sales devices has been increasing.

The characteristic of these devices is that inventory management must be carried out automatically by accurately recognizing the products placed on the shelf, calculating whether the correct product has been placed in the corresponding column, and calculating the number of products currently placed. However, at present, such technology is not available. It has not been made public.

In addition, the device that determines product placement using a weight measurement sensor only determines the placement of products of a fixed size and does not take into account changes in product size, so there is a problem in that product placement is less flexible.

In addition, the devices currently in use have malfunctions that prevent them from accurately determining the number of products sold, resulting in a problem of not obtaining accurate payment information during the product payment process.

Republic of Korea Patent Publication No. 10-1797656, (2017.11.08)

The purpose of the embodiment disclosed in this disclosure is to provide a device for calculating the position of a product placed in a product showcase based on artificial intelligence by utilizing the arrangement structure of a load cell.

The purpose of the embodiment disclosed in this disclosure is to provide a device for calculating the number of products sold based on artificial intelligence using a product recognition algorithm.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

An artificial intelligence-based product location calculation device in a product showcase according to an embodiment of the present disclosure to solve the above-described problem is for sensing the positions of products placed on the shelves of a display stand divided into a plurality of columns. Multiple load cells; And a first processor that calculates location information of the product placed on the shelf based on the sensing value sensed by the plurality of load cells and the location information of each of the plurality of load cells that sensed the sensing value, and the plurality of Two or more load cells are arranged on each line to separate the plurality of columns.

In addition, the plurality of load cells include a first load cell on one side of each line and a second load cell on the other side of each line, and the first processor includes the first load cell and the first load cell on each line. 2 The location information of the product placed on the shelf can be calculated by internal calculation of the sensing value sensed by the load cell.

In addition, the plurality of load cells include a first load cell and a second load cell provided on one side of each line, and the first processor calculates a sensing value sensed by the first load cell and the second load cell on each line. By calculating the outer product of , the location information of the product placed on the shelf can be calculated.

In addition, the first processor calculates the estimated weight and bottom shape of the product placed on the shelf based on the sensing value sensed by the plurality of load cells and the position information of each of the plurality of load cells sensing the sensing value, , the type of the product can be determined based on the calculated weight of the product and the expected lower shape of the product.

Additionally, the first processor may determine whether the product placed on the shelf is normally placed based on the calculated product location information and the determined type of the product.

In addition, the shelf includes a camera capable of photographing at least a partial area of the shelf at a plurality of different angles of view, and the first processor is configured to allow the camera to photograph the shelf at a plurality of different angles of view to produce a plurality of captured images. control to generate the product, and match the calculated location information of the product with the plurality of captured images to verify the calculation result of the location information of the product.

Additionally, the first processor may calculate image location information of the product based on the plurality of captured images and determine whether the calculated image location information matches the calculated location information.

In addition, the shelf is provided with a LiDAR sensor on each side, and the first processor can verify the calculated location information of the product based on the sensing value sensed through the LiDAR sensor.

In addition, the shelf is provided with a first LiDAR sensor on one side of the interior and a second LiDAR sensor on the other side of the interior, and the first processor, the sensing value sensed through the first LiDAR sensor and Based on the sensing value sensed through the second LIDAR sensor, location information of the product placed on the shelf is calculated, and the calculated location information of the product is compared with the calculated location information of the product to calculate the location information. You can verify the location information of the product.

In addition, the shelf is provided with a first LiDAR sensor on one side of the interior and a second LiDAR sensor on the other side of the interior, and the first processor, based on the calculated location information of the product, 1 A plurality of sensing points are assigned to each of the LiDAR sensor and the second LiDAR sensor, and the sensing values sensed through the first LiDAR sensor and the second LiDAR sensor are assigned to the assigned plurality of sensing points. Based on this, the calculated location information of the product can be verified.

In addition, an artificial intelligence-based method of calculating a product position in a product showcase according to an embodiment of the present disclosure to solve the above-described problem is performed by a device, and includes the steps of detecting that a product is placed on a shelf; And based on the sensing value sensed by a plurality of load cells for sensing the position of the product placed on the shelf of the display stand divided into a plurality of columns and the position information of each of the plurality of load cells that sensed the sensing value. It includes calculating location information on products placed on the shelf.

An artificial intelligence-based product location calculation device in a product showcase according to an embodiment of the present disclosure for solving the above-described problem (calculating the number of products sold) includes at least one memory that stores the standard weight of the product and a preset correction index. A communication module that communicates with a product location calculation device including a camera and a weight measurement sensor, and at least one processor that communicates with the memory, wherein when the door of the product storage unit is opened, the at least one processor , receiving first product information from the product location calculation device, and when the door of the product storage unit is closed, receiving second product information from the product location calculation device, and each of the first product information and the second product information Calculate the number of sales of the product by performing an absolute calculation based on the included weight information, and if it is determined that an error has occurred according to a predetermined error detection algorithm based on the second product information, stop the absolute calculation, A relative operation is performed based on the weight information difference value included in each of the first product information and the second product information to calculate the number of products sold, and the tilt information of the load cell received from the product position calculation device is calculated. If it is determined that the error has been resolved by changing the correction index based on the correction index, the relative operation is stopped and the absolute operation is performed again.

also, For the absolute calculation and the relative calculation, machine learning is performed based on the product image captured by the camera to determine the calculation value for each, and the learning model that performs the machine learning is multiclass classification. And binary classification can be implemented, wherein the multi-class classification is performed by a first convolutional neural network (CNN) model, and the binary classification is performed by a second convolutional neural network model. In the case where it is determined that there is no product in the load cell by the second convolutional neural network model, the predetermined error detection algorithm determines that the product is based on the weight information of the load cell measured by the weight measurement sensor. If it is determined that this exists, it is determined that an error has occurred, and if not, the algorithm determines that an error has not occurred. If it is determined that the error has occurred, it is determined that the preset correction index needs to be changed, It is possible to receive tilt information of the load cell from the product position calculation device, change the preset correction index based on the tilt information of the load cell, and when the preset correction index is changed, it can be determined that the error has been resolved. there is.

In an artificial intelligence-based product location calculation method in a product showcase according to an embodiment of the present disclosure to solve the above-described problem (calculating the number of products sold), the product position calculation device stores the standard weight of the product and a preset correction index. When the door of the product storage unit is opened, receiving first product information from a product location calculation device including a camera and a weight measurement sensor. When the door of the product storage unit is closed, second product information is received from the product location calculation device. Receiving, performing an absolute calculation based on weight information included in each of the first product information and the second product information to calculate the number of products sold, predetermined based on the second product information If it is determined that an error has occurred according to the error detection algorithm, the absolute operation is stopped, and a relative operation is performed based on the weight information difference value included in each of the first product information and the second product information to determine the product. In the step of calculating the number of units sold, when it is determined that the error is resolved by changing the correction index based on the tilt information of the load cell received from the product position calculation device, stopping the relative operation and performing the absolute operation again Includes steps.

In addition, the first product information is information about the product image captured by the camera and the weight of the load cell measured by the weight measurement sensor when the door of the product storage unit is opened, and the second product information is information about the weight of the load cell measured by the weight measurement sensor. When the door of the product storage unit is closed, the product image captured by the camera and information about the weight of the load cell measured by the weight measurement sensor are provided.

In addition, for the absolute calculation and the relative calculation, machine learning is performed based on the product image captured by the camera to determine the calculation value for each, and the learning model that performs the machine learning is multiclass classification (Multiclass classification) and binary classification, and the multi-class classification is performed by a first convolutional neural network (CNN) model, and the binary classification is performed by a second convolutional neural network model. It can be performed by

In addition, the predetermined error detection algorithm determines the inventory number based on the absolute calculation based on when the door is closed, and calculates the inventory number multiplied by the standard weight of the product and measured by the weight measurement sensor. The information on the weight of the load cell is compared to calculate the ratio, and if the ratio differs by more than a preset value, it is determined that the error has occurred. If the ratio differs by less than the preset value, the error occurs. It is characterized by an algorithm that determines that it has not occurred.

In addition, if it is determined that the error has occurred, it is determined that the preset correction index needs to be changed, It is possible to receive tilt information of the load cell from the product position calculation device, change the preset correction index based on the tilt information of the load cell, and when the preset correction index is changed, it can be determined that the error has been resolved. there is.

In addition, when the predetermined error detection algorithm determines that there is no product in the load cell by the second convolutional neural network model, the predetermined error detection algorithm determines that the product is based on the weight information of the load cell measured by the weight measurement sensor. If it is determined that this exists, it is determined that an error has occurred, and if not, it is characterized as an algorithm that determines that an error has not occurred.

In addition, when it is determined that the error has occurred, it is determined that the preset correction index needs to be changed, tilt information of the load cell is received from the product position calculation device, and the preset correction is made based on the tilt information of the load cell. When the index is changed and the preset correction index is changed, it can be determined that the error has been resolved.

In addition to this, a computer program stored in a computer-readable recording medium for execution to implement the present disclosure may be further provided.

In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to the means for solving the above-described problem of the present disclosure, the effect of providing a device for calculating the position of a product placed in a product showcase based on artificial intelligence by utilizing the arrangement structure of a load cell is provided.

In addition, according to the means for solving the above-described problem of the present disclosure, it provides the effect of providing a device for calculating the number of products sold based on artificial intelligence using a product recognition algorithm.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below.

Figure 1 is a diagram illustrating a conventional product location calculation device.
Figure 2 is a schematic diagram of a product location calculation system according to an embodiment of the present disclosure.
Figure 3 is a block diagram of a product location calculation device according to an embodiment of the present disclosure.
Figure 4 is a flowchart of a product location calculation method according to an embodiment of the present disclosure.
Figure 5 is a flow chart illustrating the details of S200.
Figure 6 is a diagram illustrating that a first load cell is provided on one side of each line of a shelf and a second load cell is provided on the other side.
Figure 7 is a diagram illustrating that a first load cell and a second load cell are provided on one side of each line of the shelf.
Figures 8 and 9 are diagrams illustrating verification of the calculation result of S200 using camera captured images.
Figures 10 and 11 are diagrams illustrating verification of the calculation result of S200 using the sensing value sensed through the lidar sensor.
Figure 12 is a diagram schematically showing how a product location calculation device communicates with a server according to an embodiment.
Figure 13 is a diagram schematically showing the configuration included in the product location calculation device and server according to an embodiment.
Figure 14 is a diagram showing mode switching according to a product recognition algorithm according to an embodiment.
Figure 15 is a diagram schematically showing a product location calculation device 100 including a camera and a weight measurement sensor according to an embodiment.
Figure 16 is a flowchart showing steps performed in a product location calculation device and server according to an embodiment.
Figure 17 is a diagram schematically showing measuring the inclination angle of a load cell using a tilt measurement sensor according to an embodiment.
Figure 18 is a diagram illustrating an algorithm for the process from when a mode change is made to when an error is resolved according to an embodiment.
Figure 19 is a diagram illustrating an algorithm for the process of performing an absolute operation in ABS mode according to an embodiment.
Figure 20 is a diagram illustrating an algorithm for a process in which a relative operation is performed in the REL mode according to an embodiment.
Figure 21 is a diagram illustrating an algorithm for a process for resolving an error when an error occurs according to an embodiment.

The advantages and features of the present disclosure and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present disclosure is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure is complete and to provide a general understanding of the technical field to which the present disclosure pertains. It is provided to fully inform those skilled in the art of the scope of the present disclosure, and the present disclosure is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing embodiments and is not intended to limit the disclosure. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements. Although “first”, “second”, etc. are used to describe various components, these components are of course not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may also be the second component within the technical spirit of the present disclosure.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which this disclosure pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the attached drawings.

Figure 1 is a diagram illustrating a conventional product location calculation device.

Referring to Figure 1, a device for calculating the position of products placed on a display shelf is illustrated, and this conventional device is manufactured to recognize products on a load cell line one by one.

In other words, when a product of a certain size is placed on a shelf divided into a plurality of columns at preset intervals, the device uses the sensing value of the load cell to calculate whether the product is placed and the location of the product.

Therefore, the conventional technology is only usable for arranging products of a certain size, and has a problem in that it does not work properly when the size or type of the product changes.

Accordingly, the inventor of the present disclosure seeks to provide a product position calculation device in a product showcase based on artificial intelligence according to the present disclosure to accurately calculate whether or not the product is placed and its location even when products of various shapes and weights are placed on the shelf. .

Below, the product location calculation device in the artificial intelligence-based product showcase according to the present disclosure will be described.

Figure 2 is a schematic diagram of an unmanned vending system 10 according to an embodiment of the present disclosure.

Referring to FIG. 2, the unmanned vending system 10 according to an embodiment of the present disclosure may include a location calculation device 100 and a server 200.

In an embodiment of the present disclosure, the unmanned vending system 10 includes an unmanned vending device and a server 200, and the unmanned vending device may include a location calculation device 100 according to an embodiment of the present disclosure. At this time, the unmanned vending system 10 shown in FIG. 1 is only an example and may include fewer or more components than the components shown in FIG. 1 .

The reason for this configuration is that an unmanned vending device is required for the unmanned vending system 10, and a location calculation device 100 according to an embodiment of the present disclosure is required to manage the unmanned vending device unmanned.

Therefore, in the embodiment of the present disclosure, the location calculation device 100 may be a configuration for the unmanned vending device, system 10.

The unmanned sales device is a device 100 for selling specific types of products without a seller. It is composed of a display stand (e.g., a showcase) that displays products to be sold, and detects the entry and exit of the displayed products and determines inventory. , which ultimately allows the buyer to make payment for the purchased product.

In one embodiment, the unmanned vending device detects whether the products displayed inside enter or exit as the user (i.e., the buyer) who wants to purchase the product opens and closes the door, and transmits the results to the server 200. .

Additionally, the unmanned vending device can receive inventory information generated by the location calculation device 100 according to an embodiment of the present disclosure and use this to manage the inventory of each product in the unmanned vending device.

In other words, the unmanned vending device may include a variety of devices that can perform the computational processing necessary to perform various functions for unmanned vending and provide results to the user.

For example, an unmanned vending device includes a communication device such as a communication modem for communicating with various devices or a wired or wireless network, a first memory for storing various programs and data, and a first microsecond for executing the program to perform calculations and control. It may include various computing devices including processors, etc., and the location calculation device 100 of the present disclosure.

The unmanned sales device may be configured in the form of a display stand that displays products inside, and the interior of the display shelf may be configured with a structure in which a plurality of shelves are stacked. Additionally, each of the plurality of shelves is divided into a plurality of columns, and products can be placed in each of the plurality of columns.

Products placed in an unmanned vending device may include various types of products.

In one embodiment of the present invention, the product placed in the unmanned vending device may be a product that is light in weight compared to its volume (e.g., regular cigarettes, electronic cigarettes, snacks, etc.). However, this is only an example and is not necessarily limited to this.

In one embodiment, each shelf in the unmanned vending device may be configured to have a predetermined inclination, and in this case, products placed on each column of each shelf may be pushed and moved to the front of the shelf by the predetermined inclination. make it possible

Additionally, in one embodiment, each shelf in the unmanned vending machine may further include a plurality of elastic members. That is, a plurality of elastic members may be installed corresponding to a plurality of columns separated on each shelf. For example, one elastic member may be installed in each column on the shelf, and in this case, the elastic member may be constructed using a spring or the like that is installed at the back of the shelf and has a force (i.e., elastic force) that moves toward the front of the shelf. there is. In other words, the elastic member can apply a force to push the remaining products placed on each column to the front of the shelf by elastic force whenever the frontmost product placed on each column falls off.

The position calculation device 100 may be equipped with a load cell 120 capable of measuring the weight of products placed on each shelf.

The location calculation device 100 may be equipped with a camera capable of photographing products placed on each column of each shelf. The camera can be installed on each shelf in the unmanned vending machine, and can be installed in a position to look down diagonally from the top at the products displayed on each shelf. At this time, the height between shelves in the unmanned vending machine (i.e., the height of each layer within the display shelf), the number of columns on the shelf, and the width of the product (depending on the type of product; for example, electronic cigarettes have a wider product width than regular cigarettes). The optimal camera position and/or number of cameras may be determined by considering the camera's angle of view based on the camera's angle of view (may be large), etc. In other words, the area that one camera can recognize (i.e., the number of columns) is limited depending on the height of each floor, the width of the product, etc., so in one embodiment, the camera It can be installed in a position looking down diagonally. For example, cameras (i.e., a left camera and a right camera) may be installed on the left and right sides of each shelf. However, this is only an example and is not necessarily limited to this.

The position calculation device 100 can set and store information about products to be displayed (placed) in each column for each shelf (eg, product name, product price, product size, etc.). As an example, the location calculation device 100 may receive information about products to be placed in each column for each shelf from the server 200, and information about products to be placed in each column at the request of the server 200. You can also change .

The server 200 may include a computer system and computer software (web server program) that derives and provides task results corresponding to task performance requests from clients or other web servers. In addition to the web server program described above, the server 200 may include a series of application programs running on a web server or various databases built inside the device 100. For example, the server 200 may include a computer, a server device, and a portable terminal, or may take the form of any one.

Here, the computer may include, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

The server device is a server that processes information by communicating with external devices, such as application server), computing server), database server), file server), game server), mail server), proxy server), and web server). It may include etc.

The portable terminal is, for example, a wireless communication device that guarantees portability and mobility, such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), and PDA. (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone ), all types of handheld wireless communication devices, and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-device (HMD). may include.

The server 200 may link with at least one unmanned vending device for each store via a network. As an example, the server 200 may provide information about products sold in an unmanned sales device (e.g., total number of products, price of each product, size of each product, display position of each product, identification information, top/bottom of product). /left/right images, etc.) can be managed. The server 200 can transmit information about products to be placed in each column (eg, product name, product price, product size, etc.) to the unmanned vending device to be set in the unmanned vending device.

The server 200 receives information about the purchased product (e.g., product name, number of purchased products, etc.) of the purchased product that the purchaser has taken out by opening and closing the door of the unmanned vending device, from the unmanned vending device, and based on this, the purchaser's purchase. The payment amount for the product can be calculated and transmitted to the payment device. Afterwards, the buyer can pay for the purchased product through the payment device.

Meanwhile, the network can transmit and receive various information between the server and at least one unmanned vending device. Various types of communication networks may be used, for example, wireless communication methods such as WLAN (Wireless LAN), Wi-Fi, Wibro, Wimax, and HSDPA (High Speed Downlink Packet Access). Alternatively, wired communication methods such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), and FTTH (Fiber To The Home) may be used.

The network is not limited to the communication methods presented above, and may include all other types of communication methods that are well known or will be developed in the future in addition to the communication methods described above.

As explained through FIG. 2, the location calculation device 100 according to an embodiment of the present disclosure is ultimately a device 100 that manages inventory so that the unmanned sales device can be operated unmanned. Refer to other drawings below. Hereinafter, a detailed description of the components of the location calculation device 100 and a control method of the first processor 110 for inventory management will be described.

Figure 3 is a block diagram of a product location calculation device 100 according to an embodiment of the present disclosure.

Referring to FIG. 3, the product location calculation device 100 in the artificial intelligence-based product showcase includes a first processor 110, a load cell 120, a calculation unit 130, a verification unit 140, and a first memory 150. ), a camera 160, a LiDAR sensor 170, and a first communication unit 180. However, in some embodiments, the product location calculation device 100 may include fewer or more components than the components shown in FIG. 2 .

In the device 100 according to an embodiment of the present disclosure, a plurality of load cells 120 are provided, and in detail, the device 100 senses the positions of products placed on the shelves of a display stand divided into a plurality of columns. It includes a plurality of load cells 120 to do this.

Additionally, two or more load cells 120 are disposed on each line to separate the plurality of columns.

In addition, the method of calculating the position of the product can be divided into the first embodiment and the second embodiment according to the arrangement method of the first load cell 121 and the second load cell 122 on each line and the position calculation method according to the arrangement method. You can.

The calculation unit 130 stores algorithms and commands for calculating the position of the product, and typically performs internal calculation of the sensing values sensed by the first load cell 121 and the second load cell 122 in the first embodiment. In the second embodiment, an algorithm that calculates the location information of the product may be included, and an algorithm that calculates the location information of the product by calculating the outer product of the sensing values sensed by the first load cell 121 and the second load cell 122. .

Accordingly, the first processor 110 can control the calculation unit 130 to calculate the first location information of the product placed on the shelf.

The verification unit 140 may verify the results calculated by the calculation unit 130.

At this time, the verification unit 140 may verify the calculation result using the captured image of the camera 160 that photographs the shelf and the sensing data of the LiDAR sensor 170.

The first memory 150 may store various algorithms, commands, artificial intelligence models, etc. for driving the product location calculation device 100.

In addition, the first memory 150 can store a list of products currently placed in the device 100 and the number of products, and can store a list of products placed on each shelf and the weight and shape information of each product. there is.

The first memory 150 can store data supporting various functions of the device and programs for the operation of the control unit, and stores input/output data (e.g., music files, still images, videos, etc.). A number of application programs (application programs or applications) running on the device, data for operation of the device, and commands can be stored. At least some of these applications may be downloaded from an external server via wireless communication.

The first memory 150 may be a flash first memory 150 type, a hard disk type, a solid state disk type, an SDD type, or a silicon disk drive type. Multimedia card micro type, card type first memory 150 (e.g. SD or XD first memory 150, etc.), random access memory (RAM), static random access (SRAM) memory), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic first memory 150, at least one type of magnetic disk and optical disk. It may include storage media. Additionally, the first memory 150 is separate from the main device, but may be a database connected by wire or wirelessly.

In this way, the verification unit 140 can use the weight and shape information of the product stored in the first memory 150 to verify the calculation result of the calculation unit 130.

One or more cameras 160 may be installed on each shelf, and may be installed on the upper part of the shelf so that at least a portion of the shelf can be photographed.

More specifically, in an embodiment of the present disclosure, at least one camera 160 is installed on each shelf, and can generate captured images by photographing the shelves at a plurality of different viewing angles.

The LiDAR sensor 170 may be provided on both sides of the shelf, and may generate a sensing value by sensing the direction of the shelf under the control of the first processor 110.

Among the components, the first communication unit 180 may include one or more components that enable communication with an external device, for example, a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, It may include at least one of the location information modules.

The first processor 110 is responsible for controlling the product location calculation device 100 and components within the unmanned vending device, and can execute a product location calculation method and an unmanned vending method using algorithms, commands, and artificial intelligence models.

The first processor 110 performs the above-described operations using a first memory that stores data for an algorithm for controlling the operation of components in the device or a program that reproduces the algorithm, and the data stored in the first memory. may be implemented with at least one first processor. At this time, the first memory and the first processor may each be implemented as separate chips. Alternatively, the first memory and the first processor may be implemented as a single chip.

In addition, the first processor 110 may control any one or a combination of the above-described components in order to implement various embodiments according to the present disclosure described in FIGS. 2 to N below on the present device. there is.

The sensing unit senses at least one of the device's internal information, surrounding environment information surrounding the device, and user information, and generates a corresponding sensing signal. Based on these sensing signals, the control unit can control the driving or operation of the device, or perform data processing, functions, or operations related to an application program installed on the device.

In an embodiment of the present disclosure, the load cell 120, camera 160, and lidar sensor 170 may be included in the sensing unit.

As described above, the sensing unit includes a proximity sensor, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), and a gyro. Gyroscope sensor, motion sensor, RGB sensor, IR sensor (infrared sensor), finger scan sensor, ultrasonic sensor, optical sensor, e.g. For example, a camera), a microphone, an environmental sensor (e.g., including at least one of a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, and a gas detection sensor), a chemical sensor (e.g., a healthcare sensor) , biometric sensors, etc.) may be included. Meanwhile, this device can utilize information sensed by at least two of these sensors by combining them.

In some embodiments, the first processor 110 includes an image acquisition unit that acquires an image captured by the camera 160, a product recognition unit that analyzes the acquired image to recognize the product, and detects that the location of the product has changed. It may further include a location change determination unit that determines/calculates the changed location. Additionally, the first processor 110 may use the calculation unit 130 to calculate the number of products placed on the shelf.

Figure 4 is a flowchart of a product location calculation method according to an embodiment of the present disclosure.

Figure 5 is a flow chart illustrating the details of S200.

The first processor 110 detects that a product is placed on the shelf. (S100)

In an embodiment of the present disclosure, when the zero point of the plurality of load cells 120 provided on the shelf is adjusted and no product is placed on the shelf, all load cells 120 may sense a sensing value of “0”. .

Also, when at least one product is placed on the shelf, a value other than “0” is sensed in at least one load cell 120, and at this time, the first processor 110 may determine that the product is placed on the shelf.

However, it is not limited to this, and when a sensor provided in the unmanned vending machine senses a person placing a product on a shelf, the first processor 110 may execute S100. In this case, the first processor 110 can control the load cell 120 of the lathe to start operating by switching it from sleep mode to wake up mode.

The first processor 110 calculates location information of products placed on the shelf. (S200)

The first processor 110 calculates the first position information of the product placed on the shelf based on the sensing value sensed by the plurality of load cells 120 and the position information of each of the plurality of load cells 120 that sensed the sensing values. .

In detail, when a product is placed on a shelf and a value other than “0” is sensed in at least one load cell 120, the first processor 110 detects a value other than “0” in the load cell 120. At least one of first position information and weight can be calculated based on the position information and the sensing value sensed by the corresponding load cell 120.

Below, the S200 process is explained in more detail.

Figure 6 is a diagram illustrating that a first load cell 121 is provided on one side of each line of a shelf, and a second load cell 122 is provided on the other side.

In the first embodiment of the present disclosure, the plurality of load cells 120 are provided with a first load cell 121 on one side of each line and a second load cell 122 on the other side of each line.

The first processor 110 may calculate the first position information of the product placed on the shelf by internally calculating the sensing values sensed by the first load cell 121 and the second load cell 122 on each line.

Referring to FIG. 6, eight lines are illustrated on a shelf, and a first load cell 121 is provided on one side of each of the eight lines (upper side in FIG. 6), and a first load cell 121 is provided on each of the other sides of the eight lines (lower side in FIG. 6). A second load cell 122 is provided.

The first processor 110 sensed “8” from the first load cells (121-1, 121-2, and 121-3) and “4” from the second load cells (122-1, 122-2, and 122-3). Since " was sensed, it can be determined that the product was placed on the 4th, 5th, and 6th lines from the left.

In response, the first processor 110 can calculate the placement value of the product on the x-axis, and in FIG. 6, the placement value of the product on the x-axis can be calculated as 4, 5, and 6.

In addition, the first processor 110 may calculate the first location information of the product by calculating the sensing value sensed by the product in the first load cell 121 and the second load cell 122 (internal division point formula).

In one embodiment, the first processor 110 calculates the sensing values sensed by the first load cell 121 and the second load cell 122 using an internal point formula to provide first location information (e.g., coordinates) of the product placed on the shelf. ) Any method of calculating can be applied.

As an example, the first processor 110 may calculate the first location information of the product based on Equation 1.

Product location information = (na + mb) / (m + n), (m: 1st load cell sensing value, n: 2nd load cell sensing value, a: 1st load cell coordinates, b: 2nd load cell coordinates)

At this time, when products are arranged across multiple columns, the measured/sensed weight value may be different for each column due to the characteristics of vending machine and refrigerator shelves. In one embodiment, the first processor 110 may recognize and process the objects as the same when the difference in weight between columns is less than a threshold.

Figure 7 is a diagram illustrating that the first load cell 121 and the second load cell 122 are provided on one side of each line of the shelf.

Referring to FIG. 7, eight lines on a shelf are illustrated, and a first load cell 121 and a second load cell 122 are provided on each side of the eight lines (upper side in FIG. 7).

The first processor 110 sensed “8” from the first load cells (121-1, 121-2, and 121-3) and “4” from the second load cells (122-1, 122-2, and 122-3). Since " was sensed, it can be determined that the product was placed on the 4th, 5th, and 6th lines from the left.

In response, the first processor 110 can calculate the placement value of the product on the x-axis, and in FIG. 6, the placement value of the product on the x-axis can be calculated as 4, 5, and 6.

In addition, the first processor 110 may calculate the first position information of the product by calculating the external differentiation (external point formula) of the sensing value of the product sensed by the first load cell 121 and the second load cell 122.

In one embodiment, the first processor 110 calculates the sensing values sensed by the first load cell 121 and the second load cell 122 using an extrinsic point formula to provide first location information (e.g., coordinates) of the product placed on the shelf. ) Any method of calculating can be applied.

As an example, the first processor 110 may calculate the first location information of the product based on Equation 2.

First location information of product = (mb - na) / (m - n), (m: first load cell sensing value, n: second load cell sensing value, a: first load cell coordinates, b: second load cell coordinates)

When applying the present disclosure as in the first embodiment, the load cell 120 can be placed on both sides of the line. Also, when applying the present disclosure as in the second embodiment, the load cell 120 can be placed at one end of the line.

Ultimately, the results produced in the first and second embodiments are the same, and the practitioner of the invention can easily select one by considering the characteristics of the shelf and the characteristics of the device to which the shelf is applied (refrigerator, display stand, etc.).

The first processor 110 verifies the calculation result of S200. (S300)

Figures 8 and 9 are diagrams illustrating verification of the calculation result of S200 using images captured by the camera 160.

In the embodiment of the present disclosure, various methods can be applied to verify the calculation result of S200, and a single method may be applied or a mixture of various methods may be applied.

Referring to FIGS. 8 and 9 , the first processor 110 acquires images captured by a plurality of cameras 160 of a shelf on which a product is placed. (S310)

The first processor 110 verifies the calculation result of S200 based on the plurality of captured images acquired in S310. (S330)

In one embodiment, the shelf is equipped with at least one camera 160 capable of photographing at least a portion of the shelf at a plurality of different angles of view.

The first processor 110 controls the camera 160 to generate a plurality of captured images by photographing the shelf at a plurality of different angles of view. The first processor 110 may verify the calculation result of the first location information of the product by matching the first location information of the product calculated in S200 with a plurality of captured images.

In one embodiment, the first processor 110 calculates product image second location information (location information calculated based on camera images) based on a plurality of captured images, and calculates the calculated image second location information and It can be determined whether the first location information (position information calculated based on the load cell sensing value) matches.

In one embodiment, a shelf may have markers capable of recognizing a position (eg, coordinates) on the shelf at a plurality of preset points.

The first processor 110 may recognize the image and marker of the product in a plurality of shelf images taken at different angles of view and calculate second position information on the shelf of the product.

In one embodiment, the first processor 110 may recognize that a product is placed at the location of an unidentified marker in a plurality of shelf photographed images and calculate second location information for the product.

For example, if there are markers 1 to 100 on a shelf, and markers 10 to 15, 20 to 25, and 30 to 35 are not identified in multiple shelf images, the product is considered to have been placed at the location where the corresponding markers exist. 2 Location information can be calculated.

At this time, if two or more non-adjacent markers are not identified, the first processor 110 may recognize that the camera 160 captured the product incorrectly or that multiple products are placed on the shelf and generate an error signal.

Figures 10 and 11 are diagrams illustrating verification of the calculation result of S200 using the sensing value sensed through the lidar sensor 170.

In one embodiment, a LiDAR sensor 170 is provided/installed on both sides of the shelf.

The first processor 110 acquires the sensing value sensed through the LiDAR sensor 170. (S350)

The first processor 110 verifies the calculation result of S200 based on the sensing value of the LiDAR sensor 170. (S370)

The first processor 110 may verify the third location information of the product calculated in S200 based on the sensing values sensed through the lidar sensors 170 on both sides.

Specifically, a first LiDAR sensor 171 is provided on one side of the interior of the shelf, and a second LiDAR sensor 172 is provided on the other side of the interior.

The first processor 110 provides third location information of the product placed on the shelf based on the sensing value sensed through the first LiDAR sensor 171 and the sensing value sensed through the second LiDAR sensor 172. It can be calculated. The first processor 110 may verify the calculated second location information by comparing the calculated second location information of the product and the calculated third location information.

At this time, the first processor 110 determines the appearance information of the product placed on the shelf based on the sensing value sensed through the first LiDAR sensor 171 and the sensing value sensed through the second LiDAR sensor 172. It can be created and the type of product can be judged based on appearance information.

The first processor 110 may match the appearance information with each product information in the product list stored in the first memory 150 to determine the type of product.

In one embodiment, the first processor 110 may assign a plurality of sensing points to each of the first LiDAR sensor 171 and the second LiDAR sensor 172 based on the calculated first location information of the product. there is. The first processor 110 verifies the first location information of the product based on the sensing values sensed through the first LiDAR sensor 171 and the second LiDAR sensor 172 for a plurality of assigned sensing points. You can.

The first processor 110 determines the type of product placed on the shelf. (S400)

In one embodiment, the first processor 110 calculates the weight and The expected value of the lower part shape can be calculated.

The first processor 110 may determine the type of product based on the calculated weight of the product and the expected shape of the bottom of the product.

In an embodiment of the present disclosure, the first memory 150 may store information, weight, and lower shape information of each product placed on the shelf.

The first processor 110 determines whether the product is properly placed. (S500)

The first processor 110 may determine whether the product placed on the shelf is normally placed based on the calculated first location information of the product and the determined product type.

The first memory 150 may store at least one of the type of product to be placed on each shelf, the weight of the product, the external shape of the product, and the bottom shape of the product.

The first processor 110 compares the outer shape information or bottom shape information of the product to be placed on each shelf with the first position information to determine whether the product to be placed on the corresponding shelf is correctly placed or whether the product is displayed properly. can be judged.

This means that the first processor 110 can determine whether the product type is properly placed, as well as whether the product is properly placed top and bottom and left and right.

According to the product position calculation device 100 in the artificial intelligence-based product showcase according to the embodiment of the present disclosure described above, the position can be accurately calculated even if products of various shapes and weights are placed on the shelf, and the correct product can be Not only can it be determined whether it has been properly placed, but it also has the effect of further improving accuracy by verifying the calculated location information.

Above, the product location calculation method and device 100 according to an embodiment of the present disclosure have been described. Below, an embodiment in which the product location calculation device 100 calculates the number of product sales using a product recognition algorithm is additionally described. Let me explain.

The algorithm that calculates the location of the product and the algorithm that recognizes the product and calculates the number of sales can be implemented independently or used together, so the two embodiments can be implemented separately or a mixture of the two embodiments can be used. It may be possible.

Accordingly, the product location calculation device 100 according to an embodiment of the present disclosure may include at least a portion of the configuration and functions for the algorithm for calculating the number of products sold, which will be described below.

Below, an algorithm for calculating the number of product sales using a product recognition algorithm according to an additional embodiment of the present disclosure will be described.

The product location calculation device 100 according to an embodiment of the present disclosure can calculate the number of products sold and may include the following configurations.

The product location calculation device 100 according to the present disclosure communicates with an external device 100 including a memory for storing the standard weight of the product and a preset correction index, at least one camera 160, and a weight measurement sensor. It includes a communication module that performs communication, and at least one processor that communicates with the memory, wherein the at least one processor receives first product information from the external device 100 when the door of the product storage unit is opened, When the door of the product storage unit is closed, second product information is received from the external device 100, and an absolute calculation is performed based on the weight information included in each of the first product information and the second product information to product the product. The number of sales is calculated, and if it is determined that an error has occurred according to an error detection algorithm predetermined based on the second product information, the absolute operation is stopped and included in each of the first product information and the second product information. A relative operation is performed based on the weight information difference value to calculate the number of products sold, and the correction index is changed based on the tilt information of the load cell 120 received from the external device 100 to determine the error. If it is determined that is resolved, the relative operation is stopped and the absolute operation is performed again.

In addition, for the absolute calculation and the relative calculation, machine learning is performed based on the product image captured from the camera 160 to determine the calculation value for each, and the learning model that performs the machine learning is multi-class. Multiclass classification and binary classification can be implemented, and the multiclass classification is performed by a first convolutional neural network (CNN) model, and the binary classification is performed by a second convolutional neural network (CNN) model. It is performed by a neural network model, and the predetermined error detection algorithm is, when it is determined by the second convolutional neural network model that there is no product in the load cell 120, and the load cell measured by the weight measurement sensor If it is determined that the product exists based on the weight information in (120), it is determined that an error has occurred. Otherwise, it is an algorithm that determines that an error has not occurred. If it is determined that the error has occurred, the algorithm determines that the error has occurred. Determining that the set correction index needs to be changed, receiving tilt information of the load cell 120 from the external device 100, and changing the preset correction index based on the tilt information of the load cell 120, When the preset correction index is changed, it can be determined that the error has been resolved.

Additionally, the product location calculation method according to an embodiment of the present disclosure can calculate the number of products sold and may include the following components.

The product location calculation device 100 stores the standard weight of the product and a preset correction index in the memory, and when the door of the product storage section is opened, the first product information is received from the external device 100 including the camera 160 and the weight measurement sensor. receiving, when the door of the product storage unit is closed, receiving second product information from the external device 100, absolute weight information included in each of the first product information and the second product information. Calculating the sales number of the product by performing an operation; if it is determined that an error has occurred according to a predetermined error detection algorithm based on the second product information, stopping the absolute calculation, and the first product information and the Calculating the number of products sold by performing a relative calculation based on the weight information difference value included in each second product information, based on the tilt information of the load cell 120 received from the external device 100 If it is determined that the error is resolved by changing the correction index, stopping the relative operation and performing the absolute operation again.

In addition, the first product information is information about the product image captured by the camera 160 and the weight of the load cell 120 measured by the weight measurement sensor when the door of the product storage unit is opened. 2 Product information is characterized in that it is information about the product image captured by the camera 160 and the weight of the load cell 120 measured by the weight measurement sensor when the door of the product storage unit is closed.

In addition, for the absolute calculation and the relative calculation, machine learning is performed based on the product image captured from the camera 160 to determine the calculation value for each, and the learning model that performs the machine learning is multi-class. Multiclass classification and binary classification can be implemented, and the multiclass classification is performed by a first convolutional neural network (CNN) model, and the binary classification is performed by a second convolutional neural network (CNN) model. It can be performed by a neural network model.

In addition, the predetermined error detection algorithm determines the inventory number based on the absolute calculation based on when the door is closed, and calculates the inventory number multiplied by the standard weight of the product and measured by the weight measurement sensor. A ratio is calculated by comparing the information on the weight of the load cell 120, and if the ratio differs by more than a preset value, it is determined that the error has occurred, and if the ratio differs by less than a preset value, It is characterized by an algorithm that determines that the above error has not occurred.

In addition, if it is determined that the error has occurred, it is determined that the preset correction index needs to be changed, Tilt information of the load cell 120 is received from the external device 100, the preset correction index is changed based on the tilt information of the load cell 120, and when the preset correction index is changed, the error occurs. can be judged to have been resolved.

In addition, the predetermined error detection algorithm determines that there is no product in the load cell 120 by the second convolutional neural network model, and the weight of the load cell 120 measured by the weight measurement sensor If it is determined that the product exists based on the information, it is determined that an error has occurred, and if not, it is characterized as an algorithm that determines that an error has not occurred.

In addition, when it is determined that the error has occurred, it is determined that the preset correction index needs to be changed, tilt information of the load cell 120 is received from the external device 100, and tilt information of the load cell 120 is received. The preset correction index is changed based on , and when the preset correction index is changed, it can be determined that the error has been resolved.

Comparing FIGS. 3 and 13 in the embodiment of the present disclosure, the device 100 further includes a weight measurement sensor 133, a tilt measurement sensor 137, and a door unit 190.

In this way, in order to execute the method and algorithm for calculating the number of product sales through a product recognition algorithm, some components may be omitted or some additional components may be included in the product location calculation device 100 of FIG. 3.

In addition, the load cell 120 and camera 160 of FIG. 3 may adopt the configuration of the load cell 120 and camera 160 shown in FIGS. 3 to 11, or may be used as a separate additional load cell 120 and camera. (160) may be additionally provided. In this case, the product location calculation device 100 may include at least one third load cell 120 and at least one second camera 160.

Accordingly, in an embodiment of the present disclosure, the load cell 120 may include at least one first load cell, at least one second load cell, and at least one third load cell, and the camera 160 may include at least one first camera, It may include at least one second camera.

Figure 12 is a diagram schematically showing communication between the product location calculation device 100 and the server according to an embodiment, and Figure 13 is a diagram showing the configuration included in the product location calculation device 100 and the server according to an embodiment. This is a diagram schematically showing.

Referring to Figures 12 and 13, a product location calculation device 100 and a server may be provided to perform the operation of the present invention.

In addition, in the embodiment described below, it is exemplified that data collected from the product location calculation device 100 is analyzed and processed by the server, but this is not limited to this and all operations performed by the second processor 210 of the server Can also be performed by the first processor 110 of the product location calculation device 100. In other words, the product location calculation device 100 can recognize products within the product location calculation device 100 and calculate the number of products sold without transmitting data to the server.

Referring to FIG. 13, the product position calculation device 100 includes a first processor 110, a load cell 120, a calculation unit 130, a weight measurement sensor 133, a tilt measurement sensor 137, and a verification unit 140. ), a first memory 150, a camera 160, a LiDAR sensor 170, a first communication unit 180, and a door unit 190.

The server may be provided to include a second memory 220, a second communication unit 230 capable of communicating with the product location calculation device 100, and a second processor 210.

Meanwhile, the load cell 120 may refer to a storage space in which products can be stored within the product position calculation device 100. Since the load cell 120 is inclined by the inclination angle θ1, when the user receives the product, the product can move toward the door unit 190 along the inclined plane. The load cell may be prepared/installed in a storage space that can store products within the product position calculation device 100. Additionally, in an embodiment of the present disclosure, the product location calculation device may be provided in a display/exhibition showcase to sell products.

Meanwhile, in order to measure the weight of a product stored in the product location calculation device 100, a weight measurement sensor 133 may be mounted on the load cell 120.

According to one embodiment of the present invention, the weight measurement sensor 133 is attached to the lower part of the load cell 120, and the sensing distance and sensing resistance value vary depending on the weight of the product on the load cell 120, and based on this, the weight can be measured. For example, when comparing the case where there are 10 products remaining on the load cell 120 and the case where there are 5 products remaining, in the case where there are 10 products remaining, the sensing resistance value increases as the sensing distance becomes shorter, and the product In the case where there are five remaining teeth, the sensing resistance value may decrease as the sensing distance increases, and the weight can be measured based on this.

Meanwhile, in order to measure the tilt of the load cell 120, a tilt measurement sensor 137 may be provided in the product position calculation device 100. The tilt measurement sensor 137 can be used to preset the correction index (k1) as described later, and to change the correction index (k2) by measuring a new tilt angle when an error occurs.

Meanwhile, the weight measurement sensor 133 and the tilt measurement sensor 137 sense at least one of the internal information of the device 100, the surrounding environment information surrounding the device 100, and the user information, and send a sensing signal corresponding thereto. generates Based on these sensing signals, the processor 110 or 210 may control the driving or operation of the device 100 or perform data processing, functions, or operations related to an application program installed on the device 100.

Meanwhile, the door unit 190 may be the product location calculating device 100 or a door of a product storage unit mounted on a product showcase. As described later, when performing absolute calculation in ABS mode when calculating the number of products sold, the calculation may be performed based on when the door of the door unit 190 is opened and when the door is closed.

Meanwhile, the camera 160 may refer to an artificial intelligence-based machine vision camera mounted on the product location calculation device 100. The image of the product captured by the camera 160 is transmitted to the server, and the second processor 210 can perform machine learning based on the received image.

As described above, the camera 160 of FIG. 13 may be applied in a different configuration from the camera 160 described with reference to FIGS. 3 to 11. For example, if the camera in FIG. 3 is configured to photograph each shelf, the camera in FIG. 13 may mean a device equipped to calculate the number of products.

That is, in the embodiment of the present disclosure, the device 100 may calculate the number of products sold based on the captured image of the camera 160 installed on each shelf of the product showcase, and may additionally calculate the number of products sold based on a camera installed at another location ( 160), the number of products sold can also be calculated based on the captured video.

Additionally, the first processor 110 of the device 100 may also perform machine learning based on the image captured by the camera 160.

Meanwhile, machine vision cameras can capture product videos and transmit them to the server. A machine vision camera may consist of a lens, an image sensor, a main board, and an interface board, but is not limited to this. Additionally, images created through lenses and image sensors can be corrected into an appropriate form on the motherboard as needed. The video processed on the motherboard in this way can be transmitted to the server. Machine vision cameras may include GigE Vision cameras (Gigabit Ethernet Vision Cameras), USB3.0 cameras, CameraLink cameras, and CoaXPress cameras.

Meanwhile, the memory can store the standard weight of the product and a preset correction index (k1) as described later.

The standard weight of the product is the general weight of the product on the load cell 120 and may be a fixed value stored in advance in the memory.

The standard weight of a product may be stored differently depending on the type of product in the load cell 120. For example, if the product of load cell 1 is a 185ml can of cider, the standard weight of the can of cider can be stored as 200g, and if the product of load cell 2 is a 250ml can of cola, the standard weight of the can of cola can be stored as 300g. It can be.

The memory can store data supporting various functions of the device 100 and programs for the operation of the processors 110 and 210, and can store input/output data (e.g., music files, still images, and videos). etc.), a plurality of application programs (application programs or applications) running on the device 100, data for operation of the device 100, and commands can be stored. At least some of these applications may be downloaded from an external server via wireless communication.

These memories include flash memory type, hard disk type, SSD type (Solid State Disk type), SDD type (Silicon Disk Drive type), and multimedia card micro type. , card-type memory (e.g., SD or It may include at least one type of storage medium among (only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. Additionally, the memory is separate from the device 100, but may be a database connected wired or wirelessly.

At least one processor (110 or 210) has a memory that stores data for an algorithm for controlling the operation of components in the device 100 or a program that reproduces the algorithm, and performs the above-described operations using the data stored in the memory. can be performed. At this time, the memory and processor 110 or 210 may each be implemented as separate chips. Alternatively, the memory and processor 110 or 210 may be implemented as a single chip.

In addition, the processor 110 or 210 combines any one or a plurality of the above-described components to implement various embodiments according to the present disclosure described in FIGS. 13 to 21 below on the device 100. You can control it.

Among the components, the first communication unit 180 of the product location calculation device 100 and the second communication unit 230 of the server may include one or more components that enable communication with the external device 100, For example, it may be at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

Wired communication modules include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as USB (Universal Serial Bus) modules. ), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, wireless communication modules include GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and UMTS (universal mobile telecommunications system). ), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, etc. may include a wireless communication module that supports various wireless communication methods.

The wireless communication module may include a wireless communication interface including an antenna and a transmitter that transmits a mobile communication signal. Additionally, the wireless communication module may further include a mobile communication signal conversion module that modulates a digital control signal output from the control unit through a wireless communication interface into an analog wireless signal under the control of the control unit.

The wireless communication module may include a wireless communication interface including an antenna and a receiver for receiving mobile communication signals. Additionally, the wireless communication module may further include a mobile communication signal conversion module for demodulating an analog wireless signal received through a wireless communication interface into a digital control signal.

The short-range communication module is for short-range communication, including Bluetooth®, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, and NFC (Near). Short-distance communication can be supported using at least one of (Field Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

In an embodiment of the present disclosure, the first processor 110 or the second processor 210 includes a product sales count calculation unit 213 and a machine vision camera ( It may include a machine learning unit 215 that performs machine learning of an artificial intelligence learning model based on the product image captured by 160).

Meanwhile, each component shown in FIG. 13 refers to software and/or hardware components such as Field Programmable Gate Array (FPGA) and Application Specific Integrated Circuit (ASIC).

FIG. 14 is a diagram showing mode switching according to a product recognition algorithm according to an embodiment, and FIG. 15 is a diagram showing a product location calculation device 100 including a camera 160 and a weight measurement sensor 133 according to an embodiment. This is a schematic drawing.

Referring to FIG. 14, a product can be stored in the load cell 120 with an inclination angle of θ1, and when a user receives the product 30 according to the slope, the product can be moved toward the door unit 190. The camera 160 can photograph the front part of the load cell 120 close to the door 190. Therefore, when the product image captured by the machine vision camera 160 is transmitted to the processor 110 or 210, learning is performed by the second convolutional neural network (CNN) model as described later, and the product 30 is You can determine whether it exists or not. As described later, when performing absolute calculations in ABS mode and relative calculations in REL mode, if a product exists, Nv, which is a code value based on image digital recognition, may be judged as 1, and if a product does not exist, Nv may be judged as 0. You can.

Meanwhile, the product 30 in FIG. 14 may refer to each product 30 displayed on the load cell 120.

Referring to Figure 14, the present invention is basically set to ABS mode (Absolute mode, 310), and absolute calculation can be performed when calculating the number of products sold according to ABS mode (310). At this time, if it is determined that an error has occurred according to a predetermined error detection algorithm, the absolute operation according to the ABS mode (310) is stopped, and the mode is switched to the REL mode (Relative mode, 320) to perform the relative operation (S410). In addition, when it is determined that the error has been resolved according to a predetermined correction algorithm, the relative operation according to the REL mode 320 is stopped, and the mode is switched back to the ABS mode 310 to perform an absolute operation (S420).

Referring to FIG. 15, the weight measurement sensor 133 is mounted on the lower part of the load cell 120 as described above and can measure the weight of the product on the load cell 120.

Meanwhile, the weight measurement sensor 133 can be set to measure the weight of the product when the door of the door 190 of the product location calculation device 100 is opened and to measure the weight of the product when the door of the door 190 is closed. there is.

The first product information may include weight information of the product based on when the door of the door unit 190 is opened and image information of the product captured by the camera 160, but is not limited thereto.

The second product information may include, but is not limited to, weight information of the product and image information of the product captured by the camera 160 based on when the door of the door unit 190 is closed.

Meanwhile, the first product information and the second product information are information about products in the same column on the load cell 120, and may be product information based on when the door is opened and when the door is closed.

In this way, the first product information and the second product information can be transmitted from the product location calculation device 100 to the server through communication between the first communication unit 180 of the product location calculation device 100 and the second communication unit of the server.

Figure 16 is a flowchart showing steps performed in the server and product location calculation device 100 according to an embodiment.

According to one embodiment of the present invention, the user can make the product location calculation device 100 recognize payment methods that can pay for products, including check cards and credit cards, in order to pay for products (S601). The card information recognized in this way may be transmitted to the processor 210 through the second communication unit of the server (S602). The server can receive the card information transmitted in this way (S603). If the server recognizes the card information as correct, the server may transmit a command to open the door unit 190 to the product location calculation device 100 (S604). As the door unit 190 door open command is transmitted, when the door unit 190 of the product location calculation device 100 is opened (S605), information that the door unit 190 has been opened may be transmitted to the server. There is (S606). As the door unit 190 is opened, the server can receive first product information (S607). When a product is received from the user product location calculating device 100 (S608) and the door of the door unit 190 is closed (S609), information that the door of the door unit 190 is closed may be transmitted to the server (S610). . As the door of the door unit 190 is closed, the processor 110 or 210 may receive second product information (S611). As the server receives the first product information and the second product information, it can perform calculations based on the first product information and the second product information to calculate the number of products sold by the user (S612). The calculated number of products sold is transmitted to the product location calculation device 100 (S613), and product payment according to the number of products sold can be made (S614).

FIG. 17 is a diagram schematically showing measuring the inclination angle of the load cell 120 using the inclination measurement sensor 137 according to an embodiment.

Meanwhile, the product 30 in FIG. 17 may refer to each product displayed on the load cell 120.

The load cell 120 may be inclined by an inclination angle θ1 based on a flat surface, so that when a user receives a product from the product location calculation device 100, the product on the side farthest from the door unit 190 along the inclined plane is moved to the door. You can move towards section 190.

The tilt measurement sensor 137 according to an embodiment of the present invention is attached to the bottom of the load cell 120 and can measure the tilt of the load cell 120.

Meanwhile, the measured inclination angle (θ1) can be used to determine a preset correction index (k1) stored in memory. Since the weight of the inclined load cell 120 may be different from the weight on flat ground, a correction model based on a preset correction index can be used. That is, the weight can be corrected using a correction index that can correct the weight measurement error that occurs due to the inclination of the load cell 120.

According to one embodiment, the equation of the correction model can be expressed as Equation 1 below.

Referring to Equation 1, Mb may mean the weight on the inclined load cell 120, k1 may mean the preset correction index, and Ma may mean the weight on flat ground.

Meanwhile, when the inclination angle of the load cell 120 is θ1, the preset correction index k1 may be cos θ1 or cscθ1, but is not limited thereto.

In addition, the preset correction index (k1) according to one embodiment is the actual weight value of the weight (e.g., 100 g, 200 g, etc.) when placing a unit weight (e.g., 100 g, 200 g, etc.) on the weight measurement sensor 133 at the time of installing the load cell 120. It can also be set by adjusting the zero point of the weight measurement sensor 133 so that it can be read.

Figure 18 is a diagram illustrating an algorithm for the process from when a mode change is made to when an error is resolved according to an embodiment.

The present invention basically starts in ABS mode (S810), and absolute calculation can be performed based on weight information included in each of the first product information and the second product information (S820). Accordingly, the number of products sold can be calculated (S830), and a process of determining that an error has occurred according to a predetermined error detection algorithm based on the second product information is performed (S840). If no error occurs, absolute calculation according to ABS mode is continuously performed, and if it is determined that an error has occurred, it switches to REL mode (S850). As the mode is switched to REL mode, a relative calculation can be performed based on the difference value of the weight information included in each of the first product information and the second product information (S860). Accordingly, the number of products sold can be calculated (S870), and at the same time, based on the tilt information of the load cell 120 from the tilt measurement sensor 137 included in the product position calculation device 100 according to a predetermined correction algorithm, The set correction index (k1) can be changed (S880). Accordingly, if it is determined that the error has been resolved, it switches back to ABS mode and absolute calculation can be performed. However, if it is determined that the error has not been resolved, REL mode is maintained and the number of products sold according to relative calculation can be calculated ( S890).

Figure 19 is a diagram illustrating an algorithm for the process of performing an absolute operation in ABS mode according to an embodiment.

First, the standard weight (Wm) of the product is stored in memory (S910), the first product information can be received when the door is opened (S920), and the second product information can be received when the door is closed (S930). Meanwhile, as described above, the first product information and the second product information are the weight information of the load cell 120 by the weight measurement sensor 133 and the machine vision camera 160 for when the door is opened and when the door is closed, respectively. ) may include image information of the product taken by .

The processor 110 or 210 may perform machine learning using a learning model. The calculation values of the absolute calculation in ABS mode and the relative calculation in REL mode as described later can be determined by performing machine learning using the image of the product captured by the machine vision camera 160 as data. A learning model that performs machine learning can implement multiclass classification and binary classification. Meanwhile, multiclass classification may be performed by a first convolutional neural network (CNN) model, and binary classification may be performed by a second convolutional neural network model, but are not limited thereto. For example, the learning models include Random Forest (RF), Support Vector Machine (SVC), eXtra Gradient Boost (XGB), Decision Tree (DC), Knearest Neighbors (KNN), Gaussian Naive Bayes (GNB), and Stochastic Gradient. It may include at least one algorithm among Descent (SGD), Linear Discriminant Analysis (LDA), Ridge, Lasso, and Elastic net.

Whether a product exists when the door of the door unit 190 is closed can be determined by implementing binary classification by a second convolutional neural network model.

Based on the time the door is closed after the user receives the product, if the product is recognized as existing, the digital code value Nv = 1, and if the product is recognized as not existing, the digital code value can be recognized as Nv = 0.

In addition, based on when the door is closed after the user receives the product, if it is recognized that the product exists, the standard weight Wv of the product recognized by the machine vision camera 160 can be received from the memory, and if the product does not exist, the standard weight Wv of the product can be received. If it is recognized as not being the case, the standard weight (Wv) of the product recognized by the machine vision camera 160 may be determined to be 0.

The processor 110 or 210 can perform an absolute operation according to the ABS mode (S970), and the formula for the absolute operation can be expressed as Equation 2 below.

Referring to Equation 2, Ns refers to the number of products sold, Nb refers to the number of inventory when the door is opened, Nt refers to the number of inventory when the door is closed, Wm refers to the standard weight of the product, Wc refers to the weight of the load cell 120 when the door is closed, Wv refers to the standard weight of the product recognized by the camera 160, and Nv refers to the digital code value of the product recognized by the camera 160. can do.

As the absolute operation is performed, the number of products sold (Ns) can be calculated (S980). At this time, in order to calculate the exact number of units sold, it may be determined whether an error has occurred according to a predetermined error detection algorithm based on the second product information (S990). If an error occurs, the mode is switched to REL mode and a relative operation can be performed (S9100), and if an error does not occur, the ABS mode is maintained and an absolute operation can be performed (S9110).

Figure 20 is a diagram illustrating an algorithm for a process in which a relative operation is performed in the REL mode according to an embodiment.

First, the standard weight (Wm) of the product is stored in the memory (S1010), the first product information can be received when the door is opened (S1020), and the second product information can be received when the door is closed (S1030). When the weight of the load cell 120 when the door is opened is W1 and the weight of the load cell 120 when the door is closed is W2, each load cell 120 included in the first product information and the second product information Based on the weight information, the difference value, Wd=W1-W2, can be calculated (S1040).

In the case of the relative operation in REL mode, as in the absolute operation in ABS mode, whether the product exists when the door of the door unit 190 is closed is determined by implementing binary classification by the second convolutional neural network model. You can.

Based on the time the door is closed after the user receives the product, if the product is recognized as existing, the digital code value Nv = 1, and if the product is recognized as not existing, the digital code value can be recognized as Nv = 0.

In addition, based on when the door is closed after the user receives the product, if it is recognized that the product exists, the standard weight Wv of the product recognized by the machine vision camera 160 can be received from the memory, and if the product does not exist, the standard weight Wv of the product can be received. If it is recognized as not being the case, the standard weight (Wv) of the product recognized by the machine vision camera 160 may be determined to be 0.

The processor 110 or 210 can perform a relative operation according to the REL mode (S1080), and the formula for the relative operation can be expressed as Equation 3 below.

Referring to Equation 3, Ns means the number of products sold, W1 means the weight of the load cell 120 when the door is opened, W2 means the weight of the load cell 120 when the door is closed, and Wd is It means the difference in weight of the load cell 120 when the door is opened and when the door is closed, Wm means the standard weight of the product, Wv means the standard weight of the product recognized by the camera 160, Nv may mean the digital code value of the product recognized by the camera 160.

As the relative operation is performed, the number of products sold (Ns) can be calculated (S1090). At this time, the tilt measurement sensor 137 included in the product position calculation device 100 can measure the tilt angle θ2 of the load cell 120 again. The correction index (k1) can be changed based on the tilt information of the load cell 120 received from the product position calculation device 100, and it can be determined whether the error has been resolved (S1100). Whether the error has been resolved can be determined by re-performing a predetermined error detection algorithm based on the second product information. If it is determined that the error has been resolved, the relative operation according to the REL mode is stopped and the absolute operation according to the ABS mode can be performed again (S1110). If it is determined that the error has not been resolved, the REL mode is maintained and the product according to the relative operation is At the same time as the number of sales is calculated, an error resolution algorithm can be implemented by continuously changing the correction index.

Figure 21 is a diagram illustrating an algorithm for a process for resolving an error when an error occurs according to an embodiment.

According to one embodiment, the tilt (θ1) of the inclined load cell 120 can be measured by the tilt measurement sensor 137 (S110), and a preset correction index (k1) can be calculated based on this (S1120). At this time, the preset correction index (k1) may be a cosθ1 value or cscθ1, but is not limited thereto. The preset correction index (k1) can be stored in memory (S1130). Absolute calculation according to ABS mode is performed (S1140), and if it is determined that an error has occurred according to the error detection algorithm (S1150), relative calculation according to REL mode is performed (S1160), and the number of products sold can be calculated (S1160) S1170). At the same time, according to the error occurrence signal, the tilt measurement sensor 137 of the product position calculation device 100 can measure the tilt (θ2) of the load cell 120 again. Accordingly, the preset correction index (k1) can be changed to a new correction index (k2) and corrected (S1190). At this time, the new correction index (k2) may be a cosθ2 value or cscθ2, but is not limited thereto. According to the process of determining whether the error has been resolved (S11100), if it is determined that the error has been resolved, the relative operation according to the REL mode is stopped (S1180) and the absolute operation according to the ABS mode can be performed again, and the error is not resolved. If it is determined that it is not, the relative operation according to the REL mode is maintained and the error resolution algorithm can continue to proceed.

Meanwhile, the process of determining whether an error has been resolved (S11100) may be performed by the same process as the process of determining whether an error has been detected. The predetermined error detection algorithm is a result value recognized by a second convolutional neural network model that implements binary classification that can determine the presence or absence of a product, and a weight measurement sensor that can measure the weight of the load cell 120. (133) It may be an algorithm that determines the error based on the difference in the measured values, but it is not limited to this.

The process of determining whether an error has been detected according to one embodiment is based on when the door is closed. If the inventory number is determined based on absolute calculation in ABS mode, the standard weight of the product (Wm) is added to the inventory number (Nt). ) is calculated by comparing the multiplied value with the information on the weight of the load cell 120 measured by the weight measurement sensor 133, and if the calculated ratio differs by more than a preset value, the error occurs. It may be an algorithm that determines that the error has occurred and, if the calculated ratio differs by less than a preset value, determines that the error has not occurred.

The process of determining whether an error has been detected according to one embodiment is when it is determined that there is no product in the load cell 120 (Nv=0) by a second convolutional neural network model that implements binary classification, and the weight measurement sensor When it is determined that the product exists based on the weight information of the load cell 120 measured by (133) (Wc ) may be an algorithm that determines that an error has occurred, and if not, determines that an error has not occurred.

The process of determining whether an error has been detected according to one embodiment is when it is determined that there is a product in the load cell 120 (Nv = 1) by a second convolutional neural network model that implements binary classification, and is stored in memory in advance. If the standard weight (Wm) of the stored product does not match the standard weight (Wv) of the product recognized by the machine vision camera 160, an algorithm determines that an error has occurred, otherwise, an algorithm determines that an error has not occurred. It can be.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by the processor 110 or 210, they may generate program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media storing instructions that can be decoded by a computer. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device 100, etc.

Meanwhile, the content described in FIGS. 18 to 21 is only an example for explaining the operation of the present invention, and there is no limitation to the operation in which the server performs communication with the product location calculation device 100 to calculate the number of products sold. .

The method according to an embodiment of the present disclosure described above may be implemented as a program (or application) and stored in a medium in order to be executed in combination with a server, which is hardware.

The above-described program includes C, C++, JAVA, and It may include code encoded in a computer language such as machine language. These codes may include functional codes related to functions that define the necessary functions for executing the methods, and control codes related to execution procedures necessary for the first processor of the computer to execute the functions according to predetermined procedures. may include. In addition, this code is related to a first memory reference as to which location (address address) of the internal or external first memory of the computer should be referenced by additional information or media required for the first processor of the computer to execute the functions. Additional code may be included. In addition, if the first processor of the computer needs to communicate with any other remote computer or server to execute the functions, the code can be transmitted to any other remote computer or server using the communication module of the computer. It may further include communication-related codes for how to communicate, etc., and what information or media should be transmitted and received during communication.

The storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or first memory. Specifically, examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers that the computer can access or on various recording media on the user's computer. Additionally, the medium may be distributed to computer systems connected to a network, and computer-readable code may be stored in a distributed manner.

The steps of the method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. Software modules include RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, and CD-ROM. , or it may reside on any form of computer-readable recording medium well known in the art to which this disclosure pertains.

Above, embodiments of the present disclosure have been described with reference to the attached drawings, but those skilled in the art will understand that the present disclosure can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

10: Unmanned vending system
30: Goods
100: Product location calculation device
110: first processor
120: load cell
121: first load cell
122: second load cell
130: Calculation unit
133: Weight measurement sensor
137: Tilt measurement sensor
140: verification unit
150: first memory
160: camera
170: Lidar sensor
171: First lidar sensor
172: Second lidar sensor
200: server
210: second processor
220: second memory
230: Second Communications Department

Claims (10)

A plurality of load cells for sensing the positions of products placed on the shelves of a display stand divided into a plurality of columns; and
A processor that calculates location information of the product placed on the shelf based on the sensing values sensed by the plurality of load cells and the location information of each of the plurality of load cells that sense the sensing values,
The plurality of load cells are respectively disposed on each line to separate the plurality of columns,
On one side of each line, a first load cell and a second load cell among the plurality of load cells are disposed adjacent to each other,
A device that determines whether a product is properly placed based on artificial intelligence.
According to paragraph 1,
The processor,
Characterized in calculating the position information of the product placed on the shelf by internal calculation of the sensing values sensed by the first load cell and the second load cell on each line,
A device that determines whether a product is properly placed based on artificial intelligence.
According to paragraph 2,
The processor,
Characterized in calculating the expected value of the weight and lower shape of the product placed on the shelf based on the sensing value sensed by the plurality of load cells and the location information of each of the plurality of load cells sensing the sensing value.
A device that determines whether a product is properly placed based on artificial intelligence.
According to paragraph 3,
The processor,
Characterized in determining the type of the product based on the calculated weight of the product and the expected lower shape of the product,
A device that determines whether a product is properly placed based on artificial intelligence.
According to paragraph 4,
The processor,
Characterized in determining whether the product placed on the shelf is normally placed based on the calculated location information of the product and the determined type of the product.
A device that determines whether a product is properly placed based on artificial intelligence.
According to clause 5,
The shelf includes a camera capable of photographing at least a portion of the shelf at a plurality of different angles of view,
The processor,
Controlling the camera to photograph the shelf at a plurality of different angles of view to generate a plurality of captured images,
Characterized in that the calculated location information of the product is matched with the plurality of captured images to verify the calculation result of the location information of the product.
A device that determines whether a product is properly placed based on artificial intelligence.
According to clause 6,
The processor,
Calculating image location information of the product based on the plurality of captured images,
Characterized in determining whether the calculated image location information matches the calculated location information,
A device that determines whether a product is properly placed based on artificial intelligence.
According to paragraph 1,
The shelf is provided with a LiDAR sensor on each side,
The processor,
Characterized in verifying the calculated location information of the product based on the sensing value sensed through the LiDAR sensor,
A device that determines whether a product is properly placed based on artificial intelligence.
A method performed by a product location calculation device,
A processor detecting that a product is placed on a shelf divided into a plurality of columns through a plurality of load cells; and
Comprising a step of calculating position information of the product placed on the shelf based on the sensing value sensed by the plurality of load cells and the position information of each of the plurality of load cells that sensed the sensing value,
The plurality of load cells are characterized in that a first load cell and a second load cell are provided on one side of each line to separate the plurality of columns, respectively.
A method of determining whether a product is properly placed based on artificial intelligence.
A program that is combined with a computer, which is hardware, and stored in a medium to execute the method of determining whether a product is properly placed based on the artificial intelligence of Paragraph 9.