KR102665299B1 - Product sales calculation method and calculation device through product recognition algorithm - Google Patents

Abstract

The method for calculating the number of products sold according to an embodiment of the present invention stores the standard weight of the product and a preset correction index in the memory, and stores first product information when the door of the product storage unit is opened and the door of the product storage unit is closed. Calculating the number of sales of the product by performing an absolute calculation based on the weight information included in each second product information at the time of purchase; determining that an error has occurred according to an error detection algorithm predetermined based on the second product information If determined, stopping the absolute calculation and performing a relative calculation based on the weight information difference value included in each of the first product information and the second product information to calculate the number of sales of the product, and an external device When it is determined that the error has been resolved by changing the correction index based on the tilt information of the load cell received from, stopping the relative operation and performing the absolute operation again, changing the correction index includes , It includes features determined based on the equation (Mb: weight on an inclined load cell, k1: preset correction index, Ma: weight on flat ground).

Description

Product sales quantity calculation method and calculation device through product recognition algorithm {PRODUCT SALES CALCULATION METHOD AND CALCULATION DEVICE THROUGH PRODUCT RECOGNITION ALGORITHM}

The present invention relates to a method of calculating the number of products sold, and more specifically, to a method of calculating the number of products sold using an algorithm that recognizes products when products are sold in an unmanned product sales device.

Unmanned stores are invading our daily lives. Due to the combined effects of increased non-face-to-face consumption due to the impact of COVID-19, the burden of labor costs, and the development of information and communication technology (ICT), unmanned technology is rapidly spreading throughout the retail industry.

As unmanned stores spread, demand for related technologies such as kiosks, artificial intelligence (AI) vision, and security solutions also increases, and the unmanned store industry ecosystem is growing rapidly.

Among the major technologies introduced in unmanned stores, machine vision is a technology that identifies customers' purchasing behavior through artificial intelligence-based image and video analysis. Product inventory determination is made using machine vision-based AI cameras or weight measurement sensors, but there is a problem of not obtaining accurate payment information when paying for products when the number of units sold is not accurately determined due to malfunction.

Public Patent Publication 10-2021-0141234, 2021.11.23.

The problem that the present invention aims to solve is that when an error occurs when selling a product from an unmanned product vending device and the number of products sold cannot be accurately determined, the number of products sold is determined through an alternative algorithm according to mode switching, and accurate payment information is provided. It provides a method for calculating the number of product sales to obtain.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

The method for calculating the number of products sold according to an embodiment of the present invention stores the standard weight of the product and a preset correction index in memory,

When the door of the product storage unit is opened, receiving first product information from an external device including a camera and a weight measurement sensor;

When the door of the product storage unit is closed, receiving second product information from the external device;

Calculating the number of sales of the product by performing an absolute calculation based on the weight information included in each of the first product information and the second product information;

If it is determined that an error has occurred according to an error detection algorithm predetermined based on the second product information, the absolute calculation is stopped, and the weight information difference value included in each of the first product information and the second product information is determined. A step of calculating the number of sales of the product by performing a relative operation based on

When it is determined that the error is resolved by changing the correction index based on the tilt information of the load cell received from the external device, stopping the relative operation and performing the absolute operation again.

In addition, the first product information is,

When the door of the product storage unit is opened, there may be information about the product image captured by the camera and the weight of the load cell measured by the weight measurement sensor,

The second product information is,

When the door of the product storage unit is closed, there may be information about the product image captured by the camera and the weight of the load cell measured by the weight measurement sensor.

In addition, for the absolute calculation and the relative calculation, machine learning is performed based on the product image captured by the camera to determine the calculation value for each,

The learning model that performs the machine learning can implement multiclass classification and binary classification,

The multi-class classification is performed by a first convolutional neural network (CNN) model,

The binary classification may be performed by a second convolutional neural network model.

In addition, the predetermined error detection algorithm is,

Based on when the door is closed, if the inventory number is determined based on the absolute calculation, the inventory number multiplied by the standard weight of the product is compared with the information on the weight of the load cell measured by the weight measurement sensor. The ratio is calculated, and if the ratio differs by more than a preset value, it is determined that the error has occurred,

If the ratio differs by less than a preset value, the algorithm may determine that the error has not occurred.

The method for calculating the number of products sold according to an embodiment of the present invention is,

Additionally, if it is determined that the above error has occurred,

It is determined that a change in the preset correction index is necessary,

Receiving tilt information of the load cell from the external device,

Changing the preset correction index based on the tilt information of the load cell,

When the preset correction index is changed, it can be determined that the error has been resolved.

In addition, the predetermined error detection algorithm is,

In the case where it is determined that there is no product in the load cell by the second convolutional neural network model,

*If it is determined that the product exists based on the weight information of the load cell measured by the weight measurement sensor, it is determined that an error has occurred,

Otherwise, the algorithm may determine that an error has not occurred.

The method for calculating the number of products sold according to an embodiment of the present invention is,

If it is determined that the above error has occurred,

It is determined that a change in the preset correction index is necessary,

Receiving tilt information of the load cell from the external device,

Changing the preset correction index based on the tilt information of the load cell,

When the preset correction index is changed, it can be determined that the error has been resolved.

In addition to this, a computer program stored in a computer-readable recording medium for execution to implement the present disclosure may be further provided.

In addition to this, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to one aspect, the method and calculation device for calculating the number of products sold provide a mode change for calculating the number of products sold according to an algorithm based on product recognition, so that even when an error occurs, the number of products sold according to the mode change can be calculated.

According to the method and calculation device for calculating the number of products sold according to one aspect, it is possible to obtain accurate payment information by determining when an error has occurred according to a predetermined error detection algorithm and enabling mode conversion.

According to the method and calculation device for calculating the number of products sold according to one aspect, even if the calculation of the number of products sold according to mode conversion is performed, an algorithm for resolving errors is carried out simultaneously, so that errors in the existing mode can be resolved and accurate payment information can be obtained. .

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

Figure 1 is a diagram schematically showing communication between an external device and a server according to an embodiment.
Figure 2a is a diagram schematically showing the configuration included in an external device and a server according to an embodiment.
Figure 2b is a diagram schematically showing the components included in the processor according to one embodiment.
Figure 3 is a diagram showing mode switching according to a product recognition algorithm according to an embodiment.
Figure 4 is a diagram schematically showing an external device including a camera and a weight measurement sensor according to an embodiment.
Figure 5 is a flowchart showing steps performed in an external device and server according to an embodiment.
Figure 6 is a diagram schematically showing measuring the inclination angle of a load cell using an inclination sensor according to an embodiment.
Figure 7 is a diagram illustrating an algorithm for the process from when a mode change is made to when an error is resolved according to an embodiment.
Figure 8 is a diagram illustrating an algorithm for the process of performing an absolute operation in ABS mode according to an embodiment.
Figure 9 is a diagram illustrating an algorithm for a process in which a relative operation is performed according to the REL mode according to an embodiment.
Figure 10 is a diagram illustrating an algorithm for a process for resolving an error when an error occurs according to an embodiment.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. The present embodiments are merely provided to ensure that the disclosure of the present invention is complete and to provide a general understanding of the technical field to which the present invention pertains. It is provided to fully inform the skilled person of the scope of the present invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for describing embodiments and is not intended to limit the invention. As used herein, singular forms also include plural forms, unless specifically stated otherwise in the context. As used in the specification, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other elements in addition to the mentioned elements. Like reference numerals refer to like elements throughout the specification, and “and/or” includes each and every combination of one or more of the referenced elements. Although “first”, “second”, etc. are used to describe various components, these components are of course not limited by these terms. These terms are merely used to distinguish one component from another. Therefore, it goes without saying that the first component mentioned below may also be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not to be interpreted ideally or excessively unless clearly specifically defined.

Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

Figure 1 is a diagram schematically showing communication between the external device 100 and the server 200 according to an embodiment, and Figure 2A is a diagram schematically showing communication between the external device 100 and the server 200 according to an embodiment. This is a diagram schematically showing the configuration.

Referring to Figures 1 and 2A, an external device 100 and a server 200 may be provided to perform the operation of the present invention.

The external device 100 includes a product storage unit 170 including a load cell 110, a weight measurement sensor 120, a tilt measurement sensor 130, a door unit 140, and a camera 150 that can store products. It may be provided to include a first communication module 160 capable of communicating with the server 200.

The server 200 includes a memory 210, a second communication module 230 capable of communicating with the external device 100, and at least one processor 220 capable of communicating with the memory 210. It can be arranged to include

Meanwhile, the load cell 110 may refer to a storage space in the external device 100 where products can be stored. Since the load cell 110 is inclined by the inclination angle θ1, when the user receives the product, the product can move toward the door unit 140 along the inclined plane.

Meanwhile, in order to measure the weight of a product stored in the external device 100, a weight measurement sensor 120 may be mounted on the load cell 110.

According to one embodiment of the present invention, the weight measurement sensor 120 is attached to the lower part of the load cell 110, and the sensing distance and sensing resistance value vary depending on the weight of the product on the load cell 110, and based on this, the weight can be measured. For example, when comparing the case where there are 10 products remaining on the load cell 110 and the case where there are 5 products remaining, in the case where there are 10 products remaining, the sensing resistance value increases as the sensing distance becomes shorter, and the product In the case where there are five remaining teeth, the sensing resistance value may decrease as the sensing distance increases, and the weight can be measured based on this.

Meanwhile, in order to measure the tilt of the load cell 110, a tilt measurement sensor 130 may be provided in the external device 100. The tilt measurement sensor 130 can be used to preset the correction index (k1) as described later, and to change the correction index (k2) by measuring a new tilt angle when an error occurs.

Meanwhile, the weight measurement sensor 120 and the tilt measurement sensor 130 sense at least one of the internal information of the device, the surrounding environment information surrounding the device, and the user information, and generate a sensing signal corresponding thereto. Based on these sensing signals, the processor 220 may control the driving or operation of the device, or perform data processing, functions, or operations related to an application program installed on the device.

Meanwhile, the door unit 140 may be a door of the product storage unit 170 mounted on the external device 100. As described later, when performing absolute calculation in ABS mode when calculating the number of products sold, the calculation may be performed based on when the door of the door unit 140 is opened and when the door is closed.

Meanwhile, the camera 150 may refer to an artificial intelligence-based machine vision camera mounted on the external device 100. The image of the product captured by the camera 150 is transmitted to the server 200, and the processor 220 can perform machine learning based on the received image.

Meanwhile, the machine vision camera can capture product images and transmit them to the server 200. A machine vision camera may consist of a lens, an image sensor, a main board, and an interface board, but is not limited to this. Additionally, images created through lenses and image sensors can be corrected into an appropriate form on the motherboard as needed. The image processed on the main board in this way can be transmitted to the server 200. Machine vision cameras may include GigE Vision cameras (Gigabit Ethernet Vision Cameras), USB3.0 cameras, CameraLink cameras, and CoaXPress cameras.

Meanwhile, the memory 210 can store the standard weight of the product and a preset correction index (k1) as described later.

The standard weight of the product is the general weight of the product on the load cell 110 and may be a fixed value stored in advance in the memory 210.

The standard weight of the product may be stored differently depending on the type of product in the load cell 110. For example, if the product of load cell 1 is a 185ml can of cider, the standard weight of the can of cider can be stored as 200g, and if the product of load cell 2 is a 250ml can of cola, the standard weight of the can of cola can be stored as 300g. It can be.

The memory 210 can store data supporting various functions of the device and a program for the operation of the processor 220, and input/output data (e.g., music files, still images, videos, etc.) can be stored, and a number of application programs (application programs or applications) running on the device, data for operation of the device, and commands can be stored. At least some of these applications may be downloaded from an external server via wireless communication.

The memory 210 includes a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), and a multimedia card micro type. micro type), card-type memory (e.g. SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), EEPROM (electrically erasable) It may include at least one type of storage medium among programmable read-only memory (PROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. Additionally, the memory 210 is separate from the device, but may be a database connected by wire or wirelessly.

At least one processor 220 has a memory that stores data for an algorithm for controlling the operation of components in the device or a program that reproduces the algorithm, and performs the above-described operations using the data stored in the memory 210. can do. At this time, the memory 210 and the processor 220 may each be implemented as separate chips. Alternatively, the memory 210 and processor 220 may be implemented as a single chip.

In addition, the processor 220 may control any one or a combination of the above-described components in order to implement various embodiments according to the present disclosure described in FIGS. 2B to 10 below on the present device.

Among the components, the first communication module 160 of the external device 100 and the second communication module 230 of the server may include one or more components that enable communication with the external device, for example. , it may be at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

Wired communication modules include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as USB (Universal Serial Bus) modules. ), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, wireless communication modules include GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and UMTS (universal mobile telecommunications system). ), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, etc. may include a wireless communication module that supports various wireless communication methods.

The wireless communication module may include a wireless communication interface including an antenna and a transmitter that transmits a mobile communication signal. Additionally, the wireless communication module may further include a mobile communication signal conversion module that modulates a digital control signal output from the control unit through a wireless communication interface into an analog wireless signal under the control of the control unit.

The wireless communication module may include a wireless communication interface including an antenna and a receiver for receiving mobile communication signals. Additionally, the wireless communication module may further include a mobile communication signal conversion module for demodulating an analog wireless signal received through a wireless communication interface into a digital control signal.

The short-range communication module is for short-range communication and includes Bluetooth™ RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, and NFC (Near Field Communication). , short-distance communication can be supported using at least one of Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technologies.

Figure 2b is a diagram schematically showing the components included in the processor 220 according to one embodiment.

Referring to Figure 2b, the processor 220 in the server 200 has a product sales quantity calculation unit 221 that performs absolute calculation in ABS mode and relative calculation in REL mode as described later, and product images captured by a machine vision camera. It may be composed of a machine learning unit 222 that performs machine learning of the artificial intelligence learning model based on .

At least one component may be added or deleted in response to the performance of the components shown in FIG. 2B. Additionally, it will be easily understood by those skilled in the art that the mutual positions of the components may be changed in response to the performance or structure of the system.

Meanwhile, each component shown in Figure 2b refers to software and/or hardware components such as Field Programmable Gate Array (FPGA) and Application Specific Integrated Circuit (ASIC).

Figure 3 is a diagram showing mode switching according to a product recognition algorithm according to an embodiment, and Figure 4 is a diagram schematically showing an external device 100 including a camera and a weight measurement sensor according to an embodiment.

Referring to Figure 3, the product 111 can be stored in the load cell 110 with an inclination angle of θ1, and when the user receives the product 111 according to the slope, the product 111 will be moved toward the door unit 140. You can. The camera 150 can photograph the front part of the load cell 110 close to the door 140. Therefore, when the product image captured by the machine vision camera 150 is transmitted to the processor 220, learning is performed by a second convolutional neural network (CNN) model as described later to determine whether the product 111 exists. The presence or absence can be determined. As described later, during absolute calculation in ABS mode and relative calculation in REL mode, if the product 111 exists, Nv, which is a code value based on image digital recognition, may be determined to be 1, and if the product 111 does not exist, Nv can be judged to be 0.

Meanwhile, the product 111 in FIG. 3 may refer to each product displayed on the load cell 110.

Referring to Figure 3, the present invention basically sets the ABS mode (Absolute mode) 310, and absolute calculation can be performed when calculating the number of products sold (111) according to the ABS mode (310). At this time, if it is determined that an error has occurred according to a predetermined error detection algorithm, the absolute operation according to the ABS mode (310) is stopped and switched to the REL mode (Relative mode) (320) to perform the relative operation (S410). . In addition, when it is determined that the error has been resolved according to a predetermined correction algorithm, the relative operation according to the REL mode 320 is stopped, and the mode is switched back to the ABS mode 310 to perform an absolute operation (S420).

Referring to Figure 4, the weight measurement sensor 120 is mounted on the lower part of the load cell 110 as described above and can measure the weight of the product 111 on the load cell 110.

Meanwhile, the weight measurement sensor 120 may be set to measure the weight of the product when the door of the door 140 of the external device 100 is opened and to measure the weight of the product when the door of the door 140 is closed.

The first product information may include weight information of the product based on when the door of the door unit 140 is opened and image information of the product captured by the camera 150, but is not limited thereto.

The second product information may include weight information of the product based on when the door of the door unit 140 is closed and image information of the product captured by the camera 150, but is not limited thereto.

Meanwhile, the first product information and the second product information are information about products in the same column on the load cell, and may be product information based on when the door is opened and when the door is closed.

In this way, the first product information and the second product information are transmitted from the external device 100 to the server 200 through communication between the first communication module 160 of the external device 100 and the second communication module 230 of the server 200. ) can be transmitted.

Figure 5 is a flowchart showing steps performed in the server 200 and the external device 100 according to an embodiment.

According to one embodiment of the present invention, the user can make the external device 100 recognize payment methods that can pay for products, including check cards and credit cards, in order to pay for products (S601). The card information recognized in this way can be transmitted to the processor 220 through the second communication module 230 of the server 200 (S602). The server 200 can receive the card information transmitted in this way (S603). If the server 200 recognizes the card information as correct, the server 200 can transmit a command to open the door unit 140 to the external device 100 (S604). As the door opening command to the door unit 140 is transmitted, when the door unit 140 of the external device 100 is opened (S605), information that the door of the door unit 140 has been opened will be transmitted to the server 200. (S606). As the door unit 140 is opened, the server 200 may receive first product information (S607). When the product is received from the user's external device 100 (S608) and the door of the door unit 140 is closed (S609), information that the door of the door unit 140 is closed may be transmitted to the server 200 (S610) ). As the door of the door unit 140 is closed, the processor 220 may receive second product information (S611). As the server 200 receives the first product information and the second product information, it can calculate the number of products sold by the user by performing calculations based on the first product information and the second product information (S612). . The calculated number of products sold is transmitted to the external device 100 (S613), and product payment according to the number of products sold can be made (S614).

Figure 6 is a diagram schematically showing measuring the inclination angle of the load cell 110 using the inclination sensor 130 according to an embodiment.

Meanwhile, the product 111 in FIG. 6 may refer to each product displayed on the load cell 110.

The load cell 110 can be inclined by an inclination angle θ1 based on a flat surface, so when a user receives a product from the external device 100, the product on the side far from the door unit 140 along the inclined surface is placed in the door unit ( You can move to 140).

The tilt measurement sensor 130 according to an embodiment of the present invention is attached to the bottom of the load cell 110 and can measure the tilt of the load cell 110.

Meanwhile, the measured inclination angle (θ1) can be used to determine a preset correction index (k1) stored in the memory 210. Since the weight in the inclined load cell 110 may be different from the weight on flat ground, a correction model based on a preset correction index can be used. That is, the weight can be corrected using a correction index that can correct the weight measurement error that occurs due to the inclination of the load cell 110.

According to one embodiment, the mathematical expression of the correction model may be expressed as Equation 1 below.

Equation 1

Referring to Equation 1, Mb may mean the weight on an inclined load cell, k1 may mean the preset correction index, and Ma may mean the weight on flat ground.

Meanwhile, when the inclination angle of the load cell 110 is θ1, the preset correction index k1 may be cos θ1 or cscθ1 , but is not limited thereto.

In addition, the preset correction index (k1) according to one embodiment is the actual weight value of the weight by placing a unit weight (for example, 100g, 200g, etc.) on the weight measurement sensor 120 at the time of installing the load cell 110. It can also be set by adjusting the zero point of the weighing sensor so that it can be read as .

Figure 7 is a diagram illustrating an algorithm for the process from when a mode change is made to when an error is resolved according to an embodiment.

The present invention basically starts in ABS mode (S810), and absolute calculation can be performed based on weight information included in each of the first product information and the second product information (S820). Accordingly, the number of products sold can be calculated (S830), and a process of determining that an error has occurred according to a predetermined error detection algorithm based on the second product information is performed (S840). If no error occurs, absolute calculation according to ABS mode is continuously performed, and if it is determined that an error has occurred, it switches to REL mode (S850). As the mode is switched to REL mode, a relative calculation can be performed based on the difference value of the weight information included in each of the first product information and the second product information (S860). Accordingly, the number of products sold can be calculated (S870), and at the same time, based on the tilt information of the load cell 110 from the tilt measurement sensor 130 included in the external device 100 according to a predetermined correction algorithm.

The preset correction index (k1) can be changed (S880). Accordingly, if it is determined that the error has been resolved, it switches back to ABS mode and absolute calculation can be performed. However, if it is determined that the error has not been resolved, REL mode is maintained and the number of products sold according to relative calculation can be calculated ( S890).

Figure 8 is a diagram illustrating an algorithmic process of performing an absolute operation in ABS mode according to an embodiment.

First, the standard weight (Wm) of the product is stored in the memory 210 (S910), the first product information can be received when the door is opened (S920), and the second product information can be received when the door is closed (S930) ). Meanwhile, as described above, the first product information and the second product information are the weight information of the load cell 110 by the weight measurement sensor 120 and the machine vision camera 150 for when the door is opened and when the door is closed, respectively. ) may include image information of the product taken by .

Machine learning using a learning model may be performed in the machine learning unit 222 of the processor 220. The calculation values of the absolute calculation in ABS mode and the relative calculation in REL mode as described later can be determined by performing machine learning using the image of the product captured by the machine vision camera 150 as data. A learning model that performs machine learning can implement multiclass classification and binary classification. Meanwhile, multiclass classification may be performed by a first convolutional neural network (CNN) model, and binary classification may be performed by a second convolutional neural network model, but are not limited thereto. For example, the learning models include Random Forest (RF), Support Vector Machine (SVC), eXtra Gradient Boost (XGB), Decision Tree (DC), Knearest Neighbors (KNN), Gaussian Naive Bayes (GNB), and Stochastic Gradient. It may include at least one algorithm among Descent (SGD), Linear Discriminant Analysis (LDA), Ridge, Lasso, and Elastic net.

Whether a product exists when the door of the door unit 140 is closed can be determined by implementing binary classification by a second convolutional neural network model.

Based on the time the door is closed after the user receives the product, if the product is recognized as existing, the digital code value Nv = 1, and if the product is recognized as not existing, the digital code value can be recognized as Nv = 0.

In addition, based on when the door is closed after the user receives the product, if it is recognized that the product exists, the standard weight Wv of the product recognized by the machine vision camera 150 can be received from the memory 220, and the product can be received from the memory 220. If it is recognized as not existing, the standard weight (Wv) of the product recognized by the machine vision camera 150 may be determined to be 0.

The sales quantity calculation unit 221 of the processor 220 can perform an absolute calculation according to the ABS mode (S970), and the formula for the absolute calculation can be expressed as Equation 2 below.

Equation 2

Referring to Equation 2, Ns refers to the number of products sold, Nb refers to the number of inventory when the door is opened, Nt refers to the number of inventory when the door is closed, Wm refers to the standard weight of the product, Wc refers to the weight of the load cell when the door is closed, Wv refers to the standard weight of the product recognized by the camera, and Nv refers to the digital code value of the product recognized by the camera.

As the absolute operation is performed, the number of products sold (Ns) can be calculated (S980). At this time, in order to calculate the exact number of units sold, it can be determined whether an error has occurred according to a predetermined error detection algorithm based on the second product information (S990). If an error occurs, the mode is switched to REL mode and a relative operation can be performed (S9100), and if an error does not occur, the ABS mode is maintained and an absolute operation can be performed (S9110).

Figure 9 is a diagram illustrating an algorithm for a process in which a relative operation is performed according to the REL mode according to an embodiment.

First, the standard weight (Wm) of the product is stored in the memory 210 (S1010), the first product information can be received when the door is opened (S1020), and the second product information can be received when the door is closed (S1030) ). When the weight of the load cell 110 when the door is opened is W1 and the weight of the load cell 110 when the door is closed is W2, each load cell 110 included in the first product information and the second product information Based on the weight information, the difference value, Wd=W1-W2, can be calculated (S1040).

In the case of the REL mode relative operation, as in the ABS mode absolute operation, whether the product exists when the door of the door unit 140 is closed is determined by implementing binary classification by the second convolutional neural network model. You can.

Based on the time the door is closed after the user receives the product, if the product is recognized as existing, the digital code value Nv = 1, and if the product is recognized as not existing, the digital code value can be recognized as Nv = 0.

In addition, based on when the door is closed after the user receives the product, if it is recognized that the product exists, the standard weight Wv of the product recognized by the machine vision camera 150 can be received from the memory 220, and the product can be received from the memory 220. If it is recognized as not existing, the standard weight (Wv) of the product recognized by the machine vision camera 150 may be determined to be 0.

The sales quantity calculation unit 221 of the processor 220 can perform a relative calculation according to the REL mode (S1080), and the formula for the relative calculation can be expressed as the following equation 3.

Equation 3

Referring to Equation 3, Ns refers to the number of products sold, W1 refers to the weight of the load cell when the door is opened, W2 refers to the weight of the load cell when the door is closed, and Wd refers to the weight of the load cell when the door is opened and when the door is closed. It means the weight difference value of the load cell when closed, Wm means the standard weight of the product, Wv means the standard weight of the product recognized by the camera, and Nv is the digital code value of the product recognized by the camera. It can mean.

As the relative operation is performed, the number of products sold (Ns) can be calculated (S1090). At this time, the tilt measurement sensor 130 included in the external device 100 can measure the tilt angle (θ2) of the load cell 110 again. The correction index (k1) can be changed based on the tilt information of the load cell 110 received from the external device 100, and it can be determined whether the error has been resolved (S1100). Whether the error has been resolved can be determined by re-performing a predetermined error detection algorithm based on the second product information. If it is determined that the error has been resolved, the relative operation according to the REL mode is stopped and the absolute operation according to the ABS mode can be performed again (S1110). If it is determined that the error has not been resolved, the REL mode is maintained and the product according to the relative operation is At the same time as the number of sales is calculated, an error resolution algorithm can be implemented by continuously changing the correction index.

Figure 10 is a diagram illustrating an algorithm for a process of resolving an error when an error occurs according to an embodiment.

According to one embodiment, the tilt (θ1) of the inclined load cell 110 can be measured by the tilt measurement sensor 130 (S110), and a preset correction index (k1) can be calculated based on this (S1120). At this time, the preset correction index (k1) may be a cosθ1 value or cscθ1 , but is not limited thereto. The preset correction index (k1) may be stored in the memory 210 (S1130). Absolute calculation according to ABS mode is performed (S1140), and if it is determined that an error has occurred according to the error detection algorithm (S1150), relative calculation according to REL mode is performed (S1160), and the number of products sold can be calculated (S1160) S1170). At the same time, according to the error occurrence signal, the tilt measurement sensor 130 of the external device 100 can measure the tilt (θ2) of the load cell 110 again. Accordingly, the preset correction index (k1) can be changed to a new correction index (k2) and corrected (S1190). At this time, the new correction index (k2) may be a cosθ2 value or cscθ2 , but is not limited thereto. According to the process of determining whether the error has been resolved (S11100), if it is determined that the error has been resolved, the relative operation according to the REL mode is stopped (S1180) and the absolute operation according to the ABS mode can be performed again, and the error is not resolved. If it is determined that it is not, the relative operation according to the REL mode is maintained and the error resolution algorithm can continue to proceed.

Meanwhile, the process of determining whether an error has been resolved (S11100) may be performed by the same process as the process of determining whether an error has been detected. The predetermined error detection algorithm is a result value recognized by a second convolutional neural network model that implements binary classification that can determine the presence or absence of a product, and a weight measurement sensor that can measure the weight of the load cell 110. It may be an algorithm that determines the error based on the difference between the measured values of (120), but is not limited to this.

The process of determining whether an error has been detected according to one embodiment is based on when the door is closed. If the inventory number is determined based on absolute calculation in ABS mode, the standard weight of the product (Wm) is added to the inventory number (Nt). ) is calculated by comparing the multiplied value with the information on the weight of the load cell 110 measured by the weight measurement sensor 120, and if the calculated ratio differs by more than a preset value, the error occurs. It may be an algorithm that determines that an error has occurred and, if the calculated ratio differs by less than a preset value, determines that the error has not occurred.

The process of determining whether an error has been detected according to one embodiment is when it is determined that there is no product in the load cell 110 (Nv = 0) by a second convolutional neural network model that implements binary classification, and the weight measurement sensor When it is determined that the product exists based on the weight information of the load cell 110 measured by 120 (Wc ) may be an algorithm that determines that an error has occurred, and if not, determines that an error has not occurred.

The process of determining whether an error has been detected according to one embodiment is when it is determined that there is a product in the load cell 110 (Nv = 1) by a second convolutional neural network model implementing binary classification, and the memory 120 ), if the standard weight (Wm) of the product pre-stored does not match the standard weight (Wv) of the product recognized by the machine vision camera 150, it is determined that an error has occurred, otherwise, it is determined that no error has occurred. It may be a judgment algorithm.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media storing instructions that can be decoded by a computer. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, etc.

Meanwhile, the content described in FIGS. 7 to 10 is only an example for explaining the operation of the present invention, and there are no limitations to the operation in which the server 200 performs communication with the external device 100 to calculate the number of products sold. does not exist.

The steps of the method or algorithm described in connection with embodiments of the present invention may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof. The software module may be RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, CD-ROM, or It may reside on any type of computer-readable recording medium well known in the art to which the present invention pertains.

Above, embodiments of the present invention have been described with reference to the attached drawings, but those skilled in the art will understand that the present invention can be implemented in other specific forms without changing its technical idea or essential features. You will be able to understand it. Therefore, the embodiments described above should be understood in all respects as illustrative and not restrictive.

100: external device
200: server

Claims (10)

Stores the standard weight of the product and the preset correction index in memory,
Calculating the number of products sold by performing an absolute calculation based on the weight information included in each of the first product information when the door of the product storage unit is opened and the second product information when the door of the product storage unit is closed. ;
If it is determined that an error has occurred according to an error detection algorithm predetermined based on the second product information, the absolute calculation is stopped, and the difference value of the weight information included in each of the first product information and the second product information Calculating the number of sales of the product by performing a relative operation based on;
When it is determined that the error has been resolved by changing the correction index based on the tilt information of the load cell received from an external device, stopping the relative operation and performing the absolute operation again;
Changing the correction index is,
(Mb: weight on an inclined load cell, k1: preset correction index, Ma: weight on flat ground) A method of calculating the number of product sales through a product recognition algorithm, which is determined based on the equation. According to claim 1,
The first product information is,
When the door of the product storage unit is opened, the image of the product captured by the camera and information about the weight of the load cell measured by the weight measurement sensor,
The second product information is,
When the door of the product storage unit is closed, a method of calculating the number of products sold through a product recognition algorithm, characterized in that the image of the product captured by the camera and information about the weight of the load cell measured by the weight measurement sensor. According to clause 2,
The absolute operation is, Decide based on the formula,
The relative operation is Make decisions based on consciousness
(However, Ns: Number of products sold, Nb: Number of inventory when the door is opened, Nt: Number of inventory when the door is closed, Wm: Standard weight of the product, Wc: Weight of the load cell when the door is closed, Wv: From the camera Standard weight of the recognized product, Nv: Digital code value of the product recognized by the camera, Wd: Weight difference value of the load cell when the door is opened and when the door is closed),
Based on when the door of the product storage compartment is closed,
If the camera recognizes that the product exists, Nv = 1, and if the camera recognizes that the product does not exist, Nv = 0,
If Nv=1,
The standard weight Wv of the product recognized by the camera can be received from the memory,
If Nv=0,
A method for calculating the number of product sales through a product recognition algorithm, characterized in that Wv = 0. According to clause 2,
A method for calculating the number of product sales through a product recognition algorithm, wherein the absolute calculation and the relative calculation are machine learning based on the image of the product captured by the camera, and the calculation value for each is determined. According to clause 4,
The predetermined error detection algorithm is,
Based on when the door is closed, if the inventory number is determined based on the absolute calculation, the inventory number multiplied by the standard weight of the product is compared with the information on the weight of the load cell measured by the weight measurement sensor. The ratio is calculated, and if the ratio differs by more than a preset value, it is determined that the error has occurred,
If the ratio differs by less than a preset value, the algorithm determines that the error has not occurred,
If it is determined that the above error has occurred,
It is determined that a change in the preset correction index is necessary,
Receiving tilt information of the load cell from the external device,
Changing the preset correction index based on the tilt information of the load cell,
A method of calculating the number of product sales through a product recognition algorithm, characterized in that when the preset correction index is changed, it is determined that the error has been resolved. According to clause 4,
The predetermined error detection algorithm is,
In the case where it is determined that the product is not in the load cell by a convolutional neural network model (CNN) that implements binary classification,
If the standard weight of the product pre-stored in the memory does not match the standard weight of the product recognized by the camera, it is determined that an error has occurred,
Otherwise, a method of calculating the number of product sales through a product recognition algorithm, characterized by an algorithm that determines that an error has not occurred. According to clause 6,
If it is determined that the above error has occurred,
It is determined that a change in the preset correction index is necessary,
Receiving tilt information of the load cell from the external device,
Changing the preset correction index based on the tilt information of the load cell,
A method of calculating the number of product sales through a product recognition algorithm, characterized in that when the preset correction index is changed, it is determined that the error has been resolved. A memory that stores the standard weight of the product and a preset correction index;
A communication module that communicates with an external device including at least one camera and a weight measurement sensor; and
At least one processor that communicates with the memory,
The at least one processor,
Calculate the number of products sold by performing an absolute calculation based on the weight information included in each of the first product information when the door of the product storage unit is opened and the second product information when the door of the product storage unit is closed,
If it is determined that an error has occurred according to an error detection algorithm predetermined based on the second product information,
Stop the absolute calculation and perform a relative calculation based on the difference value of the weight information included in each of the first product information and the second product information to calculate the number of sales of the product,
If it is determined that the error has been resolved by changing the correction index based on the tilt information of the load cell received from the external device, the relative operation is stopped and the absolute operation is performed again,
Changing the correction index means,
(Mb: weight on an inclined load cell, k1: preset correction index, Ma: weight on flat ground) A server that calculates the number of product sales through a product recognition algorithm, which is determined based on the equation. According to clause 8,
For the absolute calculation and the relative calculation, machine learning is performed based on the image of the product captured by the camera, and the calculation value for each is determined,
The predetermined error detection algorithm is,
Based on when the door is closed, if the inventory number is determined based on the absolute calculation, the inventory number multiplied by the standard weight of the product is compared with the information on the weight of the load cell measured by the weight measurement sensor. to calculate the ratio,
If the ratio differs by more than a preset value, it is determined that the error has occurred,
If the ratio differs by less than a preset value, the algorithm determines that the error has not occurred,
If it is determined that the above error has occurred,
It is determined that a change in the preset correction index is necessary,
Receiving tilt information of the load cell from the external device,
Changing the preset correction index based on the tilt information of the load cell,
A server for calculating the number of product sales through a product recognition algorithm, characterized in that when the preset correction index is changed, it is determined that the error has been resolved. A computer program stored on a medium for executing the method according to any one of claims 1 to 7.