WO2023234605A1 - Dispositif de calcul d'emplacement basé sur l'intelligence artificielle pour des produits dans une vitrine utilisant une structure d'agencement de cellules de charge - Google Patents

Dispositif de calcul d'emplacement basé sur l'intelligence artificielle pour des produits dans une vitrine utilisant une structure d'agencement de cellules de charge Download PDF

Info

Publication number
WO2023234605A1
WO2023234605A1 PCT/KR2023/006778 KR2023006778W WO2023234605A1 WO 2023234605 A1 WO2023234605 A1 WO 2023234605A1 KR 2023006778 W KR2023006778 W KR 2023006778W WO 2023234605 A1 WO2023234605 A1 WO 2023234605A1
Authority
WO
WIPO (PCT)
Prior art keywords
product
shelf
location information
load cell
processor
Prior art date
Application number
PCT/KR2023/006778
Other languages
English (en)
Korean (ko)
Inventor
송중석
한상진
Original Assignee
(주) 인터마인즈
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by (주) 인터마인즈 filed Critical (주) 인터마인즈
Publication of WO2023234605A1 publication Critical patent/WO2023234605A1/fr

Links

Images

Classifications

    • A—HUMAN NECESSITIES
    • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
    • A47F—SPECIAL FURNITURE, FITTINGS, OR ACCESSORIES FOR SHOPS, STOREHOUSES, BARS, RESTAURANTS OR THE LIKE; PAYING COUNTERS
    • A47F10/00—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for
    • A—HUMAN NECESSITIES
    • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
    • A47F—SPECIAL FURNITURE, FITTINGS, OR ACCESSORIES FOR SHOPS, STOREHOUSES, BARS, RESTAURANTS OR THE LIKE; PAYING COUNTERS
    • A47F10/00—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for
    • A47F10/02—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for for self-service type systems, e.g. supermarkets
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01G—WEIGHING
    • G01G19/00—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups
    • G01G19/40—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups with provisions for indicating, recording, or computing price or other quantities dependent on the weight
    • G01G19/413—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups with provisions for indicating, recording, or computing price or other quantities dependent on the weight using electromechanical or electronic computing means
    • G01G19/414—Weighing apparatus or methods adapted for special purposes not provided for in the preceding groups with provisions for indicating, recording, or computing price or other quantities dependent on the weight using electromechanical or electronic computing means using electronic computing means only
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01L—MEASURING FORCE, STRESS, TORQUE, WORK, MECHANICAL POWER, MECHANICAL EFFICIENCY, OR FLUID PRESSURE
    • G01L1/00—Measuring force or stress, in general
    • G01L1/20—Measuring force or stress, in general by measuring variations in ohmic resistance of solid materials or of electrically-conductive fluids; by making use of electrokinetic cells, i.e. liquid-containing cells wherein an electrical potential is produced or varied upon the application of stress
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
    • G01S17/06—Systems determining position data of a target
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07F—COIN-FREED OR LIKE APPARATUS
    • G07F11/00—Coin-freed apparatus for dispensing, or the like, discrete articles
    • G07F11/02—Coin-freed apparatus for dispensing, or the like, discrete articles from non-movable magazines
    • G—PHYSICS
    • G07—CHECKING-DEVICES
    • G07F—COIN-FREED OR LIKE APPARATUS
    • G07F9/00—Details other than those peculiar to special kinds or types of apparatus
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/57—Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
    • A—HUMAN NECESSITIES
    • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
    • A47F—SPECIAL FURNITURE, FITTINGS, OR ACCESSORIES FOR SHOPS, STOREHOUSES, BARS, RESTAURANTS OR THE LIKE; PAYING COUNTERS
    • A47F10/00—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for
    • A47F2010/005—Furniture or installations specially adapted to particular types of service systems, not otherwise provided for using RFID elements

Definitions

  • This disclosure relates to a product position calculation device, and more specifically, to a device that calculates the position of a product placed in a product showcase based on artificial intelligence using a load cell arrangement structure.
  • the device that determines product placement using a weight measurement sensor only determines the placement of products of a fixed size and does not take into account changes in product size, so there is a problem in that product placement is less flexible.
  • the devices currently in use have malfunctions that prevent them from accurately determining the number of products sold, resulting in a problem of not obtaining accurate payment information during the product payment process.
  • the purpose of the embodiment disclosed in this disclosure is to provide a device for calculating the position of a product placed in a product showcase based on artificial intelligence by utilizing the arrangement structure of a load cell.
  • the purpose of the embodiment disclosed in this disclosure is to provide a device for calculating the number of products sold based on artificial intelligence using a product recognition algorithm.
  • An artificial intelligence-based product position calculation device in a product showcase for achieving the above-described technical problem includes a plurality of load cells for sensing the positions of products placed on the shelves of a display stand divided into a plurality of columns; And a processor that calculates location information of the product placed on the shelf based on the sensing value sensed by the plurality of load cells and the location information of each of the plurality of load cells that sensed the sensing value, wherein the plurality of load cells are Two or more lines are disposed on each line to separate the plurality of columns, a first load cell is provided on one side of each line, a second load cell is provided on the other side of each line, and the processor
  • the sensing values sensed by the first load cell and the second load cell are internally calculated to calculate location information of the product placed on the shelf, and each of the sensing values sensed by the plurality of load cells and the plurality of load cells sensing the sensing values Based on the location information, the expected weight and bottom shape of the product placed on the shelf are calculated
  • the processor may determine whether the product placed on the shelf is normally placed based on the location information of the product and the type of the product.
  • the shelf includes a camera capable of photographing at least a portion of the shelf at a plurality of different viewing angles, and the processor controls the camera to photograph the shelf at a plurality of different viewing angles to generate a plurality of captured images. And, by matching the location information of the product with the plurality of captured images, the calculation result of the location information of the product can be verified.
  • the processor may calculate image location information of the product based on the plurality of captured images and determine whether the calculated image location information and location information of the product match.
  • the shelf is provided with a LiDAR sensor on each side, and the processor can verify the location information of the product based on the sensing value sensed through the LiDAR sensor.
  • the shelf is provided with a first LiDAR sensor on one side of the interior and a second LiDAR sensor on the other side of the interior, and the processor determines the sensing value sensed through the first LiDAR sensor and the second LiDAR sensor. Based on the sensing value sensed through the LiDAR sensor, location information (hereinafter referred to as "LiDAR-based location information") of the product placed on the shelf is calculated, and the LiDAR-based location information and the The location information of the product can be verified by comparing the location information of the product.
  • LiDAR-based location information location information of the product placed on the shelf
  • the shelf is provided with a first LiDAR sensor on one side of the interior and a second LiDAR sensor on the other side of the interior, and the processor detects the first LiDAR based on the calculated location information of the product. Allocating a plurality of sensing points to each sensor and the second LiDAR sensor, and based on the sensing values sensed through the first LiDAR sensor and the second LiDAR sensor for the assigned plurality of sensing points, You can verify the location information of the product.
  • the effect of providing a device for calculating the position of a product placed in a product showcase based on artificial intelligence by utilizing the arrangement structure of a load cell is provided.
  • Figure 1 is a diagram illustrating a conventional product location calculation device.
  • Figure 2 is a schematic diagram of a product location calculation system according to an embodiment of the present disclosure.
  • Figure 3 is a block diagram of a product location calculation device according to an embodiment of the present disclosure.
  • Figure 4 is a flowchart of a product location calculation method according to an embodiment of the present disclosure.
  • Figure 5 is a flow chart illustrating the details of S200.
  • Figure 6 is a diagram illustrating that a first load cell is provided on one side of each line of a shelf and a second load cell is provided on the other side.
  • Figure 7 is a diagram illustrating that a first load cell and a second load cell are provided on one side of each line of the shelf.
  • Figures 8 and 9 are diagrams illustrating verification of the calculation result of S200 using camera captured images.
  • Figures 10 and 11 are diagrams illustrating verification of the calculation result of S200 using the sensing value sensed through the lidar sensor.
  • Figure 12 is a diagram schematically showing how a product location calculation device communicates with a server according to an embodiment.
  • Figure 13 is a diagram schematically showing the configuration included in the product location calculation device and server according to an embodiment.
  • Figure 14 is a diagram showing mode switching according to a product recognition algorithm according to an embodiment.
  • Figure 15 is a diagram schematically showing a product location calculation device 100 including a camera and a weight measurement sensor according to an embodiment.
  • Figure 16 is a flowchart showing steps performed in a product location calculation device and server according to an embodiment.
  • Figure 17 is a diagram schematically showing measuring the inclination angle of a load cell using an inclination measurement sensor according to an embodiment.
  • Figure 18 is a diagram illustrating an algorithm for the process from when a mode change is made to when an error is resolved according to an embodiment.
  • Figure 19 is a diagram illustrating an algorithm for the process of performing an absolute operation in ABS mode according to an embodiment.
  • Figure 20 is a diagram illustrating an algorithm for a process in which a relative operation is performed in the REL mode according to an embodiment.
  • Figure 21 is a diagram illustrating an algorithm for a process for resolving an error when an error occurs according to an embodiment.
  • Figure 1 is a diagram illustrating a conventional product location calculation device.
  • a device for calculating the position of products placed on a display shelf is illustrated, and this conventional device is manufactured to recognize products on a load cell line one by one.
  • the device uses the sensing value of the load cell to calculate whether the product is placed and the location of the product.
  • the conventional technology is only usable for arranging products of a certain size, and has a problem in that it does not work properly when the size or type of the product changes.
  • the inventor of the present disclosure seeks to provide a product position calculation device in a product showcase based on artificial intelligence according to the present disclosure to accurately calculate whether or not the product is placed and its location even when products of various shapes and weights are placed on the shelf. .
  • Figure 2 is a schematic diagram of an unmanned vending system 10 according to an embodiment of the present disclosure.
  • the unmanned vending system 10 may include a location calculation device 100 and a server 200.
  • the unmanned vending system 10 includes an unmanned vending device and a server 200, and the unmanned vending device may include a location calculation device 100 according to an embodiment of the present disclosure.
  • the unmanned vending system 10 shown in FIG. 1 is only an example and may include fewer or more components than the components shown in FIG. 1 .
  • the reason for this configuration is that an unmanned vending device is required for the unmanned vending system 10, and a location calculation device 100 according to an embodiment of the present disclosure is required to manage the unmanned vending device unmanned.
  • the location calculation device 100 may be a configuration for the unmanned vending device, system 10.
  • the unmanned sales device is a device 100 for selling specific types of products without a seller. It is composed of a display stand (e.g., a showcase) that displays products to be sold, and detects the entry and exit of the displayed products and determines inventory. , which ultimately allows the buyer to make payment for the purchased product.
  • a display stand e.g., a showcase
  • the unmanned vending device detects whether the products displayed inside enter or exit as the user (i.e., the buyer) who wants to purchase the product opens and closes the door, and transmits the results to the server 200. .
  • the unmanned vending device can receive inventory information generated by the location calculation device 100 according to an embodiment of the present disclosure and use this to manage the inventory of each product in the unmanned vending device.
  • the unmanned vending device may include a variety of devices that can perform the computational processing necessary to perform various functions for unmanned vending and provide results to the user.
  • an unmanned vending device includes a communication device such as a communication modem for communicating with various devices or a wired or wireless network, a first memory for storing various programs and data, and a first microsecond for executing the program to perform calculations and control. It may include various computing devices including processors, etc., and the location calculation device 100 of the present disclosure.
  • the unmanned sales device may be configured in the form of a display stand that displays products inside, and the interior of the display shelf may be configured with a structure in which a plurality of shelves are stacked. Additionally, each of the plurality of shelves is divided into a plurality of columns, and products can be placed in each of the plurality of columns.
  • Products placed in an unmanned vending device may include various types of products.
  • the product placed in the unmanned vending device may be a product that is light in weight compared to its volume (e.g., regular cigarettes, electronic cigarettes, snacks, etc.).
  • a product that is light in weight compared to its volume e.g., regular cigarettes, electronic cigarettes, snacks, etc.
  • this is only an example and is not necessarily limited to this.
  • each shelf in the unmanned vending device may be configured to have a predetermined inclination, and in this case, products placed on each column of each shelf may be pushed and moved to the front of the shelf by the predetermined inclination.
  • each shelf in the unmanned vending machine may further include a plurality of elastic members. That is, a plurality of elastic members may be installed corresponding to a plurality of columns separated on each shelf.
  • a plurality of elastic members may be installed corresponding to a plurality of columns separated on each shelf.
  • one elastic member may be installed in each column on the shelf, and in this case, the elastic member may be constructed using a spring or the like that is installed at the back of the shelf and has a force (i.e., elastic force) that moves toward the front of the shelf. there is.
  • the elastic member can apply a force to push the remaining products placed on each column to the front of the shelf by elastic force whenever the frontmost product placed on each column falls off.
  • the position calculation device 100 may be equipped with a load cell 120 capable of measuring the weight of products placed on each shelf.
  • the location calculation device 100 may be equipped with a camera capable of photographing products placed on each column of each shelf.
  • the camera can be installed on each shelf in the unmanned vending machine, and can be installed in a position to look down diagonally from the top at the products displayed on each shelf.
  • the height between shelves in the unmanned vending machine i.e., the height of each layer within the display shelf
  • the number of columns on the shelf i.e., the width of the product (depending on the type of product; for example, electronic cigarettes have a wider product width than regular cigarettes).
  • the optimal camera position and/or number of cameras may be determined by considering the camera's angle of view based on the camera's angle of view (may be large), etc.
  • the area that one camera can recognize i.e., the number of columns
  • the camera It can be installed in a position looking down diagonally.
  • cameras i.e., a left camera and a right camera
  • the position calculation device 100 can set and store information about products to be displayed (placed) in each column for each shelf (eg, product name, product price, product size, etc.). As an example, the location calculation device 100 may receive information about products to be placed in each column for each shelf from the server 200, and information about products to be placed in each column at the request of the server 200. You can also change .
  • the server 200 may include a computer system and computer software (web server program) that derives and provides task results corresponding to task performance requests from clients or other web servers.
  • the server 200 may include a series of application programs running on a web server or various databases built inside the device 100.
  • the server 200 may include a computer, a server device, and a portable terminal, or may take the form of any one.
  • the computer may include, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.
  • the server device is a server that processes information by communicating with external devices, such as application server), computing server), database server), file server), game server), mail server), proxy server), and web server). It may include etc.
  • the portable terminal is, for example, a wireless communication device that guarantees portability and mobility, such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), and PDA. (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone ), all types of handheld wireless communication devices, and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-device (HMD). may include.
  • PCS Personal Communication System
  • GSM Global System for Mobile communications
  • PDC Personal Digital Cellular
  • PHS Personal Handyphone System
  • PDA Personal Digital Assistant
  • IMT International Mobile Telecommunication
  • CDMA Code Division Multiple Access
  • W-CDMA Wideband Code Division Multiple Access
  • WiBro Wireless Broadband Internet
  • smart phone smart phone
  • the server 200 may link with at least one unmanned vending device for each store via a network.
  • the server 200 may provide information about products sold in an unmanned sales device (e.g., total number of products, price of each product, size of each product, display position of each product, identification information, top/bottom of product). /left/right images, etc.) can be managed.
  • the server 200 can transmit information about products to be placed in each column (eg, product name, product price, product size, etc.) to the unmanned vending device to be set in the unmanned vending device.
  • the server 200 receives information about the purchased product (e.g., product name, number of purchased products, etc.) of the purchased product that the purchaser has taken out by opening and closing the door of the unmanned vending device, from the unmanned vending device, and based on this, the purchaser's purchase.
  • the payment amount for the product can be calculated and transmitted to the payment device. Afterwards, the buyer can pay for the purchased product through the payment device.
  • the network can transmit and receive various information between the server and at least one unmanned vending device.
  • Various types of communication networks may be used, for example, wireless communication methods such as WLAN (Wireless LAN), Wi-Fi, Wibro, Wimax, and HSDPA (High Speed Downlink Packet Access).
  • wireless communication methods such as WLAN (Wireless LAN), Wi-Fi, Wibro, Wimax, and HSDPA (High Speed Downlink Packet Access).
  • wired communication methods such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), and FTTH (Fiber To The Home) may be used.
  • the network is not limited to the communication methods presented above, and may include all other types of communication methods that are well known or will be developed in the future in addition to the communication methods described above.
  • the location calculation device 100 is ultimately a device 100 that manages inventory so that the unmanned sales device can be operated unmanned.
  • a device 100 that manages inventory so that the unmanned sales device can be operated unmanned.
  • Figure 3 is a block diagram of a product location calculation device 100 according to an embodiment of the present disclosure.
  • the product location calculation device 100 in the artificial intelligence-based product showcase includes a first processor 110, a load cell 120, a calculation unit 130, a verification unit 140, and a first memory 150. ), a camera 160, a LiDAR sensor 170, and a first communication unit 180.
  • the product location calculation device 100 may include fewer or more components than the components shown in FIG. 2 .
  • a plurality of load cells 120 are provided, and in detail, the device 100 senses the positions of products placed on the shelves of a display stand divided into a plurality of columns. It includes a plurality of load cells 120 to do this.
  • load cells 120 are disposed on each line to separate the plurality of columns.
  • the method of calculating the position of the product can be divided into the first embodiment and the second embodiment according to the arrangement method of the first load cell 121 and the second load cell 122 on each line and the position calculation method according to the arrangement method. You can.
  • the calculation unit 130 stores algorithms and commands for calculating the position of the product, and typically performs internal calculation of the sensing values sensed by the first load cell 121 and the second load cell 122 in the first embodiment.
  • an algorithm that calculates the location information of the product may be included, and an algorithm that calculates the location information of the product by calculating the outer product of the sensing values sensed by the first load cell 121 and the second load cell 122. .
  • the first processor 110 can control the calculation unit 130 to calculate the first location information of the product placed on the shelf.
  • the verification unit 140 may verify the results calculated by the calculation unit 130.
  • the verification unit 140 may verify the calculation result using the captured image of the camera 160 that photographs the shelf and the sensing data of the LiDAR sensor 170.
  • the first memory 150 may store various algorithms, commands, artificial intelligence models, etc. for driving the product location calculation device 100.
  • the first memory 150 can store a list of products currently placed in the device 100 and the number of products, and can store a list of products placed on each shelf and the weight and shape information of each product. there is.
  • the first memory 150 can store data supporting various functions of the device and programs for the operation of the control unit, and stores input/output data (e.g., music files, still images, videos, etc.).
  • input/output data e.g., music files, still images, videos, etc.
  • a number of application programs (application programs or applications) running on the device, data for operation of the device, and commands can be stored. At least some of these applications may be downloaded from an external server via wireless communication.
  • the first memory 150 may be a flash first memory 150 type, a hard disk type, a solid state disk type, an SDD type, or a silicon disk drive type.
  • Multimedia card micro type card type first memory 150 (e.g. SD or XD first memory 150, etc.), random access memory (RAM), static random access (SRAM) memory), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic first memory 150, at least one type of magnetic disk and optical disk. It may include storage media. Additionally, the first memory 150 is separate from the main device, but may be a database connected by wire or wirelessly.
  • the verification unit 140 can use the weight and shape information of the product stored in the first memory 150 to verify the calculation result of the calculation unit 130.
  • One or more cameras 160 may be installed on each shelf, and may be installed on the upper part of the shelf so that at least a portion of the shelf can be photographed.
  • At least one camera 160 is installed on each shelf, and can generate captured images by photographing the shelves at a plurality of different viewing angles.
  • the LiDAR sensor 170 may be provided on both sides of the shelf, and may generate a sensing value by sensing the direction of the shelf under the control of the first processor 110.
  • the first communication unit 180 may include one or more components that enable communication with an external device, for example, a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, It may include at least one of the location information modules.
  • an external device for example, a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, It may include at least one of the location information modules.
  • the first processor 110 is responsible for controlling the product location calculation device 100 and components within the unmanned vending device, and can execute a product location calculation method and an unmanned vending method using algorithms, commands, and artificial intelligence models.
  • the first processor 110 performs the above-described operations using a first memory that stores data for an algorithm for controlling the operation of components in the device or a program that reproduces the algorithm, and the data stored in the first memory.
  • a first memory that stores data for an algorithm for controlling the operation of components in the device or a program that reproduces the algorithm, and the data stored in the first memory.
  • the first memory and the first processor may each be implemented as separate chips.
  • the first memory and the first processor may be implemented as a single chip.
  • the first processor 110 may control any one or a combination of the above-described components in order to implement various embodiments according to the present disclosure described in FIGS. 2 to N below on the present device. there is.
  • the sensing unit senses at least one of the device's internal information, surrounding environment information surrounding the device, and user information, and generates a corresponding sensing signal. Based on these sensing signals, the control unit can control the driving or operation of the device, or perform data processing, functions, or operations related to an application program installed on the device.
  • the load cell 120, camera 160, and lidar sensor 170 may be included in the sensing unit.
  • the sensing unit includes a proximity sensor, an illumination sensor, a touch sensor, an acceleration sensor, a magnetic sensor, a gravity sensor (G-sensor), and a gyro.
  • Gyroscope sensor motion sensor, RGB sensor, IR sensor (infrared sensor), finger scan sensor, ultrasonic sensor, optical sensor, e.g.
  • a camera e.g., a microphone
  • an environmental sensor e.g., including at least one of a barometer, a hygrometer, a thermometer, a radiation detection sensor, a heat detection sensor, and a gas detection sensor
  • a chemical sensor e.g., a healthcare sensor
  • biometric sensors etc.
  • the first processor 110 includes an image acquisition unit that acquires an image captured by the camera 160, a product recognition unit that analyzes the acquired image to recognize the product, and detects that the location of the product has changed. It may further include a location change determination unit that determines/calculates the changed location. Additionally, the first processor 110 may use the calculation unit 130 to calculate the number of products placed on the shelf.
  • Figure 4 is a flowchart of a product location calculation method according to an embodiment of the present disclosure.
  • Figure 5 is a flow chart illustrating the details of S200.
  • the first processor 110 detects that a product is placed on the shelf. (S100)
  • all load cells 120 may sense a sensing value of “0”.
  • the first processor 110 may determine that the product is placed on the shelf.
  • the first processor 110 may execute S100.
  • the first processor 110 can control the load cell 120 of the lathe to start operating by switching it from sleep mode to wake up mode.
  • the first processor 110 calculates location information of products placed on the shelf. (S200)
  • the first processor 110 calculates the first position information of the product placed on the shelf based on the sensing value sensed by the plurality of load cells 120 and the position information of each of the plurality of load cells 120 that sensed the sensing values. .
  • the first processor 110 detects a value other than “0” in the load cell 120.
  • At least one of first position information and weight can be calculated based on the position information and the sensing value sensed by the corresponding load cell 120.
  • Figure 6 is a diagram illustrating that a first load cell 121 is provided on one side of each line of a shelf, and a second load cell 122 is provided on the other side.
  • the plurality of load cells 120 are provided with a first load cell 121 on one side of each line and a second load cell 122 on the other side of each line.
  • the first processor 110 may calculate the first position information of the product placed on the shelf by internally calculating the sensing values sensed by the first load cell 121 and the second load cell 122 on each line.
  • first load cell 121 is provided on one side of each of the eight lines (upper side in FIG. 6), and a first load cell 121 is provided on each of the other sides of the eight lines (lower side in FIG. 6).
  • a second load cell 122 is provided.
  • the first processor 110 sensed “8” from the first load cells (121-1, 121-2, and 121-3) and “4” from the second load cells (122-1, 122-2, and 122-3). Since " was sensed, it can be determined that the product was placed on the 4th, 5th, and 6th lines from the left.
  • the first processor 110 can calculate the placement value of the product on the x-axis, and in FIG. 6, the placement value of the product on the x-axis can be calculated as 4, 5, and 6.
  • the first processor 110 may calculate the first location information of the product by calculating the sensing value sensed by the product in the first load cell 121 and the second load cell 122 (internal division point formula).
  • the first processor 110 calculates the sensing values sensed by the first load cell 121 and the second load cell 122 using an internal point formula to provide first location information (e.g., coordinates) of the product placed on the shelf. ) Any method of calculating can be applied.
  • the first processor 110 may calculate the first location information of the product based on Equation 1 below.
  • m is the first load cell sensing value
  • n is the second load cell sensing value
  • a is the first load cell coordinate
  • b is the second load cell coordinate.
  • the measured/sensed weight value may be different for each column due to the characteristics of vending machine and refrigerator shelves.
  • the first processor 110 may recognize and process the objects as the same when the difference in weight between columns is less than a threshold.
  • Figure 7 is a diagram illustrating that the first load cell 121 and the second load cell 122 are provided on one side of each line of the shelf.
  • FIG. 7 eight lines on a shelf are illustrated, and a first load cell 121 and a second load cell 122 are provided on each side of the eight lines (upper side in FIG. 7).
  • the first processor 110 sensed “8” from the first load cells (121-1, 121-2, and 121-3) and “4” from the second load cells (122-1, 122-2, and 122-3). Since " was sensed, it can be determined that the product was placed on the 4th, 5th, and 6th lines from the left.
  • the first processor 110 can calculate the placement value of the product on the x-axis, and in FIG. 6, the placement value of the product on the x-axis can be calculated as 4, 5, and 6.
  • the first processor 110 may calculate the first position information of the product by calculating the external differentiation (external point formula) of the sensing value of the product sensed by the first load cell 121 and the second load cell 122.
  • the first processor 110 calculates the sensing values sensed by the first load cell 121 and the second load cell 122 using an extrinsic point formula to provide first location information (e.g., coordinates) of the product placed on the shelf. ) Any method of calculating can be applied.
  • the first processor 110 may calculate the first location information of the product based on Equation 2 below.
  • m is the first load cell sensing value
  • n is the second load cell sensing value
  • a is the first load cell coordinate
  • b is the second load cell coordinate.
  • the load cell 120 can be placed on both sides of the line. Also, when applying the present disclosure as in the second embodiment, the load cell 120 can be placed at one end of the line.
  • the results produced in the first and second embodiments are the same, and the practitioner of the invention can easily select one by considering the characteristics of the shelf and the characteristics of the device to which the shelf is applied (refrigerator, display stand, etc.).
  • the first processor 110 verifies the calculation result of S200. (S300)
  • Figures 8 and 9 are diagrams illustrating verification of the calculation result of S200 using images captured by the camera 160.
  • various methods can be applied to verify the calculation result of S200, and a single method may be applied or a mixture of various methods may be applied.
  • the first processor 110 acquires images captured by a plurality of cameras 160 of a shelf on which a product is placed. (S310)
  • the first processor 110 verifies the calculation result of S200 based on the plurality of captured images acquired in S310. (S330)
  • the shelf is equipped with at least one camera 160 capable of photographing at least a portion of the shelf at a plurality of different angles of view.
  • the first processor 110 controls the camera 160 to generate a plurality of captured images by photographing the shelf at a plurality of different angles of view.
  • the first processor 110 may verify the calculation result of the first location information of the product by matching the first location information of the product calculated in S200 with a plurality of captured images.
  • the first processor 110 calculates product image second location information (location information calculated based on camera images) based on a plurality of captured images, and calculates the calculated image second location information and It can be determined whether the first location information (position information calculated based on the load cell sensing value) matches.
  • a shelf may have markers capable of recognizing a position (eg, coordinates) on the shelf at a plurality of preset points.
  • the first processor 110 may recognize the image and marker of the product in a plurality of shelf images taken at different angles of view and calculate second position information on the shelf of the product.
  • the first processor 110 may recognize that a product is placed at the location of an unidentified marker in a plurality of shelf photographed images and calculate second location information for the product.
  • the first processor 110 may recognize that the camera 160 captured the product incorrectly or that multiple products are placed on the shelf and generate an error signal.
  • Figures 10 and 11 are diagrams illustrating verification of the calculation result of S200 using the sensing value sensed through the lidar sensor 170.
  • a LiDAR sensor 170 is provided/installed on both sides of the shelf.
  • the first processor 110 acquires the sensing value sensed through the LiDAR sensor 170. (S350)
  • the first processor 110 verifies the calculation result of S200 based on the sensing value of the LiDAR sensor 170. (S370)
  • the first processor 110 may verify the third location information of the product calculated in S200 based on the sensing values sensed through the lidar sensors 170 on both sides.
  • a first LiDAR sensor 171 is provided on one side of the interior of the shelf, and a second LiDAR sensor 172 is provided on the other side of the interior.
  • the first processor 110 provides third location information of the product placed on the shelf based on the sensing value sensed through the first LiDAR sensor 171 and the sensing value sensed through the second LiDAR sensor 172. It can be calculated.
  • the first processor 110 may verify the calculated second location information by comparing the calculated second location information of the product and the calculated third location information.
  • the first processor 110 determines the appearance information of the product placed on the shelf based on the sensing value sensed through the first LiDAR sensor 171 and the sensing value sensed through the second LiDAR sensor 172. It can be created and the type of product can be judged based on appearance information.
  • the first processor 110 may match the appearance information with each product information in the product list stored in the first memory 150 to determine the type of product.
  • the first processor 110 may assign a plurality of sensing points to each of the first LiDAR sensor 171 and the second LiDAR sensor 172 based on the calculated first location information of the product. there is.
  • the first processor 110 verifies the first location information of the product based on the sensing values sensed through the first LiDAR sensor 171 and the second LiDAR sensor 172 for a plurality of assigned sensing points. You can.
  • the first processor 110 determines the type of product placed on the shelf. (S400)
  • the first processor 110 calculates the weight and The expected value of the lower part shape can be calculated.
  • the first processor 110 may determine the type of product based on the calculated weight of the product and the expected shape of the bottom of the product.
  • the first memory 150 may store information, weight, and lower shape information of each product placed on the shelf.
  • the first processor 110 determines whether the product is properly placed. (S500)
  • the first processor 110 may determine whether the product placed on the shelf is normally placed based on the calculated first location information of the product and the determined product type.
  • the first memory 150 may store at least one of the type of product to be placed on each shelf, the weight of the product, the external shape of the product, and the bottom shape of the product.
  • the first processor 110 compares the outer shape information or bottom shape information of the product to be placed on each shelf with the first position information to determine whether the product to be placed on the corresponding shelf is correctly placed or whether the product is displayed properly. can be judged.
  • the first processor 110 can determine whether the product type is properly placed, as well as whether the product is properly placed top and bottom and left and right.
  • the position can be accurately calculated even if products of various shapes and weights are placed on the shelf, and the correct product can be Not only can it be determined whether it has been properly placed, but it also has the effect of further improving accuracy by verifying the calculated location information.
  • the product location calculation method and device 100 have been described. Below, an embodiment in which the product location calculation device 100 calculates the number of product sales using a product recognition algorithm is additionally described. Let me explain.
  • the algorithm that calculates the location of the product and the algorithm that recognizes the product and calculates the number of sales can be implemented independently or used together, so the two embodiments can be implemented separately or a mixture of the two embodiments can be used. It may be possible.
  • the product location calculation device 100 may include at least a portion of the configuration and functions for the algorithm for calculating the number of products sold, which will be described below.
  • the product location calculation device 100 can calculate the number of products sold and may include the following configurations.
  • the product location calculation device 100 communicates with an external device 100 including a memory for storing the standard weight of the product and a preset correction index, at least one camera 160, and a weight measurement sensor. It includes a communication module that performs communication, and at least one processor that communicates with the memory, wherein the at least one processor receives first product information from the external device 100 when the door of the product storage unit is opened, When the door of the product storage unit is closed, second product information is received from the external device 100, and an absolute calculation is performed based on the weight information included in each of the first product information and the second product information to product the product.
  • an external device 100 including a memory for storing the standard weight of the product and a preset correction index, at least one camera 160, and a weight measurement sensor. It includes a communication module that performs communication, and at least one processor that communicates with the memory, wherein the at least one processor receives first product information from the external device 100 when the door of the product storage unit is opened, When the door of the product storage unit is closed, second product
  • the number of sales is calculated, and if it is determined that an error has occurred according to an error detection algorithm predetermined based on the second product information, the absolute operation is stopped and included in each of the first product information and the second product information.
  • a relative operation is performed based on the weight information difference value to calculate the number of products sold, and the correction index is changed based on the tilt information of the load cell 120 received from the external device 100 to determine the error. If it is determined that is resolved, the relative operation is stopped and the absolute operation is performed again.
  • machine learning is performed based on the product image captured from the camera 160 to determine the calculation value for each, and the learning model that performs the machine learning is multi-class.
  • Multiclass classification and binary classification can be implemented, and the multiclass classification is performed by a first convolutional neural network (CNN) model, and the binary classification is performed by a second convolutional neural network (CNN) model. It is performed by a neural network model, and the predetermined error detection algorithm is, when it is determined by the second convolutional neural network model that there is no product in the load cell 120, and the load cell measured by the weight measurement sensor If it is determined that the product exists based on the weight information in (120), it is determined that an error has occurred.
  • the algorithm determines that the error has occurred. Determining that the set correction index needs to be changed, receiving tilt information of the load cell 120 from the external device 100, and changing the preset correction index based on the tilt information of the load cell 120, When the preset correction index is changed, it can be determined that the error has been resolved.
  • the product location calculation method can calculate the number of products sold and may include the following components.
  • the product location calculation device 100 stores the standard weight of the product and a preset correction index in the memory, and when the door of the product storage section is opened, the first product information is received from the external device 100 including the camera 160 and the weight measurement sensor. receiving, when the door of the product storage unit is closed, receiving second product information from the external device 100, absolute weight information included in each of the first product information and the second product information.
  • the first product information is information about the product image captured by the camera 160 and the weight of the load cell 120 measured by the weight measurement sensor when the door of the product storage unit is opened.
  • Product information is characterized in that it is information about the product image captured by the camera 160 and the weight of the load cell 120 measured by the weight measurement sensor when the door of the product storage unit is closed.
  • machine learning is performed based on the product image captured from the camera 160 to determine the calculation value for each, and the learning model that performs the machine learning is multi-class.
  • Multiclass classification and binary classification can be implemented, and the multiclass classification is performed by a first convolutional neural network (CNN) model, and the binary classification is performed by a second convolutional neural network (CNN) model. It can be performed by a neural network model.
  • the predetermined error detection algorithm determines the inventory number based on the absolute calculation based on when the door is closed, and calculates the inventory number multiplied by the standard weight of the product and measured by the weight measurement sensor.
  • a ratio is calculated by comparing the information on the weight of the load cell 120, and if the ratio differs by more than a preset value, it is determined that the error has occurred, and if the ratio differs by less than a preset value, It is characterized by an algorithm that determines that the above error has not occurred.
  • tilt information of the load cell 120 is received from the external device 100, and tilt information of the load cell 120 is received.
  • the preset correction index is changed based on , and when the preset correction index is changed, it can be determined that the error has been resolved.
  • the predetermined error detection algorithm determines that there is no product in the load cell 120 by the second convolutional neural network model, and the weight of the load cell 120 measured by the weight measurement sensor If it is determined that the product exists based on the information, it is determined that an error has occurred, and if not, it is characterized as an algorithm that determines that an error has not occurred.
  • tilt information of the load cell 120 is received from the external device 100, and tilt information of the load cell 120 is received.
  • the preset correction index is changed based on , and when the preset correction index is changed, it can be determined that the error has been resolved.
  • the device 100 further includes a weight measurement sensor 133, a tilt measurement sensor 137, and a door unit 190.
  • some components may be omitted or some additional components may be included in the product location calculation device 100 of FIG. 3.
  • the load cell 120 and camera 160 of FIG. 3 may adopt the configuration of the load cell 120 and camera 160 shown in FIGS. 3 to 11, or may be used as a separate additional load cell 120 and camera. (160) may be additionally provided.
  • the product location calculation device 100 may include at least one third load cell 120 and at least one second camera 160.
  • the load cell 120 may include at least one first load cell, at least one second load cell, and at least one third load cell
  • the camera 160 may include at least one first camera, It may include at least one second camera.
  • Figure 12 is a diagram schematically showing communication between the product location calculation device 100 and the server according to an embodiment
  • Figure 13 is a diagram showing the configuration included in the product location calculation device 100 and the server according to an embodiment. This is a diagram schematically showing.
  • a product location calculation device 100 and a server may be provided to perform the operation of the present invention.
  • data collected from the product location calculation device 100 is analyzed and processed by the server, but this is not limited to this and all operations performed by the second processor 210 of the server Can also be performed by the first processor 110 of the product location calculation device 100.
  • the product location calculation device 100 can recognize products within the product location calculation device 100 and calculate the number of products sold without transmitting data to the server.
  • the product position calculation device 100 includes a first processor 110, a load cell 120, a calculation unit 130, a weight measurement sensor 133, a tilt measurement sensor 137, and a verification unit 140. ), a first memory 150, a camera 160, a LiDAR sensor 170, a first communication unit 180, and a door unit 190.
  • the server may be provided to include a second memory 220, a second communication unit 230 capable of communicating with the product location calculation device 100, and a second processor 210.
  • the load cell 120 may refer to a storage space in which products can be stored within the product position calculation device 100. Since the load cell 120 is inclined by the inclination angle ⁇ 1, when the user receives the product, the product can move toward the door unit 190 along the inclined plane.
  • the load cell may be prepared/installed in a storage space that can store products within the product position calculation device 100.
  • the product location calculation device may be provided in a display/exhibition showcase to sell products.
  • a weight measurement sensor 133 may be mounted on the load cell 120.
  • the weight measurement sensor 133 is attached to the lower part of the load cell 120, and the sensing distance and sensing resistance value vary depending on the weight of the product on the load cell 120, and based on this, the weight can be measured. For example, when comparing the case where there are 10 products remaining on the load cell 120 and the case where there are 5 products remaining, in the case where there are 10 products remaining, the sensing resistance value increases as the sensing distance becomes shorter, and the product In the case where there are five remaining teeth, the sensing resistance value may decrease as the sensing distance increases, and the weight can be measured based on this.
  • a tilt measurement sensor 137 may be provided in the product position calculation device 100.
  • the tilt measurement sensor 137 can be used to preset the correction index (k1) as described later, and to change the correction index (k2) by measuring a new tilt angle when an error occurs.
  • the weight measurement sensor 133 and the tilt measurement sensor 137 sense at least one of the internal information of the device 100, the surrounding environment information surrounding the device 100, and the user information, and send a sensing signal corresponding thereto. generates Based on these sensing signals, the processor 110 or 210 may control the driving or operation of the device 100 or perform data processing, functions, or operations related to an application program installed on the device 100.
  • the door unit 190 may be the product location calculation device 100 or a door of a product storage unit mounted on a product showcase. As described later, when performing absolute calculation in ABS mode when calculating the number of products sold, the calculation may be performed based on when the door of the door unit 190 is opened and when the door is closed.
  • the camera 160 may refer to an artificial intelligence-based machine vision camera mounted on the product location calculation device 100.
  • the image of the product captured by the camera 160 is transmitted to the server, and the second processor 210 can perform machine learning based on the received image.
  • the camera 160 of FIG. 13 may be applied in a different configuration from the camera 160 described with reference to FIGS. 3 to 11.
  • the camera in FIG. 13 may mean a device equipped to calculate the number of products.
  • the device 100 may calculate the number of products sold based on the captured image of the camera 160 installed on each shelf of the product showcase, and may additionally calculate the number of products sold based on a camera installed at another location ( 160), the number of products sold can also be calculated based on the captured video.
  • the first processor 110 of the device 100 may also perform machine learning based on the image captured by the camera 160.
  • Machine vision cameras can capture product videos and transmit them to the server.
  • a machine vision camera may consist of a lens, an image sensor, a main board, and an interface board, but is not limited to this. Additionally, images created through lenses and image sensors can be corrected into an appropriate form on the motherboard as needed. The video processed on the motherboard in this way can be transmitted to the server.
  • Machine vision cameras may include GigE Vision cameras (Gigabit Ethernet Vision Cameras), USB3.0 cameras, CameraLink cameras, and CoaXPress cameras.
  • the memory can store the standard weight of the product and a preset correction index (k1) as described later.
  • the standard weight of the product is the general weight of the product on the load cell 120 and may be a fixed value stored in advance in the memory.
  • the standard weight of a product may be stored differently depending on the type of product in the load cell 120. For example, if the product of load cell 1 is a 185ml can of cider, the standard weight of the can of cider can be stored as 200g, and if the product of load cell 2 is a 250ml can of cola, the standard weight of the can of cola can be stored as 300g. It can be.
  • the memory can store data supporting various functions of the device 100 and programs for the operation of the processors 110 and 210, and can store input/output data (e.g., music files, still images, and videos). etc.), a plurality of application programs (application programs or applications) running on the device 100, data for operation of the device 100, and commands can be stored. At least some of these applications may be downloaded from an external server via wireless communication.
  • card-type memory e.g., SD or It may include at least one type of storage medium among (only memory), PROM (programmable read-only memory), magnetic memory, magnetic disk, and optical disk. Additionally, the memory is separate from the device 100, but may be a database connected wired or wirelessly.
  • At least one processor (110 or 210) has a memory that stores data for an algorithm for controlling the operation of components in the device 100 or a program that reproduces the algorithm, and performs the above-described operations using the data stored in the memory. can be performed.
  • the memory and processor 110 or 210 may each be implemented as separate chips. Alternatively, the memory and processor 110 or 210 may be implemented as a single chip.
  • processor 110 or 210 combines any one or a plurality of the above-described components to implement various embodiments according to the present disclosure described in FIGS. 13 to 21 below on the device 100. You can control it.
  • the first communication unit 180 of the product location calculation device 100 and the second communication unit 230 of the server may include one or more components that enable communication with the external device 100, For example, it may be at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.
  • Wired communication modules include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as USB (Universal Serial Bus) modules. ), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).
  • LAN Local Area Network
  • WAN Wide Area Network
  • VAN Value Added Network
  • USB Universal Serial Bus
  • HDMI High Definition Multimedia Interface
  • DVI Digital Visual Interface
  • RS-232 Recommended standard 232
  • power line communication or POTS (plain old telephone service).
  • wireless communication modules include GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and UMTS (universal mobile telecommunications system). ), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, etc. may include a wireless communication module that supports various wireless communication methods.
  • GSM Global System for Mobile Communication
  • CDMA Code Division Multiple Access
  • WCDMA Wideband Code Division Multiple Access
  • UMTS universal mobile telecommunications system
  • TDMA Time Division Multiple Access
  • LTE Long Term Evolution
  • 4G, 5G, 6G, etc. may include a wireless communication module that supports various wireless communication methods.
  • the wireless communication module may include a wireless communication interface including an antenna and a transmitter that transmits a mobile communication signal. Additionally, the wireless communication module may further include a mobile communication signal conversion module that modulates a digital control signal output from the control unit through a wireless communication interface into an analog wireless signal under the control of the control unit.
  • the wireless communication module may include a wireless communication interface including an antenna and a receiver for receiving mobile communication signals. Additionally, the wireless communication module may further include a mobile communication signal conversion module for demodulating an analog wireless signal received through a wireless communication interface into a digital control signal.
  • the short-range communication module is for short-range communication and includes Bluetooth (Bluetooth), RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, and NFC (Near Field). Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology can be used to support short-distance communication.
  • the first processor 110 or the second processor 210 includes a product sales count calculation unit 213 and a machine vision camera ( It may include a machine learning unit 215 that performs machine learning of an artificial intelligence learning model based on the product image captured by 160).
  • each component shown in FIG. 13 refers to software and/or hardware components such as Field Programmable Gate Array (FPGA) and Application Specific Integrated Circuit (ASIC).
  • FPGA Field Programmable Gate Array
  • ASIC Application Specific Integrated Circuit
  • FIG. 14 is a diagram showing mode switching according to a product recognition algorithm according to an embodiment
  • FIG. 15 is a diagram showing a product location calculation device 100 including a camera 160 and a weight measurement sensor 133 according to an embodiment. This is a schematic drawing.
  • a product can be stored in the load cell 120 with an inclination angle of ⁇ 1, and when a user receives the product 30 according to the slope, the product can be moved toward the door unit 190.
  • the camera 160 can photograph the front part of the load cell 120 close to the door 190. Therefore, when the product image captured by the machine vision camera 160 is transmitted to the processor 110 or 210, learning is performed by the second convolutional neural network (CNN) model as described later, and the product 30 is You can determine whether it exists or not.
  • CNN convolutional neural network
  • Nv which is a code value based on image digital recognition
  • the product 30 in FIG. 14 may refer to each product 30 displayed on the load cell 120.
  • the present invention is basically set to ABS mode (Absolute mode, 310), and absolute calculation can be performed when calculating the number of products sold according to ABS mode (310).
  • ABS mode Absolute mode, 310
  • absolute calculation can be performed when calculating the number of products sold according to ABS mode (310).
  • the absolute operation according to the ABS mode 310) is stopped, and the mode is switched to the REL mode (Relative mode, 320) to perform the relative operation (S410).
  • the relative operation according to the REL mode 320 is stopped, and the mode is switched back to the ABS mode 310 to perform an absolute operation (S420).
  • the weight measurement sensor 133 is mounted on the lower part of the load cell 120 as described above and can measure the weight of the product on the load cell 120.
  • the weight measurement sensor 133 can be set to measure the weight of the product when the door of the door 190 of the product location calculation device 100 is opened and to measure the weight of the product when the door of the door 190 is closed. there is.
  • the first product information may include weight information of the product based on when the door of the door unit 190 is opened and image information of the product captured by the camera 160, but is not limited thereto.
  • the second product information may include, but is not limited to, weight information of the product and image information of the product captured by the camera 160 based on when the door of the door unit 190 is closed.
  • the first product information and the second product information are information about products in the same column on the load cell 120, and may be product information based on when the door is opened and when the door is closed.
  • the first product information and the second product information can be transmitted from the product location calculation device 100 to the server through communication between the first communication unit 180 of the product location calculation device 100 and the second communication unit of the server.
  • Figure 16 is a flowchart showing steps performed in the server and product location calculation device 100 according to an embodiment.
  • the user can make the product location calculation device 100 recognize payment methods that can pay for products, including check cards and credit cards, in order to pay for products (S601).
  • the card information recognized in this way may be transmitted to the processor 210 through the second communication unit of the server (S602).
  • the server can receive the card information transmitted in this way (S603). If the server recognizes the card information as correct, the server may transmit a command to open the door unit 190 to the product location calculation device 100 (S604).
  • the door unit 190 door open command is transmitted, when the door unit 190 of the product location calculation device 100 is opened (S605), information that the door unit 190 has been opened may be transmitted to the server. There is (S606).
  • the server can receive first product information (S607).
  • first product information S607
  • the server S610
  • the processor 110 or 210 may receive second product information (S611).
  • the server receives the first product information and the second product information, it can perform calculations based on the first product information and the second product information to calculate the number of products sold by the user (S612).
  • the calculated number of products sold is transmitted to the product location calculation device 100 (S613), and product payment according to the number of products sold can be made (S614).
  • FIG. 17 is a diagram schematically showing measuring the inclination angle of the load cell 120 using the inclination measurement sensor 137 according to an embodiment.
  • the product 30 in FIG. 17 may refer to each product displayed on the load cell 120.
  • the load cell 120 may be inclined by an inclination angle ⁇ 1 based on a flat surface, so that when a user receives a product from the product location calculation device 100, the product on the side farthest from the door unit 190 along the inclined plane is moved to the door. You can move towards section 190.
  • the tilt measurement sensor 137 is attached to the bottom of the load cell 120 and can measure the tilt of the load cell 120.
  • the measured inclination angle ( ⁇ 1) can be used to determine a preset correction index (k1) stored in memory. Since the weight of the inclined load cell 120 may be different from the weight on flat ground, a correction model based on a preset correction index can be used. That is, the weight can be corrected using a correction index that can correct the weight measurement error that occurs due to the inclination of the load cell 120.
  • Equation 3 the equation of the correction model may be expressed as Equation 3 below.
  • Mb may mean the weight on the inclined load cell 120
  • k1 may mean the preset correction index
  • Ma may mean the weight on flat ground.
  • the preset correction index k1 may be cos ⁇ 1 or csc ⁇ 1, but is not limited thereto.
  • the preset correction index (k1) is the actual weight value of the weight (e.g., 100 g, 200 g, etc.) when placing a unit weight (e.g., 100 g, 200 g, etc.) on the weight measurement sensor 133 at the time of installing the load cell 120. It can also be set by adjusting the zero point of the weight measurement sensor 133 so that it can be read.
  • Figure 18 is a diagram illustrating an algorithm for the process from when a mode change is made to when an error is resolved according to an embodiment.
  • the present invention basically starts in ABS mode (S810), and absolute calculation can be performed based on weight information included in each of the first product information and the second product information (S820). Accordingly, the number of products sold can be calculated (S830), and a process of determining that an error has occurred according to a predetermined error detection algorithm based on the second product information is performed (S840). If no error occurs, absolute calculation according to ABS mode is continuously performed, and if it is determined that an error has occurred, it switches to REL mode (S850). As the mode is switched to REL mode, a relative calculation can be performed based on the difference value of the weight information included in each of the first product information and the second product information (S860).
  • the number of products sold can be calculated (S870), and at the same time, based on the tilt information of the load cell 120 from the tilt measurement sensor 137 included in the product position calculation device 100 according to a predetermined correction algorithm,
  • the set correction index (k1) can be changed (S880). Accordingly, if it is determined that the error has been resolved, it switches back to ABS mode and absolute calculation can be performed. However, if it is determined that the error has not been resolved, REL mode is maintained and the number of products sold according to relative calculation can be calculated (S890 ).
  • Figure 19 is a diagram illustrating an algorithm for the process of performing an absolute operation in ABS mode according to an embodiment.
  • the standard weight (Wm) of the product is stored in memory (S910), the first product information can be received when the door is opened (S920), and the second product information can be received when the door is closed (S930) ).
  • the first product information and the second product information are the weight information of the load cell 120 by the weight measurement sensor 133 and the machine vision camera 160 for when the door is opened and when the door is closed, respectively.
  • the processor 110 or 210 may perform machine learning using a learning model.
  • the calculation values of the absolute calculation in ABS mode and the relative calculation in REL mode as described later can be determined by performing machine learning using the image of the product captured by the machine vision camera 160 as data.
  • a learning model that performs machine learning can implement multiclass classification and binary classification. Meanwhile, multiclass classification may be performed by a first convolutional neural network (CNN) model, and binary classification may be performed by a second convolutional neural network model, but are not limited thereto.
  • the learning models include Random Forest (RF), Support Vector Machine (SVC), eXtra Gradient Boost (XGB), Decision Tree (DC), Knearest Neighbors (KNN), Gaussian Naive Bayes (GNB), and Stochastic Gradient. It may include at least one algorithm among Descent (SGD), Linear Discriminant Analysis (LDA), Ridge, Lasso, and Elastic net.
  • Whether a product exists when the door of the door unit 190 is closed can be determined by implementing binary classification by a second convolutional neural network model.
  • the standard weight Wv of the product recognized by the machine vision camera 160 can be received from the memory, and if the product does not exist, the standard weight Wv of the product can be received. If it is recognized as not being the case, the standard weight (Wv) of the product recognized by the machine vision camera 160 may be determined to be 0.
  • the processor 110 or 210 can perform an absolute operation according to the ABS mode (S970), and the formula for the absolute operation can be expressed as Equation 4 below.
  • Ns refers to the number of products sold
  • Nb refers to the number of inventory when the door is opened
  • Nt refers to the number of inventory when the door is closed
  • Wm refers to the standard weight of the product
  • Wc refers to the weight of the load cell 120 when the door is closed
  • Wv refers to the standard weight of the product recognized by the camera 160
  • Nv refers to the digital code value of the product recognized by the camera 160. can do.
  • the number of products sold can be calculated (S980).
  • Ns the number of products sold
  • Figure 20 is a diagram illustrating an algorithm for a process in which a relative operation is performed in the REL mode according to an embodiment.
  • the standard weight (Wm) of the product is stored in the memory (S1010), the first product information can be received when the door is opened (S1020), and the second product information can be received when the door is closed (S1030).
  • W1 weight of the load cell 120 when the door is opened
  • W2 weight of the load cell 120 when the door is closed
  • the standard weight Wv of the product recognized by the machine vision camera 160 can be received from the memory, and if the product does not exist, the standard weight Wv of the product can be received. If it is recognized as not being the case, the standard weight (Wv) of the product recognized by the machine vision camera 160 may be determined to be 0.
  • the processor 110 or 210 can perform a relative operation according to the REL mode (S1080), and the formula for the relative operation can be expressed as Equation 5 below.
  • Ns means the number of products sold
  • W1 means the weight of the load cell 120 when the door is opened
  • W2 means the weight of the load cell 120 when the door is closed
  • Wd is It means the difference in weight of the load cell 120 when the door is opened and when the door is closed
  • Wm means the standard weight of the product
  • Wv means the standard weight of the product recognized by the camera 160
  • Nv may mean the digital code value of the product recognized by the camera 160.
  • the number of products sold can be calculated (S1090).
  • the tilt measurement sensor 137 included in the product position calculation device 100 may measure the tilt angle ⁇ 2 of the load cell 120 again.
  • the correction index (k1) can be changed based on the tilt information of the load cell 120 received from the product position calculation device 100, and it can be determined whether the error has been resolved (S1100). Whether the error has been resolved can be determined by re-performing a predetermined error detection algorithm based on the second product information. If it is determined that the error has been resolved, the relative operation according to the REL mode is stopped and the absolute operation according to the ABS mode can be performed again (S1110). If it is determined that the error has not been resolved, the REL mode is maintained and the product according to the relative operation is At the same time as the number of sales is calculated, an error resolution algorithm can be implemented by continuously changing the correction index.
  • Figure 21 is a diagram illustrating an algorithm for a process for resolving an error when an error occurs according to an embodiment.
  • the tilt ( ⁇ 1) of the inclined load cell 120 can be measured by the tilt measurement sensor 137 (S110), and a preset correction index (k1) can be calculated based on this (S1120).
  • the preset correction index (k1) may be a cos ⁇ 1 value or csc ⁇ 1, but is not limited thereto.
  • the preset correction index (k1) can be stored in memory (S1130). Absolute calculation according to ABS mode is performed (S1140), and if it is determined that an error has occurred according to the error detection algorithm (S1150), relative calculation according to REL mode is performed (S1160), and the number of products sold can be calculated (S1160) S1170).
  • the tilt measurement sensor 137 of the product position calculation device 100 can measure the tilt ( ⁇ 2) of the load cell 120 again. Accordingly, the preset correction index (k1) can be changed to a new correction index (k2) and corrected (S1190). At this time, the new correction index (k2) may be a cos ⁇ 2 value or csc ⁇ 2, but is not limited thereto.
  • the process of determining whether the error has been resolved S11100
  • the relative operation according to the REL mode is stopped (S1180) and the absolute operation according to the ABS mode can be performed again, and the error is not resolved. If it is determined that it is not, the relative operation according to the REL mode is maintained and the error resolution algorithm can continue to proceed.
  • the process of determining whether an error has been resolved may be performed by the same process as the process of determining whether an error has been detected.
  • the predetermined error detection algorithm is a result value recognized by a second convolutional neural network model that implements binary classification that can determine the presence or absence of a product, and a weight measurement sensor that can measure the weight of the load cell 120. (133) It may be an algorithm that determines the error based on the difference in the measured values, but it is not limited to this.
  • the process of determining whether an error has been detected is based on when the door is closed. If the inventory number is determined based on absolute calculation in ABS mode, the standard weight of the product (Wm) is added to the inventory number (Nt). ) is calculated by comparing the multiplied value with the information on the weight of the load cell 120 measured by the weight measurement sensor 133, and if the calculated ratio differs by more than a preset value, the error occurs. It may be an algorithm that determines that the error has occurred and, if the calculated ratio differs by less than a preset value, determines that the error has not occurred.
  • the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by the processor 110 or 210, they may generate program modules to perform operations of the disclosed embodiments.
  • the recording medium may be implemented as a computer-readable recording medium.
  • Computer-readable recording media include all types of recording media storing instructions that can be decoded by a computer. For example, there may be read only memory (ROM), random access memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device 100, etc.
  • ROM read only memory
  • RAM random access memory
  • magnetic tape magnetic tape
  • magnetic disk magnetic disk
  • flash memory optical data storage device 100, etc.
  • FIGS. 18 to 21 the content described in FIGS. 18 to 21 is only an example for explaining the operation of the present invention, and there is no limitation to the operation in which the server performs communication with the product location calculation device 100 to calculate the number of products sold. .
  • the method according to an embodiment of the present disclosure described above may be implemented as a program (or application) and stored in a medium in order to be executed in combination with a server, which is hardware.
  • the above-described program includes C, C++, JAVA, and It may include code encoded in a computer language such as machine language. These codes may include functional codes related to functions that define the necessary functions for executing the methods, and control codes related to execution procedures necessary for the first processor of the computer to execute the functions according to predetermined procedures. may include. In addition, this code is related to a first memory reference as to which location (address address) of the internal or external first memory of the computer should be referenced by additional information or media required for the first processor of the computer to execute the functions. Additional code may be included. In addition, if the first processor of the computer needs to communicate with any other remote computer or server to execute the functions, the code can be transmitted to any other remote computer or server using the communication module of the computer. It may further include communication-related codes for how to communicate, etc., and what information or media should be transmitted and received during communication.
  • the storage medium refers to a medium that stores data semi-permanently and can be read by a device, rather than a medium that stores data for a short period of time, such as a register, cache, or first memory.
  • examples of the storage medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., but are not limited thereto. That is, the program may be stored in various recording media on various servers that the computer can access or on various recording media on the user's computer. Additionally, the medium may be distributed to computer systems connected to a network, and computer-readable code may be stored in a distributed manner.
  • the steps of the method or algorithm described in connection with the embodiments of the present disclosure may be implemented directly in hardware, implemented as a software module executed by hardware, or a combination thereof.
  • Software modules include RAM (Random Access Memory), ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), Flash Memory, hard disk, removable disk, and CD-ROM. , or it may reside on any form of computer-readable recording medium well known in the art to which this disclosure pertains.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Electromagnetism (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)

Abstract

La présente divulgation concerne un dispositif de calcul d'emplacement basé sur l'intelligence artificielle pour des produits dans une vitrine utilisant une structure d'agencement de cellules de charge, le dispositif comprenant : une pluralité de cellules de charge pour détecter l'emplacement de produits placés sur un rayonnage d'un présentoir divisé en colonnes multiples ; et un processeur qui calcule des informations d'emplacement des produits placés sur le rayonnage, en fonction de valeurs de détection obtenues par la pluralité de cellules de charge et d'informations d'emplacement de chacune des cellules de charge détectant les valeurs de détection.
PCT/KR2023/006778 2022-06-02 2023-05-18 Dispositif de calcul d'emplacement basé sur l'intelligence artificielle pour des produits dans une vitrine utilisant une structure d'agencement de cellules de charge WO2023234605A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
KR1020220067664A KR102430486B1 (ko) 2022-06-02 2022-06-02 로드셀 배치 구조를 활용한 인공지능 기반의 상품 쇼케이스 내의 상품 위치 산출 장치, 방법 및 프로그램
KR10-2022-0067664 2022-06-02

Publications (1)

Publication Number Publication Date
WO2023234605A1 true WO2023234605A1 (fr) 2023-12-07

Family

ID=82845320

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/KR2023/006778 WO2023234605A1 (fr) 2022-06-02 2023-05-18 Dispositif de calcul d'emplacement basé sur l'intelligence artificielle pour des produits dans une vitrine utilisant une structure d'agencement de cellules de charge

Country Status (2)

Country Link
KR (3) KR102430486B1 (fr)
WO (1) WO2023234605A1 (fr)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102430486B1 (ko) * 2022-06-02 2022-08-08 (주) 인터마인즈 로드셀 배치 구조를 활용한 인공지능 기반의 상품 쇼케이스 내의 상품 위치 산출 장치, 방법 및 프로그램

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150055341A (ko) * 2013-11-13 2015-05-21 현대자동차주식회사 보행 로봇 제어방법
KR20180105779A (ko) * 2017-03-16 2018-10-01 주식회사 인바디 낙상 위험 평가 시스템
KR20190017635A (ko) * 2017-08-11 2019-02-20 (주) 인터마인즈 전경 영상 추출 장치 및 방법
KR102179614B1 (ko) * 2020-08-24 2020-11-18 (주) 인터마인즈 무인 판매 방법 및 시스템
KR102430486B1 (ko) * 2022-06-02 2022-08-08 (주) 인터마인즈 로드셀 배치 구조를 활용한 인공지능 기반의 상품 쇼케이스 내의 상품 위치 산출 장치, 방법 및 프로그램

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101797656B1 (ko) 2015-12-09 2017-11-15 롯데알미늄 주식회사 멀티자판기의 상품위치 검출장치

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR20150055341A (ko) * 2013-11-13 2015-05-21 현대자동차주식회사 보행 로봇 제어방법
KR20180105779A (ko) * 2017-03-16 2018-10-01 주식회사 인바디 낙상 위험 평가 시스템
KR20190017635A (ko) * 2017-08-11 2019-02-20 (주) 인터마인즈 전경 영상 추출 장치 및 방법
KR102179614B1 (ko) * 2020-08-24 2020-11-18 (주) 인터마인즈 무인 판매 방법 및 시스템
KR102430486B1 (ko) * 2022-06-02 2022-08-08 (주) 인터마인즈 로드셀 배치 구조를 활용한 인공지능 기반의 상품 쇼케이스 내의 상품 위치 산출 장치, 방법 및 프로그램

Also Published As

Publication number Publication date
KR102669653B1 (ko) 2024-05-28
KR102430486B1 (ko) 2022-08-08
KR20240078426A (ko) 2024-06-03
KR20230167681A (ko) 2023-12-11

Similar Documents

Publication Publication Date Title
WO2019164280A1 (fr) Dispositif électronique flexible comprenant un capteur optique et son procédé de fonctionnement
WO2017065475A1 (fr) Dispositif électronique et procédé de traitement de gestes associé
WO2023234605A1 (fr) Dispositif de calcul d'emplacement basé sur l'intelligence artificielle pour des produits dans une vitrine utilisant une structure d'agencement de cellules de charge
WO2020149454A1 (fr) Dispositif électronique destiné à effectuer une authentification d'utilisateur et procédé d'opération associé
WO2019231042A1 (fr) Dispositif d'authentification biométrique
WO2021054589A1 (fr) Appareil électronique et procédé de commande associé
WO2020054973A1 (fr) Dispositif électronique de détection d'objet externe à l'aide d'un réseau d'antennes et procédé de fonctionnement associé
WO2022139461A1 (fr) Capacité tridimensionnelle d'angle d'arrivée dans des dispositifs électroniques
WO2019164290A1 (fr) Procédé d'authentification biométrique utilisant une pluralité de caméras avec différents champs de vision et appareil électronique associé
WO2022010122A1 (fr) Procédé pour fournir une image et dispositif électronique acceptant celui-ci
WO2022039560A1 (fr) Capacité d'angle d'arrivée dans des dispositifs électroniques avec fusion de capteur de mouvement
WO2020130301A1 (fr) Dispositif électronique permettant de suivre l'activité d'un utilisateur et son procédé de fonctionnement
WO2022025729A1 (fr) Détermination d'angle d'arrivée dans des dispositifs électroniques avec décision fusionnée à partir d'un mouvement
WO2021230485A1 (fr) Procédé et appareil de fourniture d'image
WO2021101006A1 (fr) Dispositif électronique pour fournir un contenu sur la base de l'emplacement d'une image réfléchissante d'un objet externe, et procédé de fonctionnement de dispositif électronique
WO2015199354A1 (fr) Procédé d'extraction de pupille utilisant une binarisation par agrégation de voisins et appareil de commande à extraction de pupille l'utilisant
WO2020017827A1 (fr) Dispositif électronique et procédé de commande pour dispositif électronique
WO2016143964A1 (fr) Terminal et procédé de fonctionnement associé
WO2021112490A1 (fr) Dispositif électronique pliable comprenant une caméra
WO2024075947A1 (fr) Dispositif électronique permettant de générer une interrogation d'image pour une recherche d'objet, et procédé pour faire fonctionner un dispositif électronique
WO2019124775A1 (fr) Dispositif électronique et procédé de fourniture des informations de service associées à un contenu de diffusion dans un dispositif électronique
WO2024025176A1 (fr) Procédé de comptage d'exercices et dispositif électronique le prenant en charge
WO2022177101A1 (fr) Dispositif électronique comprenant un capteur d'empreinte digitale et procédé associé de mise à jour d'informations d'empreinte digitale
WO2024117730A1 (fr) Dispositif électronique permettant d'identifier un objet et son procédé de commande
WO2020190110A1 (fr) Dispositif électronique contenant un capteur d'image et procédé de fonctionnement de celui-ci

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23816268

Country of ref document: EP

Kind code of ref document: A1