KR20230166854A - Method and apparatus for recognizing based on camera arranged on both sides of the front for each floor of showcase - Google Patents

Abstract

A camera-based recognition method and device placed on both sides of the front of each showcase floor is provided. The method includes acquiring a plurality of photographic data captured by a plurality of cameras respectively disposed on both sides of the front of each showcase floor, and using the plurality of photographic data to recognize the type of product placed in the central area of the corresponding floor. The recognition step may include extracting duplicate columns between the plurality of photographed data and recognizing the type of product placed in the extracted column using a product-related database.

Description

Camera-based recognition method and device placed on both sides of the front of each showcase floor {METHOD AND APPARATUS FOR RECOGNIZING BASED ON CAMERA ARRANGED ON BOTH SIDES OF THE FRONT FOR EACH FLOOR OF SHOWCASE}

This disclosure relates to a camera-based recognition method and device placed on both sides of the front of each showcase floor.

Generally, in order to purchase a product at a store, the buyer selects the product he or she wants to purchase, takes the product to a checkout location such as a checkout counter to pay for the selected product, and then pays the seller for the product to purchase it. It is being adopted.

For example, when purchasing coffee at a convenience store, the buyer obtains the coffee he/she wishes to purchase from the refrigerator or other product storage department at the convenience store, carries it to the cash register, hands it to the seller, and the seller provides price information for the product. After determining the price using a barcode, etc., the purchase stage of the product is completed by paying the determined price and handing it over to the buyer.

However, in the case of the product purchase method described above, since the seller must stand by at the stand, there is a problem in that the product cannot be purchased if the seller is absent for some reason. Additionally, when a large number of buyers flock in a short period of time or when there is only one counter for checkout, there is a problem in which buyers have to wait for a long time to purchase a product. In addition, there is a problem of increased labor cost burden because sales stores must hire sellers.

Recently, in order to overcome this problem, product sales outlets have installed various unmanned sales devices such as kiosks, and the number of situations where buyers can purchase products directly using unmanned sales devices without taking the product directly to the seller is increasing.

Public Patent Publication No. 10-2021-0141234, 2021.11.23.

The purpose of the embodiment disclosed in this disclosure is to provide a camera-based recognition method and device placed on both sides of the front of each showcase floor.

The problems to be solved by the present disclosure are not limited to the problems mentioned above, and other problems not mentioned can be clearly understood by those skilled in the art from the description below.

A recognition method based on cameras disposed on both sides of the front of each showcase floor according to one aspect of the present disclosure for achieving the above-described technical problem includes acquiring a plurality of captured data captured by a plurality of cameras disposed on both sides of the front of each showcase floor. A step of recognizing the type of product placed in the central area of the corresponding floor using the plurality of photographed data, wherein the recognition step includes extracting an overlapping column between the plurality of photographed data, , the type of product placed in the extracted column can be recognized using a product-related database.

In addition, each layer of the showcase includes n columns (where n is a natural number), and the first captured data captured by the first camera among the plurality of cameras is n-p in sequential order among the n columns. It includes columns (where p is a natural number smaller than n), and the second captured data captured by the second camera among the plurality of cameras may include n-q columns in reverse order among the n columns (where p is a natural number smaller than n). , q is a natural number less than n).

Additionally, each of the first camera and the second camera may be arranged symmetrically on both sides such that m columns overlap between the n-p columns and the n-q columns.

In addition, in the recognition step, for products placed in the overlapping column between the first and second photographed data, the first photographed data and the second photographed data are combined to generate combined data, and the database A product matching the combination data can be searched, and the type of the searched product can be determined as the type of product placed in the overlapping column.

In addition, in the recognition step, for products placed in the overlapping column between the first photographed data and the second photographed data, products whose matching rate with the first photographed data is greater than or equal to a preset first value in the database Extract, search for products whose matching rate with the second shooting data is more than a preset second value among the extracted products, and determine the type of the searched product as the type of product placed in the overlapping column. .

In addition, the method may further include the step of analyzing the plurality of photographed data based on a pre-built learning model to recognize misplaced products among the plurality of products placed on the corresponding floor.

In addition, the learning model is constructed by learning learning data for each misplacement type for the display state of the product, and the misplacement type for the display state is that the product moves on two axes among the x-axis, y-axis, and z-axis. This may include cases where the object is rotated based on the reference.

In addition, the learning model is constructed by learning learning data of misplacement for the display position of the product, and the misplacement for the display position may indicate a case where the product is not placed in a column assigned to the product. there is.

In addition, a camera-based recognition device disposed on both sides of the front of each showcase floor according to another aspect of the present disclosure includes a communication unit that communicates with a plurality of cameras arranged on each showcase floor, and products displayed in the showcase based on the plurality of cameras. a memory storing at least one process for recognition and a processor operating according to the process, wherein the processor is disposed on both sides of the front for each showcase floor through the communication unit based on the process. Obtain a plurality of shooting data captured by a plurality of cameras, recognize the type of product placed in the central area of the floor using the plurality of shooting data, and the processor performs the recognition, Overlapping columns between data can be extracted and the type of product placed in the extracted column can be recognized using a product-related database.

In addition to this, a computer program stored in a computer-readable recording medium for execution to implement the present disclosure may be further provided.

In addition, a computer-readable recording medium recording a computer program for executing a method for implementing the present disclosure may be further provided.

According to the means for solving the above-described problem of the present disclosure, by increasing the recognition rate for products displayed in the middle among products displayed in each of a plurality of columns, it becomes possible to recognize all products displayed in all columns, and thus the displayed products Management of all products can be done efficiently.

In addition, according to the means for solving the above-described problem of the present disclosure, it is possible to determine whether or not each product is misplaced using shooting data for each floor, thereby enabling real-time monitoring of product display.

In addition, according to the means for solving the above-described problem of the present disclosure, it is effective compared to the existing method in terms of the position of the camera installed in the showcase. In other words, by installing the camera in a position where it can look down diagonally at the products placed on the shelf, a sufficient distance between the camera lens and the object can be secured even if the margin between shelves (i.e. between each layer of the shelf) is small. As a result, more shelves can be installed within the display stand, thereby increasing efficiency in terms of floor area ratio.

The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned may be clearly understood by those skilled in the art from the description below.

1 is a diagram schematically showing an unmanned vending system according to the present disclosure.
Figure 2 is a diagram schematically showing the configuration included in the unmanned vending device and server according to the present disclosure.
Figure 3 is a view looking down from above of products displayed on shelves divided into a plurality of columns according to the present disclosure and a camera provided to photograph the products.
Figure 4 is a flowchart of a camera-based recognition method placed on both sides of the front of each showcase floor according to the present disclosure.
Figure 6 is an example diagram for explaining overlapping columns between a plurality of captured data according to the present disclosure.
FIGS. 7A to 7D are exemplary diagrams for explaining types of misplacement of products in a display state according to the present disclosure.

Like reference numerals refer to like elements throughout this disclosure. The present disclosure does not describe all elements of the embodiments, and general content or overlapping content between the embodiments in the technical field to which the present disclosure pertains is omitted. The term 'unit, module, member, block' used in the specification may be implemented as software or hardware, and depending on the embodiment, a plurality of 'unit, module, member, block' may be implemented as a single component, or It is also possible for one 'part, module, member, or block' to include multiple components.

Throughout the specification, when a part is said to be “connected” to another part, this includes not only direct connection but also indirect connection, and indirect connection includes connection through a wireless communication network. do.

Additionally, when a part "includes" a certain component, this means that it may further include other components rather than excluding other components, unless specifically stated to the contrary.

Throughout the specification, when a member is said to be located “on” another member, this includes not only cases where a member is in contact with another member, but also cases where another member exists between the two members.

Terms such as first and second are used to distinguish one component from another component, and the components are not limited by the above-mentioned terms.

Singular expressions include plural expressions unless the context clearly makes an exception.

The identification code for each step is used for convenience of explanation. The identification code does not explain the order of each step, and each step may be performed differently from the specified order unless a specific order is clearly stated in the context. there is.

Hereinafter, the operating principle and embodiments of the present disclosure will be described with reference to the attached drawings.

In this specification, 'camera-based recognition devices placed on both sides of the front of each showcase floor according to the present disclosure' includes various devices that can perform computational processing and provide results to the user. For example, the camera-based recognition device disposed on both sides of the front of each showcase floor according to the present disclosure may include all of a computer, a server device, and a portable terminal, or may take the form of any one.

Here, the computer may include, for example, a laptop, desktop, laptop, tablet PC, slate PC, etc. equipped with a web browser.

The server device is a server that processes information by communicating with external devices, and may include an application server, computing server, database server, file server, game server, mail server, proxy server, and web server.

The portable terminal is, for example, a wireless communication device that guarantees portability and mobility, such as PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), and PDA. (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), WiBro (Wireless Broadband Internet) terminal, smart phone ), all types of handheld wireless communication devices, and wearable devices such as watches, rings, bracelets, anklets, necklaces, glasses, contact lenses, or head-mounted-device (HMD). may include.

1 is a diagram schematically showing an unmanned vending system according to the present disclosure.

Referring to FIG. 1, the unmanned vending system 10 according to an embodiment of the present invention includes an unmanned vending device 100 and a server for recognizing products displayed in the showcase based on cameras placed on both sides of the front of each showcase floor. It may include (200). Here, the unmanned vending system 10 shown in FIG. 1 is only an example and may include fewer or more components than the components shown in FIG. 1 .

The unmanned sales device 100 is a device for selling specific types of products without a seller. It is configured in the form of a display stand (i.e., a showcase) to display products to be sold, and detects the entry and exit of the displayed products and determines inventory. , which ultimately allows the buyer to make payment for the purchased product.

In one embodiment, the unmanned vending device 100 detects whether the products displayed inside enter or exit as the user (i.e., the buyer) who wants to purchase the product opens or closes the door, and sends the results to the server 200. Can be transmitted.

In other words, the unmanned vending device 100 may include various devices that can perform computational processing necessary to perform various functions for unmanned vending and provide results to the user. For example, the unmanned vending device 100 includes a communication device such as a communication modem for communicating with various devices or a wired or wireless network, a memory for storing various programs and data, and a microprocessor for calculating and controlling the program by executing it. It may include various computing devices including the like.

The unmanned vending device 100 may be configured in the form of a display stand that displays products therein, and the interior of the display stand may be configured in a structure in which a plurality of shelves are stacked. Additionally, each of the plurality of shelves is divided into a plurality of columns, and products can be placed in each of the plurality of columns. Products placed in the unmanned vending device 100 may include various types of products. In one embodiment of the present invention, the product placed in the unmanned vending device 100 may be a product that is light in weight compared to its volume (e.g., regular cigarettes, electronic cigarettes, snacks, etc.). However, this is only an example and is not necessarily limited to this.

In one embodiment, each shelf in the unmanned vending device 100 may be configured to have a predetermined inclination, and in this case, products placed on each column of each shelf are moved to the front of the shelf by the predetermined inclination. Allow it to be pushed and moved.

Additionally, in one embodiment, each shelf in the unmanned vending device 100 may further include a plurality of elastic members. That is, a plurality of elastic members may be installed corresponding to a plurality of columns separated on each shelf. For example, one elastic member may be installed in each column on the shelf, and in this case, the elastic member may be constructed using a spring or the like that is installed at the back of the shelf and has a force (i.e., elastic force) that moves toward the front of the shelf. there is. In other words, the elastic member can apply a force to push the remaining products placed on each column to the front of the shelf by elastic force whenever the frontmost product placed on each column falls off.

The unmanned vending device 100 may be equipped with a camera capable of photographing products placed on each column of each shelf. The camera may be installed on each shelf in the unmanned vending device 100, and may be installed in a position to view the products displayed on each shelf diagonally from the top. At this time, the height between shelves in the unmanned vending device 100 (i.e., the height of each layer within the shelf), the number of columns on the shelf, and the width of the product (depending on the type of product, for example, electronic cigarettes are higher than regular cigarettes). The optimal camera position and/or number of cameras can be determined by considering the camera's angle of view based on the product width (the width of the product may be large), etc. That is, the area that one camera can recognize (i.e., the number of columns) is limited depending on the height of each floor, the width of the product, etc., so in one embodiment, the camera is installed on both sides of the front for each shelf in the unmanned vending device 100. can be installed in More specifically, the camera may be installed in a position looking diagonally downward from the front upper left and upper right for each shelf within the unmanned vending device 100. For example, cameras (i.e., a left camera and a right camera) may be installed on the left and right sides of each shelf. However, this is only an example and is not necessarily limited to this.

The unmanned vending device 100 can set and store information (eg, product name, product price, product size, etc.) about products to be displayed (placed) in each column for each shelf. As an example, the unmanned vending device 100 may receive information about products to be placed in each column for each shelf from the server 200, and information about products to be placed in each column at the request of the server 200. You can also change .

The server 200 may include a computer system and computer software (web server program) that derives and provides task results corresponding to task performance requests from clients or other web servers. In addition to the web server program described above, the server 200 may include a series of application programs running on a web server or various databases built inside the device. For example, the server 200 may be in the form of a camera-based recognition device placed on both sides of the front of each floor of the showcase according to the present disclosure.

The server 200 may link with at least one unmanned vending device 100 for each store via a network. As an example, the server 200 may provide information about products sold in the unmanned vending device 100 (e.g., total number of products, price of each product, size of each product, display position of each product, identification information, product top/bottom/left/right images, etc.) can be managed. The server 200 may transmit information about products to be placed in each column (eg, product name, product price, product size, etc.) to the unmanned vending device 100 to be set in the unmanned vending device 100.

The server 200 can receive a plurality of shooting data from a plurality of cameras installed on each floor of the showcase from the unmanned vending device 100 to identify and manage the display status of the products displayed on each floor of the showcase.

The server 200 receives information about the purchased product (e.g., product name, number of purchased products, etc.) that the buyer has taken out by opening and closing the door of the unmanned vending device 100, and receives this from the unmanned vending device 100. Based on this, the payment amount for the purchaser's purchased product can be calculated and transmitted to the payment device. Afterwards, the buyer can pay for the purchased product through the payment device.

Meanwhile, the network can transmit and receive various information between the server 200 and at least one unmanned vending device 100. Various types of communication networks may be used, for example, wireless communication methods such as WLAN (Wireless LAN), Wi-Fi, Wibro, Wimax, and HSDPA (High Speed Downlink Packet Access). Alternatively, wired communication methods such as Ethernet, xDSL (ADSL, VDSL), HFC (Hybrid Fiber Coax), FTTC (Fiber to The Curb), and FTTH (Fiber To The Home) may be used.

The network is not limited to the communication methods presented above, and may include all other types of communication methods that are well known or will be developed in the future in addition to the communication methods described above.

Figure 2 is a diagram schematically showing the configuration included in the unmanned vending device and server according to the present disclosure.

The unmanned vending device 100 is a first device capable of communicating between the server 200 and a product storage unit 140 including a load cell 110 capable of storing products, a door unit 120, and a camera 130. It may be provided to include a communication unit 150, etc.

The server 200 includes a memory 210, a second communication unit 230 capable of communicating with the unmanned vending device 100, and at least one processor 220 capable of communicating with the memory 210. It can be arranged to include

Meanwhile, the load cell 110 may refer to a storage space in which products can be stored within the unmanned vending device 100. Since the load cell 110 is inclined by the inclination angle θ1, when the user receives the product, the product can move toward the door unit 120 along the inclined plane.

Meanwhile, the door unit 120 may be a door of the product storage unit 140 mounted on the unmanned vending device 100. As described later, when performing absolute calculation in ABS mode when calculating the number of products sold, the calculation may be performed based on when the door of the door unit 120 is opened and when the door is closed.

Meanwhile, the camera 130 is installed on each showcase floor of the unmanned sales device 100 and can capture images of displayed products. The photographing data of the product captured by the camera 150 is transmitted to the server 200, and the processor 220 can monitor the displayed product based on the received photographing data.

In one embodiment, the camera according to the present disclosure may refer to an artificial intelligence-based machine vision camera. A machine vision camera may consist of a lens, an image sensor, a main board, and an interface board, but is not limited to this. Additionally, images created through lenses and image sensors can be corrected into an appropriate form on the motherboard as needed. The image processed on the main board in this way can be transmitted to the server 200. Machine vision cameras may include GigE Vision cameras (Gigabit Ethernet Vision Cameras), USB3.0 cameras, CameraLink cameras, and CoaXPress cameras.

The memory 210 can store data supporting various functions of the device and a program for the operation of the processor 220, and input/output data (e.g., music files, still images, videos, etc.) can be stored, and a number of application programs (application programs or applications) running on the device, data for operation of the device, and commands can be stored. At least some of these applications may be downloaded from an external server via wireless communication.

The memory 210 may store at least one process for recognizing products displayed in the showcase based on a plurality of cameras.

The memory 210 includes a flash memory type, a hard disk type, a solid state disk type, an SDD type (Silicon Disk Drive type), and a multimedia card micro type. micro type), card type memory (e.g. SD or XD memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), EEPROM (electrically erasable) It may include at least one type of storage medium among programmable read-only memory (PROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, and optical disk. Additionally, the memory 210 is separate from the device, but may be a database connected by wire or wirelessly.

At least one processor 220 has a memory that stores data for an algorithm for controlling the operation of components in the device or a program that reproduces the algorithm, and performs the above-described operations using the data stored in the memory 210. can do. At this time, the memory 210 and the processor 220 may each be implemented as separate chips. Alternatively, the memory 210 and processor 220 may be implemented as a single chip.

In addition, the processor 220 may control any one or a combination of the above-described components in order to implement various embodiments according to the present disclosure described in FIGS. 4 to 7 below on the present device.

Among the components, the first communication unit 150 of the unmanned vending device 100 and the second communication unit 230 of the server 200 may include one or more components that enable communication with an external device, e.g. For example, it may be at least one of a broadcast reception module, a wired communication module, a wireless communication module, a short-range communication module, and a location information module.

Wired communication modules include various wired communication modules such as Local Area Network (LAN) modules, Wide Area Network (WAN) modules, or Value Added Network (VAN) modules, as well as USB (Universal Serial Bus) modules. ), HDMI (High Definition Multimedia Interface), DVI (Digital Visual Interface), RS-232 (recommended standard 232), power line communication, or POTS (plain old telephone service).

In addition to Wi-Fi modules and WiBro (Wireless broadband) modules, wireless communication modules include GSM (global System for Mobile Communication), CDMA (Code Division Multiple Access), WCDMA (Wideband Code Division Multiple Access), and UMTS (universal mobile telecommunications system). ), TDMA (Time Division Multiple Access), LTE (Long Term Evolution), 4G, 5G, 6G, etc. may include a wireless communication module that supports various wireless communication methods.

The wireless communication module may include a wireless communication interface including an antenna and a transmitter that transmits a mobile communication signal. Additionally, the wireless communication module may further include a mobile communication signal conversion module that modulates a digital control signal output from the control unit through a wireless communication interface into an analog wireless signal under the control of the control unit.

The wireless communication module may include a wireless communication interface including an antenna and a receiver for receiving mobile communication signals. Additionally, the wireless communication module may further include a mobile communication signal conversion module for demodulating an analog wireless signal received through a wireless communication interface into a digital control signal.

The short-range communication module is for short-range communication and includes Bluetooth™, RFID (Radio Frequency Identification), Infrared Data Association (IrDA), UWB (Ultra Wideband), ZigBee, and NFC (Near Field). Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and Wireless USB (Wireless Universal Serial Bus) technology can be used to support short-distance communication.

Figure 3 is a view looking down from above of products displayed on shelves divided into a plurality of columns according to the present disclosure and a camera provided to photograph the products.

Referring to Figure 3, each shelf of the display stand (i.e., each floor of the showcase) may be divided into a plurality of columns (e.g., column 1, column 2, column 3, ...., column n), and each column (e.g. : Products can be placed on column 1, column 2, column 3, …., column n). At this time, different types of products may be placed in each column, and the same products may be placed in the same column (i.e., one column). For example, A products may be placed in column 1, B products may be placed in column 2, C products may be placed in column 3, and D products may be placed in column 4. In addition, the products placed on each column can be arranged and arranged from the front of the shelf. For example, a buyer can open the door, select the product placed at the front of each column on the shelf within the display shelf, and take it out of the display shelf. You can.

In one embodiment, as shown in FIG. 3, the camera 130 may be installed inside the display stand to view the products displayed on the shelves of the display stand diagonally from the top, and may be installed on each shelf within the display stand. . At this time, the height between shelves within the display stand (i.e., the height of each layer within the display stand), the number of columns on the shelf, and the width of the product (depending on the type of product. For example, electronic cigarettes may have a larger product width than regular cigarettes. Based on at least one of the following, the optimal camera position and/or number of cameras may be determined by considering the angle of view of the camera 130. As an example, as shown in Figure 3, considering the area that one camera can recognize (i.e., the number of columns) depending on the height of each layer, the width of the product, etc., in the upper left and upper right of the shelf Two cameras (i.e., a left camera and a right camera) 130 may be installed in a position looking down diagonally. In addition, cameras (i.e., left camera and right camera) 130 may be installed at the upper left and upper right sides of the front end of the shelf, which are positions that enable recognition of products placed in the front area of the shelf.

Figure 4 is a flowchart of a camera-based recognition method placed on both sides of the front of each showcase floor according to the present disclosure. The method of FIG. 4 may be performed by the server 200 disclosed in FIG. 2. Additionally, as described above, the server 200 disclosed in FIG. 2 may be in the form of a camera-based recognition device disposed on both sides of the front of each floor of the showcase according to the present disclosure. In addition, hereinafter, the method of FIG. 4 will be described as being performed by the server 200, but in some cases, the method of FIG. 4 may be performed by the unmanned vending device 100 disclosed in FIG. 2.

Referring to FIG. 4, the processor 220 of the server 200 may acquire a plurality of captured data captured by a plurality of cameras disposed on both sides of the front of each showcase floor (S110).

As described above, the unmanned vending device 100 may place one camera on the front left side of each floor of the showcase (i.e., each shelf of the display case) and another camera on the front right side. Additionally, as described above, the unmanned vending machine includes n columns on each floor of the showcase (where n is a natural number), and each product can be displayed in each column.

As shown in Figure 5, a plurality of cameras arranged facing inward at both ends of each floor can each capture images of displayed products. The plurality of captured data generated in this way can be received by the server 200 through the second communication unit 230.

In more detail, the first captured data captured by the first camera 130 installed on the left among the plurality of cameras may include n-p columns in sequential order among the n columns. Here, p may be a natural number smaller than n.

Additionally, the second captured data captured by the second camera 130 installed on the right side among the plurality of cameras may include n-q columns in reverse order among the n columns. Here, q may be a natural number smaller than n.

Referring to FIG. 5, when n is 10, p is 4, and q is 4, the first captured data captured by the first camera 130 disposed to the left of column 1 is only from column 1 to column 6. The second captured data captured by the second camera 130 disposed to the right of column 10 may include only columns 10 to 5. That is, the first camera 130 can only photograph products 1 to 6, and the second camera 130 can only photograph products 10 to 5. In the above, p and q are described as being the same value, but the present invention is not limited thereto and p and q may be different values.

At this time, each of the first camera 130 and the second camera 130 may be arranged symmetrically on both sides such that m columns overlap between the n-p columns and the n-q columns. This is because the cameras are placed on both sides of the front for each floor, and the central area (i.e., if there are a total of 10 columns, the 5th and 6th columns, which are the central area) may have a small image and lower the recognition rate, so the middle numbered columns are duplicated. The purpose is to place cameras as much as possible and utilize shooting data from overlapping areas to increase the recognition rate of the central area.

The number (m) of columns to be photographed repeatedly may be set in advance by the server 200. The unmanned vending device 100 can generate shooting data by changing the arrangement of the camera according to this setting information.

Referring to FIG. 6, when n is 10, p is 4, q is 4, and m is set to 2, the corresponding portions of columns 5 and 6 of the first captured data and the column of the second captured data The corresponding parts of columns 5 and 6 may overlap.

Referring to FIG. 4, the processor 220 of the server 200 can recognize the type of product placed in the central area of the corresponding floor using a plurality of captured data for each floor (S120).

More specifically, the processor 220 of the server 200 can extract duplicate columns between a plurality of shooting data and recognize the type of product placed in the extracted column using a product-related database. .

As described above, since the number of duplicated columns is preset by the server 100, the server 100 stores this setting information. And, when a plurality of captured data is acquired, duplicate columns can be extracted based on the stored setting information.

The processor 220 of the server 200 can identify the type of the product by finding the product image included in the duplicate column in the product-related database. Here, the product-related database may be included in the server 200, or may be separate from the server 200, but may be connected by wire or wirelessly. Additionally, the product-related database may include information on all products displayed on the unmanned vending device 100.

According to the embodiment, the processor 220 of the server 200 combines the first photographed data and the second photographed data for products placed in the overlapping column between the first photographed data and the second photographed data. Data can be generated, products matching the combination data can be searched in a product-related database, and the type of the searched product can be determined as the type of product placed in the overlapping column. At this time, when there are multiple overlapping columns, combination data can be generated between product images in the same column in each shooting data.

Referring to FIG. 6, when the duplicated columns are column 5 and column 6, the part corresponding to column 5 in the first shooting data and the part corresponding to column 5 in the second shooting data are combined to create combined data for product 5. , and combined data for product 6 can be generated by combining the part corresponding to column 6 in the second shooting data and the part corresponding to column 6 in the second shooting data. Then, using the combination data for Product 5, a matching product can be searched within the database, and the type of the searched product can be determined to be the type of Product 5. Additionally, the combination data for product 6 can be used to search for matching products in the database, and the type of the searched product can be determined to be the type of product 6.

Combination data can be created by overlapping two pieces of data with low recognition rates. Because of this, the recognition rate of objects in the combined data can be further increased.

In addition, the processor 220 of the server 200 searches for a product that matches the image of the object in the combination data in the product-related database, and determines that the type of the searched product is the type of the product in the extracted column. .

According to the embodiment, the processor 220 of the server 200 determines that, for products placed in overlapping columns between first and second photographed data, a matching rate with the first photographed data is set in the product-related database. Extract products with a preset first value or more, search for products whose matching rate with the second shooting data is more than a preset second value among the extracted products, and place the types of the searched products in overlapping columns. It can be determined by the type of product.

Referring to FIG. 6, when the duplicated columns are column 5 and column 6, the processor 220 of the server 200 selects column 5 in the first shooting data based on product information stored in the product-related database. Extract products whose matching rate with the image is more than a first value (e.g., more than 80 percent), and match the image corresponding to column 5 in the second shooting data based on the product information of the extracted products. Products whose ratio is greater than or equal to a second value (for example, 80 percent or more) may be searched, and the type of the searched product may be determined to be the type of product 5 displayed in column 5. In addition, based on product information stored in the product-related database, products whose matching rate with the image corresponding to column 6 is more than the first value (e.g., 80 percent or more) are extracted from the first shooting data, and extracted again. Based on the product information of these products, search for products whose matching rate with the image corresponding to column 6 in the second shooting data is more than a second value (for example, 80 percent or more), and the type of the searched product can be determined as the type of product 6 displayed in column 6.

According to the embodiment, product search using the second shooting data extracts only one product with the highest matching rate among the plurality of products searched using the first shooting data to determine the type of product displayed in the corresponding column. You can.

Meanwhile, the camera-based recognition method placed on both sides of the front of each showcase floor according to the present disclosure analyzes the plurality of shooting data based on a pre-built learning model to recognize misplaced products among the plurality of products placed on the corresponding floor. Additional steps may be included.

Depending on the embodiment, the learning model may be constructed by learning learning data for each misplacement type regarding the display state of the product. In other words, it is possible to determine whether any product is displayed crookedly among the products displayed on each floor from a plurality of photographic data received for each floor through a learning model that learns cases where products are displayed crookedly by type.

Here, the misplacement type for the display state may include a case where the product is rotated based on two axes of the x-axis, y-axis, and z-axis.

The first learning data may be data when the product is placed correctly, as shown in FIG. 7A.

As shown in FIG. 7B, the second learning data may include data when the product is rotated 180 degrees based on the xz plane and when the product is rotated 180 degrees based on the yz plane.

The third learning data may include data when the product is rotated 0 to 90 degrees based on the xz plane and when the product is rotated 90 to 180 degrees based on the xz plane, as shown in FIG. 7C.

The fourth learning data, as shown in FIG. 7D, may include data when the product is rotated 0 degrees to 90 degrees inward based on the yz plane and when the product is rotated 0 degrees to 90 degrees outward based on the yz plane. .

Depending on the embodiment, the learning model may be constructed by learning learning data about misplacement of products. Here, misplacement of the display position may indicate a case where the product is not placed in the column assigned to the product.

In other words, it is possible to determine whether any of the products displayed on each floor are in the wrong display position from the plurality of shooting data received for each floor through a learning model learned by matching each product with the location where the product should be placed.

FIG. 4 depicts steps S110 and S120 as being sequentially executed, but this is merely an illustrative explanation of the technical idea of this embodiment, and those skilled in the art will understand the steps of this embodiment. Since it is possible to apply various modifications and modifications by changing the order shown in FIG. 4 or executing steps S110 to S120 in parallel as long as it does not deviate from the essential characteristics, FIG. 4 is not limited to a time-series order. no.

Meanwhile, in the above description, steps S110 and S120 may be further divided into additional steps or combined into fewer steps, depending on the implementation of the present disclosure. Additionally, some steps may be omitted or the order between steps may be changed as needed.

Meanwhile, the disclosed embodiments may be implemented in the form of a recording medium that stores instructions executable by a computer. Instructions may be stored in the form of program code, and when executed by a processor, may create program modules to perform operations of the disclosed embodiments. The recording medium may be implemented as a computer-readable recording medium.

Computer-readable recording media include all types of recording media storing instructions that can be decoded by a computer. For example, there may be Read Only Memory (ROM), Random Access Memory (RAM), magnetic tape, magnetic disk, flash memory, optical data storage device, etc.

As described above, the disclosed embodiments have been described with reference to the attached drawings. A person skilled in the art to which this disclosure pertains will understand that the present disclosure may be practiced in forms different from the disclosed embodiments without changing the technical idea or essential features of the present disclosure. The disclosed embodiments are illustrative and should not be construed as limiting.

10: Unmanned vending system
100: Unmanned vending device
110: load cell
120: Door part
130: camera
140: Product storage department
150: 1st Communications Department
200: server
210: memory
220: processor
230: Second Communications Department

Claims (10)

In a method performed by a device,
Obtaining a plurality of captured data captured by a plurality of cameras disposed on both sides of the front of each showcase floor; and
Recognizing the type of product placed in the central area of the corresponding floor using the plurality of photographed data,
The recognition step is,
Extracting duplicate columns between the plurality of shooting data and recognizing the type of product placed in the extracted column using a product-related database,
Camera-based recognition method placed on both sides of the front of each showcase floor. According to claim 1,
Each layer of the showcase includes n columns (where n is a natural number),
The first captured data captured by the first camera among the plurality of cameras includes np columns in sequential order among the n columns (where p is a natural number smaller than n), and the first captured data captured by the first camera among the plurality of cameras includes np columns in sequential order among the n columns (where p is a natural number smaller than n). The second captured data captured by the camera includes nq columns in reverse order among the n columns (where q is a natural number smaller than n),
Camera-based recognition method placed on both sides of the front of each showcase floor. According to clause 2,
Each of the first camera and the second camera,
arranged symmetrically on both sides so that m columns overlap between the np number of columns and the nq number of columns,
Camera-based recognition method placed on both sides of the front of each showcase floor. According to clause 2,
The recognition step is,
For products placed in the overlapping column between the first and second photographed data, combination data is generated by combining the first and second photographed data, and matched with the combination data in the database. Searching for a product and determining the type of the searched product as the type of product placed in the duplicate column,
Camera-based recognition method placed on both sides of the front of each showcase floor. According to clause 2,
The recognition step is,
For products placed in the overlapping column between the first photographed data and the second photographed data, products whose matching rate with the first photographed data is equal to or higher than a preset first value are extracted from the database, and the extracted products Searching for products whose matching rate with the second shooting data is more than a preset second value, and determining the type of the searched product as the type of product placed in the overlapping column,
Camera-based recognition method placed on both sides of the front of each showcase floor. According to claim 1,
Analyzing the plurality of photographed data based on a pre-built learning model to recognize misplaced products among the plurality of products placed on the corresponding floor; further comprising:
Camera-based recognition method placed on both sides of the front of each showcase floor. According to clause 6,
The learning model is,
It is constructed by learning learning data for each type of misplacement about the display status of the product,
The misplacement type for the display state includes a case where the product is rotated about two axes of the x-axis, y-axis, and z-axis.
Camera-based recognition method placed on both sides of the front of each showcase floor. According to clause 6,
The learning model is,
It is constructed by learning misplacement learning data about product display positions,
Misplacement of the display position refers to a case where the product is not placed in the column assigned to the product.
Camera-based recognition method placed on both sides of the front of each showcase floor. A program coupled to a computer and stored in a computer-readable recording medium to execute the method of any one of claims 1 to 8. A communication unit that communicates with a plurality of cameras arranged on each showcase floor;
a memory storing at least one process for recognizing products displayed in the showcase based on the plurality of cameras; and
Including a processor operating according to the process,
The processor, based on the process,
Obtaining a plurality of captured data captured by the plurality of cameras disposed on both sides of the front of each showcase floor through the communication unit,
Recognize the type of product placed in the central area of the floor using the plurality of shooting data,
Upon recognition, the processor
Extracting duplicate columns between the plurality of shooting data and recognizing the type of product placed in the extracted column using a product-related database,
Camera-based recognition devices placed on both sides of the front of each showcase floor.