KR102233126B1 - System and method for verifying barcode scanning - Google Patents

Abstract

According to one embodiment of the present invention, provided is a method of verifying whether a barcode is scanned, which comprises the steps of: a) acquiring, through an image acquisition module, an image of a product payment area including a barcode scanning area for product recognition and a product holding area in which barcode-scanned products are mounted; b) extracting a product object image from the barcode scanning area from the image through an image-based product recognition module, and recognizing a product under barcode scanning based on the product object image; c) calculating, through a total volume change amount calculation module, the total volume change amount of the product for which barcode scanning in the product holding area is completed based on the image; and d) through a barcode scan error determination module, determining the abnormal barcode scan or barcode scan omission of the product based on the total volume change of the product, the product recognized from the product object image, and the product recognized by the barcode scan.

Description

Barcode scan verification method and system {SYSTEM AND METHOD FOR VERIFYING BARCODE SCANNING}

The present invention relates to a method and system for verifying whether a barcode is scanned.

Most of the calculations of products purchased by customers at stores such as supermarkets and discount stores are automatically calculated by reading barcodes attached to the products, and a POS system is used as a system connected to a scanner and a computer calculator for calculating such purchased products. The POS system is linked to a computer, and a very useful system is widely used by calculating the price of a product purchased by a customer very quickly and accurately. In this POS system, the price of each product is settled by reading information from the barcode attached to the product through an input device called a scanner. Therefore, products that have not been read by the scanner are excluded from the calculation, which is an important factor in the sales loss of the store.

Recently, each store has an unmanned checkout counter (SCO, Self-Check Out) installed, and in the case of such an unmanned checkout counter, customers perform barcode scans themselves. In this process, there may be an omission of barcode scanning or the scanning of barcodes of other products.

Each store usually has a manager to monitor these activities, but one manager monitors several unmanned checkout counters, so it is practically difficult to monitor all of the above activities. The burden can be increased.

Accordingly, the necessity of developing a technology capable of monitoring and preventing the act of scanning barcodes of other products or missing barcode scans in a payment manager through an unmanned checkout counter (SCO, Self-Check Out) is gradually increasing.

An embodiment of the present invention is to determine the abnormal barcode scan or missing barcode scan of the product based on the total volume change amount of the product calculated using the depth image, the product recognized using the color image, and the product recognized by the barcode scan. It is an object of the present invention to provide a method and system for verifying whether a barcode can be scanned.

On the other hand, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned are clearly to those of ordinary skill in the technical field to which the present invention belongs from the following description. It will be understandable.

A method of verifying whether a barcode is scanned according to an embodiment of the present invention is: a) an image of a product payment region including a barcode scanning region for product recognition and a product mounting region in which the barcode-scanned product is mounted through an image acquisition module. Obtaining a; b) extracting a product object image from the barcode scanning area from the image through an image-based product recognition module, and recognizing a product being scanned barcode based on the product object image; c) calculating a total volume change amount of the product on which the barcode scan has been completed in the product storage area based on the image through the total volume change amount calculation module; And d) determining an abnormal barcode scan or missing barcode scan of the product based on the total volume change of the product, the product recognized from the product object image, and the product recognized by the barcode scan through a barcode scan error determination module. Includes.

The step d) includes: d-1) comparing the total volume change amount with a set reference volume change amount through a unit for determining whether a new product is placed, and determining whether a new product is placed in the product storage area; d-2) determining whether a barcode scan has been performed on the new product before the new product is installed, through a barcode scan determination unit; And d-3) when it is determined that the barcode scan has not been performed before the new product is placed in the product mounting area through the barcode scan error determination unit, determining that the barcode scan is omitted.

The step d) includes: d-1) comparing the total volume change amount with a set reference volume change amount through a unit for determining whether a new product is placed, and determining whether a new product is placed in the product storage area; d-2) determining whether a barcode scan has been performed on the new product before the new product is installed, through a barcode scan determination unit; And d-4) when a barcode scan is performed on the new product through the barcode scan error determination unit, but the product recognized based on the product object image and the product recognized by the barcode scan do not match, the abnormal barcode It may include the step of determining by scanning.

The step a) includes: a-1) obtaining a depth image of the product mounting area by a depth camera included in the image acquisition unit; a-2) obtaining a color image for the barcode scan area by a digital camera included in an image acquisition unit; a-3) extracting a motion region of an object from the color image through a motion region extraction unit; And a-4) determining whether the motion degree of the object in the motion region is less than a set motion reference value through a motion degree determination unit, wherein the step b) is performed in the step a-4). It may be performed when it is less than the motion reference value.

The step c) is: c-1) through a difference image acquisition unit, a depth difference image is obtained by comparing the depth image acquired in the product mounting area with a background depth image, and the color image acquired in the product mounting area Comparing a background color image to obtain a color difference image; c-2) obtaining a depth binarized image by binarizing the depth difference image through a binarized image acquisition unit, and binarizing the color difference image to obtain a color binarized image; c-3) extracting a first image region having a first pixel value from the depth binarized image through a first image region extraction unit; c-4) extracting a second image region having a second pixel value from the color binarized image through a second image region extraction unit; c-5) calculating a first partial volume change amount of the first image region based on the width of the first image region and a depth image corresponding to the first image region through a first partial volume change amount calculation unit ; c-6) extracting a third image region that does not overlap with the first image region from among the second image regions through a third image region extraction unit; c-7) calculating a second partial volume change amount by integrating a reference constant with respect to the area of the third image area through a second partial volume change amount calculating unit; And c-8) calculating a total volume change amount of the product based on the first partial volume change amount and the second partial volume change amount through the total volume change amount calculation unit.

e) updating a color image and a depth image obtained after the product is placed in the product mounting area as the background color image and the background depth image when it is determined that normal barcode scanning has been performed through the background image update module It may further include.

f) If it is determined that the normal barcode scan has been performed through the purchase list update module, adding the product identified by the normal barcode scan to the purchase list.

A recording medium according to another embodiment of the present invention is a computer-readable recording medium in which a program for executing a method of verifying whether to scan a barcode is recorded.

A system for verifying whether to scan a barcode according to another embodiment of the present invention includes an image acquisition module for acquiring an image of a product payment area including a barcode scan area for product recognition and a product placement area in which a product on which the barcode scan has been completed is mounted; An image-based product recognition module for extracting a product object image from the barcode scanning area from the image and recognizing a product being scanned with a barcode based on the product object image; A total volume change calculation module for calculating a total volume change amount of the product on which the barcode scan has been completed in the product storage area based on the image; And determining an abnormal barcode scan or missing barcode scan of a product based on the total volume change of the product, the product recognized from the product object image, and the product recognized by the barcode scan. do.

The barcode scan error determination module includes: a new product placement determination unit that compares the total volume change amount with a set reference volume change amount to determine whether a new product is placed in the product placement area; A bar code scan determination unit for determining whether a bar code scan has been performed on the new product before the new product is placed; And a barcode scan error determination unit that determines that the barcode scan is missing when it is determined that the barcode scan has not been performed before the new product is placed in the product mounting area.

The barcode scan error determination module includes: a new product placement determination unit that compares the total volume change amount with a set reference volume change amount to determine whether a new product is placed in the product placement area; A bar code scan determination unit for determining whether a bar code scan has been performed on the new product before the new product is placed; And a barcode scan error determination unit that determines as the abnormal barcode scan when the barcode scan for the new product is performed but the product recognized based on the product object image and the product recognized by the barcode scan do not match. I can.

The image acquisition module includes: an image acquisition unit that acquires a depth image of the product mounting area by a depth camera and acquires a color image of the barcode scan area by a digital camera; A motion region extraction unit for extracting a motion region of an object from the color image; And a motion degree determination unit that determines whether a motion degree of the object in the motion region is less than a set motion reference value, and generates a motion signal of the image-based product recognition module when the motion degree is less than the motion reference value.

The total volume change calculation module: Comparing the depth image and background depth image acquired in the product mounting area to obtain a depth difference image, and comparing the color image and background color image acquired in the product mounting area to obtain a color difference image A difference image acquiring unit for acquiring; A binarized image acquisition unit for binarizing the depth difference image to obtain a depth binarized image, and binarizing the color difference image to obtain a color binarized image; A first image region extraction unit extracting a first image region having a first pixel value from the depth binarized image; A second image region extraction unit extracting a second image region having a second pixel value from the color binarized image; A first partial volume change amount calculation unit for calculating a first partial volume change amount of the first image region based on the area of the first image region and a depth image corresponding to the first image region; A third image region extraction unit for extracting a third image region not overlapping with the first image region from among the second image regions; A second partial volume change amount calculating unit for calculating a second partial volume change amount by integrating a reference constant with respect to the area of the third image area; And a total volume change amount calculating unit calculating a total volume change amount of the product based on the first partial volume change amount and the second partial volume change amount.

When it is determined that the normal barcode scan has been performed, a background image update module for updating a color image and a depth image obtained after the product is mounted in the product mounting area to the background color image and the background depth image may be further included. have.

When it is determined that the normal barcode scan has been performed, a purchase list update module for updating the purchase list by adding a product identified by the normal barcode scanning to a purchase list may be further included.

The method and system for verifying whether a barcode is scanned according to an embodiment of the present invention is based on the product's total volume change calculated using the depth image, the product recognized using the color image, and the product recognized by the barcode scan. It is possible to determine a barcode scan or a barcode scan omission.

On the other hand, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description. I will be able to.

1 is a block diagram schematically showing the configuration of a system for verifying whether a barcode is scanned according to an embodiment of the present invention.
2 is a flowchart of a method for verifying whether a barcode is scanned according to an embodiment of the present invention.
3 is a block diagram schematically showing the configuration of an image acquisition module.
4 is a flowchart showing step S100 in more detail.
5 is a block diagram schematically showing the configuration of a total volume change calculation module.
6 is a flowchart showing step S300 in more detail.
7 is a diagram schematically showing the configuration of a barcode scan error determination module.
8A is a flowchart of step S400 according to the first embodiment.
8B is a flowchart of step S400 according to the second embodiment.

Other advantages and features of the present invention, and a method of achieving them will become apparent with reference to embodiments to be described later in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. It is provided to completely inform the scope of the invention to the possessor, and the invention is only defined by the scope of the claims.

Even if not defined, all terms (including technical or scientific terms) used herein have the same meaning as commonly accepted by universal technology in the prior art to which this invention belongs. Terms defined by general dictionaries may be construed as having the same meaning as the related description and/or the text of this application, and not conceptualized or excessively formalized, even if not clearly defined herein. Won't.

The terms used in this specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification,'include' and/or various conjugated forms of this verb, for example,'including','including','including','including', and the like refer to the mentioned composition, ingredient, component, Steps, operations and/or elements do not exclude the presence or addition of one or more other compositions, components, components, steps, operations and/or elements. In the present specification, the term'and/or' refers to each of the listed components or various combinations thereof.

Meanwhile, terms such as'~ unit','~ group','~ block', and'~ module' used throughout this specification may mean a unit that processes at least one function or operation. For example, it can mean software, hardware components such as FPGAs or ASICs. However,'~bu','~gi','~block', and'~module' are not meant to be limited to software or hardware. The'~ unit','~ group','~ block', and'~ module' may be configured to be in an addressable storage medium, or may be configured to reproduce one or more processors.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings of the present specification.

1 is a block diagram schematically showing a configuration of a system 10 for verifying whether a barcode is scanned according to an embodiment of the present invention.

Referring to FIG. 1, a barcode scan verification system 10 includes an image acquisition module 100, an image-based product recognition module 200, a total volume change calculation module 300, a barcode scan error determination module 400, and a background An image update module 500 and a purchase list update module 600 may be included.

The image acquisition module 100 may acquire an image of a product payment area including a barcode scan area for product recognition and a product placement area in which a product on which the barcode scan has been completed is mounted. The image-based product recognition module 200 extracts a product object image from the barcode scanning area from the image, and recognizes a product being scanned for a barcode based on the product object image. Based on the image, it is possible to calculate the total volume change amount of the product on which the barcode scan has been completed in the product mounting area. The barcode scan error determination module 400 may determine an abnormal barcode scan or an omission of barcode scan of a product based on the total volume change amount of the product, the product recognized from the product object image, and the product recognized by the barcode scan. When it is determined that the normal barcode scan has been performed, the background image update module 500 may update the color image and the depth image acquired after the product is placed in the product mounting area as the background color image and the background depth image. have. When it is determined that the normal barcode scan has been performed, the purchase list update module 600 may update the purchase list by adding a product identified by the normal barcode scanning to a purchase list.

2 is a flowchart of a method S10 for verifying whether a barcode is scanned according to an embodiment of the present invention.

Referring to FIG. 2, a method for verifying whether a barcode is scanned (S10) is performed in a product payment area including a barcode scanning area for product recognition and a product mounting area in which a product having completed barcode scanning is mounted, through the image acquisition module 100. Obtaining an image for (S100), extracting a product object image from the barcode scanning area from the image through the image-based product recognition module 200, and recognizing the product being scanned barcode based on the product object image. Step (S200), through the total volume change calculation module 300, calculating the total volume change amount of the product for which the barcode scan has been completed in the product storage area based on the image (S300), a barcode scan error determination module ( 400), determining an abnormal barcode scan or missing barcode scan of a product based on the total volume change amount of the product, the product recognized from the product object image, and the product recognized by the barcode scan (S400), a background image When it is determined that the normal barcode scan has been performed through the update module 500, updating the color image and the depth image obtained after the product is placed in the product mounting area to the background color image and the background depth image ( S500) and the purchase list update module 600, when it is determined that the normal barcode scan has been performed, adding a product identified by the normal barcode scan to the purchase list (S600).

Hereinafter, a method (S10) for verifying whether to scan a barcode through the system 10 for verifying whether to scan a barcode will be described in more detail.

3 is a block diagram schematically showing the configuration of the image acquisition module 100.

Referring to FIG. 3, the image acquisition module 100 may include an image acquisition unit 110, a motion region extraction unit 120, and a motion degree determination unit 130.

With the introduction of an unmanned checkout counter (SCO, Self-Check Out), a buyer who wants to purchase a product selects the product to purchase from the counter, then brings it to an unmanned cash register and scans the barcode directly.

The unmanned checkout counter is provided with a product payment area including an area for placing products before performing barcode scanning, a barcode scanning area for performing product recognition through a barcode scanner, and a product placing area for placing products on which barcode scanning has been completed.

An image acquisition unit 110 capable of acquiring an image of a product payment area and an image of a product existing in the product payment area is provided in an area where the product payment area can be viewed.

The image acquisition unit 110 may acquire a depth image of the product mounting area by a depth camera, and acquire a color image of the barcode scan area by a digital camera. Of course, in addition to this, the depth camera and digital camera included in the image acquisition unit 110 may be provided to acquire a depth image and a color image over the entire area of the product payment area.

The camera can be provided as a depth camera and a digital camera. The depth camera includes a photodiode (PD) type camera, a vidicon type camera, a triangulation type 3D laser scanner, a depth camera using a structured ray pattern, and a TOF using the difference in reflection time of infrared rays. A depth camera of the (Time-Of-flight) method can be applied. However, the present invention is not limited thereto, and any depth camera capable of obtaining 3D distance information of a product may be applied regardless of the type. Digital cameras can also be applied irrespective of the type as long as they can acquire color images.

The motion region extraction unit 120 may extract a motion region of an object from the color image. When a color image for the barcode scan area is acquired by the digital camera, the motion area extraction unit 120 may extract a motion area, which is an area in which motion of an object exists, from the color image for the barcode scan area. Here, the object may be a product on which barcode scanning is in progress, and in this case, only an image in which the motion of the product object exists may be extracted through the motion region extraction unit 120.

The motion degree determination unit 130 determines whether the motion degree of the object in the motion area is less than a set motion reference value, and generates an operation signal of the image-based product recognition module 200 when the motion degree is less than the motion reference value. I can.

Since the product recognition rate decreases when a blur effect occurs in the image due to the movement of the product object, in order to extract the product object image from the image through the image-based product recognition module 200 and recognize the product based on the product object image, , An image higher than the quality required by the image-based product recognition module 200 is required. To this end, the motion degree determination unit 130 determines whether the motion degree of the product object in the motion region extracted through the motion region extraction unit 120 is less than a motion reference value. If the degree of movement of the product object is less than the motion reference value, it is determined that an image higher than the quality required by the image-based product recognition module 200 is obtained, and an operation signal for the operation of the image-based product recognition module 200 may be generated. have.

For example, considering a process in which a buyer takes a product to a barcode scanner and scans it, the degree of movement of the product object may be greater than or equal to the motion reference value while approaching the barcode scanner or moving away from the barcode scanner. However, while the barcode attached to the product is scanned with the barcode scanner, the movement of the product object is reduced, so that the degree of movement of the product object may be less than the motion reference value. Through this, the image-based product recognition module 200 may recognize the product being scanned barcode based on the image of the product object during the time during which the barcode scan is being performed by the product object through the barcode scanner.

4 is a flowchart showing step S100 in more detail.

Referring to FIG. 4, the step S100 is a step of acquiring a depth image for the product mounting area by a depth camera included in the image acquisition unit 110 (S110), and a digital camera included in the image acquisition unit 110 By, obtaining a color image for the barcode scan area (S120), extracting a motion area of an object from the color image through the motion area extraction unit 120 (S130), and a motion degree determination unit ( Through 130), a step S140 of determining whether the degree of motion of the object in the motion region is less than a set motion reference value may be included.

In step S200, through the image-based product recognition module 200, a product object image may be extracted from the barcode scanning area from the image, and a product being scanned barcode may be recognized based on the product object image.

As mentioned above, the image-based product recognition module 200 can operate only when an operation signal is generated from the motion degree determination unit 130.

In the database, image information of each product sold in the store is stored. The image-based product recognition module 200 may compare the image information of each product stored in the database with the product object image extracted from the image acquired in the barcode scanning area to recognize the product being scanned by the barcode.

In addition, the image-based product recognition module 200 performs machine learning using the image information of each product, not the image information of each product stored in the database, as learning data, and then in the barcode scan area based on the learning result. The product being scanned by barcode can be recognized from the product object image extracted from the acquired image. At this time, machine learning can also be performed in a deep learning method through deep neural networks.

5 is a block diagram schematically showing the configuration of the total volume change calculation module 300.

5, the total volume change calculation module 300 includes a difference image acquisition unit 310, a binarization image acquisition unit 320, a first image region extraction unit 330, and a second image region extraction unit 340. , A first partial volume change amount calculation unit 350, a third image region extraction unit 360, a second partial volume change amount calculation unit 370, and a total volume change amount calculation unit 380.

The difference image acquisition unit 310 obtains a depth difference image by comparing the depth image acquired in the product mounting area with a background depth image, and compares the color image acquired in the product mounting area with a background color image to obtain a color difference image. Can be obtained. The binarized image acquisition unit 320 may obtain a depth binarized image by binarizing the depth difference image, and binarize the color difference image to obtain a color binarized image. The first image region extraction unit 330 may extract a first image region having a first pixel value from the depth binarized image. The second image region extraction unit 340 may extract a second image region having a second pixel value from the color binarized image. The first partial volume change amount calculation unit 350 may calculate a first product volume change amount of the first image region based on the first image region and a depth image corresponding to the first image region. The third image region extraction unit 360 may extract a third image region that does not overlap with the first image region from among the second image regions. The second partial volume change amount calculation unit 370 may calculate a second product volume change amount by integrating a reference constant with respect to the third image region. The total volume change amount calculation unit 380 may calculate a total volume change amount of the product based on the first partial volume change amount and the second product volume change amount.

In more detail with respect to the difference image acquisition unit 310, the difference image acquisition unit 310 may acquire a depth difference image and a color difference image using a background depth image and a background color image stored in a database. That is, the depth difference image is obtained by excluding the background depth image for the product mounting area from the depth image in the product mounting area acquired by the depth camera, and the product is mounted in the color image in the product mounting area acquired by the digital camera. A color difference image may be obtained by excluding a background color image for an area.

As mentioned above, the binarized image acquisition unit 320 may obtain a depth binarized image by binarizing the depth difference image, and binarize the color difference image to obtain a color binarized image.

The first image region extraction unit 330 may extract a first image region having a first pixel value from the depth binarized image obtained through the binarized image acquisition unit 320, and the second image region extraction unit 340 May extract a second image region having a second pixel value from the color binarized image acquired through the binarized image acquisition unit 320.

For example, one part of an object mounted in the product mounting area is a part that can be detected in height through a depth camera (the first part), and the rest of the same object is a part in which the height cannot be detected through a depth camera (the second part). ) Can be.

In this case, the first image region extraction unit 330 extracts a first image region having a first pixel value from the depth binarized image obtained by binarizing the depth difference image. It may correspond to a portion (the first portion) capable of detecting a height through a depth camera among the mounted objects.

In addition, the second image region extraction unit 340 extracts a second image region having a second pixel value from the color binarized image obtained by binarizing the color difference image. It may correspond to the entire part of the mounted object.

The third image region extraction unit 360 extracts only a third image region that does not overlap with the first image region from among the second image regions. That is, the third image region may correspond to a portion (the second portion) in which height detection through a depth camera is impossible among objects mounted on the product mounting region.

The first partial volume change amount calculation unit 350 includes a first partial volume that is a volume change amount of a first part corresponding to the first image region through the height of the first image region obtained through the width and depth image of the first image region. The amount of change can be calculated.

The second partial volume change amount calculation unit 370 may calculate a second partial volume change amount, which is a volume change amount of the second part corresponding to the third image region, by integrating a reference constant with respect to the area of the third image region. In this case, the reference constant may be adjusted according to the user's setting. Since it is impossible to measure the height of the second part through the depth image, in the present invention, the volume of the second part is measured by assuming the reference constant as the height of the second part.

The total volume change amount calculation unit 380 calculates the total volume change amount of the product placed in the product storage area by summing the first partial volume change amount and the second partial volume change amount.

For example, it may be assumed that a customer places a new product in the product storage area while several products are not already installed in the product storage area. In this case, the new product may be a product including both a portion capable of detecting the height through a depth camera (a first portion) and a portion incapable of detecting the height through a depth camera (a second portion), as exemplified above.

In this case, the first partial volume change amount calculation unit 350 calculates a volume change amount, that is, a first partial volume change amount, due to a part (first part) capable of detecting height through a depth camera in the new product.

In addition, the second partial volume change amount calculation unit 370 calculates a volume change amount, that is, a second partial volume change amount, due to a part (second part) in which the height cannot be detected through the depth camera in the new product.

And the total volume change calculation unit 380 calculates the total volume change by summing the first partial volume change amount and the second partial volume change amount, through which the total volume change amount generated by additionally placing new products in the product storage area is calculated. Can be.

6 is a flowchart showing step S300 in more detail.

Referring to FIG. 6, in step S300, a depth difference image is obtained by comparing a depth image acquired in the product mounting area with a background depth image through the difference image acquisition unit 310, and a color image acquired in the product mounting area. Comparing the color difference image with the background color image (S310), the depth difference image is obtained by binarizing the depth difference image through the binarization image acquisition unit 320, and the color difference image is binarized to perform color binarization. Acquiring an image (S320), extracting a first image region having a first pixel value from the depth binarized image through a first image region extraction unit 330 (S330), a second image region extraction unit ( Extracting a second image region having a second pixel value from the color binarized image through 340 (S340), the first image region and the first image region through a first partial volume change calculation unit 350 Calculating a first partial volume change amount of the first image region based on the depth image corresponding to (S350), and the first image region among the second image regions through the third image region extraction unit 360 Extracting a non-overlapping third image region (S360), calculating a second partial volume change amount by integrating a reference constant for the third image region through the second partial volume change amount calculation unit 370 (S370) ) And calculating the total volume change amount of the product based on the first partial volume change amount and the second product volume change amount through the total volume change amount calculation unit 380 (S380).

7 is a diagram schematically showing the configuration of a barcode scan error determination module 400.

Referring to FIG. 7, the barcode scan error determination module 400 may include a new product placement determination unit 410, a barcode scan determination unit 420, and a barcode scan error determination unit 430.

The new product placement determination unit 410 may compare the total volume change amount with a set reference volume change amount to determine whether a new product is placed in the product placement area. The barcode scan determination unit 420 may determine whether a barcode scan for the new product has been performed before the new product is installed. When it is determined that the barcode scan has not been performed before the new product is placed in the product mounting area, the barcode scan error determination unit 430 may determine that the barcode scan is omitted. In addition, when the product recognized based on the product object image and the product recognized by the barcode scan do not match, the barcode scan error determination unit 430 may determine the abnormal barcode scan.

8A is a flowchart of step S400 according to the first embodiment.

Referring to FIG. 8A, according to the first embodiment of step S400, in step S400, the total volume change amount is compared with a set reference volume change amount through the unit 410 for determining whether to place a new product. Determining whether the new product has been scanned through the step (S410) of determining whether or not the new product has been subjected to a barcode scan through the bar code scan determination unit 420 (S420), and a barcode scan error determination unit ( When it is determined through 430 that the barcode scan has not been performed before the new product is mounted in the product mounting area, determining that the barcode scan is omitted (S430).

8B is a flowchart of step S400 according to the second embodiment.

Referring to FIG. 8B, according to the second embodiment of step S400, in step S400, the total volume change amount is compared with a set reference volume change amount through the new product placement determination unit 410, and the new product is placed in the product placement area. Determining whether the new product has been scanned through the step (S410) of determining whether or not the new product has been subjected to a barcode scan through the bar code scan determination unit 420 (S420), and a barcode scan error determination unit ( In step 430, when the barcode scan for the new product is performed, but the product recognized based on the product object image and the product recognized by the barcode scan do not match, determining as the abnormal barcode scan (S440) It may include.

The first embodiment of step S400 performed by the barcode scan error determination module 400 and the second embodiment of step S400 are summarized as follows.

The new product placement determination unit 410 compares the total volume change amount in the product placement area obtained through the total volume change amount calculation module 300 with a reference volume change amount to determine whether a new product is placed in the product placement area. More precisely, when the total volume change amount exceeds the reference volume change amount, the new product deferment determination unit 410 may determine that the new product is placed in the product deferment area. In this case, the reference volume change amount may be set to correspond to the minimum volume of the product sold in the store. However, the present invention is not limited thereto, and the amount of change in the reference volume may be adjusted according to the user's selection.

The bar code scan determination unit 420 determines whether the new product has been scanned by a barcode scanner before the new product is placed in the product mounting area. Specifically, when it is determined that a new product has been placed in the product placement area through the new product placement determination unit 410, it is detected whether a barcode scan has been performed by a bar code scanner at a time before it is determined that the new product has been placed. By doing so, it can be determined whether the barcode scan of the new product has been performed.

According to the first embodiment of the barcode scanning error determination unit 430, the barcode scanning error determination unit 430 determines that a new product has been installed through the new product placement determination unit 410, and the barcode scan whether or not a determination unit ( If it is determined through 420) that the barcode scan for the new product has not been performed, it is determined that the barcode scan for the new product is omitted.

For example, when a customer moves a product to a product storage area without performing a barcode scan, the barcode scan error determination unit 430 may determine that the customer misses the barcode scan.

According to the second embodiment of the barcode scanning error determination unit 430, the barcode scanning error determination unit 430 determines that a new product has been installed through the new product placement determination unit 410, and the barcode scan whether or not a determination unit ( If it is determined that the barcode scan for the new product has been performed through 420, but the product recognized based on the product object image and the product recognized by the barcode scan are inconsistent, the product to be viewed by the image acquisition unit 110 is It is determined that barcode scanning of other products has been performed, that is, abnormal barcode scanning has been performed.

For example, if a customer places a low-priced product under a high-priced product while allowing the high-priced product to be reflected on the camera and then only scans the barcode on the low-priced product, the barcode scan error determination unit 430 It can be determined that the customer has performed an abnormal barcode scan.

In the case of the above two embodiments of the barcode scanning error determination unit 430, the barcode scanning error determination unit 430 may stop an additional barcode scanning operation, and furthermore, a communication signal indicating that a barcode scan is missing or an abnormal barcode scan has been performed, The manager can be notified by means of light emission or an alarm.

According to another embodiment of the barcode scanning error determination unit 430, the barcode scanning error determination unit 430 is the volume of the product recognized based on the product object image or the volume and total volume change of the product recognized by barcode scanning. It may be determined whether there is another type of barcode scanning error by comparing the total volume change amount calculated through the calculation module 300.

More specifically, the volume of each product sold in the store may be stored in the database, and accordingly, the product recognized based on the product object image or the product recognized by barcode scanning can be matched with the product volume stored in the database. I can. When the volume of the product recognized based on the product object image or the product recognized by the barcode scan and the volume change amount calculated by the total volume change amount calculation module 300 are different from each other, the barcode scan error determination unit 430 It may be determined that the product recognized based on the image or another product hidden under the product recognized by barcode scanning and moved to the product storage area together.

For example, after a customer hides a high-priced product under a low-priced product, only the low-priced product is projected on the image acquisition unit 110 to scan the barcode on the low-priced product, and then the low-cost object and the high-priced object At the same time, it can be assumed that the product is mounted in the product storage area. In this case, the barcode scan error determination unit 430 is the total volume change calculated through the total volume change calculation module 300 compared to the volume of the product recognized based on the product object image or the product volume recognized by the barcode scan. It can be determined that this is larger, and through this, it can be determined that a barcode scan omission has occurred for an expensive product.

Referring back to FIGS. 1 and 2, the background image update module 500 performs a normal barcode scan, rather than an abnormal barcode scan, an omission of barcode scan, and another type of barcode scan error, when the barcode scan of a new product through a barcode scanner Determine if you have lost. If it is determined that a normal barcode scan has been performed, the color image acquired after the new product is placed in the product mounting area is updated as a background color image, and the depth image acquired after the new product is placed in the product mounting area is used as a background depth image. To update. Through this, it is possible to determine whether or not another new product has been placed in the product receiving area after the new product has been placed in the product receiving area.

When it is determined that the normal barcode scan has been performed, the purchase list update module 600 performs a task of adding the product identified by the normal barcode scan to the purchase list. Through this, all products scanned by the buyer can be registered in the purchase list.

The recording medium according to another embodiment of the present invention may be a computer-readable recording medium in which a program for executing the method S10 for verifying whether to scan a barcode is recorded.

The method (S10) and the system 10 for verifying whether a barcode is scanned according to an embodiment of the present invention include a total volume change of a product calculated using a depth image, a product recognized using a color image, and a product recognized by a barcode scan. Based on this, it is possible to determine an abnormal barcode scan of a product or an omission of a barcode scan.

In the above, the present invention has been described through the embodiments, but the above embodiments are merely for explaining the spirit of the present invention and are not limited thereto. One of ordinary skill in the art will understand that various modifications may be made to the above-described embodiments. The scope of the present invention is determined only through the interpretation of the appended claims.

10: Barcode scan verification system
100: image acquisition module 110: image acquisition unit
120: motion region extraction unit 130: motion degree determination unit
200: image-based product recognition module 300: total volume change calculation module
310: difference image acquisition unit 320: binarization image acquisition unit
330: first image region extraction unit 340: second image region extraction unit
350: first partial volume change amount calculation unit 360: third image region extraction unit
370: second partial volume change amount calculation unit 380: total volume change amount calculation unit
400: barcode scan error determination module
410: new product deferment determination unit
420: bar code scan determination unit
430: barcode scan error determination unit 500: background image update module
600: purchase list update module

Claims (15)

a) obtaining, through an image acquisition module, an image of a product payment area including a barcode scanning area for product recognition and a product placement area in which a product having completed barcode scanning is mounted;
b) extracting a product object image from the barcode scanning area from the image through an image-based product recognition module, and recognizing a product being scanned barcode based on the product object image;
c) calculating a total volume change amount of the product on which the barcode scan has been completed in the product storage area based on the image through the total volume change amount calculation module; And
d) Determining an abnormal barcode scan or missing barcode scan of a product based on the total volume change amount of the product, the product recognized from the product object image, and the product recognized by the barcode scan through the barcode scan error determination module. Including,
Step c) is:
c-1) Through the difference image acquisition unit, a depth difference image is obtained by comparing the depth image acquired in the product mounting area and the background depth image, and the color image acquired in the product mounting area and the background color image are compared. Obtaining a color difference image;
c-2) obtaining a depth binarized image by binarizing the depth difference image through a binarized image acquisition unit, and binarizing the color difference image to obtain a color binarized image;
c-3) extracting a first image region having a first pixel value from the depth binarized image through a first image region extraction unit;
c-4) extracting a second image region having a second pixel value from the color binarized image through a second image region extraction unit;
c-5) calculating a first partial volume change amount of the first image region based on the width of the first image region and a depth image corresponding to the first image region through a first partial volume change amount calculation unit ;
c-6) extracting a third image region that does not overlap with the first image region from among the second image regions through a third image region extraction unit;
c-7) calculating a second partial volume change amount by integrating a reference constant with respect to the area of the third image area through a second partial volume change amount calculating unit; And
c-8) A method of verifying whether to scan a barcode comprising the step of calculating a total volume change amount of the product based on the first partial volume change amount and the second partial volume change amount through a total volume change amount calculation unit. The method of claim 1,
Step d) is:
d-1) determining whether a new product has been placed in the product receiving area by comparing the total volume change amount with a set reference volume change amount through a new product placement determination unit;
d-2) determining whether a barcode scan has been performed on the new product before the new product is installed, through a barcode scan determination unit; And
d-3) When it is determined that the barcode scan has not been performed before the new product is placed in the product mounting area through the barcode scanning error determination unit, verification of whether the barcode is scanned, including determining that the barcode scan is missing Way. The method of claim 1,
Step d) is:
d-1) determining whether a new product has been placed in the product receiving area by comparing the total volume change amount with a set reference volume change amount through a new product placement determination unit;
d-2) determining whether a barcode scan has been performed on the new product before the new product is installed, through a barcode scan determination unit; And
d-4) When the barcode scan for the new product is performed through the barcode scan error determination unit, but the product recognized based on the product object image and the product recognized by the barcode scan do not match, scan the abnormal barcode A method of verifying whether to scan a barcode comprising the step of determining as. The method of claim 1,
The step a) is:
a-1) obtaining a depth image of the product mounting area by a depth camera included in the image acquisition unit;
a-2) obtaining a color image for the barcode scan area by a digital camera included in an image acquisition unit;
a-3) extracting a motion region of an object from the color image through a motion region extraction unit; And
a-4) determining, through a motion degree determination unit, whether the motion degree of the object in the motion region is less than a set motion reference value,
The step b) is a method of verifying whether to scan a barcode performed when the degree of motion is less than the motion reference value in step a-4). delete The method of claim 1,
e) updating a color image and a depth image obtained after the product is placed in the product mounting area as the background color image and the background depth image when it is determined that normal barcode scanning has been performed through the background image update module A method of verifying whether to scan a barcode further comprising a. The method of claim 6,
f) when it is determined that the normal barcode scan has been performed through a purchase list update module, adding a product identified by the normal barcode scan to a purchase list. A computer-readable recording medium in which a program for executing the method for verifying whether to scan a barcode according to any one of claims 1 to 4, 6, and 7 is recorded. An image acquisition module for acquiring an image of a product payment area including a barcode scan area for product recognition and a product placement area in which a product on which the barcode scan is completed is mounted;
An image-based product recognition module for extracting a product object image from the barcode scanning area from the image and recognizing a product being scanned with a barcode based on the product object image;
A total volume change calculation module for calculating a total volume change amount of the product on which the barcode scan has been completed in the product storage area based on the image; And
A barcode scan error determination module comprising the step of determining an abnormal barcode scan or missing barcode scan of a product based on the total volume change amount of the product, the product recognized from the product object image, and the product recognized by the barcode scan, ,
The total volume change calculation module is:
A difference image acquisition unit that compares the depth image acquired in the product mounting area with a background depth image to obtain a depth difference image, and compares the color image acquired in the product mounting area with a background color image to obtain a color difference image;
A binarized image acquisition unit for binarizing the depth difference image to obtain a depth binarized image, and binarizing the color difference image to obtain a color binarized image;
A first image region extraction unit extracting a first image region having a first pixel value from the depth binarized image;
A second image region extraction unit extracting a second image region having a second pixel value from the color binarized image;
A first partial volume change amount calculation unit for calculating a first partial volume change amount of the first image region based on the area of the first image region and a depth image corresponding to the first image region;
A third image region extraction unit for extracting a third image region not overlapping with the first image region from among the second image regions;
A second partial volume change amount calculating unit for calculating a second partial volume change amount by integrating a reference constant with respect to the area of the third image area; And
And a total volume change amount calculation unit that calculates a total volume change amount of the product based on the first partial volume change amount and the second partial volume change amount. The method of claim 9,
The barcode scan error determination module:
A new product placement determination unit that compares the total volume change amount with a set reference volume change amount to determine whether a new product is placed in the product placement area;
A bar code scan determination unit for determining whether a bar code scan has been performed on the new product before the new product is placed; And
When it is determined that the barcode scan has not been performed before the new product is placed in the product mounting area, a barcode scan verification system including a barcode scan error determination unit that determines that the barcode scan is missing. The method of claim 9,
The barcode scan error determination module:
A new product placement determination unit that compares the total volume change amount with a set reference volume change amount to determine whether a new product is placed in the product placement area;
A bar code scan determination unit for determining whether a bar code scan has been performed on the new product before the new product is placed; And
A barcode including a barcode scan error determination unit that determines as the abnormal barcode scan when a barcode scan is performed on the new product but the product recognized based on the product object image and the product recognized by the barcode scan do not match. Scanning Verification System. The method of claim 9,
The image acquisition module:
An image acquisition unit that acquires a depth image of the product mounting area by a depth camera, and acquires a color image of the barcode scan area by a digital camera;
A motion region extraction unit for extracting a motion region of an object from the color image; And
It is determined whether the motion degree of the object in the motion area is less than a set motion reference value, and if the motion degree is less than the motion reference value, verification of a barcode scan including a motion degree determination unit that generates an operation signal of the image-based product recognition module system. delete The method of claim 9,
When it is determined that the normal barcode scan has been performed, a barcode further comprising a background image update module for updating a color image and a depth image obtained after the product is mounted in the product mounting area into the background color image and the background depth image. Scanning Verification System. The method of claim 14,
When it is determined that the normal barcode scan has been performed, the system further comprises a purchase list update module for updating the purchase list by adding a product identified by the normal barcode scanning to a purchase list.

Images