KR102260355B1 - System and method for recognizing purchase behavior based on image - Google Patents

Abstract

An image-based purchase behavior recognition system according to an embodiment of the present invention includes: an image acquisition module for acquiring an input image from a photographing device; an inventory management module for generating an out-of-stock notification when the input image and a reference model, which is an image when the item does not exist, match; a hand motion detection module for detecting a motion of a customer's hand in the input image; a change region detection module for detecting a region in which the foreground is changed in the input image; a customer behavior recognition module for recognizing the purchase or return behavior of the customer; and a background model management module for updating the input image after the customer's purchase or return of the product is completed as a background model.

Description

Image-based purchasing behavior recognition system and method {SYSTEM AND METHOD FOR RECOGNIZING PURCHASE BEHAVIOR BASED ON IMAGE}

The present invention relates to an image-based purchasing behavior recognition system and method.

Retailers, wholesalers, and other product distributors generally maintain an inventory of various products that can be ordered and purchased by customers or business partners. For example, in a typical mart, a user (customer) enters the mart, selects a product, and lines up at the checkout counter holding a cart containing the product. Thereafter, the mart employee recognizes the barcode for each product selected by the user and calculates the product.

The 4th industrial revolution, which has recently been spotlighted, includes representative keywords such as big data, Internet of Things, artificial intelligence, and deep learning. The technology of the 4th industrial revolution can have a direct impact on our real life, and technology is evolving through learning and analysis beyond being a convenient tool in our lives.

According to these technological developments, interest in offline shopping to which the 4th industrial revolution is applied is increasing recently.

In response to these technological changes, Amazon has opened a new offline mart, 'Amazon Go'. Amazon Go is not much different from regular supermarkets that sell groceries and miscellaneous goods.

However, users in Amazon Go do not have to perform separate calculations after selecting a product. Amazon Go automatically recognizes the product selected by the user, and automatically charges payment for the recognized product by the payment method corresponding to the user.

As such, as unmanned stores are gradually introduced, various technologies are being developed to recognize the purchase behavior of the customer and automatically detect the item purchased by the customer.

In the case of the existing technology for recognizing a customer's purchasing behavior, a method using the difference in the customer's hand shape has been developed. However, the difference in hand shape is not easy to detect and is highly error prone. Therefore, the development of customer purchasing behavior recognition technology through a method other than hand shape difference is required.

In addition, although an article recognition technology, which is a technology for automatically detecting a product purchased by a customer, has been developed, there is a problem in that the performance of the recognizer is low or the product cannot be recognized when the product sensed by the recognizer is not a target product.

An object of the present invention is to provide a shelf product inventory monitoring system and method capable of recognizing a customer's product purchase or return behavior through easy measurement of a customer's hand movement and a previous shelf state.

On the other hand, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned are clearly to those of ordinary skill in the technical field to which the present invention belongs from the following description. It will be understandable.

An image-based purchase behavior recognition system according to an embodiment of the present invention includes: an image acquisition module for acquiring an input image from a photographing device; an inventory management module for generating an out-of-stock notification when the input image and a reference model, which is an image when the item does not exist, match; a hand motion detection module for detecting a motion of a customer's hand in the input image; a change region detection module for detecting a region in which the foreground is changed in the input image; a customer behavior recognition module for recognizing the purchase or return behavior of the customer; and a background model management module for updating the input image after the customer's purchase or return of the product is completed as a background model.

The hand motion detection module includes: a hand position detection unit for detecting the customer's hand in each frame of the input image and setting a bounding box including the hand and a neighboring area of the hand; a hand position tracking unit for tracking the trajectory of the bounding box by analyzing the correlation between the previous frame and the current frame; and a hand movement learning unit for updating data related to the hand movement acquired by the hand position detection unit and the hand position tracking unit.

The change area detection module includes: an input image foreground extractor for extracting a region corresponding to the foreground from the current input image; a background model foreground extracting unit for extracting a region corresponding to the foreground from the background model; and a change region for extracting a changed region, which is a region in which the foreground is changed, by comparing the region corresponding to the foreground of the current input image extracted by the input image foreground extractor with the region corresponding to the foreground of the background model extracted by the background model foreground extractor. It may include an extractor.

The customer behavior recognition module includes: a product change area extraction unit for extracting a product change area that is a change area by the movement of an item in the change area; a similarity measuring unit for measuring a degree of similarity between the product change area and an area corresponding to the product change area in the reference model; and if the similarity value measured by the similarity measurement unit is greater than a reference value after the hand trajectory sensed by the hand motion detection module overlaps the product change area, it is determined that the customer has purchased the first product, and the hand When the similarity value measured by the similarity measurement unit after the trajectory of the hand sensed by the motion detection module overlaps the product change area is smaller than a reference value, a customer behavior determination unit that determines that the customer has returned the second product; , the first article is an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the background model, or an article recognized through pre-registered shelf-loaded article information, and the second article is It may be an item recognized through an image-based item recognizer by extracting an area corresponding to the item change area from the current input image, or an item recognized through pre-registered shelf loading item information.

If the customer behavior recognition module determines that the customer has purchased the product, the POS management module adds the purchased product to the customer purchase list, and removes the returned product from the customer purchase list if it is determined that the customer has returned the product may further include.

An image-based purchase behavior recognition method according to an embodiment of the present invention includes the steps of: (a) acquiring an input image from a photographing device; (b) generating an out-of-stock notification when the input image and the reference model, which is an image when the item does not exist, match; (c) detecting a motion of the customer's hand in the input image; (d) detecting an area in which the foreground is changed in the input image; (e) recognizing the purchase or return behavior of the customer; and (f) updating the input image after the customer's purchase or return of the product is completed as a background model.

The step (c) includes: (c-1) detecting the customer's hand in each frame of the input image, and then setting a bounding box including the hand and an area adjacent to the hand; (c-2) tracing the trajectory of the bounding box by analyzing the correlation between the previous frame and the current frame; and (c-3) updating data related to hand motion acquired by the hand position detection unit and the hand position tracking unit.

The step (d) includes: (d-1) extracting a region corresponding to the foreground from the current input image; (d-2) extracting a region corresponding to the foreground from the background model; and (d-3) a region in which the foreground is changed by comparing the region corresponding to the foreground of the current input image extracted in step (d-1) with the region corresponding to the foreground of the background model extracted in step (d-2). It may include the step of extracting the change region.

The step (e) includes: (e-1) extracting an article change area, which is a change area by movement of an article in the change area; (e-2) measuring a degree of similarity between the product change area and an area corresponding to the product change area in the reference model; and (e-3) when the similarity value measured in step (e-2) is greater than a reference value after the hand trajectory detected in step (c) overlaps the product change area, the customer purchases the first product After the hand trajectory detected in step (c) overlaps with the product change area, if the similarity value measured in step (e-2) is smaller than the reference value, the customer returns the second product. and determining that the first article is an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the background model, or an article recognized through pre-registered shelf loading article information. and the second article may be an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the current input image, or an article recognized through pre-registered shelf loading article information.

(g) A customer who adds the purchased item to the customer's purchase list when it is determined that the customer has purchased the item in step (e), and removes the item from the customer's purchase list when it is determined that the customer has returned the item It may further include a purchase list update step.

In the recording medium according to an embodiment of the present invention, a program for executing the above-described image-based purchasing behavior recognition method by a computer is recorded.

The image-based purchasing behavior recognition system and method according to an embodiment of the present invention may recognize the customer's purchase or return behavior through easy-to-measure the customer's hand movement and the previous shelf state.

On the other hand, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description. I will be able to.

1 is a block diagram showing a schematic configuration of an image-based purchasing behavior recognition system according to an embodiment of the present invention.
2 is a photograph showing an input image according to an embodiment of the present invention.
3 is a photograph showing a reference model according to an embodiment of the present invention.
4 is a photograph showing a background model according to an embodiment of the present invention.
5 is a block diagram showing a schematic configuration of a hand motion detection module according to an embodiment of the present invention.
6 is a block diagram showing a schematic configuration of a change area detection module according to an embodiment of the present invention.
7 is a block diagram showing a schematic configuration of a customer behavior recognition module according to an embodiment of the present invention.
8 is a block diagram illustrating a schematic flow of an image-based purchasing behavior recognition method according to an embodiment of the present invention.
9 is a block diagram illustrating a schematic flow of a step of detecting a motion of a customer's hand in an input image according to an embodiment of the present invention.
10 is a block diagram illustrating a schematic flow of a step of detecting a region in which a foreground is changed in an input image according to an embodiment of the present invention.
11 is a block diagram schematically illustrating a step of recognizing a customer's product purchase or return behavior according to an embodiment of the present invention.

Other advantages and features of the present invention, and a method of achieving them will become apparent with reference to embodiments to be described later in detail together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms. It is provided to completely inform the scope of the invention to the possessor, and the invention is only defined by the scope of the claims.

Even if not defined, all terms (including technical or scientific terms) used herein have the same meaning as commonly accepted by universal technology in the prior art to which this invention belongs. Terms defined by general dictionaries may be construed as having the same meaning as the related description and/or the text of this application, and not conceptualized or excessively formalized, even if not clearly defined herein. Won't.

The terms used in this specification are for describing exemplary embodiments and are not intended to limit the present invention. In this specification, the singular form also includes the plural form unless specifically stated in the phrase. As used in the specification,'include' and/or various conjugated forms of this verb, for example,'including','including','including','including', and the like refer to the mentioned composition, ingredient, component, Steps, operations and/or elements do not exclude the presence or addition of one or more other compositions, components, components, steps, operations and/or elements. In the present specification, the term'and/or' refers to each of the listed components or various combinations thereof.

Meanwhile, terms such as'~ unit','~ group','~ block', and'~ module' used throughout this specification may mean a unit that processes at least one function or operation. For example, it can mean software, hardware components such as FPGAs or ASICs. However,'~bu','~gi','~block', and'~module' are not meant to be limited to software or hardware. '~ unit', '~ group', '~ block', and '~ module' may be configured to reside in an addressable storage medium or may be configured to reproduce one or more processors.

Accordingly, as an example, '~ part', '~ group', '~ block', and '~ module' are components such as software components, object-oriented software components, class components, and task components. fields, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and include variables. The functions provided in the components and'~bu','~gi','~block', and'~module' include a smaller number of components and'~bu','~gi', and'~block. It may be combined into','~modules' or further separated into additional components and'~unit','~group','~block', and'~module'.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings of the present specification.

1 is a block diagram showing a schematic configuration of an image-based purchase behavior recognition system 10 according to an embodiment of the present invention.

1, the image-based purchase behavior recognition system 10 includes an image acquisition module 100, an inventory management module 200, a hand motion detection module 300, a change area detection module 400, and a customer behavior recognition module. 500 , a background model management module 600 and a POS management module 700 .

The image acquisition module 100 acquires an input image from a photographing device (not shown).

2 is a photograph showing an input image according to an embodiment of the present invention.

Referring to FIG. 2 , an input image refers to an image acquired by an operation of a photographing apparatus.

When a subject is present in the input image, the area corresponding to the foreground, which is the subject part, becomes the foreground area, and the area corresponding to the background excluding the foreground becomes the background area. That is, when a subject exists in the photographing range of the photographing apparatus, the input image includes both the foreground area and the background area.

The input image includes coordinate values that are positional information of each pixel constituting the input image, and color information about hue, saturation, and value of each pixel.

3 is a photograph showing a reference model according to an embodiment of the present invention.

Referring to FIG. 3 , the reference model refers to an image obtained by photographing only the background in which there is no product among the input images obtained from the photographing device.

The inventory management module 200 stores a reference model, which is an image when the product does not exist. Therefore, it is possible to check whether the current input image acquired by the image acquisition module 100 matches the reference model, and if they match, a no-stock notification is generated.

For example, if there is an item on the shelf (the place where a product is placed and sold in a store), the current input image and the reference model have a similarity less than or equal to the reference value, and in this case, the inventory management module 200 determines that the product currently exists on the shelf. do. On the other hand, when there is no item on the shelf, the current input image and the reference model have a similarity greater than or equal to the reference value, and in this case, the inventory management module 200 determines that the item does not currently exist on the shelf.

When it is determined that there is no item on the shelf, the inventory management module 200 generates a no-stock notification.

When the material management server (not shown) receives the out-of-stock notification generated from the inventory management module 200, it is recommended to replenish the items on the shelf by identifying the remaining number of the items in the warehouse and delivering them to the manager. can do.

4 is a photograph showing a background model according to an embodiment of the present invention.

Referring to FIG. 4 , the background model refers to an input image after the customer's purchase or return of the product is completed. In the case of the background model of FIG. 4 , it can be easily seen that the input image is after the customer purchases a Post-it in the state of FIG. 3 .

The background model management module 600 acquires the background model through the input image after the customer's purchase or return of the product is completed. That is, a new background model is updated in the background model management module 600 whenever the customer's purchase or return of the product is completed.

5 is a block diagram showing a schematic configuration of a hand motion detection module 300 according to an embodiment of the present invention.

Referring to FIG. 5 , the hand motion detection module 300 includes a hand position detection unit 310 , a hand position tracking unit 320 , and a hand motion learning unit 330 .

The hand motion detection module 200 detects the motion of the customer's hand in the input image.

According to an embodiment, the hand motion detection module 200 sets a bounding box including a hand and a neighboring region of the hand in the input image. Thereafter, the movement trajectory of the bounding box is traced to determine the trajectory of the hand movement. That is, it is relatively easy to measure and can reduce measurement errors because it is relatively easy to measure the trajectory of the bounding box including the hand, rather than grasping the fine movement of the finger.

The hand position detection unit 310 detects the customer's hand in each frame of the input image, and then sets a bounding box including the hand and a neighboring region of the hand.

The hand position detection unit 310 first detects the position of the customer's hand in the input image. The customer's hand position detection may use a method using the customer's skin color. In addition, a method of obtaining a human body region using an RGB-D image, grouping them according to similarity, and then detecting a hand region at the distal end may also be used.

For example, in the case of using the skin color of the customer, first, the skin color of the customer is detected from the photographing device.

If a region having a color similar to the detected skin color is extracted, regions such as the customer's face, arms, legs, and hands can be detected. In addition, when the customer's face region is recognized through the detected skin color, a candidate region in which the hand can be located can be calculated by using the body ratio determined in consideration of the position and size information of the face.

That is, the customer's hand position can be detected by extracting the intersection of the candidate region where the hand can be located and the region similar to the skin color.

In addition, it is also possible to detect the customer's hand position using RGB-D images. In the case of RGB-D method, it can be used when the customer is wearing gloves and it is impossible to detect the hand through the skin color.

A depth information image corresponding to the customer's human body can be acquired through the RGB-D image, and pixels can be grouped based on the similarity between pixels through the acquired depth information image. Thereafter, hand detection is possible by extracting the extremity of the human body based on the clustered pixel information.

In addition, it is possible to detect the customer's hand position using various methods that have already been disclosed.

After detecting the position of the customer's hand, the hand position detection unit 310 sets a bounding box including the hand and a neighboring region of the hand.

The hand position tracking unit 320 tracks the movement trajectory of the hand by analyzing the correlation between the previous frame and the current frame and tracking the movement trajectory of the bounding box set in the hand position detection unit 310 .

By tracking the bounding box including the hand and the neighboring area of the hand rather than tracking the hand, it is possible to reduce recognition errors due to differences in the shape of each customer's hand.

In addition, it is possible to improve the data processing speed for hand motion detection because it is relatively easy to grasp the movement and the trajectory by grasping the movement trajectory of the bounding box rather than grasping the movement of the hand including the movement of the finger.

The hand motion learning unit 330 continuously updates the data obtained from the hand position detecting unit 310 and the hand position tracking unit 320 .

That is, through continuous learning using the data generated by the hand position detection unit 310 and the hand position tracking unit 320, it is possible to cope with various hand shapes and hand movements of customers, thereby reducing errors related to hand movement detection. .

6 is a block diagram showing a schematic configuration of a change area detection module 400 according to an embodiment of the present invention.

Referring to FIG. 6 , the change region detection module 400 includes an input image foreground extractor 410 , a background model foreground extractor 420 , and a change region extractor 430 .

The change region detection module 400 detects a region whose foreground is changed in the input image.

According to an embodiment, the change region detection module 400 extracts a foreground region from each of the input image and the background model, compares them, and extracts a change region, which is a region in which the foreground is changed.

The input image foreground extraction unit 410 extracts a region corresponding to the foreground from the input image obtained from the image acquisition module 100 .

The method of extracting the area corresponding to the foreground from the input image is to extract the foreground by dividing the input image and the reference model in pixel units and comparing each pixel with the difference in color information including color, brightness, and saturation, or to extract the boundary from the input image. After extraction, a method of extracting the inner region of the boundary line as the foreground may be used.

After separating the foreground from the input image, it is converted into a binary image through image thresholding.

The converted binary image is numbered by grouping adjacent pixel values through labeling, and the smallest label area is deleted among them.

The background model foreground extraction unit 420 extracts a region corresponding to the foreground from the background model.

The process of extracting the region corresponding to the foreground from the background model is the same as the process of extracting the region corresponding to the foreground from the input image foreground extracting unit 410 .

The changed area extraction unit 430 compares the area corresponding to the foreground of the current input image extracted by the input image foreground extraction unit 410 with the area corresponding to the foreground of the background model extracted by the background model foreground extraction unit 420, and the changed Determine if there is an area.

If there is a changed area, the change area extraction unit 430 designates the area as a change area and extracts it.

7 is a block diagram showing a schematic configuration of a customer behavior recognition module 500 according to an embodiment of the present invention.

Referring to FIG. 7 , the customer behavior recognition module 500 includes an article change region extraction unit 510 , a similarity measurement unit 520 , and a customer behavior determination unit 530 .

According to one embodiment, the customer behavior recognition module 500 is a customer through the movement of the customer's hand detected through the hand motion detection module 300 and the foreground change of the input image detected through the change area detection module 400 . Recognize whether this item has been purchased or returned.

The article change region extraction unit 510 extracts an article change region that is a change region due to the movement of the article.

That is, the change area extraction unit 430 extracts an item change area, which is a change area generated by the movement of an article, rather than a change area generated by a hand movement, among the designated change areas.

The similarity measuring unit 520 measures the similarity between the product change area and the area corresponding to the product change area in the reference model.

In the case of similarity measurement, the higher the similarity between the two areas to be compared, the higher the similarity value.

For example, when an item disappears from the input image, the area where the existing item was located corresponds to the item change area. In this case, since the product change area is in a state where only the background exists without the product, a large similarity value is displayed when measuring the similarity with the reference model.

On the other hand, when a new item appears in the input image, the area in which the new item appears corresponds to the item change area. In this case, since the item change area is filled with newly appeared items, a small similarity value is shown when measuring the similarity with the reference model.

That is, by measuring the similarity between the item change area and the area corresponding to the item change area in the reference model, it is possible to know whether the item disappeared or appeared in the input image.

The customer behavior determining unit 530 determines whether the customer has purchased or returned the product based on the motion of the customer's hand detected through the hand motion detection module 300 and the similarity value measured by the similarity measurement unit 520 .

For example, if the similarity value measured by the similarity measurement unit 520 is greater than the reference value after the hand trajectory detected by the motion detection module 300 overlaps the product change area, it is determined that the customer has purchased the first product can do.

On the other hand, when the similarity value measured by the similarity measurement unit 520 after the hand trajectory detected by the hand motion detection module 300 overlaps the product change area is smaller than the reference value, it is determined that the customer has returned the second product can do.

Here, the first article refers to an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the background model, or an article recognized through pre-registered shelf-loaded article information.

In addition, the second article refers to an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the current input image, or an article recognized through pre-registered shelf-loaded article information.

That is, the customer behavior determining unit 530 determines whether the customer purchases or returns the product based on the trajectory of the hand detected by the hand motion detection module 300 and the similarity value measured by the similarity measurement unit 520 .

Referring to FIG. 1 , the image-based purchase behavior recognition system 10 further includes a POS management module 700 .

When it is determined that the customer has purchased the product through the customer behavior recognition module 500, the POS management module 700 adds the purchased product to the customer purchase list, and when it is determined that the customer has returned the product, the customer purchase list Remove the returned item from

8 is a diagram illustrating a schematic flow of a method (S10) for recognizing an image-based purchase behavior according to an embodiment of the present invention.

Referring to FIG. 8 , the image-based purchase behavior recognition method ( S10 ) includes the step of acquiring an input image from a photographing device ( S100 ), and when the input image and the reference model, which is an image when the item does not exist, match the stock out notification step of generating (S200), detecting the movement of the customer's hand in the input image (S300), detecting an area with a changed foreground in the input image (S400), recognizing the customer's purchase or return behavior (S500), updating the input image as a background model after the customer's purchase or return action is completed (S600) and if it is determined that the customer has purchased the item in step S500, add the purchased item to the customer purchase list, and a customer purchase list update step (S700) of removing the product from the customer purchase list in which it is determined that the customer has returned the product.

In step S100, an input image is obtained from a photographing device.

In step S200, when it is determined that there is no stock because the input image and the reference model match, an out-of-stock notification is generated.

The reference model refers to an image when an article is not present in a previously stored shelf.

The out-of-stock notification in step S200 is transmitted to the material management server. The material management server determines the number of remaining items in the warehouse and delivers them to the manager, and recommends replenishing the items on the shelf.

9 is a view showing a schematic flow of the step (S300) of detecting the motion of the customer's hand in the input image according to an embodiment of the present invention.

Referring to FIG. 9 , in step S300 , after detecting the customer's hand in each frame of the input image, setting a bounding box including the hand and a neighboring region of the hand ( S310 ), and analyzing the correlation between the previous frame and the current frame and tracking the trajectory of the bounding box (S320) and learning the movement of the hand (S330).

According to one embodiment, in step S300, after setting a bounding box including a hand and a neighboring region of the hand in the input image, the movement of the hand is sensed by grasping the trajectory of the bounding box. In addition, various hand shapes and hand movements can be detected without errors through continuous updates of images related to hand movements.

10 is a diagram schematically illustrating a flow of detecting a region in which a foreground is changed in an input image (S400) according to an embodiment of the present invention.

Referring to FIG. 10 , step S400 includes extracting a region corresponding to the foreground from the current input image ( 410 ), extracting a region corresponding to the foreground from the background model ( S420 ) and the current input image extracted in steps S410 . and comparing the foreground area with the foreground area of the background model extracted in step S420 to extract a change area (S430).

According to an embodiment, in step S400 , a foreground region is extracted from each of the input image and the background model, and then, a change region, which is a region in which the foreground is changed, is extracted by comparing them.

11 is a view showing a schematic flow of the step (S500) of recognizing the customer's purchase or return behavior of the product according to an embodiment of the present invention.

Referring to FIG. 11 , step S500 is a step of extracting an article change region that is a change region due to the movement of an article in the change region 510, measuring the similarity between the article change region and the region corresponding to the article change region in the reference model. If the similarity value measured in step S520 is greater than the reference value after the hand trajectory detected in steps S520 and S300 overlaps with the product change area, it is determined that the customer has purchased the first product, and detected in step S300 and when the similarity value measured in step S520 is smaller than the reference value after the trajectory of the hand overlaps the product change area, determining that the customer has returned the second product (S530).

Here, the first article refers to an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the background model or an article recognized through pre-registered shelf-loaded article information.

In addition, the second article refers to an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the current input image, or an article recognized through pre-registered shelf-loaded article information.

Referring back to FIG. 8 , in step S600 , the input image after the customer's purchase or return of the product is completed is updated as a background model.

Further, in step S700, when it is determined that the customer has purchased the product in step S500, the purchased product is added to the customer purchase list, and when it is determined that the customer has returned the product, the product is removed from the customer purchase list.

That is, the new background model is updated whenever the customer's purchase or return of the product is completed, and the customer's purchase list is updated according to the customer's purchase or return of the product.

According to an embodiment of the present invention, the present invention includes a computer-readable recording medium in which a program for executing the above-described image-based purchasing behavior recognition method is recorded.

The image-based purchasing behavior recognition system according to an embodiment of the present invention recognizes the customer's purchase or return of the product by detecting the motion of the customer's hand and the movement of the product.

Although the present invention has been described by way of examples above, the above examples are merely for explaining the spirit of the present invention and are not limited thereto. Those skilled in the art will understand that various modifications may be made to the above-described embodiments. The scope of the present invention is determined only through interpretation of the appended claims.

10: Image-based purchase behavior recognition system
100: image acquisition module 200: inventory management module
300: hand motion detection module 310: hand position detection unit
320: hand position tracking unit 330: hand movement learning unit
400: change area detection module 410: input image foreground extraction unit
420: background model foreground extraction unit 430: change area extraction unit
500: customer behavior recognition module 510: product change area extraction unit
520: similarity measuring unit 530: customer behavior determining unit

Claims (11)

an image acquisition module for acquiring an input image from a photographing device;
an inventory management module for generating an out-of-stock notification when the input image and a reference model, which is an image when the item does not exist, match;
a hand motion detection module for detecting a motion of a customer's hand in the input image;
a change region detection module for detecting a region in which the foreground is changed in the input image;
a customer behavior recognition module for recognizing the purchase or return behavior of the customer; and
and a background model management module for updating the input image after the customer's purchase or return of the product is completed as a background model,
The hand motion detection module includes:
a hand position detection unit for detecting the customer's hand in each frame of the input image and setting a bounding box including the hand and an area adjacent to the hand;
a hand position tracking unit for tracking the trajectory of the bounding box by analyzing the correlation between the previous frame and the current frame; and
and a hand movement learning unit for updating data related to the movement of the hand obtained from the hand position detection unit and the hand position tracking unit,
The change area detection module includes:
an input image foreground extractor for extracting a region corresponding to the foreground from the current input image;
a background model foreground extracting unit for extracting a region corresponding to the foreground from the background model; and
A change area for extracting a change area, which is an area in which the foreground is changed, by comparing the area corresponding to the foreground of the current input image extracted by the input image foreground extraction unit with the area corresponding to the foreground of the background model extracted by the background model foreground extraction unit comprising an extractor;
The customer behavior recognition module includes:
an article change region extraction unit for extracting an article change region that is a change region due to movement of an article in the change region;
a similarity measuring unit for measuring a degree of similarity between the product change area and an area corresponding to the product change area in the reference model; and
When the similarity value measured by the similarity measurement unit is greater than the reference value after the hand trajectory sensed by the hand motion detection module overlaps the product change area, it is determined that the customer has purchased the first product, and the hand motion detection When the similarity value measured by the similarity measurement unit after the trajectory of the hand sensed by the module overlaps the product change area is smaller than a reference value, a customer behavior determination unit that determines that the customer has returned the second product;
Image-based purchasing behavior recognition system.
delete delete The method of claim 1,
The first article is an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the background model, or an article recognized through pre-registered shelf loading article information,
The second article is an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the current input image, or an article recognized through pre-registered shelf loading article information.
Image-based purchasing behavior recognition system.
The method of claim 1,
When it is determined that the customer has purchased the product in the customer behavior recognition module, a POS management module that adds the purchased product to the customer purchase list and removes the returned product from the customer purchase list when it is determined that the customer has returned the product containing
Image-based purchasing behavior recognition system.
(a) acquiring an input image from a photographing device;
(b) generating an out-of-stock notification when the input image and the reference model, which is an image when the item does not exist, match;
(c) detecting a motion of the customer's hand in the input image;
(d) detecting an area in which the foreground is changed in the input image;
(e) recognizing the purchase or return behavior of the customer; and
(f) updating the input image after the customer's purchase or return of the product is completed as a background model,
Step (c) is:
(c-1) detecting the customer's hand in each frame of the input image and setting a bounding box including the hand and a neighboring region of the hand;
(c-2) tracing the trajectory of the bounding box by analyzing the correlation between the previous frame and the current frame; and
(c-3) updating data related to the movement of the hand,
Step (d) is:
(d-1) extracting a region corresponding to the foreground from the current input image;
(d-2) extracting a region corresponding to the foreground from the background model; and
(d-3) A change that is an area in which the foreground is changed by comparing the area corresponding to the foreground of the current input image extracted in step (d-1) with the area corresponding to the foreground of the background model extracted in step (d-2) extracting a region,
Step (e) is:
(e-1) extracting an article change region that is a change region due to movement of an article in the change region;
(e-2) measuring a degree of similarity between the product change area and an area corresponding to the product change area in the reference model; and
(e-3) If the similarity value measured in step (e-2) is greater than the reference value after the hand trajectory detected in step (c) overlaps with the product change area, it is determined that the customer has purchased the first product It is determined that the customer returned the second product when the similarity value measured in step (e-2) is smaller than the reference value after the hand trajectory detected in step (c) overlaps with the product change area comprising the steps of
Image-based purchasing behavior recognition method.
delete delete The method of claim 6,
The first article is an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the background model, or an article recognized through pre-registered shelf loading article information,
The second article is an article recognized through an image-based article recognizer by extracting a region corresponding to the article change region from the current input image, or an article recognized through pre-registered shelf loading article information.
Image-based purchasing behavior recognition method.
The method of claim 6,
(g) A customer purchase that adds the purchased item to the customer purchase list when it is determined that the customer has purchased the item in step (e), and removes the item from the customer purchase list when it is determined that the customer has returned the item list update step
Image-based purchasing behavior recognition method.
In a computer-readable recording medium,
A recording medium recording a program for executing the image-based purchasing behavior recognition method according to claim 6 by a computer.

Images