JP2022036983A - Self-register system, purchased commodity management method and purchased commodity management program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a self-register system capable of properly managing commodities to be purchased by customers without previously preparing images of individual commodities.

SOLUTION: Change detection means 840 detects a change region of a commodity shelf on which commodities are arranged on the basis of images in which the commodities are imaged. Classification means 850 classifies a change in the commodity shelf in the detected change region. Shopping list generation means 860 generates a shopping list of a customer on the basis of classification of the change in the commodity shelf and shelving allocation information for the commodity shelf.

SELECTED DRAWING: Figure 31

COPYRIGHT: (C)2022,JPO&INPIT

Description

The present invention relates to a self-register (self-registration) system that automatically manages products purchased by a customer, a purchased product management method, and a purchased product management program.

At convenience stores, supermarkets, and other stores that sell products, self-checkouts have been introduced in which customers themselves operate register (hereinafter referred to as cashiers) terminals. In a general self-checkout, a customer makes a checkout terminal read a product identification code (for example, a barcode) to make a payment. However, since it takes time and effort for the customer to determine the position of the identification code, various methods for automatically identifying the product purchased by the customer have been proposed.

For example, Patent Document 1 describes a product monitoring device that monitors the display status of products. The apparatus described in Patent Document 1 detects a product from an image obtained by photographing the display area, and evaluates the product display state in the display area based on the detection result.

Further, Patent Document 2 describes a POS (Point Of Sales) system for settlement of goods. The POS system described in Patent Document 2 detects a customer's flow line using a captured image, identifies a customer who intends to settle the product, and displays the product at a position corresponding to the customer's flow line. Recognize the product to be settled with.

In addition, Non-Patent Document 1 describes a method of subtracting a background image using an adaptive Gaussian mixture model.

International Publication No. 2015/079622 International Publication No. 2015/140853

Zoran Zivkovic, "Improved Adaptive Gaussian Mixture Model for Background Subtraction", Proceedings of the 17th International Conference on Pattern Recognition (ICPR'04), the United States, IEEE Computer Society, 8 May 2004, Volume2-Volume02, p. 28-31

The product monitoring device described in Patent Document 1 extracts an object from an image by using a general image recognition method. However, in order to extract an object from an image using a general image recognition method, an image (reference image) representing the object to be extracted is required. However, convenience stores and supermarkets may have thousands to tens of thousands of products, and these products are generally replaced on a regular basis. Therefore, collecting or updating reference images for all products has a problem that the cost becomes very high.

Further, in the system described in Patent Document 2, a product associated with the flow line data is set as a candidate for payment processing, but it is difficult to specify a product purchased by a customer from the candidates.

Therefore, an object of the present invention is to provide a self-registration system, a purchased product management method, and a purchased product management program that can appropriately manage products purchased by a customer without preparing images of individual products in advance.

The self-registration system of the present invention classifies a change detecting means for detecting a change region of a product shelf on which a product is placed and a change in the product shelf in the detected change region based on an captured image in which the product is imaged. It is provided with a classification means to generate a shopping list, a shopping list generation means for generating a customer's shopping list based on the classification of changes in the product shelves, and the shelving allocation information of the product shelves. It is characterized by classifying the changes in the product shelves from the images of the change areas by using the shelf change model that models the changes in the product shelves.

The purchased product management method of the present invention detects a change area of the product shelf on which the product is placed based on the captured image in which the product is imaged, classifies the change of the product shelf in the detected change area, and classifies the change. A shelf change model that generates a customer's shopping list based on the classification of changes in the product shelves and the shelf allocation information of the product shelves, and models the changes in the product shelves before and after the change in time during classification. It is characterized in that the change of the product shelf is classified from the image of the change area by using.

The purchased product management program of the present invention performs a change detection process for detecting a change area of a product shelf on which the product is placed based on an image captured by the computer, and a product shelf in the detected change area. Based on the classification process for classifying changes in the product shelves, the classification of changes in the product shelves, and the shelving allocation information for the product shelves, the shopping list generation process for generating the customer's shopping list is executed. Using a shelf change model that models changes in the product shelves before and after the change, it is characterized in that the changes in the product shelves are classified from the image of the change area.

According to the present invention, it is possible to appropriately manage the products purchased by the customer without preparing images of the individual products in advance.

It is a block diagram which shows the structural example of the 1st Embodiment of the self-registration system by this invention. It is explanatory drawing which shows the example of the use scene of the self-registration system 1. It is a block diagram which shows the structural example of the image processing apparatus 100. It is a block diagram which shows the structural example of the 1st change detection part and the 1st storage part. It is explanatory drawing which shows the operation example of the foreground area detection part. It is explanatory drawing which shows the example of the classification result output by the area change classification part. It is explanatory drawing which shows the example of the relationship information between a product and a person. It is explanatory drawing which shows the example which integrated the relationship information between a product and a person. It is a block diagram which shows the configuration example of the shopping list management apparatus. It is explanatory drawing which shows the example of the shelf allocation information. It is a flowchart which shows the operation example of the image processing apparatus 100 of 1st Embodiment. It is a flowchart which shows the operation example of the self-registration system of 1st Embodiment. It is explanatory drawing which shows the structural example of the image processing apparatus 200. It is a block diagram which shows the structural example of the 2nd change detection part and the 2nd storage part. It is a flowchart which shows the operation example of the image processing apparatus 200 of 2nd Embodiment. It is explanatory drawing which shows the structural example of the 3rd Embodiment of the self-registration system by this invention. It is explanatory drawing which shows the structural example of the image processing apparatus 300. It is explanatory drawing which shows the example of the flow line data. It is explanatory drawing which shows the example of the relation information of a product and a person. It is explanatory drawing which shows the example which integrated the relationship between a product and a person. It is a flowchart which shows the operation example of the image processing apparatus 300 of 3rd Embodiment. It is a block diagram which shows the modification of the self-registration system of 3rd Embodiment. It is a block diagram which shows the structural example of the image processing apparatus 300a. It is explanatory drawing which shows the other example of the flow line data. It is explanatory drawing which shows the operation example which detects the foreground area. It is explanatory drawing which shows the operation example which detects the foreground area. It is explanatory drawing which shows the operation example which detects the foreground area. It is explanatory drawing which shows the operation example of the self-registration system in 1st specific example. It is explanatory drawing which shows the operation example of the self-registration system in the 2nd specific example. It is a block diagram which shows the hardware composition example of the information processing apparatus which realizes the component element of each apparatus. It is a block diagram which shows the outline of the self-registration system by this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[First Embodiment]
In the first embodiment, a self-registration system 1 that manages a list of products that a customer plans to purchase (hereinafter referred to as a shopping list) based on a captured image will be described.

As will be described later, the self-registration system 1 of the present embodiment detects changes in the product shelves 3 based on the captured image captured by the image pickup device 2, and also detects a region of a person or an object included in the captured image. Then, the self-registration system 1 associates the change in the product shelf 3 with the extracted person, and integrates the information associated with the person as a reference. In this way, the self-registration system 1 is configured to correlate changes in the product shelves 3 based on the extracted persons. As a result, the self-registration system 1 detects the product that the extracted person intends to purchase.

FIG. 1 is a block diagram showing a configuration example of a first embodiment of the self-registration system according to the present invention. The self-registration system 1 of the present embodiment includes a terminal 10, an image processing device 100, an image pickup device 2, a shopping list management device 20, an output device 30, a payment device 40, and a person identification device 50 owned by a customer. It is equipped with. The terminal 10, the image processing device 100, the image pickup device 2, the shopping list management device 20, the output device 30, the payment device 40, and the person identification device 50 are connected to each other via, for example, a network N. To.

In this embodiment, the image processing device 100, the image pickup device 2, and the shopping list management device 20 will be described as having separate configurations. However, each device may be configured to be built in another device. For example, the image processing device 2 may have a function as an image processing device 100 described later, or the image processing device 100 may have a function as an image processing device 2. Further, for example, the image processing device 100 and the shopping list management device 20 may be realized by the same hardware, receive an image captured by the image pickup device 2, and perform each process described later.

Further, FIG. 1 illustrates a case where the image pickup device 2 is one. However, the number of image pickup devices 2 included in the self-registration system 1 is not limited to one, and may be two or more.

FIG. 2 is an explanatory diagram showing an example of a usage scene of the self-registration system 1. Referring to FIG. 2, in the self-registration system 1, the image pickup device 2 takes an image of the product shelf 3 of the store. Then, the image pickup device 2 transmits a video signal indicating the captured captured image to the image processing device 100.

The image pickup device 2 is, for example, a surveillance camera installed in a store or the like. The image pickup device 2 is installed at a predetermined position where the product shelf 3 can be imaged, for example, in a store. Further, a camera ID or the like for identifying the image pickup device 2 is assigned to the image pickup device 2 in advance. The image pickup device 2 acquires a captured image. At this time, the image pickup apparatus 2 refers to, for example, a clock owned by the image pickup device 2 and associates the image pickup time, which is the time when the image capture image is acquired, with the acquired image. In this way, the image pickup device 2 acquires a captured image showing the state of the product shelf 3 and the like.

The image captured by the image pickup apparatus 2 may be a moving image or a continuous still image. Further, in the present embodiment, the captured image acquired by the image pickup apparatus 2 is at least one of a color image (hereinafter, referred to as an RGB (Red Green Blue) image) and a distance image. The captured image acquired by the image pickup apparatus 2 may be, for example, an image in a color space other than the RGB image.

As described above, the image pickup apparatus 2 transmits a video signal indicating the acquired captured image to the image processing apparatus 100. The image pickup device 2 may store the captured image inside the image pickup device 2 or in a storage device different from the image processing device 100.

The image processing device 100 is an information processing device that classifies changes in the product shelves by analyzing the captured image in which the product is captured.

FIG. 3 is an explanatory diagram showing a configuration example of the image processing apparatus 100. The image processing apparatus 100 of the present embodiment includes a first acquisition unit 110, a first change detection unit 120, a first storage unit 130, a first relevance generation unit 140, and a relevance integration unit 150. Specifically, the image processing device 100 includes, for example, an arithmetic unit and a storage device (not shown). Then, the image processing device 100 realizes each of the above processing units by, for example, executing a program stored in the storage device by the arithmetic unit.

The image processing apparatus 100 illustrated in FIG. 3 shows a configuration peculiar to the present disclosure. The image processing device 100 may have a member not shown in FIG. This point is the same for the second and subsequent embodiments.

The first acquisition unit 110 acquires a video signal indicating an captured image obtained by the imaging device 2 imaging the product shelf 3. The first acquisition unit 110 may receive the video signal transmitted from the image pickup apparatus 2. The first acquisition unit 110 may acquire a video signal converted based on an image captured inside the image pickup device 2 or stored in a storage device different from the image pickup device 2 and the image processing device 100.

As described above, the image processing device 100 may be configured to be built in the image pickup device 2. In the case of such a configuration, the first acquisition unit 110 may be configured to acquire the captured image itself.

The first acquisition unit 110 converts the acquired video signal into at least one or both of the RGB image and the distance image constituting the video signal. Then, the first acquisition unit 110 supplies the converted image (at least one or both of the RGB image and the distance image) to the first change detection unit 120. The RGB image and the distance image obtained by the first acquisition unit 110 converting the video signal are simply referred to as an captured image because the image pickup device 2 represents an image captured by the product shelf 3.

FIG. 4 is a block diagram showing a configuration example of the first change detection unit 120 and the first storage unit 130. Hereinafter, the first change detection unit 120 and the first storage unit 130 will be described with reference to FIG.

The first storage unit 130 is a storage device such as a hard disk or a memory. The first storage unit 130 stores the background information 131, the shelf change model 132, the foreground information 133, and the person information 134. The first storage unit 130 may be realized by a storage device different from the image processing device 100, or may be built in the first change detection unit 120. Further, the background information 131, the shelf change model 132, the foreground information 133, and the person information 134 may be stored in the same storage device or may be stored in different storage devices.

The background information 131 is an image that serves as a reference for the first change detection unit 120 to compare with the captured image. The background information 131 is also referred to as a background image. The background information 131 is preferably, for example, an image of the same type as the captured image. For example, when the captured image is an RGB image, it is preferable that the background information 131 is also an RGB image. The background information 131 may be an image captured first supplied from the first acquisition unit 110 to the first change detection unit 120, or may be an image given in advance.

Further, as will be described later, the background information 131 is updatable information. Details of the process for updating the background information 131 will be described later.

The shelf change model 132 is a model of a change in the product shelf 3 that has been learned in advance. The shelf change model 132 is stored in, for example, in the first storage unit 130 in advance. The shelf change model 132 may be obtained, for example, by learning using machine learning such as the generally widely known Convolutional Neural Network.

The shelf change model 132 is, for example, a “change due to no product being included in the product shelf 3” learned by using an image in which the product is included in the product shelf 3 and an image in which the product is not included in the product shelf 3. Or, it represents "change due to the new inclusion of the product in the product shelf 3". Further, the shelf change model 132 represents "changes due to changes in the appearance of the products displayed on the product shelves 3" learned using images of a plurality of products and a plurality of images in which the shape of each product has changed. .. Further, the shelf change model 132 is an image captured in a state where no object exists in front of the product shelf 3 and an image captured in a state where an object such as a person exists in front of the product shelf 3. It represents "changes due to the presence of a person in front of the product shelves 3", "changes due to the presence of a shopping cart in front of the product shelves 3", etc. learned using images. In addition, the shelf change model 132 may represent, for example, "changes due to changes in lighting" learned using images under various environments.

The training data of the shelf change model 132 may be, for example, a 6-channel image in which two RGB images before and after the change are combined, or one of the R component, the G component, or the B component of the two RGB images before and after the change. It may be a two-channel image in which the above are combined. Further, the training data may be, for example, a 4-channel image in which any two of the R component, the G component, and the B component of the two RGB images before and after the change are combined, or the two RGB images before and after the change are grayscaled. It may be a two-channel image that has been converted into an image and then combined. Further, in the training data, the RGB images before and after the change are converted into another color space such as the HSV (Hue Saturation Value) color space, and one or more channels in the color space after the conversion to the other color space are combined. It may be an image.

Further, the training data of the shelf change model 132 may be generated from a color image such as an RGB image, or may be generated by using both a color image and a distance image. ..

The foreground information 133 is information stored by the foreground area detection unit 121. The foreground information 133 is a foreground region (change region) which is a region different from the background image in the RGB image, which is detected as a result of comparing the background information 131 which is the background image and the RGB image which is the captured image by the foreground region detection unit 121. ) Is included. Specifically, the foreground information 133 is, for example, a binary image associated with the imaging time of the captured image. The details of the process for storing the foreground information 133 will be described later.

The person information 134 is information stored by the foreground area tracking unit 123. The person information 134 is generated, for example, by associating the person area extracted by the foreground area tracking unit 123 with the ID (camera ID) of the image pickup device, the person ID, the position on the captured image, the imaging time of the captured image, and the like. As will be described later, the person area is, for example, an RGB image. In other words, the person information 134 can include, for example, information indicating the color, area, shape, aspect ratio of the circumscribed rectangle, and the like of the stored person area. The details of the process for storing the person information 134 will also be described later.

The first change detection unit 120 detects a change area related to the product shelf 3.

For example, when the product displayed on the product shelf 3 included in the captured image is not included in the image acquired before the captured image (for example, the background image), the first change detection unit 120 is the area of the product. Is detected. Further, the first change detection unit 120 detects, for example, the area of the product when the product displayed on the product shelf 3 included in the background image is not included in the captured image. Further, the first change detection unit 120 detects, for example, the area of the product when the product displayed on the product shelf 3 included in the captured image and the product included in the background image look different. In this way, the first change detection unit 120 displays the products such as the number of products has decreased (disappeared), the number of products has increased (newly appeared), and the appearance of the products has changed, based on the captured image. Detect changes in state.

Further, the first change detection unit 120 is, for example, when the captured image is captured when a person or an object is present between the product shelf 3 and the image pickup device 2, the person included in the captured image captured by the product shelf 3. And detect the area of the object.

As described above, the first change detection unit 120 depends on the change area inside the product shelf 3, which is the area where the display state of the product has changed, and the person or object existing between the product shelf 3 and the image pickup device 2. A change area related to the product shelf 3, such as a change area of the captured image, is detected.

As illustrated in FIG. 4, the first change detection unit 120 includes a foreground area detection unit 121, a background information update unit 122, a foreground area tracking unit 123, a first extraction unit 124, and a second extraction unit 125. It has a region change classification unit 126.

The foreground region detection unit 121 receives the captured image supplied from the first acquisition unit 110. Further, the foreground region detection unit 121 acquires the background information 131 corresponding to the captured image from the first storage unit 130. As described above, the background information 131 is an RGB image. The foreground region detection unit 121 compares the captured image, which is two RGB images, with the background information 131. Then, the foreground region detection unit 121 detects the region changed between the two compared RGB images as the change region. It can be said that the foreground region detection unit 121 performs a process of detecting a foreground region, which is a region different from the background image, in order to compare the background information 131, which is a background image, with the RGB image, which is an captured image.

In the present embodiment, the method by which the foreground region detection unit 121 detects the change region is not particularly limited. The foreground region detection unit 121 may detect a change region using existing technology. The foreground region detection unit 121 may detect the changing region by using, for example, the background subtraction method. The foreground region detection unit 121 may generate a binary image in which the pixel value of the detected change region is 255 and the other pixel values are 0.

Here, an example of the operation of the foreground region detection unit 121 will be described more specifically with reference to FIG. FIG. 5 is an explanatory diagram showing an operation example of the foreground region detection unit 121. FIG. 5A shows an example of a captured image, and FIG. 5B shows an example of background information 131 corresponding to the captured image stored in the first storage unit 130. Further, FIG. 5C shows an example of a binary image which is a detection result of a change region.

Referring to FIGS. 5A and 5B, there are differences in the three regions of the product G1, the product G2, and the product G3 between the captured image and the background information 131. For example, in the case of the examples shown in FIGS. 5A and 5B, the product G1 is not included in the background information 131, but is included in the captured image. Further, the product G3 is included in the background information 131, but is not included in the captured image. Further, at the position of the product G2 included in the captured image, another product is displayed on the background information 131. Therefore, the foreground region detection unit 121 also detects the region of the product G2 as a region where the product G2 has changed. In such a case, the foreground region detection unit 121 represents, for example, the portion corresponding to the region of the product G1, the product G2, and the product G3 in white and the other portion in black as illustrated in FIG. 5C. Generates a binary image expressed in.

In the following description, the change region refers to each white portion exemplified in FIG. 5 (c). That is, the change region is, for example, a set of pixels having a pixel value of 255 and any pixel value adjacent to the pixel having a pixel value of 255. In the example shown in FIG. 5 (c), the foreground region detection unit 121 detects three change regions.

As described above, the foreground region detection unit 121 generates, for example, a binary image having the same size as the captured image. Further, the foreground region detection unit 121 associates the binary image, which is the detection result, with the imaging time of the captured image used for generating the binary image. The foreground region detection unit 121 may associate the binary image with information indicating the captured image used for generating the binary image, information indicating the background information 131, and the like. Then, the foreground region detection unit 121 supplies the detection result associated with the imaging time of the captured image to the background information updating unit 122 and the foreground region tracking unit 123. Further, the foreground region detection unit 121 stores the detection result associated with the imaging time of the captured image in the first storage unit 130 as the foreground information 133.

The detection result by the foreground region detection unit 121 may include information indicating the detected change region. The foreground region detection unit 121 uses, for example, information indicating the position of the detected change region (region having a pixel value of 255) and the size thereof as information indicating an image captured image and information indicating a background image used for detecting the change region. It may be associated and output as a detection result. As described above, the detection result output by the foreground region detection unit 121 may be in any format.

Further, the foreground region detection unit 121 may associate the binary image which is the detection result with the imaging time of the captured image and also associate the color information included in the change region extracted from the captured image. The foreground area detection unit 121 may associate the image of the change area with the detection result instead of the color information of the change area. In this way, the foreground region detection unit 121 may associate information other than the imaging time with the detection result.

Further, as illustrated in FIG. 5C, the binary image generated by the foreground region detection unit 121 may include a plurality of change regions. In such a case, the foreground region detection unit 121 may generate a binary image for each change region. The method by which the foreground region detection unit 121 generates a binary image for each change region will be described later as a modification of the foreground region detection unit 121.

The foreground area tracking unit 123 tracks the change area detected by the foreground area detection unit 121 among a plurality of captured images. The foreground area tracking unit 123 supplies a binary image to the first extraction unit 124 and the second extraction unit 125, or extracts a person area, depending on the tracking result. Further, the foreground area tracking unit 123 supplies an update signal indicating the update of the background information 131 to the background information update unit 122.

The foreground area tracking unit 123 receives, for example, a detection result (binary image) supplied from the foreground area detection unit 121. Further, the foreground region tracking unit 123 is a binary image generated from the captured image associated with the binary image as the detection result and captured before the capture time of the captured image related to the binary image. A certain foreground information 133 is acquired from the first storage unit 130. Then, the foreground area tracking unit 123 tracks the changed area by performing a process of associating each changed area represented by each binary image.

The foreground area tracking unit 123 can track the changing area using various methods. The foreground area tracking unit 123 has, for example, the area, shape, and circumscribed rectangle of the change area represented by the binary image supplied from the foreground area detection unit 121 and the foreground information 133 acquired from the first storage unit 130. The similarity is calculated based on at least one aspect ratio. Then, the foreground area tracking unit 123 tracks the changed area by associating the changed areas having the highest calculated similarity with each other. Further, in the case of a configuration in which the color information is associated with the detection result, the foreground area tracking unit 123 may perform tracking using the color information. The foreground area tracking unit 123 may perform tracking based on the image of the change area associated with the detection result.

The foreground area tracking unit 123 confirms whether the tracked result is equal to or longer than a predetermined time, or whether the movement amount of the changing region is equal to or longer than a predetermined threshold value. The predetermined time and the predetermined threshold value used by the foreground area tracking unit 123 for confirmation are arbitrary.

When the amount of movement of the changing region is less than a predetermined threshold value and the traced result is equal to or longer than a predetermined time, the foreground region tracking unit 123 uses the binary image which is the detection result supplied from the foreground region detection unit 121. It is supplied to the first extraction unit 124 and the second extraction unit 125. At this time, the foreground region tracking unit 123, for example, imparts information indicating the captured image used for generating the binary image and information indicating the background information 131 to the binary image, and first extracts the binary image. It is supplied to the unit 124 and the second extraction unit 125. The foreground region tracking unit 123 may supply, for example, a corresponding captured image and background information 131 to the first extraction unit 124 and the second extraction unit 125 together with the binary image. Further, when the binary image contains a plurality of change regions and any of the change regions has not been tracked for a predetermined time or longer, the foreground region tracking unit 123 includes information indicating the change region traced for a predetermined time or longer. Binary images may be supplied to the first extraction unit 124 and the second extraction unit 125.

When the binary image includes a plurality of change areas, the foreground area tracking unit 123 may generate a plurality of binary images so that one binary image includes one change area. For example, a binary image containing only a change region tracked for a predetermined time or longer is supplied to the first extraction unit 124 and the second extraction unit 125, and a binary image including a change region not tracked for a predetermined time or longer is included. May be discarded. The foreground region tracking unit 123 may receive a binary image for each change region from the foreground region detection unit 121 as a detection result.

Further, when the movement amount of the changing region is equal to or more than a predetermined threshold value, the foreground region tracking unit 123 determines that the object included in the changing region is a moving object. When it is determined that the object included in the change region is a moving object in this way, the foreground region tracking unit 123 suppresses the supply of the change region to the first extraction unit 124 and the second extraction unit 125. As a result, the image processing device 100 can delete changes related to the product shelf 3 irrelevant to the increase or decrease of the product, such as "change due to the presence of a person in front of the product shelf 3". This makes it possible to more accurately monitor the display status of products.

The foreground region tracking unit 123 may supply the determination result that the object included in the change region is a moving object to the first extraction unit 124 and the second extraction unit 125 in association with the change region. Then, when the determination result is associated with the change area, the area change classification unit 126 classifies the changes related to the product shelf 3 in this change area into the types related to the changes other than the products displayed on the product shelf 3. You may. For example, the area change classification unit 126 describes changes related to the product shelf 3 in the change area as "changes due to the presence of a person in front of the product shelves 3" and "changes due to the presence of a shopping cart in front of the product shelves 3". Etc., it may be classified into types related to changes other than products.

The foreground area tracking unit 123 extracts a change area determined to be a moving object from the captured image as a person area. Specifically, the foreground area tracking unit 123 uses a captured image and a binary image having the same size as the captured image, and a region on the captured image corresponding to a region having a pixel value of 255 in the binary image. The image of is extracted as the first attention image. For example, when the captured image is an RGB image, the extracted person area is also an RGB image.

The foreground area tracking unit 123 may extract a person area having the same shape as the change area for each change area determined to be a moving object, or the same as a frame having a predetermined shape circumscribing the change area. The area surrounded by the frame of the shape may be extracted as a person area. The shape of the frame circumscribing the change region may be any shape such as a rectangle or an ellipse. Further, the foreground area tracking unit 123 may extract an area surrounded by a frame circumscribing the change area by a predetermined size as a person area.

Next, the foreground area tracking unit 123 gives an ID (camera ID) of the image pickup device 2 to the extracted person area, for example, a person ID assigned to each extracted person area, a position on the captured image, an imaging time of the captured image, and the like. To associate. Then, the foreground area tracking unit 123 stores the associated information as the person information 134 in the first storage unit. The position on the captured image may be represented by, for example, the coordinate values of the four corners of the circumscribed rectangle of the change region determined to be a moving object, the coordinate values of at least one of the four corners, and the width and height of the circumscribed rectangle. It may be expressed as a rectangle.

Further, the foreground area tracking unit 123 supplies an update signal indicating the update of the background information 131 to the background information update unit 122.

For example, when the foreground region tracking unit 123 supplies a detection result indicating a change region to the first extraction unit 124 and the second extraction unit 125 after tracking the change region, the foreground region tracking unit 123 sends an update signal having a value of 1 to the change region. The background information update unit 122 is supplied together with the indicated information. The update signal having a value of 1 indicates that the image of the portion corresponding to the change region in the background information 131 is updated. Further, when the detection result is not supplied to the first extraction unit 124 and the second extraction unit 125, the foreground area tracking unit 123 sends an update signal having a value of 0 to the background information update unit 122 together with information indicating the change area. May be supplied. The update signal having a value of 0 indicates that the image of the portion corresponding to the change region in the background information 131 is not updated. The case where the detection result is not output to the first extraction unit 124 and the second extraction unit 125 is, for example, when the tracking result is less than a predetermined time or when the movement amount of the change region is equal to or more than a predetermined threshold value. This is the case.

The foreground area tracking unit 123 may supply an update signal indicating the update of the background information 131 to the background information update unit 122 at a timing other than the above-exemplified timing. The foreground area tracking unit 123 includes, for example, products included in the change area based on purchase information, purchase information, work information of a clerk, etc. of the product transmitted from an external device (not shown) of the image processing device 100. When it is determined that there is a high possibility that the product has been purchased or replenished, an update signal having a value of 1 may be output so as to update the background of the product shelf 3. The foreground area tracking unit 123 may supply an update signal indicating an update of the background information 131 to the background information update unit 122 based on the tracking time or the like included in the tracking result.

The background information updating unit 122 includes an captured image supplied from the first acquisition unit 110, a detection result supplied from the foreground region detection unit 121, and background information 131 (for example, RGB) stored in the first storage unit 130. The background information 131 is updated based on the image) and the update signal supplied from the foreground area tracking unit 123. The method by which the background information updating unit 122 updates the background information 131 is not particularly limited. The background information updating unit 122 may update the background information 131 by using, for example, the same method as in Non-Patent Document 1.

The background information updating unit 122 does not have to update the image of the portion of the image indicated by the background information 131 that corresponds to the change region indicated by the detection result supplied from the foreground region detection unit 121, for example. .. For example, when the background information update unit 122 receives the update signal having the above-mentioned value of 0 from the foreground area tracking unit 123, the background information update unit 122 does not update the background information of the region corresponding to the change region.

As described above, when the foreground area tracking unit 123 does not output the detection result to the first extraction unit 124 and the second extraction unit 125, the foreground region tracking unit 123 supplies the update signal having a value of 0 to the background information update unit 122. As described above, when the tracking result satisfies the first predetermined condition, the background information update unit 122 receives the update signal having a value of 0 and does not update the background information of the region corresponding to the change region. In other words, when the tracking result satisfies the first predetermined condition, the background information update unit 122 updates the background information 131 other than the area corresponding to the change area. As a result, the region corresponding to the region of the image captured next acquired by the first acquisition unit 110 that has not been updated is easily detected by the foreground region detection unit 121 as a change region.

Further, the background information update unit 122 is supplied from the foreground area detection unit 121 among the RGB images shown by the background information 131 when the value of the update signal supplied from the foreground area tracking unit 123 is 1, for example. The image of the part corresponding to the change area indicated by the detection result is updated. As described above, when the tracking result is longer than a predetermined time, the foreground region tracking unit 123 supplies the detection result representing the traced change region to the first extraction unit 124 and the second extraction unit 125, and the value is set. The update signal of 1 is supplied to the background information update unit 122. That is, when the second predetermined condition that the tracking result is the result of being tracked for a predetermined time or longer is satisfied, the background information update unit 122 receives an update signal having a value of 1 from the foreground area tracking unit 123, and the background information. The image of the part corresponding to the change area in 131 is updated. As a result, the background information updating unit 122 can bring the background information 131 stored in the first storage unit 130 closer to the captured image acquired by the first acquisition unit 110 at that time. Therefore, the image processing apparatus 100 can prevent the foreground region detection unit 121 from detecting the region on the captured image that is subsequently acquired by the first acquisition unit 110 corresponding to the change region as the change region.

The first extraction unit 124 receives a binary image which is a detection result from the foreground region tracking unit 123. Further, the first extraction unit 124 acquires the captured image used for generating the binary image from the first acquisition unit 110. The first extraction unit 124 may receive the captured image together with the binary image from the foreground region tracking unit 123.

The first extraction unit 124 extracts an image of the change region from the captured image. Specifically, the first extraction unit 124 uses the captured image and the binary image having the same size as the captured image, and the region on the captured image corresponding to the region where the pixel value in the binary image is 255. The image of is extracted as the first attention image. For example, when the binary image is shown in FIG. 5C, the first extraction unit 124 extracts three first attention images from the captured image. For example, when the captured image is an RGB image, the extracted first attention image is also an RGB image.

The first extraction unit 124 may extract the first attention image of a region having the same shape as the change region for each change region, or the frame having the same shape as the frame having a predetermined shape circumscribing the change region. The image of the area surrounded by may be extracted as the first attention image. The shape of the frame circumscribing the change region may be any shape such as a rectangle or an ellipse. Further, the first extraction unit 124 may extract an image of a region surrounded by a frame having a predetermined size larger than the frame circumscribing the change region as the first attention image.

The first extraction unit 124 supplies the extracted first attention image to the region change classification unit 126. The region on the captured image of the first attention image extracted by the first extraction unit 124 is also referred to as a first attention region. Further, the first extraction unit 124 acquires the position information of the first attention region, associates the position information with the imaging time, and supplies them to the first relevance generation unit 140. The position information of the first attention area may be, for example, the coordinate values of the four corners of the circumscribed rectangle of the first attention area, or may be expressed by the coordinate values of at least one of the four corners and the width and height of the circumscribed rectangle. May be good. Further, if the circumscribed rectangle is a circle, the position information of the first attention region may be, for example, the center coordinates of the circle and the radius of the circle. Further, if the circumscribing rectangle is an ellipse, the position information of the first attention region may be, for example, the center coordinates of the ellipse and the major axis and the minor axis of the ellipse.

The second extraction unit 125 receives the binary image which is the detection result from the foreground area tracking unit 123. Further, the second extraction unit 125 acquires the background information 131 used for generating the binary image from the first storage unit 130. The second extraction unit 125 may receive the background information 131 together with the binary image from the foreground area tracking unit 123.

The second extraction unit 125 extracts an image of the change region from the background information 131. Specifically, the second extraction unit 125 uses the background information 131 as the background image and the binary image to obtain an image of the region on the background information 131 corresponding to the region having a pixel value of 255 in the binary image. It is extracted as a second attention image. The method for extracting the second attention image is the same as the method for extracting the first attention image. The second extraction unit 125 supplies the extracted second attention image to the region change classification unit 126. The area on the background information 131 of the second attention image extracted by the second extraction unit 125 is also referred to as a second attention area.

The area change classification unit 126 classifies the changes related to the product shelf 3 in the change area, and supplies the classification result to the first association generation unit 140. The area change classification unit 126 is based on the first attention area and the second attention area supplied from the first extraction unit 124 and the second extraction unit 125, and the shelf change model 132 stored in the first storage unit 130. Then, the change from the state of the image in the region corresponding to the detected change region on the background image to the state of the image in the region corresponding to the change region on the captured image is classified.

The state of the image is, for example, the state where the image contains or does not include the product, the state where the image contains or does not include the customer, the state where the image contains or does not include the shopping basket, and the state where the image contains the shopping cart. The state of being included or not included, etc. Based on the shelf change model 132, the area change classification unit 126 changes the product shelf 3 in the change area, for example, "change due to the product shelf 3 not including the product" and "the product is stored in the product shelf 3". "Changes due to new inclusion", "Changes due to changes in the appearance of the products displayed on the product shelves 3", "Changes due to the presence of a person in front of the product shelves 3", "Changes due to the presence of a person in front of the product shelves 3", "Changes due to the presence of a person in front of the product shelves 3" Classify into types of changes, such as "changes due to having a shopping cart in front" and "changes due to changes in lighting". The type in which the region change classification unit 126 classifies changes in the state in the change region is an example, and is not limited thereto.

For example, "change due to the fact that the product is no longer included in the product shelf 3" can be said to be "change due to the product being taken", and "change due to the new inclusion of the product in the product shelf 3". Can also be said to be "changes due to the placement of products on the product shelves." Further, for example, "change due to change in lighting" can be said to be "change in environment of product shelves". Further, the area change classification unit 126 may classify the change of the product shelf into "change due to the change in the appearance of the product". Whether the "change in appearance due to the change in the appearance of the product shelf 3" is, for example, "change in appearance due to the arrangement of different products" or "change in appearance due to the change in the attitude of the product". Etc., and may be further classified.

More specifically, the region change classification unit 126 receives the first attention image from the first extraction unit 124. Further, the region change classification unit 126 receives the second attention image from the second extraction unit 125. Then, the area change classification unit 126 changes from the state of the second attention image to the state of the first attention image corresponding to the second attention image based on the shelf change model 132 stored in the first storage unit 130. Changes are classified, for example, into the types described above. In other words, the region change classification unit 126 classifies the change from the state of the second attention image to the state of the first attention image based on the result of comparison with the shelf change model 132.

FIG. 6 is an explanatory diagram showing an example of the classification result output by the area change classification unit 126. The region change classification unit 126 outputs, for example, the classification result 90 illustrated in FIG.

As illustrated in FIG. 6, the classification result 90 includes, for example, a second attention image 91, a first attention image 92, and a type of change 93. The classification result 90 shown in FIG. 6 is an example, and the classification result 90 may include information other than the information exemplified in FIG. The classification result 90 may include, for example, information regarding the captured image (identifier, imaging time, etc.), information indicating the position of the first attention image 92 in the captured image, and the like.

The area change classification unit 126 may classify the changes related to the product shelf 3 into any of the above-mentioned types by using, for example, a machine learning method (Convolutional Neural Network or the like) for which the shelf change model 132 is created.

The above is an example of the configuration of the first change detection unit 120.

The first relevance generation unit 140 receives the classification result of the change region and the position information of the change region from the first change detection unit 120. Further, the first relevance generation unit 140 acquires the person information 134 from the first storage unit 130. Then, the first relevance generation unit 140 corresponds to the change area based on the image pickup time of the change area corresponding to the position information of the change area and the image pickup time of the person associated with the person information 134. Generates relationship information between a product and a person, which shows the relationship between the product (change in the display state of the product) and the person. After that, the first relevance generation unit 140 supplies the relevance information between the generated product and the person to the relevance integration unit 150.

Specifically, the first relevance generation unit 140 extracts, for example, a person who intersects the change area from the persons who were imaged at a time before the image pickup time when the change area was detected. Then, the first relevance generation unit 140 associates the person imaged at the time closest to the imaging time of the changed area among the extracted persons with the changed area.

FIG. 7 is an explanatory diagram showing an example of relationship information between a product and a person generated by the first relationship generation unit 140. FIG. 7 exemplifies a camera ID indicating an ID of an image pickup device, a person ID indicating a person reflected in the image pickup device, position information of a change area of a product shelf, and a classification result of the change. In FIG. 7, the position information of the change area is represented by the coordinate value of one corner of the circumscribed rectangle of the change area and the width and height of the circumscribed rectangle. Further, as for the types of changes, "changes due to the acquisition of goods" are shown as "decrease in goods", and "changes due to the new inclusion of goods in the goods shelf 3" are shown as "increase in goods".

However, the relevance information may include the content itself classified by the area change classification unit 126. In that case, the shopping list update unit 22, which will be described later, determines the content for updating the shopping list based on the classification content.

The first relevance generation unit 140, for example, associates the generated relevance information of the person and the product with the person information 134 stored in the first storage unit 130, and supplies the relevance integration unit 150 to the relevance integration unit 150. You may. Further, the first relevance generation unit 140 may add information (identifier, imaging time, etc.) regarding the captured image to the relevance information.

The relevance integration unit 150 receives the relevance information between the product and the person from the first relevance generation unit 140. Then, when the relevance information of the same person exists in the received relevance information, the relevance integration unit 150 integrates them into one. After that, the relevance integration unit 150 supplies the integrated relevance information to the shopping list management device 20.

The relevance integration unit 150 calculates the similarity based on, for example, at least one of the color, area, shape, and aspect ratio of the circumscribed rectangle of the person area stored in the person information 134 of the first storage unit 130. Then, the relevance integration unit 150 determines that the person areas having the highest calculated similarity are the same person. As described above, the relevance integration unit 150 integrates the relevance information determined to be the same person.

FIG. 8 is an explanatory diagram showing an example in which the relationship information between the product and the person exemplified in FIG. 7 is integrated. In the example shown in FIG. 8, the relationship information of the person ID = 1 and the person ID = 4 illustrated in FIG. 7 is integrated into one. Further, the relationship information of the person ID = 2 and the person ID = 3 illustrated in FIG. 7 is also integrated into one. That is, in FIG. 8, when the person ID = 1 and the person ID = 4 exemplified in FIG. 7 are the same person, and the person ID = 2 and the person ID = 3 exemplified in FIG. 7 are the same person. An example is shown.

In the example shown in FIG. 8, as an example of integration, two person IDs are compared when the relevance information is integrated, and a person ID having a small value is adopted as the person ID of the relevance information after integration. However, a person ID with a large value may be adopted. Further, for example, after integrating the relevance information, the person ID may be reassigned. Further, the person ID used when the person identification device 50 described later identifies the person may be adopted, or the identification information for identifying the terminal 10 carried by the person may be adopted.

Further, the relevance integration unit 150 may detect that the goods have been returned to a place different from the place where the goods were taken, based on the relevance information. The relevance integration unit 150 may detect that the product has been returned to a place different from the place where the product was taken, for example, by comparing the place where the product was acquired and the place where the product was returned.

Hereinafter, the operation of the relevance integration unit 150 will be specifically described with reference to FIG. The relevance integration unit 150 compares the position information and the type of change for each person ID. In the example shown in FIG. 7, a person with a person ID = 1 acquires a product from the location (10, 0) on the product shelf reflected in the image pickup device with the camera ID = 1, and places the product at the location (250, 300). I'm putting it back. Further, the acquired product and the returned product have both the width and the height of the circumscribed rectangle (30, 50), and are judged to be the same product. From this, the relevance integration unit 150 detects that the person with the person ID = 1 has returned to a different place on the same shelf as the shelf from which the product was acquired. The person with the person ID = 2 is returned to the same place where the same product was acquired. Therefore, the relevance integration unit 150 does not detect that this behavior has been returned to a location different from where the goods were taken.

In this way, the relevance integration unit 150 detects that the product has been returned to a place different from the place where the product was taken, for example, by detecting that the same person has returned the product to a different place. In addition, the relevance integration unit 150 may detect that the product has been returned to a place different from the place where the product was taken, for example, even when the product has been returned to the same place but the appearance of the product has changed.

The above is an example of each configuration of the image processing apparatus 100.

The terminal 10 is a device carried by a customer, and is realized by, for example, a mobile terminal, a tablet terminal, or the like. The terminal 10 stores information for identifying a customer, and is used when a person identification device 50, which will be described later, associates the terminal 10 with a customer (person). The terminal 10 may display, for example, information identifying a customer as a label (bar code or the like), or may transmit the information by short-range wireless communication.

Further, the terminal 10 notifies the customer of various information in a mode (display, vibration, light, voice, etc.) that can be perceived by a person in response to a notification from the notification unit 23 described later. The specific content notified by the notification unit 23 will be described later.

The person identification device 50 is a device for identifying a person. In the present embodiment, the person identification device 50 does not need to identify the characteristics of the person itself (for example, gender, age, height, etc.), and may be distinguishable from other persons. For example, in the example shown in FIG. 8, it is sufficient that the person with the person ID = 1, the person with the ID = 2, and the person with the ID = 5 can be identified as different people. The person identification device 50 is provided at the entrance of a store, for example, to identify a person.

In the present embodiment, the person identification device 50 identifies a person based on a captured image. The method by which the person identification device 50 identifies a person is arbitrary. The person identification device 50 is, for example, information (color, area, shape, and circumscribing rectangle of a person area) used by the above-mentioned relationship integration unit 150 to calculate the similarity of a person from an image of a person who has entered the store. Aspect ratio, etc.) may be obtained. Then, the person identification device 50 may identify a person by using the acquired information.

Further, the person identification device 50 may associate the identified person with the device (terminal 10) carried by the person. Specifically, the person identification device 50 captures a person when the terminal 10 is detected by a sensor (not shown) provided at the entrance of the store, and the person and the terminal 10 are identified by the captured image. It may be associated with the identification information. In this case, the person identification device 50 is realized by a device including an image pickup device and a sensor. It should be noted that the image pickup device and the sensor may be realized by different hardware. Further, the device carried by a person is not limited to hardware such as a mobile phone, and may be a medium such as an IC card, for example.

Specifically, when the customer starts the application program installed on the terminal 10 to display the customer's identification information and the customer causes the person identification device 50 to identify the identification information, the person identification device 50 However, the person and the terminal 10 may be associated with each other.

FIG. 9 is a block diagram showing a configuration example of the shopping list management device 20. The shopping list management device 20 manages the shopping list for each person. The shopping list management device 20 includes a shopping list generation unit 21, a shopping list update unit 22, a notification unit 23, and a shopping list storage unit 24.

The shopping list storage unit 24 stores a shopping list for each person. The shopping list storage unit 24 may store the shopping list in association with the person ID described above, for example. Further, the shopping list storage unit 24 may store the shopping list in association with the identifier given by the person identification device 50 described above when the person is identified. Further, when the person and the terminal 10 are associated with each other, the shopping list storage unit 24 may store the person and the terminal 10 in association with the shopping list.

The shopping list generation unit 21 generates a shopping list and registers it in the shopping list storage unit 24. For example, when the person identification device 50 identifies a person, the shopping list generation unit 21 may generate a shopping list corresponding to the person. Further, the shopping list generation unit 21 may generate a shopping list corresponding to the person, for example, when the person identification device 50 associates the person with the terminal 10. In this case, since the shopping list is associated with the terminal 10 owned by the person, the notification unit 23, which will be described later, can notify the terminal 10 of the change that has occurred in the shopping list.

Further, when the self-registration system 1 does not include the person identification device 50 (that is, when the terminal 10 and the shopping list are not associated with each other), the shopping list generation unit 21 has the first association generation unit 140 as a person and a product. You may generate a shopping list when you generate the relevance information for. In this case, the shopping list generation unit 21 may integrate the shopping list when the relevance integration unit 150 integrates the relevance information. In this way, the shopping list is managed in association with each person.

The shopping list update unit 22 receives the integrated relevance information from the relevance integration unit 150, and updates the contents of the shopping list based on the relevance information integrated by the relevance integration unit 150. Specifically, the shopping list update unit 22 updates the contents of the shopping list corresponding to each person based on the classification of changes in the product shelves included in the relevance information and the shelf allocation information of the product shelves.

In the present embodiment, the relevance information includes the position information of the change area of the product shelf and the classification of the change of the product shelf. Therefore, the shopping list update unit 22 identifies the products for which the change in the display state is detected based on the shelf allocation information of the product shelves on which the products are arranged.

The shopping list update unit 22 may receive only the relevance information newly integrated by the relevance integration unit 150 (that is, the difference information), or may receive all of the integrated relevance information. When the difference information is received, the shopping list update unit 22 updates the already generated shopping list for the products included in the difference information. When all of the integrated relevance information is received, the shopping list update unit 22 updates all of the person's shopping list based on the received relevance information.

The shelving allocation information is information indicating the arrangement position of products prepared in advance for the product shelves of each store. The shelving allocation information is, for example, information in which a product shelf number (column) and a column number are associated with a product name arranged at the position of the number. Further, the position of the image pickup device 2 and the position of the product shelf are managed in association with each other in advance. Therefore, it is possible to specify the product by associating the area of the product shelf of the image captured by the image pickup device 2 with the shelf allocation information.

FIG. 10 is an explanatory diagram showing an example of shelf allocation information. In the example shown in FIG. 10, it is assumed that the image I1 is an image captured by a camera having a camera ID = 1 which is an image pickup device. Further, the shelf allocation information I2 exemplified in FIG. 10 represents products arranged in a range specified by columns and columns of product shelves. For example, it is assumed that the position information of the product I3 included in the image I1 matches the position information (that is, the position information (10, 0, 30, 50) in the first line of the relevance information exemplified in FIG. 8). In this case, the shopping list update unit 22 identifies the product I3 as the product a associated with the first column of the shelf allocation information I2.

Further, as described above, the relevance information includes the classification result of the change area of the product shelf. That is, it is possible to identify the classification of changes in the product shelves from the relevance information. Therefore, the shopping list update unit 22 identifies the products classified as the changes in the product shelves caused by the person picking up the products based on the shelf allocation information of the product shelves on which the products are arranged. Then, the shopping list update unit 22 performs a registration process for registering the specified product in the shopping list corresponding to the person. The change in the product shelves caused by the person picking up the product corresponds to the "change due to the product being picked up" in the above example, and corresponds to the "decrease in the product" in the type of change.

The shopping list update unit 22 performs a process of registering the product in the shopping list corresponding to the person as a registration process for registering the product in the shopping list. In addition, the shopping list update unit 22 may notify the terminal 10 of information that the product has been registered in the notification unit 23, which will be described later, as a registration process. At that time, the notification unit 23 may inquire the customer via the terminal 10 whether or not the correct product has been added to the shopping list.

Further, the shopping list update unit 22 specifies a product classified as a change in the product shelf due to the person returning the product based on the shelf allocation information of the product shelf on which the product is arranged. Then, the shopping list update unit 22 performs a deletion process for deleting the specified product from the shopping list corresponding to the person. The change in the product shelf caused by the person returning the product corresponds to the "change due to the product being placed on the product shelf" in the above example, and corresponds to the "product increase" in the type of change. ..

The shopping list update unit 22 performs a process of deleting the product from the shopping list corresponding to the person as a deletion process for deleting the product from the shopping list. Further, the shopping list updating unit 22 may notify the terminal 10 of the information that the product has been deleted to the notification unit 23, which will be described later, as a deletion process. At that time, the notification unit 23 may inquire the customer via the terminal 10 whether or not the correct product has been deleted from the shopping list.

In addition, the shopping list update unit 22 does not immediately delete the product from the shopping list, but includes a deletion flag in the shopping list that identifies the product returned to a place different from the place where it was taken as a deletion process. It may be set for the target product. Then, the shopping list update unit 22 may notify the terminal 10 of the information that the deletion flag is set in the notification unit 23, which will be described later. At that time, similarly, the notification unit 23 may inquire the customer via the terminal 10 whether or not the product for which the deletion flag is set is a product to be deleted. Then, the shopping list update unit 22 receives an instruction indicating whether or not to delete the product for which the deletion flag is set via the terminal 10 carried by the notified person, and receives an instruction to delete the product. If so, you may remove the item from your shopping list.

Further, the shopping list updating unit 22 may cause the notification unit 23 to send the shopping list to the terminal 10 every time the contents of the shopping list are updated.

The notification unit 23 notifies the terminal 10 of the information of the shopping list. As described above, the notification unit 23 may notify the terminal 10 that the product registration and deletion and the deletion flag have been set according to the processing of the shopping list update unit 22.

Further, the notification unit 23 may notify the payment device 40, which will be described later, to stop the payment processing when there is a product for which the deletion flag is set in the shopping list. By giving such a notification, it is possible to prevent the unapproved product from being settled.

The output device 30 outputs the contents of the shopping list. The output device 30 may be installed in the vicinity of the payment device 40, which will be described later, for example, and may output the contents of the shopping list at the time of payment processing. If the contents of the shopping list can be output to the terminal 10, the self-registration system 1 does not have to include the output device 30. The output device 30 may be, for example, a display device such as a display, or may be a POS (point of sales) terminal. Further, the output device 30 is not limited to these, and may be, for example, a speaker or a mobile terminal.

The payment device 40 performs payment processing based on the contents of the shopping list. For example, when the shopping list is associated with the terminal 10, the payment device 40 may notify the terminal 10 associated with the shopping list of the total amount and perform payment processing. It should be noted that a method in which the payment device 40 makes a payment via an individual terminal 10 is widely known, and detailed description thereof will be omitted here.

On the other hand, when the shopping list is not associated with the terminal 10, the payment device 40 causes the output device 30 to display the contents of the shopping list and the total amount of money, and accepts payment processing such as payment from a customer or card payment. good. Since payment methods based on deposits and card payments are also widely known, detailed explanations will be omitted here.

Further, when the payment device 40 receives a notification from the above-mentioned notification unit 23 that the payment process is stopped (specifically, a notification that a product for which the deletion flag is set remains), the payment device 40 receives the notification. You may stop the payment process based on the shopping list and display various alerts. The payment device 40 may, for example, notify that the deletion flag remains in the terminal 10. Further, the payment device 40 may prompt the customer for confirmation by displaying a product for which the deletion flag is set on the output device 30 or providing voice guidance, for example.

The image processing device 100 (more specifically, the first acquisition unit 110, the first change detection unit 120, the first association generation unit 140, and the association integration unit 150) is a CPU of a computer that operates according to a program. Realized by. For example, the program is stored in a storage unit (not shown) included in the image processing device 100, the CPU reads the program, and according to the program, the first acquisition unit 110, the first change detection unit 120, and the first storage unit. It may operate as 130, the first association generation unit 140, and the association integration unit 150.

Further, the first acquisition unit 110, the first change detection unit 120, the first storage unit 130, the first association generation unit 140, and the association integration unit 150 included in the image processing device 100 are each dedicated hardware. It may be realized.

Further, the shopping list management device 20 (more specifically, the shopping list generation unit 21, the shopping list update unit 22, and the notification unit 23) is also realized by the CPU of the computer that operates according to the program. For example, the program is stored in a storage unit (not shown) included in the shopping list management device 20, and the CPU reads the program and, according to the program, serves as a shopping list generation unit 21, a shopping list update unit 22, and a notification unit 23. It may work. Further, the shopping list generation unit 21, the shopping list update unit 22, and the notification unit 23 included in the shopping list management device 20 may be realized by dedicated hardware, respectively.

Next, the operation of the image processing apparatus 100 of the present embodiment will be described with reference to FIG. FIG. 11 is a flowchart showing an operation example of the image processing apparatus 100 of the present embodiment.

The first acquisition unit 110 acquires an captured image which is an RGB image from the video signal obtained by capturing the image of the product shelf 3 (step S1001). The first acquisition unit 110 supplies the acquired captured image to the first change detection unit 120.

The foreground region detection unit 121 of the first change detection unit 120 captures an captured image which is an RGB image supplied from the first acquisition unit 110 and background information 131 which is an RGB image stored in the first storage unit 130. The region changed between the two RGB images is detected as a change region (foreground region) (step S1002). Then, the foreground area detection unit 121 supplies the detection result of the change area to the background information update unit 122 and the foreground area tracking unit 123. The foreground area detection unit 121 generates, for example, a binary image in which the detected pixel of the change area is 255 and the pixel of the other area is 0, and the binary image is used as the detection result of the change area to update the background information. It supplies to the unit 122 and the foreground area tracking unit 123.

Further, the foreground region detection unit 121 stores the foreground information 133 in the first storage unit 130 (step S1003). As described above, the foreground information 133 is a detection result associated with the imaging time.

The foreground area tracking unit 123 tracks the change area based on the detection result supplied from the foreground area detection unit 121 and the foreground information 133 (step S1004). The foreground region tracking unit 123 supplies a binary image showing a change region traced for a predetermined time or longer to the first extraction unit 124 and the second extraction unit 125. The foreground area tracking unit 123 supplies an update signal indicating the update of the background information 131 to the background information update unit 122.

As a result of tracking, the foreground area tracking unit 123 determines that the object included in the change area is a moving object when the movement amount of the change area is equal to or more than a predetermined threshold value, and uses the determined change area as the person area. Extract. After that, the foreground area tracking unit 123 associates predetermined information with the person area and stores it as the person information 134 in the first storage unit.

The background information update unit 122 is supplied from the captured image supplied from the first acquisition unit 110, the detection result of the change region supplied from the foreground region detection unit 121, the background information 131, and the foreground region tracking unit 123. The background information 131 is updated based on the update signal (step S1005). Note that step S1005 may be performed at any timing after step S1004.

The first extraction unit 124 determines the detection result on the captured image based on the captured image supplied from the first acquisition unit 110 and the detection result related to the captured image supplied from the foreground region tracking unit 123. The image of the region (first attention region) corresponding to the indicated change region is extracted as the first attention image (step S1006). The first extraction unit 124 supplies the extracted first attention image to the region change classification unit 126.

The second extraction unit 125 is a first extraction unit based on the detection result supplied from the foreground area tracking unit 123 and the background information 131 acquired from the first storage unit 130 and used to obtain the detection result. The second attention image is extracted from the background information 131 by the same operation as 124 (step S1007). The second extraction unit 125 supplies the extracted second attention image to the region change classification unit 126. In addition, step S1006 and step S1007 may be performed at the same time, or may be performed in the reverse order.

The area change classification unit 126 includes a first attention image supplied from the first extraction unit 124, a second attention image supplied from the second extraction unit 125, and a shelf change model 132 stored in the first storage unit 130. Based on the above, the changes related to the product shelf 3 are classified (step S1008). Specifically, the change regarding the product shelf 3 is a change from the state in the second attention image to the state in the first attention image.

The first relevance generation unit 140 receives the classification result of the change region and the position information of the change region from the region change classification unit 126 of the first change detection unit 120. Further, the first relevance generation unit 140 acquires the person information 134 from the first storage unit 130. Then, the first relevance generation unit 140 extracts a person who intersects the change area from the persons imaged at a time before the image pickup time when the change area is detected. After that, the first relevance generation unit 140 associates the person imaged at the time closest to the imaging time of the changed area among the extracted persons with the changed area (step S1009). As a result, the first relevance generation unit 140 generates relevance information.

The relevance integration unit 150 receives the relevance information of the product and the person from the first relevance generation unit 140, and if the relevance information of the same person exists, integrates them into one. The relevance integration unit 150 calculates the similarity based on, for example, at least one of the color, area, shape, and aspect ratio of the circumscribed rectangle of the person area stored in the person information 134 of the first storage unit 130. Then, the relevance integration unit 150 determines that the person areas having the highest calculated similarity are the same person. After that, the relevance integration unit 150 integrates the relevance information including the person determined to be the same person into one (step S1010).

The image processing device 100 determines whether or not the first acquisition unit 110 has received the next video signal (whether or not there is the next captured image) (step S1011). If there is a next captured image (YES in step S1011), the process proceeds to step S1001. On the other hand, when there is no next captured image (NO in step S1011), the image processing apparatus 100 ends the operation.

Next, the operation of the self-registration system 1 of the present embodiment will be described with reference to FIG. FIG. 12 is a flowchart showing an operation example of the self-registration system 1 of the present embodiment. First, when a customer (person) visits a store, the person identification device 50 identifies the person and associates the person with the terminal 10 (step S1101). Further, the shopping list generation unit 21 generates a shopping list corresponding to the person (step S1102). After the customer enters the store, the image pickup device 2 acquires the captured image (step S1103). The image processing device 100 detects a change region of the product shelf based on the captured image (step S1104). Then, the image processing apparatus 100 classifies the change of the product shelf in the detected change region (step S1105). The image processing apparatus 100 updates the shopping list corresponding to the person based on the classification of changes in the product shelves and the shelf allocation information of the product shelves (step S1106).

After that, the payment device 40 performs payment processing based on the contents of the shopping list (step S1107).

As described above, in the present embodiment, the first change detection unit 120 (more specifically, the foreground area detection unit 121) detects the change area of the product shelf from the image taken by the product shelf. In addition, the first change detection unit 120 (more specifically, the area change classification unit 126) classifies the changes in the product shelves in the detected change area. Then, the shopping list update unit 22 generates a customer's shopping list based on the classification of changes in the product shelves and the shelf allocation information of the product shelves. Therefore, it is possible to appropriately manage the products purchased by the customer without preparing images of the individual products in advance.

Further, for example, in the generally known background subtraction method, other information can be detected only by defining the information that suppresses detection in the background, so it is necessary to collect the reference image of the object in advance. do not have. However, there are multiple changes in the product shelves that are detected as image differences. Therefore, when only the general background subtraction method is used, it is not possible to accurately recognize changes in the product shelves.

For example, "the product was taken", "the product was placed (replenished)", "the product was picked up and then returned", "the appearance and position changed", "the lighting changed", etc. Are both detected as changes in the product shelves. However, the change due to "the goods were taken" indicates that the goods have decreased from the goods shelves, and "the goods have been placed (replenished)" indicates that the goods have increased on the goods shelves. In addition, changes due to "changes in appearance and position" and "changes in lighting" do not mean that the number of products has increased or decreased from the product shelves.

Therefore, there is a problem that the type of change in the product shelf detected as an image difference cannot be specified and the display state of the product cannot be accurately grasped only by using a general technique.

However, in the present embodiment, the area change classification unit 126 classifies the change of the product shelf in the detected change area. Specifically, the area change classification unit 126 classifies the change of the product shelf from the image of the change area by using the shelf change model 132 that models the change of the product shelf 3 before and after the change of time. .. This shelf change model 132 classifies changes due to the product being taken, changes in the store environment of the product shelf, changes due to the product being placed on the product shelf, changes due to the change in the appearance of the product, and the like. Will be possible. As a result, the image processing device 100 can more accurately determine the state of the product shelf 3, such as whether the product is in the picked state or the product shelf 3 is in the replenished state.

Further, the image processing apparatus 100 of the present embodiment can detect that the product has been returned to a place different from the place where the product was taken, as described above. For example, if a refrigerated product is returned to a place different from the place where the product was taken, such as returning it to a normal temperature shelf, a loss of sales opportunity or a loss of product disposal will occur, which will greatly affect the sales of the store. Therefore, when such a situation occurs, it is preferable to promptly carry out product management work to solve it. By using the image processing device 100 of the present embodiment, it is possible to reduce the occurrence of sales opportunity loss, product disposal loss, etc. due to the product being returned to a place different from the place where the product was taken.

In the present embodiment, the case where the image pickup device 2 takes an image of the product shelf 3 has been described. However, the target to be imaged by the image pickup device 2 is not limited to the product shelf 3. The image pickup device 2 may, for example, take an image of a product stacked on a wagon. That is, the image captured by the image pickup device 2 may be an image captured by capturing the product stacked on the wagon. The image processing device 100 may detect a change region by comparing an captured image of a product stacked on a wagon with a background image. As described above, the image processing device 100 is not limited to the product shelves displayed so that all the faces of the products can be seen, and can use captured images obtained by capturing images of the products displayed by various display methods.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the second embodiment, a self-registration system 1 for acquiring different types of images by a plurality of image pickup devices 2 and monitoring the product shelves 3 will be described. In this embodiment, a case where an RGB image and a distance image are acquired by a plurality of image pickup devices 2 will be described. The image pickup apparatus 2 may be, for example, an RGB camera for acquiring an RGB image and a depth camera for acquiring a distance image.

In this case, the RGB camera and the depth camera are provided at close positions and image the same object (product shelf 3). Further, the RGB camera and the depth camera are time-synchronized, and it is preferable to take an image of the product shelf 3 at substantially the same time. That is, the depth camera is preferably a camera that outputs a distance image in which the imaging range of the RGB image is captured within a predetermined time from the shooting time of the RGB image captured by the RGB camera. Further, the image pickup apparatus 2 may be a sensor capable of acquiring a plurality of types of images (for example, RGB images and distance images). The image pickup apparatus 2 may be, for example, an RGBD camera.

The image processing device 200 in the present embodiment is the same as the image processing device 100 shown in FIG. 1 described in the first embodiment, that is, the terminal 10, the image pickup device 2, the shopping list management device 20, the output device 30, and the payment device 40. And is communicably connected to the person identification device 50. The terminal 10, the shopping list management device 20, the output device 30, the payment device 40, and the person identification device 50 of this embodiment are the same as those of the first embodiment.

FIG. 13 is an explanatory diagram showing a configuration example of the image processing apparatus 200. The image processing device 200 of the present embodiment is related to the first acquisition unit 110, the second acquisition unit 210, the second change detection unit 220, the second storage unit 230, and the second relevance generation unit 240. Including the integration unit 150. As described above, the image processing device 200 includes the second acquisition unit 210 in addition to the configuration of the image processing device 100. Further, instead of the first change detection unit 120, the first storage unit 130, and the first association generation unit 140 of the image processing apparatus 100, the second change detection unit 220, the second storage unit 230, and the second association generation unit 140 are replaced. Including part 240.

In the above, the same reference numerals are given to the elements having the same functions as the elements included in the drawings described in the first embodiment described above. Hereinafter, a configuration characteristic of the present embodiment will be described.

FIG. 14 is a block diagram showing a configuration example of the second change detection unit 220 and the second storage unit 230. As illustrated in FIG. 14, the second change detection unit 220 includes a foreground area detection unit 121, a background information update unit 122, a foreground area tracking unit 123, a first extraction unit 124, and a second extraction unit 125. It has a region change classification unit 126 and a distance information update unit 127. Further, the second storage unit 230 stores the background information 131, the shelf change model 132, the foreground information 133, the person information 134, and the distance information 135.

The first acquisition unit 110 acquires an captured image which is an RGB image, as in the first embodiment. Hereinafter, the captured image which is the RGB image acquired by the first acquisition unit 110 is referred to as the first captured image.

Similar to the first acquisition unit 110, the second acquisition unit 210 receives a video signal indicating the captured image obtained by the image pickup device 2 capturing the product shelf 3, and acquires a distance image from the video signal. The second acquisition unit 210 receives a video signal of a type different from the video signal acquired by the first acquisition unit 110. For example, when the video signal acquired by the first acquisition unit 110 is a video signal representing an RGB image, the second acquisition unit 210 acquires a video signal representing a distance image. Here, the distance image may mean, for example, an image having a value of the distance from the image pickup apparatus 2 to the object. Further, each pixel of the distance image may be represented by a value of 0 to 255, for example. At this time, the value of each pixel (that is, the distance value) may approach 0 if the object is close to the image pickup apparatus 2, and may approach 255 if the object is far away. The value of each pixel of the distance image is not limited to this. In the present embodiment, the captured image acquired by the second acquisition unit 210 will be described as a grayscale distance image.

The second acquisition unit 210, like the first acquisition unit 110, is an image converted based on an image captured inside the image pickup device 2 or stored in a storage device different from the image pickup device 2 and the image processing device 100. You may get the signal. Further, when the image processing device 100 is built in the image pickup device 2, the second acquisition unit 210 may acquire the captured image itself.

The second acquisition unit 210 converts the acquired video signal into a distance image representing the video signal, and supplies the distance image to the first extraction unit 124, the second extraction unit 125, and the distance information update unit 127. The distance image obtained by converting the video signal by the second acquisition unit 210 or the captured image acquired from the image pickup device 2 is hereinafter referred to as a second captured image.

The first acquisition unit 110 and the second acquisition unit 210 may be integrally formed. Further, the first captured image and the second captured image are related to each other based on the information indicating the captured position and the imaging time.

The second storage unit 230 of the present embodiment stores the distance information 135. The distance information 135 will be described later.

The first extraction unit 124 extracts an image of the change region from the second captured image. Specifically, the first extraction unit 124 uses a second captured image which is a distance image supplied from the second acquisition unit 210 and a binary image which is a detection result supplied from the foreground region detection unit 121. Then, the image of the region on the second captured image corresponding to the region having the pixel value of 255 in the binary image is extracted as the first attention image. The first extraction unit 124 may extract the first attention image by the same method as in the first embodiment. Then, the first extraction unit 124 supplies the extracted first attention image to the region change classification unit 126.

The second extraction unit 125 is a distance image before the time when the distance image, which is the second captured image associated with the first captured image used by the foreground region detection unit 121 to generate the binary image, is captured. The image of the change area is extracted from. Specifically, the second extraction unit 125 receives the binary image which is the detection result from the foreground region detection unit 121. Further, the second extraction unit 125 acquires the distance information 135, which is the second captured image captured before the imaging time of the first captured image used for generating the binary image, from the second storage unit 230. .. The distance information 135 is a second captured image updated by the second acquisition unit 210, which will be described later, and is a distance image acquired by the second acquisition unit 210. The distance information 135 is associated with the imaging time. As described above, since the first captured image and the second captured image are time-synchronized, the captured time of the first captured image and the captured time of the second captured image associated with the first captured image Will be about the same. Therefore, the second extraction unit 125 pays second attention to the image of the change region from the (past) second captured image captured before the second captured image of the object to be extracted by the first extraction unit 124. It can be said that it is extracted as an image.

The second extraction unit 125 extracts the second attention image by the same method as the method in which the first extraction unit 124 extracts the first attention image. The second extraction unit 125 supplies the extracted second attention image to the region change classification unit 126.

The distance information updating unit 127 updates the distance information 135 based on the distance image supplied from the second acquisition unit 210 and the distance information 135 stored in the second storage unit 230. The distance information updating unit 127 may update the distance information 135 by the same operation as the background information updating unit 122, for example.

The area change classification unit 126 classifies changes regarding the product shelf 3 in the change area based on the distance information 135 in the change area. First, the first attention image is received from the first extraction unit 124. Further, the region change classification unit 126 receives the second attention image from the second extraction unit 125. The area change classification unit 126 classifies changes regarding the product shelf 3 in the change area based on the first attention image and the second attention image.

The region change classification unit 126 is based on, for example, a calculation result obtained by subtracting the value (distance value) of each pixel in the second attention image from the value (distance value) of each pixel in the first attention image. Changes may be categorized. For example, when the calculation result is a value equal to or higher than the first predetermined threshold value, that is, when the object included in the first attention image is deeper than the object included in the second attention image, the region change classification unit. 126 may classify the change regarding the product shelf 3 in the change area as "change due to the product being no longer included in the product shelf 3" ("change due to the product being taken"). Further, for example, when the calculation result is a value equal to or less than the second predetermined threshold value, that is, the object included in the first attention image is closer to the image pickup device 2 than the object included in the second attention image. In some cases, the area change classification unit 126 may classify the change regarding the product shelf 3 in the change area as "change due to the new inclusion of the product in the product shelf 3". In other cases, the area change classification unit 126 describes the change regarding the product shelf 3 in the change area as "a change due to a change in the appearance of the product displayed on the product shelf 3" or "a change due to a change in lighting". ] Etc. may be classified.

Further, the region change classification unit 126 clusters, for example, the distance value of each pixel in the first attention image and the distance value of each pixel in the second attention image, and sets the distance value of the class having the largest number of elements to each attention. A distance value representing an image may be used, and the coordinates of clusters of each class of attention images may be obtained. Then, in the region change classification unit 126, for example, the absolute value of the difference between the distance values representing each attention image is within the third predetermined threshold value, and the coordinates of the cluster of the class of each attention image are the fourth. When the product is separated by a predetermined threshold value or more, the change regarding the product shelf 3 in the change area may be classified as "change due to a change in the appearance of the product displayed on the product shelf 3".

Further, the area change classification unit 126 uses, for example, the result of subtracting the distance value representing the first attention image from the preset distance value from the image pickup device 2 to the product shelf 3, and the product shelf in the change area. The changes relating to 3 may be classified into types related to changes in the products displayed on the product shelves 3 or types related to changes due to objects other than the products displayed on the product shelves 3. For example, if the above result is a positive value, that is, if the object included in the image of the change region portion of the captured image exists between the image pickup device 2 and the product shelf 3, the region change classification unit. 126 may classify the change regarding the product shelf 3 in the change area into the type related to "change due to an object other than the product displayed on the product shelf 3". The types related to "changes due to objects other than the products displayed on the product shelves 3" are, for example, "changes due to the presence of a person in front of the product shelves 3" and "there is a shopping cart in front of the product shelves 3". It is at least one of "change due to" and "change due to having a person and a shopping basket in front of the product shelf 3". Further, if the above result is other than a positive value, the area change classification unit 126 may classify the change regarding the product shelf 3 in the change area into the type related to "change of the product displayed on the product shelf 3". good. As described above, the region change classification unit 126 can reduce the processing cost required for the classification process by performing the classification process using the first attention image and the preset distance value.

The image processing unit 200 (more specifically, the first acquisition unit 110, the second acquisition unit 210, the second change detection unit 220, the second relevance generation unit 240, and the relevance integration unit 150). Is realized by the CPU of the computer that operates according to the program.

The above is an example of the configuration characteristic of the image processing apparatus 200.

Next, the operation of the image processing apparatus 200 of the present embodiment will be described with reference to FIG. FIG. 15 is a flowchart showing an operation example of the image processing apparatus 200 of the present embodiment.

The first acquisition unit 110 acquires the first captured image, which is an RGB image, from the video signal obtained by capturing the image of the product shelf 3 (step S2001). In addition, the second acquisition unit 210 acquires a second captured image, which is a distance image, from the video signal captured by the product shelf 3 (step S2002). The imaging time of the second captured image may be within a predetermined time from the imaging time of the first captured image, and the imaging range of the second captured image may be the imaging range of the first captured image. Further, the timing at which the second acquisition unit 210 acquires the second captured image may be before step S2005.

The foreground area detection unit 121 detects the change area (step S2003), and the background information update unit 122 updates the background information 131 (step S2004). The first extraction unit 124 is indicated by the detection result on the second captured image based on the second captured image supplied from the second acquisition unit 210 and the detection result supplied from the foreground region detection unit 121. The image of the region (first attention region) corresponding to the change region is extracted as the first attention image (step S2005). The first extraction unit 124 supplies the extracted first attention image to the region change classification unit 126.

Further, the second extraction unit 125 is before the detection result supplied from the foreground region detection unit 121 and the imaging time of the second captured image supplied to the first extraction unit 124 acquired from the second storage unit 230. Based on the distance information 135 indicating the second captured image captured in the above, the second attention image is extracted from the distance information 135 by the same operation as that of the first extraction unit 124 (step S2006). The second extraction unit 125 supplies the extracted second attention image to the region change classification unit 126. In addition, step S2005 and step S2006 may be performed at the same time, or may be performed in the reverse order. Then, the area change classification unit 126 classifies the change regarding the product shelf 3 in the change area based on the comparison result between the value of each pixel of the first attention image and the value of each pixel of the second attention image (step S2007). ). Further, the distance information updating unit 127 updates the distance information 135 based on the second captured image supplied from the second acquisition unit 210 and the distance information 135 (step S2008).

After that, the image processing apparatus 200 performs the same processing as the processing from step S1009 to step S1010 exemplified in FIG. 11, so that the second association generation unit 240 generates the association (step S2009), and the association is integrated. Unit 150 integrates the relevance information (step S2010).

The image processing device 200 determines whether or not the first acquisition unit 110 and the second acquisition unit 210 have received the next video signal (whether or not there is the next captured image) (step S2011). If there is a next captured image (YES in step S2011), the process proceeds to step S2001. On the other hand, when there is no next captured image (NO in step S2011), the image processing apparatus 200 ends the operation.

As described above, in the present embodiment, the foreground region detection unit 121 arranges the product as the captured image in which the product is captured, based on the RGB image and the distance image obtained by capturing the imaging range of the RGB image. Detect the changing area of the product shelf. Therefore, since changes in the product shelves can be determined more accurately, the products purchased by the customer can be managed more appropriately.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. In the third embodiment, in addition to the configuration of the image processing apparatus 100 described in the first embodiment, the image processing apparatus 300 having a configuration for generating customer flow line data will be described. As will be described later, the image processing apparatus 300 generates relevance information using the generated flow line data. As a result, the customer's flow line data can be acquired with higher accuracy, and it becomes possible to appropriately manage the products purchased by the customer.

FIG. 16 is an explanatory diagram showing a configuration example of a third embodiment of the self-registration system according to the present invention. The self-registration system 4 of the present embodiment includes an image pickup device 2 that captures an image of a passage in a store in the same manner as a general surveillance camera, in addition to an image pickup device 2 that captures an image of a product shelf 3. Further, the image pickup device 5 is communicably connected to the terminal 10, the shopping list management device 20, the output device 30, the payment device 40, the person identification device 50, and the image processing device 300 via the network N, similarly to the image pickup device 2. Has been done. The terminal 10, the image pickup device 2, the shopping list management device 20, the output device 30, the payment device 40, and the person identification device 50 of the present embodiment are also the same as those of the first embodiment.

The image pickup device 5 is, for example, a surveillance camera installed in a store or the like. The image pickup device 5 is installed, for example, at a position where each passage in the store can be imaged. The configuration of the image pickup device 5 may be the same as that of the image pickup device 2.

The self-registration system 4 may have one image pickup device 5 or a plurality of image pickup devices 5. That is, the self-registration system 4 may image each aisle in the store with one image pickup device 5, or may image each aisle in the store with a plurality of image pickup devices 5. Further, in the present embodiment, a case where the self-registration system 4 has an image pickup device 2 and another image pickup device 5 will be described. However, the self-registration system 4 may acquire the customer's flow line data based on the captured images captured by the plurality of image pickup devices 2, for example. That is, the self-registration system 4 may acquire the customer's flow line data based on the captured images captured by the plurality of image pickup devices 2 that image the product shelves 3.

The image processing device 300 generates flow line data indicating a movement route in the customer's store from the RGB image taken by the image pickup device 5. Then, the image processing device 300 generates the relationship information between the product and the person based on the generated flow line data and the change area of the product shelf 3.

FIG. 17 is a block diagram showing a configuration example of the image processing device 300. As shown in FIG. 17, the image processing apparatus 300 of the present embodiment has a first acquisition unit 110, a first change detection unit 120, a first storage unit 130, a first relevance generation unit 340, and a third acquisition unit. A unit 310, a flow line data generation unit 320, and a relevance integration unit 150 are included. As described above, the image processing apparatus 300 of the present embodiment includes the first association generation unit 340 instead of the first association generation unit 140 of the image processing apparatus 100. Further, the image processing device 300 includes a third acquisition unit 310 and a flow line data generation unit 320 in addition to the configuration of the image processing device 100.

In the above, the same reference numerals are given to the elements having the same functions as the elements included in the drawings described in the first embodiment and the second embodiment described above. Hereinafter, a configuration characteristic of the present embodiment will be described.

The third acquisition unit 310 acquires an RGB image from the image pickup apparatus 5 by the same operation as the first acquisition unit 110, and supplies the RGB image to the flow line data generation unit 320. The RGB image acquired by the third acquisition unit 310 is an image of the aisle of the store, similar to a general surveillance camera.

The flow line data generation unit 320 generates the flow line data of a person in the store by using the RGB images captured by at least one image pickup device 5. The flow line data generated by the flow line data generation unit 320 is information including, for example, a person ID for identifying a customer and a product shelf ID which is an identifier of the product shelf 3 visited by the customer. The method by which the flow line data generation unit 320 generates the flow line data is not particularly limited. The flow line data generation unit 320 may generate flow line data by using, for example, the method described in Patent Document 2. That is, for example, the movement line data generation unit 320 detects a person who can obtain the same data by performing face recognition of the person as the same person, or extracts information indicating customer characteristics such as clothes to obtain the same person. By detecting it, the customer in the captured image is identified. Further, in the flow line data generation unit 320, the customer stays in front of the product shelf 3 for a certain period of time, the distance between the customer and the product shelf 3 becomes equal to or less than a predetermined distance, and the customer can use the product shelf. Judge that you have visited 3. Then, the flow line data generation unit 320 generates the flow line data by associating the person ID for identifying the customer with the product shelf ID visited by the customer. The flow line data generation unit 320 may generate flow line data by using a method other than those exemplified above.

FIG. 18 is an explanatory diagram showing an example of flow line data generated by the flow line data generation unit 320. The flow line data shown in FIG. 18 includes a person ID indicating a person, a product shelf ID indicating a product shelf visited by the person, and a camera ID indicating an image pickup device that images the product shelf. In the example shown in FIG. 18, it is shown that three people have visited the product shelf A, and it is shown that a person with a person ID = 1 and a person with a person ID = 2 have visited the product shelf A twice. In the example shown in FIG. 18, the time when the person visited the product shelf is not specified, but the flow line data may include the time when the person visited each product shelf. In the example shown in FIG. 18, it is shown that the product shelves A are visited in order from the top row to the oldest.

The flow line data generation unit 320 generates, for example, the flow line data as shown in FIG. Then, the flow line data generation unit 320 associates the generated flow line data with the time of the captured image of the flow line data, and supplies the generated flow line data to the first relationship generation unit 340. In other words, the flow line data generation unit 320 includes the time when each person in the flow line data visits each product shelf in the flow line data and supplies it to the first relevance generation unit 340.

The first relevance generation unit 340 receives the classification result of the change region and the position information of the change region from the first change detection unit 120. Further, the first relevance generation unit 340 acquires the flow line data from the flow line data generation unit 320. Then, the first relevance generation unit 340 is the product and the person corresponding to the change area based on the image pickup time of the change area linked to the position information of the change area and the image pickup time linked to the flow line data. Generate relevance information. After that, the first relevance generation unit 340 supplies the relevance information between the generated product and the person to the relevance integration unit 150.

Specifically, the first relevance generation unit 340 puts the change region on the product shelf at the time closest to the imaging time of the change region from the flow line data generated at the time before the imaging time when the change region was detected. Associate the visitor with the area of change.

FIG. 19 is an explanatory diagram showing an example of relationship information between a product and a person generated by the first relationship generation unit 340. FIG. 19 shows a camera ID indicating an ID of an image pickup device, a person ID indicating a person reflected in the image pickup device, a product shelf ID indicating a product shelf, position information of a change area of the product shelf, and a change thereof. The classification result of is illustrated.

In this way, the first relevance generation unit 340 generates relevance information by using the flow line data instead of the person information 134. The first relevance generation unit 340 may also use the person information 134 when generating the relevance information.

The relevance integration unit 150 receives the relationship between the product and the person from the first relevance generation unit 340, and if there is a relationship of the same person, integrates them into one. Then, the relevance integration unit 150 supplies the integrated relevance to the shopping list management device 20.

Specifically, the relevance integration unit 150 integrates the relevance of the same person based on the person ID. FIG. 20 is an explanatory diagram showing an example in which the relationship between the product illustrated in FIG. 19 and the person is integrated. In the example shown in FIG. 20, the relevance information of the person with the person ID = 1 in FIG. 19 is integrated into one, and the relevance information of the person with the person ID = 2 is also integrated into one.

The above is an example of the configuration characteristic of the image processing apparatus 300.

Next, the operation of the image processing apparatus 300 of the present embodiment will be described with reference to FIG. 21. FIG. 21 is a flowchart showing an operation example of the image processing apparatus 300 of the present embodiment.

Referring to FIG. 21, the area change classification unit 126 classifies changes by performing the same processing as the processing from step S1001 to step S1008 shown in FIG. 11 by the image processing apparatus 300 (step S3001).

The third acquisition unit 310 acquires an captured image (third captured image) which is an RGB image from the video signal transmitted from the image pickup device 5 that images the aisle of the store (step S3002). The third acquisition unit 310 supplies the acquired captured image to the flow line data generation unit 320.

The flow line data generation unit 320 generates flow line data of a person in a store using RGB images captured by at least one image pickup device (step S3003). Then, the flow line data generation unit 320 associates the generated flow line data with the time of the captured image of the flow line data and supplies it to the first relevance generation unit 340.

The processes of steps S3002 and S3003 may be executed in the same order as the processes of step S3001.

The first relevance generation unit 340 receives the classification result of the change region and the position information of the change region from the first change detection unit 120. Further, the first relevance generation unit 340 acquires the flow line data from the flow line data generation unit 320. Then, the first relevance generation unit 340 is a product and a person corresponding to the change area based on the image pickup time of the change area linked to the position information of the change area and the image pickup time linked to the flow line data. Relevance information is generated (step S3004). Specifically, the first relevance generation unit 340 puts the change region on the product shelf at the time closest to the imaging time of the change region from the flow line data generated at the time before the imaging time when the change region was detected. Associate the visitor with the area of change. Then, the first relevance generation unit 340 supplies the relevance information between the generated product and the person to the relevance integration unit 150.

After that, the image processing apparatus 300 performs the processes from step S3005 to step S3006, which are the same processes as those from step S1010 to step S1011 shown in FIG.

As described above, the image processing apparatus 300 of the present embodiment includes the third acquisition unit 310 and the flow line data generation unit 320. With such a configuration, the flow line data generation unit 320 generates the flow line data based on the RGB image supplied from the third acquisition unit 310. Then, the image processing device 300 is based on the classification result of classifying the change region detected from the RGB image of the product shelf 3 and the change region, and the flow line data generated from the RGB image of the passage of the store. , Generates product-person relevance information. In the captured image of the aisle of the store used for generating the flow line data, which is captured by the imaging device 5, the whole body of the person is captured. Therefore, it is easier to identify the person in the captured image than the RGB image captured by the product shelf 3. That is, it is expected that when the relevance information is generated using the flow line data as in the image processing device 300, the relevance information is generated with higher accuracy than when the relevance information is generated using the person information 134. can. Therefore, the image processing device 300 according to the present embodiment can integrate the relevance information with higher accuracy than the image processing device 100 of the first embodiment. As a result, the customer's flow line data can be acquired with higher accuracy, and it becomes possible to appropriately manage the products purchased by the customer.

Next, a modified example of the third embodiment will be described. In the third embodiment, the image pickup device 5 images the aisle in the store and generates the customer's flow line data. However, the method of generating the flow line data is not limited to the method of using an image. FIG. 22 is a block diagram showing a modified example of the self-registration system of the third embodiment. The self-registration system 4a exemplified in FIG. 22 includes a flow line detection device 6 for detecting a customer's flow line instead of the image pickup device 5 included in the self-registration system 4 of the third embodiment. Further, the self-registration system 4a exemplified in FIG. 22 includes an image processing device 300a instead of the image processing device 300 included in the self-registration system 4 of the third embodiment.

The flow line detection device 6 is a device that detects the position of the terminal 10 carried by the customer. The flow line detection device 6 detects, for example, the short-range radio from the terminal 10 to detect the position of the customer. For example, when the person identification device 50 authenticates the terminal 10 at a predetermined place (for example, an entrance) in the store, the flow line detection device 6 tracks the authenticated person and detects the flow line of the person. do. The shopping list generation unit 21 may generate a shopping list corresponding to a person who carries the authenticated terminal 10 at the timing when this authentication is performed.

FIG. 23 is a block diagram showing a configuration example of the image processing device 300a. As shown in FIG. 23, the image processing apparatus 300a of the present embodiment includes a first acquisition unit 110, a first change detection unit 120, a first storage unit 130, a third acquisition unit 310a, and a flow line data generation unit. It includes 320a, a first association generation unit 340, and an association integration unit 150. As described above, the image processing device 300a of the present modification includes the third acquisition unit 310a and the flow line data generation unit 320a in place of the third acquisition unit 310 and the flow line data generation unit 320 of the image processing device 300.

The third acquisition unit 310a acquires the customer's position information from the flow line detection device 6. The customer's location information includes, for example, the identification information of the terminal 10. The flow line data generation unit 320a generates the flow line data of a person in the store. The flow line data generated by the flow line data generation unit 320a is information including, for example, identification information of the terminal 10 carried by the customer and a product shelf ID which is an identifier of the product shelf 3 visited by the customer. The method by which the flow line data generation unit 320a generates the flow line data is not particularly limited. Similar to the third embodiment, the flow line data generation unit 320a is, for example, such that the customer stays in front of the product shelf 3 for a certain period of time, the distance between the customer and the product shelf 3 becomes a predetermined distance or less, and the like. By the method, it is determined that the customer has visited the product shelf 3. Then, the flow line data generation unit 320a generates the flow line data by associating the identification information of the terminal 10 carried by the customer with the product shelf ID visited by the customer.

FIG. 24 is an explanatory diagram showing another example of the flow line data. As shown in FIG. 24, for example, the flow line data generation unit 320a generates flow line data in which the identification information of the terminal 10 and the product shelf ID are associated with each other. Further, the flow line data generation unit 320a associates the generated flow line data with the acquisition time of the flow line data and supplies the generated flow line data to the first relevance generation unit 340. In other words, the flow line data generation unit 320a includes the time when each person in the flow line data visits each product shelf in the flow line data and supplies it to the first relevance generation unit 340. Subsequent processing is the same as that of the third embodiment.

As described above, in this modification, the flow line data is generated based on the position information of the person detected by the flow line detection device 6. As a result, as in the third embodiment, the customer's flow line data can be acquired with higher accuracy, so that the product purchased by the customer can be appropriately managed.

As shown in the second embodiment, the self-registration system of the third embodiment includes the second acquisition unit 210, and is configured to classify changes in the product shelves using an RGB image and a distance image. May be good.

<Modification example of foreground area detection unit>
Next, a modified example of the processing of the foreground region detection unit included in the first change detection unit 120 or the second change detection unit 220 included in the image processing apparatus of each of the above-described embodiments will be described.

In this modification, the foreground area detection unit 121 included in the first change detection unit 120 or the second change detection unit 220 further uses the shelf area information registered in advance, and the object included in the change area is a product shelf. Specify that it is not a product in 3.

In this modification, a modification of the foreground region detection unit 121 of the image processing apparatus 100 in the first embodiment will be described. However, this modification can be applied to both the image processing device 200 and the image processing device 300.

25, 26, and 27 are explanatory views showing an operation example in which the foreground region detection unit 121 detects the foreground region in this modification.

The foreground region detection unit 121 detects a change region by comparing, for example, the captured image supplied from the first acquisition unit 110 with the background information 131, and a binary image representing the change region exemplified in FIG. 25. It is assumed that the detection result 71 is generated. It is assumed that the detection result includes three change regions, a change region 72, a change region 73, and a change region 74. In such a case, the foreground region detection unit 121 generates the detection result 71A, the detection result 71B, and the detection result 71C, which are separate binary images for each change region, with respect to the detection result 71 by a general labeling method. ..

That is, when the detection result includes a plurality of change regions, the foreground region detection unit 121 generates a plurality of binary images such that each change region is included in a separate binary image.

Then, the foreground area detection unit 121 determines whether or not the change area is the area where the change related to the change of the product is detected, based on the shelf area information registered in advance and each of the plurality of binary images. Here, the shelf area information indicates an area in which the products on the product shelf 3 are displayed.

The self-registration system 1 monitors the products on the product shelves 3. Therefore, the area where the products are displayed, which is indicated by the shelf area information, can be referred to as a monitored area. Further, the shelf area information can also be referred to as monitoring area information. The shelf area information is, for example, an image having the same size as the captured image acquired by the first acquisition unit 110, and the pixel value of the monitored area of the product shelf 3 to be monitored is represented by 255, and the other areas are represented by 0. It may be a binary image. Further, the monitoring target area included in the shelf area information may be, for example, one or a plurality. The shelf area information may be stored in the first storage unit 130 in advance, for example. The shelf area information includes information for specifying the product shelf 3 included in the captured image acquired by the first acquisition unit 110.

The foreground area detection unit 121 calculates the logical product for each corresponding pixel by using the shelf area information related to the product shelf 3 included in the captured image acquired by the first acquisition unit 110. For example, when the shelf area information 75 illustrated in FIG. 26 is used, the foreground area detection unit 121 calculates the logical product of the detection result 71A, the detection result 71B, or the detection result 71C for each corresponding pixel. In the example shown in FIG. 26, the monitored area in the shelf area information 75 is represented in white, so that the shelf area information 75 includes six monitored areas.

The operation result 76A exemplified in FIG. 27 is the result of the logical product operation of the shelf area information 75 and the detection result 71A. Further, the calculation result 76B is the result of the calculation of the logical product of the shelf area information 75 and the detection result 71B. Further, the calculation result 76C is the result of the calculation of the logical product of the shelf area information 75 and the detection result 71C.

Non-commodity objects such as people and carts span multiple shelves. Therefore, as a result of calculating the logical product of the detection result 71A and the shelf area information 75, as shown in the calculation result 76A exemplified on the left side of FIG. 27, the portion having a pixel value of 255 representing the change region (white portion). ) Is divided into a plurality of areas. On the other hand, the portion (white portion) representing the change region of the calculation result 76B and the calculation result 76C has no change from the detection result 71B and the detection result 71C, respectively, and is a continuous region (pixels having a pixel value of 255). Moreover, it is a set of pixels in which the pixel value of any of the pixels adjacent to the pixel is 255).

The products displayed in the display area (monitoring target area) of the product shelf 3 do not span a plurality of monitoring target areas. Therefore, when the change region is divided into a plurality of regions as in the calculation result 76A, the foreground region detection unit 121 determines that the change with respect to this change region is due to a change other than the product. In such a case, the foreground region detection unit 121 does not include this change in the detection result supplied to the foreground region tracking unit 123. In other words, the foreground region detection unit 121 supplies the detection result 71B and the detection result 71C to the foreground region tracking unit 123.

With such a configuration, the foreground region tracking unit 123 can supply changes to the products displayed on the product shelf 3 to the first extraction unit 124 and the second extraction unit 125. In other words, since the area change classification unit 126 can perform the change classification process for the products displayed on the product shelf 3, it is possible to prevent the deterioration of the classification accuracy of the changes for the products due to the influence of objects other than the products. can. Further, since the foreground region detection unit 121 can classify the change in the change region as a change due to an object other than the product before performing the classification processing of the region change classification unit 126, the processing amount of the region change classification unit 126. Can be reduced.

Next, a specific example of the present invention will be described. In the first specific example, an operation example of the self-registration system when the customer carries the terminal 10 will be described, and in the second specific example, an operation example of the self-registration system when the customer does not carry the terminal 10 will be described.

FIG. 28 is an explanatory diagram showing an operation example of the self-registration system in the first specific example. First, when the customer C visits the store, the person identification device 50 identifies the person via the terminal 10 carried by the customer (step S11). After that, the person is uniquely managed by using the identification ID that identifies the terminal 10. Further, the shopping list generation means (shopping list generation unit 21 in the embodiment) generates a shopping list L corresponding to the identification ID (step S12). The notification unit 23 transmits the generated shopping list L to the terminal 10 (step S13). Further, the notification unit 23 stores the generated shopping list L in the shopping list storage unit 24 (step S14).

After that, the change detection means (first change detection unit 120 in the embodiment) detects the change region based on the captured image in which the product is captured, and the classification means (region change classification unit 126 in the embodiment) detects. Classify the changes in the product shelves in the changed area (step S15). Here, it is assumed that the customer picks up the product. The shopping list generation means (shopping list update unit 22 in the embodiment) is based on the classification of changes in the product shelves (specifically, “changes due to the product being taken”) and the shelving allocation information of the product shelves. The product is specified, and a registration process for registering the specified product in the shopping list L corresponding to the person is performed (step S16). The notification unit 23 notifies the terminal 10 of the contents of the shopping list L in which the product is registered (step S17).

Further, the change detecting means detects the change region based on the captured image in which the product is captured, and the classification means classifies the change of the product shelf in the detected change region (step S18). Here, it is assumed that the customer returns the product. The shopping list generation means identifies the product based on the classification of changes in the product shelves (specifically, "changes due to the new inclusion of products in the product shelves 3") and the shelf allocation information of the product shelves. Then, a deletion process for deleting the specified product from the shopping list L corresponding to the person is performed (step S19). The notification unit 23 notifies the terminal 10 of the contents of the shopping list L from which the product has been deleted (step S20). The notification unit 23 may notify the terminal 10 of the shopping list L in which the deletion flag is set for the product.

When the customer intends to leave the store, the payment device 40 performs payment processing based on the contents of the shopping list L corresponding to the person (step S21). At this time, the output device 30 may display the contents of the shopping list L.

FIG. 29 is an explanatory diagram showing an operation example of the self-registration system in the second specific example. First, when the customer visits the store, the person identification device 50 identifies the person C from the image captured by the image pickup device 2 (step S31). After that, the person is identified at the position by using the identification ID for identifying the person. Further, the shopping list generation means (shopping list generation unit 21 in the embodiment) generates a shopping list L corresponding to the identification ID (step S32). Further, the notification unit 23 stores the generated shopping list L in the shopping list storage unit 24 (step S33).

After that, as in the first specific example, the change detection means (first change detection unit 120 in the embodiment) detects the change region based on the captured image in which the product is captured, and the classification means (in the embodiment). The area change classification unit 126) classifies the changes in the product shelves in the detected change area (step S34). Here, it is assumed that the customer picks up the product. The shopping list generation means (shopping list update unit 22 in the embodiment) is based on the classification of changes in the product shelves (specifically, “changes due to the product being taken”) and the shelving allocation information of the product shelves. The product is specified, and a registration process for registering the specified product in the shopping list corresponding to the person is performed (step S35). Specifically, the shopping list generation means adds a product to the shopping list L stored in the shopping list storage unit 24.

Then, suppose the customer returns the item. Similar to the first specific example, the shopping list generation means is based on the classification of changes in the product shelves (specifically, "changes due to the placement of the products on the product shelves") and the shelving allocation information of the product shelves. The product is specified based on the above, and a deletion process for deleting the specified product from the shopping list L corresponding to the person is performed (step S36). Specifically, the shopping list generation means deletes the product from the shopping list L stored in the shopping list storage unit 24.

When the customer intends to leave the store, the payment device 40 performs payment processing based on the contents of the shopping list L corresponding to the person C (step S37). Specifically, the output device 30 displays the contents of the shopping list and the total amount, and the payment device 40 performs payment processing based on payment from the person or card payment.

<About hardware configuration>
In each of the above-described embodiments, the components of each device (more specifically, image processing devices 100 to 500, shopping list management device 20) indicate, for example, a block of functional units. Some or all of the components of each device are realized by any combination of the information processing device 600 illustrated in FIG. 30 and the program. FIG. 30 is a block diagram showing a hardware configuration example of the information processing device 600 that realizes the components of each device. As an example, the information processing apparatus 600 includes the following configuration.
-CPU (Central Processing Unit) 601
-ROM (Read Only Memory) 602
-RAM (Random Access Memory) 603
-Program group 604 loaded in RAM 603
A storage device 605 for storing the program group 604.
Information processing device 600 A drive device 606 that reads and writes to an external recording medium 610.
Information processing device 600 Communication interface 607 for connecting to an external communication network 611
-I / O interface 608 for data input / output
-Bus 609 connecting each component

The components of each device in each of the above-described embodiments are realized by the CPU 601 acquiring and executing the program group 604 that realizes these functions. The program group 604 that realizes the functions of the components of each device is stored in, for example, a storage device 605 or ROM 602 in advance, and the CPU 601 loads the program group 604 into the RAM 603 and executes the program as needed. The program group 604 may be supplied to the CPU 601 via the communication network 611, or may be stored in the recording medium 610 in advance, and the drive device 606 may read the program and supply the program to the CPU 601.

It should be noted that there are various modifications in the method of realizing each device. Each device may be realized, for example, by any combination of the information processing device 600 and the program, which are separate for each component. Further, a plurality of components included in each device may be realized by any combination of one information processing device 600 and a program.

Further, a part or all of each component of each image processing device can be realized by other general-purpose or dedicated circuits, processors and the like, or a combination thereof. These may be composed of a single chip or may be composed of a plurality of chips connected via a bus.

Some or all of the components of each device may be realized by a combination of the above-mentioned circuit or the like and a program.

When a part or all of the components of each device are realized by a plurality of information processing devices, circuits, etc., the plurality of information processing devices, circuits, etc. may be centrally arranged or distributed. May be good. For example, the information processing device, the circuit, and the like may be realized as a form in which each is connected via a communication network, such as a client-and-server system and a cloud computing system.

Next, the outline of the present invention will be described. FIG. 31 is a block diagram showing an outline of the self-registration system according to the present invention. The self-registration system 800 (for example, the self-registration system 1) according to the present invention is a change detection means 840 (for example, first) that detects a change area of a product shelf on which a product is placed based on an image captured by the product. The change detection unit 120), the classification means 850 for classifying the changes in the product shelves in the detected change area (for example, the area change classification unit 126), the classification of the changes in the product shelves, and the shelving allocation information of the product shelves. Based on this, it is provided with a shopping list generation means 860 (for example, a shopping list update unit 22) that generates a customer's shopping list.

With such a configuration, it is possible to appropriately manage the products purchased by the customer without preparing images of individual products in advance.

Specifically, the classification of changes in the product shelves is the change due to the product being taken, the change in the store environment of the product shelf, the change due to the product being placed on the product shelf, or the appearance of the product. It may be any of the changes due to the fact.

Further, the shopping list generation means 860 may register the product in the shopping list when the classification of the change in the product shelf is the change due to the product being taken.

Further, the shopping list generation means 860 may delete the product from the shopping list when the classification of the change in the product shelf is the change due to the product being placed on the product shelf.

In addition, the change due to the change in the appearance of the product, which is the classification of the change in the product shelf, includes the change in the appearance due to the arrangement of different products and the change in the appearance due to the change in the attitude of the product. It may be.

Further, the self-registration system 800 may include a relationship generation means (for example, a first relationship generation unit 140) that generates relationship information indicating the relationship between the change area and the person. The relationship generation means associates the change area with the person based on the image pickup time and the person information acquired from the captured image and the position information and the image pickup time of the change area acquired from the captured image, and the relevance information. May be generated. Then, the shopping list generation means 860 may generate a shopping list corresponding to a person based on the classification of changes in the product shelves in the change area and the shelf allocation information of the product shelves. According to such a configuration, since the flow line of the person can be obtained from the image, the product purchased by the customer can be appropriately managed.

Similarly, the self-registration system 800 may include a relevance generation means (for example, a second relevance generation unit 240) that generates relevance information indicating the relationship between the change area and the person. The association generation means is acquired from the acquisition time and position information of the person information detected by the flow line detecting means (for example, the flow line detecting device 6) that detects the flow line of the person in the store, and the captured image. Relevance information may be generated by associating the change area with a person based on the position information of the change area and the imaging time. Then, the shopping list generation means may generate a shopping list corresponding to a person based on the classification of changes in the product shelves in the change area and the shelf allocation information of the product shelves. Since the flow line of a person can be acquired even with such a configuration, it is possible to appropriately manage the products purchased by the customer.

Further, specifically, the classification means 850 uses a shelf change model (for example, a shelf change model 132) that models changes in the product shelves before and after the change in time, and uses an image of the change region to obtain products. You may classify changes in the shelves.

Further, the change detecting means 840 detects the change area of the product shelf on which the product is arranged based on the RGB image and the distance image obtained by capturing the imaging range of the RGB image as the captured image in which the product is captured. May be good. With such a configuration, changes in the product shelves can be determined more accurately, so that the products purchased by the customer can be managed more appropriately.

Some or all of the above embodiments may also be described, but not limited to:

(Appendix 1) A change detecting means for detecting a change area of a product shelf on which the product is placed based on an image captured by the product, and a classification for classifying changes in the product shelf in the detected change area. A self-registration system including means, a shopping list generation means for generating a customer's shopping list based on the classification of changes in the product shelves, and the shelf allocation information of the product shelves.

(Appendix 2) The classification of changes in the product shelves is as follows: changes due to the products being taken, changes in the store environment of the product shelves, changes due to the products being placed on the product shelves, or changes in the appearance of the products. The self-registration system described in Appendix 1, which is one of the changes due to the above.

(Appendix 3) The shopping list generation means is the self-registration system described in Appendix 2 for registering the product in the shopping list when the classification of the change in the product shelf is the change due to the product being picked up.

(Appendix 4) The shopping list generation means is the self-registration system according to Appendix 2, which deletes the product from the shopping list when the classification of the change in the product shelf is a change due to the product being placed on the product shelf.

(Appendix 5) Changes in the appearance of products, which is a classification of changes in product shelves, include changes in appearance due to the placement of different products and changes in appearance due to changes in the attitude of products. The self-registration system according to any one of Supplementary note 2 to Supplementary note 4, which includes the above.

(Appendix 6) A relevance generation means for generating relevance information indicating a relationship between a change area and a person is provided, and the relevance generation means is obtained from the imaging time and person information acquired from the captured image and the captured image. Based on the acquired position information of the change area and the imaging time, the change area is associated with the person to generate the relevance information, and the shopping list generation means classifies the change of the product shelf in the change area and the change. The self-registration system according to any one of Supplementary note 1 to Supplementary note 5, which generates a shopping list corresponding to the person based on the shelf allocation information of the product shelf.

(Appendix 7) A relevance generation means for generating relevance information indicating a relationship between a change area and a person is provided, and the relevance generation means is detected by a movement line detecting means for detecting a movement line of a person in a store. Based on the acquisition time and position information of the information of the person concerned and the position information and the imaging time of the change area acquired from the captured image, the change area and the person are associated with each other to generate the relevance information, and the shopping list is generated. The generation means is one of Appendix 1 to Appendix 5 that generates a shopping list corresponding to the person based on the classification of changes in the product shelves in the change area and the shelf allocation information of the product shelves. Described self-registration system.

(Appendix 8) The classification means uses a shelf change model that models changes in the product shelves before and after changes in time, and classifies changes in the product shelves from images of the change area. The self-registration system described in any one of them.

(Appendix 9) The change detecting means detects the change area of the product shelf on which the product is arranged based on the RGB image and the distance image obtained by capturing the imaging range of the RGB image as the captured image in which the product is captured. The self-registration system according to any one of Supplementary note 1 to Supplementary note 8.

(Appendix 10) Based on the captured image in which the product is imaged, the change area of the product shelf on which the product is placed is detected, the change of the product shelf in the detected change area is classified, and the change of the product shelf is classified. A purchased merchandise management method, characterized in that a customer's shopping list is generated based on the classification of changes and the shelving allocation information of the merchandise shelves.

(Appendix 11) A change detection process for detecting a change area of a product shelf on which the product is placed, and a change in the product shelf in the detected change area are classified by a computer based on an image captured by the product. A purchase product management program for executing a shopping list generation process for generating a customer's shopping list based on the classification process, the classification of changes in the product shelf, and the shelving allocation information of the product shelf.

In addition, the program described in each of the above-described embodiments and appendices may be stored in a storage device or recorded in a recording medium readable by a computer. The recording medium is a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, and a semiconductor memory.

1, 4, 4a Self-registration system 2, 5 Imaging device 3 Product shelf 6 Flow line detection device 10 Terminal 20 Shopping list management device 30 Output device 40 Payment device 50 Person identification device 71 Detection result 72 Change area 73 Change area 74 Change area 75 Shelf area information 76 Calculation result 90 Classification result 91 Second attention image 92 First attention image 93 Change type 100 Image processing device 110 First acquisition unit 120 First change detection unit 121 Foreground area detection unit 122 Background information update unit 123 Foreground Area tracking unit 124 1st extraction unit 125 2nd extraction unit 126 Area change classification unit 127 Distance information update unit 130 1st storage unit 131 Background information 132 Shelf change model 133 Foreground information 134 Person information 135 Distance information 140 1st relevance generation Part 150 Relevance integration unit 200 Image processing device 210 2nd acquisition unit 220 2nd change detection unit 230 2nd storage unit 240 2nd relevance generation unit 250 Relevance integration unit 300, 300a Image processing device 310, 310a 3rd acquisition Unit 320, 320a Flow line data generation unit 340 First relevance generation unit 600 Information processing device 601 CPU
602 ROM
603 RAM
604 Program group 605 Storage device 606 Drive device 607 Communication interface 608 I / O interface 609 Bus 610 Recording medium 611 Communication network

Claims (9)

A change detection means for detecting a change area of a product shelf on which a product is placed based on a captured image in which the product is imaged, and a change detection means.
A classification means for classifying changes in the product shelves in the detected change area, and
A shopping list generation means for generating a customer's shopping list based on the classification of changes in the product shelves and the shelf allocation information of the product shelves is provided.
The classification means is a self-registration system characterized by classifying changes in the product shelves from an image of the change area by using a shelf change model that models changes in the product shelves before and after changes in time. .. The classification of changes in the product shelves is the change due to the product being taken, the change in the store environment of the product shelf, the change due to the product being placed on the product shelf, or the change due to the change in the appearance of the product. The self-registration system according to claim 1, which is either. The self-registration system according to claim 2, wherein the shopping list generation means registers the product in the shopping list when the classification of the change in the product shelf is the change due to the product being taken. The self-registration system according to claim 2, wherein the shopping list generation means deletes the product from the shopping list when the classification of the change in the product shelf is a change due to the product being placed on the product shelf. It is equipped with a relevance generation means that generates relevance information indicating the relationship between the change area and the person.
The association generation means associates the change area with the person based on the image capture time and person information acquired from the captured image and the position information and the image capture time of the change area acquired from the image. Generate sexual information,
The shopping list generation means is any of claims 1 to 4, which generates a shopping list corresponding to the person based on the classification of changes in the product shelves in the change area and the shelf allocation information of the product shelves. The self-registration system described in item 1. It is equipped with a relevance generation means that generates relevance information indicating the relationship between the change area and the person.
The association generating means includes acquisition time and position information of the person's information detected by the flow line detecting means for detecting the flow line of the person in the store, and position information and imaging of a change area acquired from the captured image. Based on the time of day, the change area is associated with the person to generate relevance information.
The shopping list generation means is any of claims 1 to 5 that generates a shopping list corresponding to the person based on the classification of changes in the product shelves in the change area and the shelf allocation information of the product shelves. The self-registration system described in item 1. The change detecting means detects a change area of a product shelf on which the product is arranged, based on an RGB image and a distance image obtained by capturing the imaging range of the RGB image as an image captured by the product. The self-registration system according to any one of claims 6. Based on the captured image in which the product is imaged, the change area of the product shelf on which the product is placed is detected.
Classify the changes in the product shelves in the detected change area and classify them.
A customer's shopping list is generated based on the classification of changes in the product shelves and the shelving allocation information of the product shelves.
At the time of the classification, the purchased product is characterized in that the change of the product shelf is classified from the image of the change area by using the shelf change model that models the change of the product shelf before and after the change of time. Management method. On the computer
Change detection processing that detects the change area of the product shelf on which the product is placed based on the captured image in which the product is imaged.
Classification processing for classifying changes in the product shelves in the detected change area, and
A shopping list generation process for generating a customer's shopping list is executed based on the classification of changes in the product shelves and the shelf allocation information of the product shelves.
A purchased product management program for classifying changes in the product shelves from images in the change region using a shelf change model that models changes in the product shelves before and after changes in time in the classification process.

Images