JP2016201105A - Information processor and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To finally enable the acquisition of marketing information that is not accompanied with a purchasing behavior without attaching an RF tag to each commodity.SOLUTION: An information processor includes an imaging part 101 for sequentially imaging an image partially including a commodity shelf 1, a commodity specification part for specifying commodity 4 included in the image picked up by the imaging part 101, a movement direction specification part for specifying a movement direction of the commodity 4 specified by the commodity specification part, a marketing information determination part for determining whether the commodity 4 is taken up from the commodity shelf 1 or the commodity 4 is returned onto the commodity shelf 1 on the basis of the movement direction specified by the movement direction specification part, and creating marketing information including this determination result, and a data storage part for storing the marketing information created by the marketing information determination part.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus and an information processing method, and more particularly, to an information processing apparatus and an information processing method that collect marketing information about products.

Patent Document 1 describes a system that utilizes general object recognition technology as a POS (Point Of Sales) system for collecting marketing information. This POS system extracts the feature quantity of the target product from the image data obtained by capturing the target product with the imaging device, calculates the degree of similarity by comparing with the previously prepared product data, and calculates the calculated similarity. In response, the type of the target product is detected. Such a system can easily collect information on commodities for which barcodes cannot be added, such as fresh foods such as fruits and vegetables.
There is also a system for collecting marketing information in the vicinity of a product shelf using an RF (Radio Frequency) tag (for example, Patent Document 2).
The general object recognition technique is disclosed in Non-Patent Document 1.

JP 2014-206988 A (paragraphs 0008 to 0065, FIG. 2) International Publication No. 2014/087559 (paragraphs 0019-0065, FIG. 1)

Oncel Tuzel, Fatih Porikli, Peter Meer, “Region Covariance: A Fast Descriptor for Detection and Classification” European Conference on Computer, June 2010, Vis.

Since the system described in Patent Document 1 collects marketing information at the cash register at the time of accounting, for example, marketing information that does not ultimately involve purchasing behavior, such as picking up a product near the product shelf, etc. Cannot be collected.
On the other hand, the system described in Patent Document 2 can collect marketing information even near the product shelf, but it is necessary to affix an RF tag for each product, which increases the work burden on employees in the store.

  Accordingly, the present invention has been made to solve the above-described problems, and is intended to be able to finally acquire marketing information that does not involve purchasing behavior without attaching an RF tag for each product. Objective.

  An information processing apparatus according to an aspect of the present invention includes an imaging unit that sequentially captures an image including a product shelf in part, a product identification unit that identifies a product included in the image captured by the imaging unit, The product is picked up from the product shelf from the movement direction specifying unit that specifies the movement direction of the product specified by the product specifying unit and the movement direction specified by the movement direction specifying unit, or the product is A marketing information determination unit that determines whether the product has been returned to the product shelf and generates marketing information including a determination result, and a data storage unit that stores the marketing information generated by the marketing information determination unit. And

  An information processing method according to an aspect of the present invention sequentially captures an image partially including a product shelf, identifies a product included in the captured image, and identifies a movement direction of the identified product. Then, it is determined whether the product has been picked up from the product shelf or the product has been returned to the product shelf from the identified movement direction, and generates marketing information including a determination result, the generated It is characterized by memorizing marketing information.

  According to one aspect of the present invention, it is possible to finally obtain marketing information that does not involve purchasing behavior without attaching an RF tag for each product.

It is the schematic which shows the example of installation of the marketing information collection apparatus which concerns on Embodiment 1-6. 1 is a block diagram schematically showing a configuration of a marketing information collection device according to Embodiment 1. FIG. 4 is a flowchart illustrating processing at the time of motion direction calculation in a motion direction calculation unit according to the first embodiment. (A) And (B) is the schematic for demonstrating the block matching method in Embodiment 1. FIG. 5 is a schematic diagram illustrating an example of an image in Embodiment 1. FIG. It is a block diagram which shows roughly the structure of the marketing information collection apparatus which concerns on Embodiment 2. FIG. It is a block diagram which shows roughly the structure of the marketing information collection apparatus 300 which concerns on Embodiment 3. FIG. 10 is a flowchart illustrating a process of calculating a motion area in a motion area estimation unit according to the third embodiment. 10 is a schematic diagram illustrating an example of a circumscribed rectangle in Embodiment 3. FIG. 14 is a flowchart illustrating processing at the time of motion direction calculation in a motion direction calculation unit according to the third embodiment. It is a block diagram which shows roughly the structure of the marketing information collection apparatus which concerns on Embodiment 4. FIG. 10 is a schematic diagram illustrating an example of an image in Embodiment 4. FIG. It is the schematic which shows arrangement | positioning of the goods of the goods shelf in Embodiment 4. FIG. FIG. 16 is a schematic diagram illustrating an example of storage position information in the fourth embodiment. FIG. 16 is a schematic diagram illustrating an example of an image in a fourth embodiment. It is a block diagram which shows roughly the structure of the marketing information collection apparatus which concerns on Embodiment 5. FIG. It is a block diagram which shows roughly the structure of the marketing information collection apparatus which concerns on Embodiment 6. FIG. 20 is a flowchart showing a flow of processing of an attribute determination unit in the sixth embodiment. 18 is a flowchart showing an estimation process of a product estimation unit in the sixth embodiment. It is a block diagram which shows roughly the structure of the marketing information collection apparatus which concerns on Embodiment 7. FIG.

Embodiment 1 FIG.
FIG. 1 is a schematic diagram illustrating an installation example of marketing information collection devices 100 to 700 as information processing devices according to the first to seventh embodiments.
Figure 1 is modeled on a convenience store store in Japan. Products are arranged in the product shelf 1, and marketing information collection devices 100 to 700 are attached to the upper ceiling 3 between the product shelf 1 and the customer 2. The marketing information collection devices 100 to 700 are attached so that an image of the product 4 obtained by the customer 2 can be acquired.

FIG. 2 is a block diagram schematically showing the configuration of the marketing information collecting apparatus 100 according to the first embodiment.
The marketing information collecting apparatus 100 according to the first embodiment includes an imaging unit 101, an image capturing unit 102, a frame memory 103, a product data storage unit 104, a product estimation unit 105, a motion direction calculation unit 106, A frame memory 107, a marketing information determination unit 108, and a data recording unit 109 as a data storage unit are provided. The information processing method according to the first embodiment is a method executed by the marketing information collecting apparatus 100 according to the first embodiment.

The imaging unit 101 captures a target image and generates image data of the image. Here, the imaging unit 101 sequentially captures images including a part of the product shelf 1 shown in FIG.
For example, the imaging unit 101 includes an optical lens, an optical filter, an image sensor, and a processor that controls the image sensor if necessary, and captures image data of a temporally continuous digital image (hereinafter also simply referred to as an image). Sequentially supplied to the embedding unit 102. The image data supplied from the imaging unit 101 may be color or monochrome. However, in order to improve the estimation accuracy in the product estimation unit 105, color image data is more desirable. For this reason, in this embodiment, it is assumed that the image data is color image data. The color information of the color image data may be RGB, YCbCr, HSV, or the like, but is assumed to be RGB hereinafter. Furthermore, the image size is (w, h).

  The image capturing unit 102 supplies the image data given from the imaging unit 101 to the product estimation unit 105 and the frame memory 103. Here, the operation of the image capturing unit 102 will be described. The image capturing unit 102 stores the image data sent from the imaging unit 101 in the frame memory 103. This operation is generally performed in synchronization with the timing of the imaging unit 101. In parallel with this image storage operation, the image capture unit 102 reads image data from the frame memory 103 and supplies the image data to the product estimation unit 105.

The frame memory 103 is a memory that temporarily stores the image data given from the image capturing unit 102. The frame memory 103 is generally a FIFO (First In First Out) system, but may be a FILO (First In Last Out) system. Further, the frame memory 103 may read the image data in the order instructed from the product estimation unit 105.
Further, the marketing information collecting apparatus 100 may include a line memory instead of the frame memory 103. In this case, a line memory may be incorporated in the image capturing unit 102. In the following, it is assumed that the frame memory 103 performs a FIFO operation.

  The product data storage unit 104 is a product data storage unit that stores information necessary for specifying the products displayed on the product shelf 1. For example, the product data storage unit 104 stores feature quantities and product names calculated in advance for each product.

The product estimation unit 105 is a product identification unit that identifies a product included in the image captured by the imaging unit 101. For example, the product estimation unit 105 sequentially specifies the product names of the products included in the images sequentially captured by the imaging unit 101 and the positions of the products in the images.
Specifically, the product estimation unit 105 receives the image data of the current frame from the image capturing unit 102, and estimates the product in the image of the received image data. For example, the product estimation unit 105 calculates the feature amount of the product region in the image data using a general object recognition technique. Then, the product estimation unit 105 calculates the similarity between the calculated feature amount and the feature amount stored in the product data storage unit 104, and sets the one with the highest similarity as a product candidate. However, since the product area does not always have a product due to a disturbance or the like, the product estimation unit 105 determines that there is no corresponding product when the calculated similarity is less than a predetermined threshold. Details of the product estimation unit 105 including the general object recognition technique will be described later.

Finally, the product estimation unit 105 supplies product information indicating the product name to the marketing information determination unit 108 when it is determined that there is a detected product and there is a product. At this time, the presence or absence of the product can be omitted by setting the product name to “No applicable product”, for example.
Further, the product estimation unit 105 supplies the product area information and the current frame image data when it is determined that there is a product to the motion direction calculation unit 106. The product area information is information indicating an area (rectangular area Rect _c ) in which the product is detected.
Here, as a final output of the marketing information collection device 100, when a thumbnail of the detected product is handled, the product estimation unit 105 extracts the current frame image data based on the product region information. The partial image data may be supplied to the marketing information determination unit 108. Hereinafter, it is assumed that the product information indicating the presence or absence of the detected product and / or the product name is supplied to the marketing information determination unit 108, and the product region information and the current frame image data are supplied to the motion direction calculation unit 106. explain.

  Here, it supplements about the information of this product area. The information on the product area is a candidate area for the product picked up by the customer 2. This region is handled as a rectangle for simplicity, but may be a circle, an ellipse, or a free shape. In that case, the product estimation unit 105 supplies an amount such as a radius to the marketing information determination unit 108 or supplies a mask image indicating a region to the marketing information determination unit 108.

The movement direction calculation unit 106 is a movement direction specifying unit that specifies the movement direction of the product specified by the product estimation unit 105.
Specifically, the movement direction calculation unit 106 specifies the moving direction of the product based on the image data and the product area information provided from the product estimation unit 105. For example, the motion direction calculation unit 106 receives the image data from the product estimation unit 105 and stores it in the frame memory 107. The motion direction calculation unit 106 controls the frame memory 107 to control the latest image data sent from the product estimation unit 105 and image data one frame time before the latest image data (hereinafter, current frame image data). And two image data of image data two frames before (referred to as past frame image data). One frame time is a time indicating an imaging interval. For example, if the imaging speed of the imaging unit 101 is 30 fps (frame per second: equivalent to the number of imaging per second), one frame time is 1/30 second.
The movement direction calculation unit 106 calculates the direction of movement of the product picked up by the customer 2 from the current frame image data and the past frame image data, and a detailed algorithm will be described later.
The motion direction calculation unit 106 supplies motion vector information indicating the motion direction of the product to the marketing information determination unit 108. At the same time, the movement direction calculation unit 106 also supplies the product area information and the current frame image data to the marketing information determination unit 108.

Based on the motion vector information given from the motion direction calculation unit 106, the marketing information determination unit 108 determines whether the product has been picked up from the product shelf 1 or returned to the product shelf 1, and the determination result Is generated and stored in the data recording unit 109.
Specifically, the marketing information determination unit 108 receives product information from the product estimation unit 105 and product region information, motion vector information, and current frame image data from the motion direction calculation unit 106. The marketing information determination unit 108 records nothing in the data recording unit 109 when there is no product taken by the customer 2 according to the presence or absence of the product indicated by the product information supplied from the product estimation unit 105. Information indicating that the process is completed or there is no product is recorded in the data recording unit 109. On the other hand, when there is a product taken by the customer 2, the marketing information determination unit 108 generates marketing information including the product name, product region information, and product movement determination result, and the data recording unit 109. To record. Here, the motion determination based on the motion vector information from the motion direction calculation unit 106 will be described later.

  The data recording unit 109 stores the marketing information given from the marketing information determination unit 108. For example, the data recording unit 109 may be a built-in HDD (Hard Disk Drive) or an external HDD, or a removable medium such as an SD card. Further, the data recording unit 109 may be a rewritable memory such as a RAM (Random Access Memory) or an EEPROM (Electrically Erasable Programmable Read-Only Memory). Further, the marketing information to be recorded may include thumbnail image data obtained by cutting out a product portion from current frame image data or product area information. The product data storage unit 104 can also be configured with an HDD, a RAM, a ROM, and the like, like the data recording unit 109.

  The image capturing unit 102, the product estimating unit 105, the motion direction calculating unit 106, and the marketing information determining unit 108 of the marketing information collecting apparatus 100 configured as described above are configured such that a CPU (Central Processing Unit) is a memory or an HDD. This can be realized by executing a program stored in an external storage device. However, these are not limited to those realized as software on a computer system. For example, all or a part of these may be realized in hardware by an integrated logic IC such as ASIC (Application Specific Integrated Circuits) or FPGA (Field Programmable Gate Array). Some or all of these may be implemented in software by a DSP (Digital Signal Processor) or the like. Alternatively, a component specialized for a specific application may be realized by hardware such as an expansion board mounted on the computer system, and other general-purpose components may be realized by software on the computer system. Good.

  Details of the operation of the marketing information collecting apparatus 100 configured as described above will be described.

The product estimation unit 105 estimates a product included in the image data of the current frame using a sliding window method. The sliding window method is a method of performing a full search by sliding an area having a predetermined window size (w _c , h _c ) over the entire area of the image indicated by the image data of the current frame. In addition, it is better to define a plurality of types of window sizes (w _c , h _c ). For example, (w _c , h _c ) = (16, 16), (32, 32), (48, 48), (64, 64), etc. As shown in FIG. 1, the positional relationship between the imaging unit 101 and the customer 2 and the product 4 is different for each store or installation position, or the optical part of the imaging unit 101, for example, This is because the size of the product 4 in the image is different due to the fact that the magnification of the lens is different.
The product estimation unit 105 calculates the feature amount for each sliding window, calculates the similarity with the feature amount of the product stored in the product data storage unit 104, and the customer 2 picks up or returns it. Estimate the product. As the general object recognition method, the method disclosed in Non-Patent Document 1 described above may be employed.

Here, the general object recognition method disclosed in Non-Patent Document 1 will be briefly described. In this method, a feature amount is calculated from the luminance, color and luminance first derivative, second derivative of luminance, and the like of a rectangular area in an image, and a distance dist of the feature amount calculated from two rectangles is obtained. Since the distance dist indicates the distance of the feature amount calculated from the two rectangles, the reciprocal of the distance dist is the similarity. That is, the similarity is calculated by the following equation (1).

但し、閾値Ｄ_Ｒは、「０」よりも大きいものとする。
この処理は、ノイズに対する頑健性を担保するものである。最終的に、商品推定部１０５は、矩形領域Ｒｅｃｔ_ｃの左端位置（ｘ_ｃ，ｙ_ｃ）と、矩形サイズ（ｗ_ｃ，ｈ_ｃ）と、現フレームの画像データとを動き方向算出部１０６に供給する。また、商品推定部１０５は、推定された商品の商品名を、マーケティング情報判定部１０８に供給する。但し、商品推定部１０５は、商品が見つからない矩形領域Ｒｅｃｔ_ｃの場合は、商品名を商品なしとするか、その矩形領域Ｒｅｃｔ_ｃを取り除くこととする。 In the product data storage unit 104, feature quantities for each product calculated in advance by the above method are stored. The product estimation unit 105 obtains a feature amount for each rectangular area Rect _c that is a sliding window. However, c = 0,..., K (k is a positive integer). This is because in the sliding window method, there are a plurality of rectangular regions for obtaining feature amounts. Then, the product estimation unit 105 obtains a similarity R _p between these feature values and the feature values stored in the product data storage unit 104. Product estimation unit 105 estimates a product having a maximum feature quantity similarity R _p satisfies the following formula (2), the products included in the rectangular area Rect _c.

However, the threshold value D _R is larger than that "0".
This process ensures robustness against noise. Finally, the product estimation unit 105 sends the left end position (x _c , y _c ) of the rectangular region Rect _c , the rectangular size (w _c , h _c ), and the image data of the current frame to the motion direction calculation unit 106. Supply. Further, the product estimation unit 105 supplies the product name of the estimated product to the marketing information determination unit 108. However, in the case of a rectangular region Rect _c where no product is found, the product estimation unit 105 determines that the product name is no product or removes the rectangular region Rect _c .

Next, details of the calculation of the movement direction in the movement direction calculation unit 106 will be described.
FIG. 3 is a flowchart showing processing at the time of motion direction calculation in the motion direction calculation unit 106.
As described above, the motion direction calculation unit 106 takes in the current frame image data and the past frame image data from the product estimation unit 105 or the frame memory 107 (S10). As described above, the current frame image data and the past frame image data are RGB color image data.

Next, the movement direction calculation unit 106 estimates the movement direction of the product based on the captured current frame image data and past frame image data (S11). For example, the motion direction calculation unit 106 transmits the current frame image data and the past for each rectangular area Rect _c having the left end position (x _c , y _c ) and the rectangular size (w _c , h _c ) sent from the product estimation unit 105. A motion vector is obtained by a block matching method with frame image data.

4A and 4B are schematic diagrams for explaining the block matching method.
As shown in FIG. 4 (B), the upper left of the rectangle coordinates indicating the search range in the image Im2 of the past frame image data ^{_{(x win c, y win c}} ), a rectangle of size ^{(w win} _c , h ^win _c ). These shall satisfy the following formulas (3) to (6).

The motion direction calculation unit 106 includes a rectangle having a size (w _c , h _c ) included in the search range shown in FIG. 4B and the current frame shown in FIG. 4A. An absolute difference value sum with the rectangular area Rect _c of the coordinate position (x _c , y _c ) and size (w _c , h _c ) of the image Im1 of the image data is obtained, and the coordinate position with the smallest absolute difference value sum is obtained. Let (x ′ _c , y ′ _c ). Therefore, the motion vector of the c-th rectangular area Rect _c is obtained by (x _c −x ′ _c , y _c −y ′ _c ). Here, in the above description, the search range is set to three times the size of the rectangular region Rect _c , but may be larger or smaller than this. Finally, the motion direction calculation unit 106 supplies the motion vector (v ^c _x , v ^c _y ) = (x _c −x ′ _c , y _c −y ′ _c ) and the current frame image data to the marketing information determination unit 108. To do.

一方、動きベクトルＶのｙ成分ｖ^ｃ _ｙが下記の（８）式を満たす場合に、マーケティング情報判定部１０８は、商品を戻したと判定する。

また、下記の（９）式を満たす時は、先に（６）式が満たされた場合に限り、マーケティング情報判定部１０８は、客２が商品を見定めていると判定することができる。

同様に、客２と商品棚１が、図５とは異なる配置の場合でも、マーケティング情報判定部１０８は、動きベクトルＶの傾きにより動きを判定することができる。 Next, determination of the movement direction in the marketing information determination unit 108 will be described.
The marketing information determination unit 108 determines that the customer 2 picks up or returns the product based on the motion vector (v ^c _x , v ^c _y ) indicated by the vector information given from the motion direction calculation unit 106. To do. In the arrangement as shown in FIG. 1, it is assumed that the image Im3 of the current frame image data is photographed with the positional relationship as shown in FIG. 5 and the motion vector V is given from the motion direction calculation unit 106. As shown in Figure 5, with respect to customer 2, if the shelves 1 are arranged in the upper in the image Im3 of the current frame image data, y component v ^c _y of the motion vector V is below ( When the expression 7) is satisfied, the marketing information determination unit 108 determines that the customer 2 picks up the product.

On the other hand, it is determined that the y component v ^c _y of the motion vector V in the case that satisfies the following equation (8), marketing information determination unit 108, returned goods.

Further, when the following expression (9) is satisfied, the marketing information determination unit 108 can determine that the customer 2 is looking for the product only when the expression (6) is satisfied first.

Similarly, even when the customer 2 and the product shelf 1 are arranged differently from those in FIG. 5, the marketing information determination unit 108 can determine the movement based on the inclination of the motion vector V.

  In the first embodiment, as shown in FIG. 1, the marketing information collection device 100 is a single device, but the imaging unit 101 and other portions may be configured as separate units. For example, the marketing information collection apparatus 100 may not include the imaging unit 101 and may receive image data from the imaging unit 101 of another unit. Further, the frame memory 103 and the frame memory 107 may be configured as one memory. The product data storage unit 104 and the data recording unit 109 may be combined into one, or may be combined with the frame memory 103 and the frame memory 107 described above.

  Since the marketing information collecting apparatus 100 according to the first embodiment is configured as described above, marketing such as picking up or returning a product that does not directly lead to purchase without attaching an RF tag to the product. Information can be collected.

Embodiment 2. FIG.
FIG. 6 is a block diagram schematically showing the configuration of the marketing information collection apparatus 200 according to the second embodiment.
The marketing information collecting apparatus 200 according to the second embodiment includes an imaging unit 101, an image capturing unit 102, a frame memory 103, a product data storage unit 104, a product estimation unit 105, a motion direction calculation unit 106, A frame memory 107, a marketing information determination unit 208, a data recording unit 109, and a clock unit 210 are provided. Note that the information processing method according to the second embodiment is a method executed by the marketing information collecting apparatus 200 according to the second embodiment.
The marketing information collection device 200 according to the second embodiment is configured in the same manner as the marketing information collection device 100 according to the first embodiment, except for the marketing information determination unit 208 and the clock unit 210. Hereinafter, marketing information determination unit 208 and clock unit 210 that are different from those in the first embodiment will be mainly described.

  The clock unit 210 measures the date and time. Note that the date and time measured by the clock unit 210 are treated as the current date and time. When the clock unit 210 receives the request from the marketing information determination unit 208, the clock unit 210 supplies the date and time of the request timing.

  The marketing information determination unit 208 has substantially the same function as the marketing information determination unit 108 of the first embodiment. However, when recording the marketing information in the data recording unit 109, the marketing information determination unit 208 acquires the date and time from the clock unit 210, and performs marketing. Add date and time information to the information and record it.

  Since the marketing information collection device 200 according to the second embodiment is configured as described above, there is an effect that marketing information of the entire store such as date and time can be added.

  In the configuration of the second embodiment, the clock unit 210 illustrated in FIG. 6 is added, but the marketing information determination unit 208 may include the clock unit 210.

  Note that the clock unit 210 may be realized by the CPU executing a program, or may be realized in hardware by an integrated logic IC such as an ASIC or FPGA. Further, the clock unit 210 may be realized as software by a DSP or the like.

Embodiment 3 FIG.
FIG. 7 is a block diagram schematically showing the configuration of the marketing information collecting apparatus 300 according to the third embodiment.
The marketing information collection device 300 according to the third embodiment includes an imaging unit 101, an image capture unit 102, a frame memory 103, a product data storage unit 104, a product estimation unit 305, a motion direction calculation unit 306, A frame memory 107, a marketing information determination unit 308, a data recording unit 109, a moving region estimation unit 311, and a frame memory 312 are provided. Note that the information processing method according to the third embodiment is a method executed by the marketing information collecting apparatus 300 according to the third embodiment.
The marketing information collecting apparatus 300 according to the third embodiment is the marketing information according to the first embodiment except for the product estimation unit 305, the motion direction calculation unit 306, the marketing information determination unit 308, the motion region estimation unit 311 and the frame memory 312. The configuration is the same as that of the information collecting apparatus 100. In the following, the product estimation unit 305, the motion direction calculation unit 306, the marketing information determination unit 308, the motion region estimation unit 311 and the frame memory 312 that are different from those in the first embodiment will be mainly described. The image capturing unit 102 reads image data from the frame memory 103 and supplies the image data to the product estimation unit 305 and the moving region estimation unit 311.

The moving area estimation unit 311 is a moving area specifying unit that sequentially specifies moving areas in images sequentially captured by the imaging unit 101. For example, the moving region estimation unit 311 receives the image data from the image capturing unit 102, and estimates a range in which there has been motion in the image data using a method described later.
In addition, the moving region estimation unit 311 controls the frame memory 312 so as to hold the latest image data, the current frame image data, and the past frame image data transmitted from the image capturing unit 102.
The moving area estimation unit 311 estimates the area of the product 4 picked up by the customer 2 from the current frame image data and the past frame image data. Although a detailed algorithm will be described later, the moving area estimation unit 311 has a coordinate position (hereinafter, the area) in the image of an area that is equal to or larger than a predetermined first size and equal to or smaller than a predetermined second size. And the moving area information indicating the vertical and horizontal widths of the area is supplied to the movement direction calculation unit 306 and the product estimation unit 305. The moving region estimation unit 311 also supplies the latest image data sent from the image capturing unit 102 to the motion direction calculation unit 306. In the third embodiment, the region estimated to have motion is handled as a rectangle, but may be a circle, an ellipse, or a free shape. In this case, an amount such as a radius is supplied to the motion direction calculation unit 306 and the product estimation unit 305, but a mask image indicating a region may be supplied to the motion direction calculation unit 306 and the product estimation unit 305.

The movement direction calculation unit 306 is a movement direction specification unit that specifies the movement direction of the product included in the region from the region specified by the movement region estimation unit 311.
Specifically, the motion direction calculation unit 306 receives the latest image data and motion region information from the motion region estimation unit 311. Here, the motion direction calculation unit 306 holds the latest image data, current frame image data, and past frame image data sent from the motion region estimation unit 311, as with the motion region estimation unit 311. The frame memory 107 is controlled. Here, since the frame memory 312 and the frame memory 107 hold similar image data, they can be combined into one, but in the following description, they are assumed to be separate frame memories.
The motion direction calculation unit 306 calculates the motion vector of the product from the vertical and horizontal widths of the product region, the coordinate position, the current frame image data, and the past frame image data indicated by the motion region information. Then, the motion direction calculation unit 306 supplies the motion vector, the motion area information, and the current frame image data to the marketing information determination unit 308. A method for calculating the motion vector will be described later.

The product estimation unit 305 is a product specification unit that specifies the product name of the product included in the area specified by the dynamic region estimation unit 311.
Specifically, the product estimation unit 305 receives the motion region information from the motion region estimation unit 311 and estimates the product name by the same method as the product estimation unit 105 in the first and second embodiments. The product estimation unit 305 supplies product information and product area information to the marketing information determination unit 308 according to the estimation result.

The marketing information determination unit 308 receives a motion vector, moving region information, and current frame image data from the motion direction calculation unit 306, and receives product information and product region information from the product estimation unit 305. Then, the marketing information determination unit 308 determines a motion vector by collating the product estimation result of the product estimation unit 305 with the motion area information of the motion direction calculation unit 306.
When the product information from the product estimation unit 305 indicates that there is no product that has been picked up by the customer 2, the marketing information determination unit 308 does not record anything and terminates the process or there is no product. Is recorded in the data recording unit 109. On the other hand, when the product information indicates that there is a product, the marketing information determination unit 308 records the product name and product area information in the data recording unit 109.

  Note that the product estimation unit 305, the motion direction calculation unit 306, the marketing information determination unit 308, and the motion area estimation unit 311 may be realized by the CPU executing a program. Also, some or all of these may be implemented in hardware by an integrated logic IC such as an ASIC or FPGA. Furthermore, some or all of these may be implemented by software such as a DSP.

  The operation of marketing information collection apparatus 300 according to Embodiment 3 configured as described above will be described below.

FIG. 8 is a flowchart showing processing for calculating a moving area in the moving area estimation unit 311.
As described above, the moving region estimation unit 311 captures the current frame image data and the past frame image data from the image capture unit 102 or the frame memory 312 (S20). As described above, the current frame image data and the past frame image data are image data of RGB color images.

  Next, the moving region estimation unit 311 generates absolute difference image data from the image of the current frame image data and the image of the past frame image data (S21). Here, a method of generating an absolute difference image will be described.

但し、ｘは、水平方向の画素位置、ｙは、垂直方向の画素位置、Ｄ_ｘｙは、（ｘ，ｙ）に位置する画素の絶対差分値である。Ｒ_ｘｙは、現フレーム画像データの（ｘ，ｙ）に位置する画素のコンポーネントＲの値、Ｇ_ｘｙは、現フレーム画像データの（ｘ，ｙ）に位置する画素のコンポーネントＧの値、Ｂ_ｘｙは、現フレーム画像データの（ｘ，ｙ）に位置する画素のコンポーネントＢの値である。また、Ｒ’_ｘｙは、過去フレーム画像データの（ｘ，ｙ）に位置する画素のコンポーネントＲの値、Ｇ’_ｘｙは、過去フレーム画像データの（ｘ，ｙ）に位置する画素のコンポーネントＧの値、Ｂ’_ｘｙは、過去フレーム画像データの（ｘ，ｙ）に位置する画素のコンポーネントＢの値である。
なお、動領域推定部３１１は、画像データをＲＧＢ系からＹＣｂＣｒ系に変換し、輝度（Ｙ）の絶対差分値を求めてもよく、また、ＲＧＢから代表して例えばＧ値を用いて絶対差分値を求めてもよい。以降では、上記（１０）式に基づいて算出したものとして説明する。 The absolute difference image data is calculated in units of pixels according to the following equation (10).

However, x is a pixel position in the horizontal direction, y is a pixel position in the vertical direction, and D _xy is an absolute difference value of a pixel located at (x, y). R _xy is the value of the component R of the pixel located at (x, y) of the current frame image data, G _xy is the value of the component G of the pixel located at (x, y) of the current frame image data, and B _xy Is the value of component B of the pixel located at (x, y) of the current frame image data. R ′ _xy is the value of the component R of the pixel located at (x, y) in the past frame image data, and G ′ _xy is the value of the component G of the pixel located at (x, y) in the past frame image data. The value B ′ _xy is the value of the component B of the pixel located at (x, y) of the past frame image data.
Note that the moving region estimation unit 311 may convert the image data from the RGB system to the YCbCr system to obtain the absolute difference value of the luminance (Y), and represent the absolute difference using, for example, the G value as a representative of RGB. A value may be obtained. In the following description, it is assumed that the calculation is based on the above equation (10).

但し、Ｄ’_ｘｙは、（ｘ，ｙ）に位置する画素の閾値処理後の値、Ｔ_Ｄは、予め定められた閾値である。
ここでは、Ｄ’_ｘｙは、８ｂｉｔ整数型としているが、０又は１をとる２値としてもよい。また、動領域推定部３１１は、ステップＳ２１及びＳ２２での計算を便宜上分離して行っているが、まとめて計算してもよい。
動領域推定部３１１は、このようにすべての画素について計算する。そして、計算されたＤ’_ｘｙにより構成される画像データを閾値画像データとする。 Next, the moving region estimation unit 311 performs threshold processing on the absolute difference image data generated in step S20 (S22). The threshold processing in step S22 is performed using the following equation (11).

However, D _'xy is (x, y) values after threshold processing of pixels positioned, _{T D} is a predetermined threshold.
Here, D ′ _xy is an 8-bit integer type, but it may be a binary value that takes 0 or 1. In addition, although the moving region estimation unit 311 performs the calculations in steps S21 and S22 separately for convenience, the calculation may be performed collectively.
The moving region estimation unit 311 calculates for all the pixels in this way. Then, the image data constituted by the calculated D ′ _xy is set as threshold image data.

Next, the moving region estimation unit 311 applies the expansion and contraction of the image of the morphological operation to the generated threshold image data. This calculation may be performed by a general method in the field of image processing, and is not particularly defined here. This process has an effect of connecting nearby areas and is a filter process for noise. For this reason, the expansion and contraction processing of the image may be omitted.
Next, the moving region estimation unit 311 obtains a group of rectangles circumscribing a portion having a pixel value of 255 for the threshold image data. FIG. 9 shows an example of a circumscribed rectangle. The hatched portion in FIG. 9 is a portion where D ′ _xy = 0, and the blank portion is a portion where D ′ _xy = 255. The circumscribed rectangle is a rectangle RE that is in contact with the outside of the blank portion as shown in FIG. Note that, as shown in FIG. 9, a part of the circumscribed rectangle RE includes a hatched portion, but when it is in contact with the blank portion, it is regarded as one lump.
One or more circumscribed rectangles may be found in the threshold image data. If the circumscribed rectangle at this time is RE _n , n is the number of circumscribed rectangles found. However, when n = 0, the circumscribed rectangle is not found.

第２の閾値処理は、下記の（１３）式に従う。

ここで、ｗ_ｎは、外接矩形ＲＥ_ｎの横幅を示し、ｈ_ｎは、外接矩形ＲＥ_ｎの縦幅を示す。Ｔ_ｗ１及びＴ_ｈ１は、０よりも大きい第１の閾値、Ｔ_ｗ２及びＴ_ｈ２は、０よりも大きい第２の閾値である。これら２つの式を満たす外接矩形を閾値外接矩形ＲＥ_ｃとする。但し、ｃは、ｃ＝０，・・・，ｋ（ｋ≦ｎ）を満たす正の整数である。この場合、閾値処理を満たした外接矩形がｋ個あることになる。
次に、ｋ個の閾値外接矩形ＲＥ_ｃのそれぞれの現フレーム画像データにおける左上の座標位置を（ｘ_ｃ，ｙ_ｃ）、矩形の大きさを（ｗ_ｃ，ｈ_ｃ）とする。最終的に動領域推定部３１１は、閾値外接矩形ＲＥｃを示す（ｘ_ｃ，ｙ_ｃ）及び（ｗ_ｃ，ｈ_ｃ）を動領域情報として、商品推定部３０５及び動き方向算出部３０６に供給し、画像取込部１０２から送られてきた最新の画像データを動き方向算出部３０６に供給する。 Next, the process proceeds to step S23. In step S23, the moving region estimation unit 311 filters these circumscribed rectangles by the first threshold processing and the second threshold processing.
The first threshold processing follows the following equation (12).

The second threshold processing follows the following equation (13).

Here, _{w n} denotes the width of the circumscribed rectangle RE _{n, h n} denotes a vertical width of the circumscribed rectangle RE _n. T _w1 and T _h1 are first threshold _values greater than 0, and T _w2 and T _h2 are second threshold _values greater than 0. A circumscribed rectangle satisfying these two expressions is defined as a threshold circumscribed rectangle RE _c . However, c is a positive integer satisfying c = 0,..., K (k ≦ n). In this case, there are k circumscribed rectangles that satisfy the threshold processing.
Next, assume that the upper left coordinate position of each of the k threshold circumscribing rectangles RE _c in the current frame image data is (x _c , y _c ), and the size of the rectangle is (w _c , h _c ). Finally, the moving region estimation unit 311 supplies (x _c , y _c ) and (w _c , h _c ) indicating the threshold circumscribed rectangle REc to the product estimation unit 305 and the movement direction calculation unit 306 as moving region information. The latest image data sent from the image capturing unit 102 is supplied to the motion direction calculating unit 306.

Next, calculation of the movement direction in the movement direction calculation unit 306 will be described. However, the operation of the frame memory 107 is as described above.
FIG. 10 is a flowchart showing processing at the time of motion direction calculation in the motion direction calculation unit 306.
As described above, the motion direction calculation unit 306 takes in the current frame image data and the past frame image data from the motion region estimation unit 311 or the frame memory 107 (S30). As described above, the current frame image data and the past frame image data are image data of RGB color images.

Next, the motion direction calculation unit 306 has a coordinate position (x _c , y _c ) and a size (w _c , h _c ) of the threshold circumscribed rectangle RE _c indicated by the motion region information sent from the motion region estimation unit 311. ) To obtain a motion vector by a block matching method between the current frame image data and the past frame image data. The block matching method is the same as the method described above with reference to FIG. However, in the third embodiment, a threshold circumscribed rectangle is used as the rectangular region Rect _c instead of the sliding window.
Finally, the motion direction calculation unit 306 sends the motion vector (v ^c _x , v ^c _y ) = (x _c −x ′ _c , y _c −y ′ _c ) and the current frame image data to the marketing information determination unit 308. Supply.

Next, the product estimation unit 305 will be described. Here, for each threshold circumscribed rectangle indicated by the moving area information passed from the moving area estimation unit 311, the similarity with the product data stored in the product data storage unit 104 is calculated, and the customer 2 picks up. Or estimate the returned merchandise. This product estimation is performed in the same manner as the estimation in the product estimation unit 105 according to the first and second embodiments. However, in Embodiment 3, a threshold circumscribed rectangle is used as the rectangular region Rect _c instead of the sliding window. For example, the product estimation unit 305 obtains a distance dist between the feature amount calculated from the threshold circumscribed rectangle and the feature amount stored in the product data storage unit 104. Since the distance dist indicates the distance of the feature amount calculated from the two rectangles, the reciprocal of the distance dist is the similarity. The similarity is R = 1 / dist.

Specifically, the product estimation unit 305 obtains a feature amount for each threshold circumscribed rectangle RE _c given from the moving region estimation unit 311. Next, the product estimation unit 305 obtains the similarity R _p between these feature amounts and the feature amounts in the product data storage unit 104. A product whose similarity R _p satisfies the above-described expression (2) is estimated to be a product included in the threshold circumscribed rectangle RE _c .
Finally, the product estimation unit 305 includes moving area information indicating the left end position (x _c , y _c ) and size (w _c , h _c ) of the threshold circumscribed rectangle RE _c and the product name of the estimated product. Are supplied to the marketing information determination unit 308. However, in the case of the threshold circumscribed rectangle RE _c where no product is found, the product estimation unit 305 determines that the product name is no product or removes the rectangle.

Next, the determination of the movement direction in the marketing information determination unit 308 will be described.
The marketing information determination unit 308 determines whether the customer 2 has picked up or returned the product based on the motion vector (v ^c _x , v ^c _y ) passed from the motion direction calculation unit 306. In the arrangement as shown in FIG. 1, it is assumed that the image Im3 of the current frame image data is photographed with the positional relationship as shown in FIG. As shown in Figure 5, with respect to customer 2, if the shelves 1 are arranged in the upper in the image Im3 of the current frame image data, y component v ^c _y of the motion vector V is the ( When the expression 7) is satisfied, the marketing information determination unit 308 determines that the customer 2 picks up the product. On the other hand, it is determined that the y component v ^c _y of the motion vector V when meeting the above (8), marketing information determination unit 308, returned goods. Furthermore, when the above expression (9) is satisfied, the marketing information determination unit 308 can determine that the customer 2 has determined the product only when the expression (6) is satisfied first. Similarly, even when the customer 2 and the product shelf 1 are arranged differently from those in FIG. 5, the marketing information determination unit 308 can determine the movement based on the inclination of the motion vector V.

（１４）式を満たす場合に、マーケティング情報判定部３０８は、商品名並びに動領域情報（ｘ_ｃ２，ｙ_ｃ２）及び（ｗ_ｃ２，ｈ_ｃ２）を、マーケティング情報としてデータ記録部１０９に保存するか、外部へ配信するか、又は、その両方を実施する。また、マーケティング情報判定部３０８は、実施の形態２で述べたように日付及び時刻情報を付加することもできる。 Next, the marketing information determination unit 308 includes a threshold circumscribed rectangle (here, (x _c1 , y _c1 ) and (w _c1 , h _c1 )) indicated by the movement area information from the movement direction calculation unit 306, and a product Information determined by the product estimation unit 305 that there is no product from the threshold circumscribed rectangles (here, (x _c2 , y _c2 ) and (w _c2 , h _c2 )) indicated by the moving region information from the estimation unit 305 Filter. Basically, (x _c1 , y _c1 ) and (x _c2 , y _c2 ), (w _c1 , h _c1 ) and (w _c2 , h _c2 ) should be the same value, but in the product estimation unit 305 If it is determined that there is no product by the threshold value processing, C2 <C1. Therefore, the marketing information determination unit 308 determines whether or not the following expression (14) is satisfied.

When the expression (14) is satisfied, the marketing information determination unit 308 saves the product name and the movement area information (x _c2 , y _c2 ) and (w _c2 , h _c2 ) in the data recording unit 109 as marketing information. , Deliver to the outside, or both. The marketing information determination unit 308 can also add date and time information as described in the second embodiment.

  In the third embodiment, as shown in FIG. 1, the marketing information collection device 300 is described as one device, but the imaging unit 101 and other portions may be configured as separate units. Good. For example, the marketing information collection device 300 may not include the imaging unit 101 and may receive image data from the imaging unit 101 of another unit. Further, the frame memory 103, the frame memory 312 and the frame memory 107 may be configured as one memory. The product data storage unit 104 and the data recording unit 109 may be combined into one, or may be combined into the frame memory 103, the frame memory 312 and the frame memory 107. Furthermore, the motion area estimation unit 311 and the motion direction calculation unit 306 may be combined into one.

  Since the marketing information collecting apparatus 300 according to the third embodiment is configured as described above, marketing such as picking up or returning a product that does not directly lead to purchase without attaching an RF tag to the product. Information can be collected. Furthermore, since the moving region estimation unit 311 is provided, marketing information can be collected at a higher speed than the sliding window method of the first embodiment.

Embodiment 4 FIG.
FIG. 11 is a block diagram schematically showing the configuration of the marketing information collecting apparatus 400 according to the fourth embodiment.
The marketing information collection device 400 according to Embodiment 4 includes an imaging unit 101, an image capture unit 102, a frame memory 103, product data and storage position storage unit 404, a product estimation unit 405, and a motion direction calculation unit. 306, a frame memory 107, a marketing information determination unit 308, a data recording unit 109, a moving region estimation unit 311, and a frame memory 312. Note that the information processing method according to the fourth embodiment is a method executed by the marketing information collecting apparatus 400 according to the fourth embodiment.
The marketing information collection device 400 according to the fourth embodiment is configured in the same manner as the marketing information collection device 300 according to the third embodiment except for the product data and storage position storage unit 404 and the product estimation unit 405. . In the following, product data and storage position storage unit 404 and product estimation unit 405 that are different from those in Embodiment 3 will be mainly described. The image capturing unit 102 reads image data from the frame memory 103 and supplies the image data to the product estimation unit 405 and the moving region estimation unit 311.

Similar to the product data storage unit 104 in the first embodiment, the product data and storage position storage unit 404 stores product feature quantities required when the product estimation unit 105 estimates a product, This is a product data storage unit for storing storage position information indicating the storage position (display position).
It is assumed that the image data of the current frame is an image Im4 as shown in FIG. In addition, it is assumed that the products in the product shelf 1 are displayed as shown in FIG. In such a case, from the image Im4, for example, it is difficult to distinguish the position of the product itself such as the product A, the product D, or the product G. However, as in the group 11, the group 12, or the group 13, It is possible to identify the merchandise county in the direction. Therefore, the product data and storage location storage unit 404 records storage location information indicating the storage location of the product in addition to the product name and the feature amount of the product. Here, for example, as shown in FIG. 14, the storage position information includes the coordinates (x _g , y _g ) and size (h _g , w _g ) of the end points of the areas where the respective groups are arranged. To do. Here, g is an identifier for identifying each group, and g = 11 to 13 in FIG.

The product estimation unit 405 is a product specification unit that specifies a product included in the image captured by the imaging unit 101.
Specifically, the product estimation unit 405 receives the image data of the current frame from the image capture unit 102 and estimates the product in the image data. For example, the product estimation unit 405 calculates the feature amount of the product area in the image data by using the general object recognition technology, and the feature amount and the product data and the product stored in the storage position storage unit 404. The product is estimated based on the similarity with the feature amount. In the fourth embodiment, the feature amount of the product for which the similarity is calculated is narrowed down based on the storage position (display position) of the product.
Hereinafter, the product estimation in the product estimation unit 405 will be specifically described. Here, it is assumed that the threshold circumscribed rectangles indicated by the moving region information passed from the moving region estimation unit 311 are (x _c , y _c ) and (w _c , h _c ). For example, as shown in FIG. 12, when the product 4 is picked up by the customer 2, from the image Im4 and the upper left coordinates (x _c , y _c ) of the threshold circumscribing rectangle, the product shelf 1 The group cannot be determined. However, as shown in FIG. 15, if the customer 2 is exactly the image Im5 where the product 4 is being taken out of the product shelf 1, the upper left coordinates (x _c , y _c ) of the threshold circumscribed rectangle, Based on the coordinates (x _g , y _g ) and the size (h _g , w _g ) indicated by the product data and the storage position information stored in the storage position storage unit 404, the product 4 from the group 11 or the group 12 Can be determined to have been removed.

For example, the product estimation unit 405 specifies the upper left coordinates (x _c , y _c ) of the threshold circumscribed rectangle by the coordinates (x _g , y _g ) and the size (h _g , w _g ) indicated by the storage position information. The group may be determined by whether or not it is included in the area where the group is arranged. In this case, the region where the group is located, the coordinates _(x g, _{y g)} and size _{(h g, w} g) and the like comprising a region specified by coordinates _(x g, _{y g)} and The area specified by the size (h _g , w _g ) may be an enlarged area. Further, the product estimation unit 405 determines that the distance between the upper left coordinates (x _c , y _c ) of the threshold circumscribed rectangle and the coordinates (x _g , y _g ) indicated by the storage position information is a predetermined threshold value. If it is shorter than that, it can be determined that the group is the group from which the product is taken out.

  Then, the product estimation unit 405 may estimate the product based on the similarity between the feature amount of the product belonging to the group in which the product is determined to be extracted and the feature amount of the product area in the image data. In this way, it is possible to narrow down the feature quantities of products to be collated from the positional relationship, instead of collating with the feature quantities of all the products stored in the product data and storage position storage unit 404. The product estimation after narrowing down the product candidates may be the same method as the product estimation unit 305 of the third embodiment. At this time, for example, as shown in FIG. 12, it is difficult to narrow down from the group information, and all the products stored in the product data and storage position storage unit 404 may be candidates, and tracking may be performed. An implementation that holds group information using technology may be used.

  Note that the product estimation unit 405 may be realized by the CPU executing a program. Further, the product estimation unit 405 may be realized in hardware by an integrated logic IC such as ASIC or FPGA. Furthermore, the product estimation unit 405 may be realized in software by a DSP or the like.

  In the fourth embodiment, as shown in FIG. 1, the marketing information collection device 400 is a single device, but the imaging unit 101 and other portions may be configured as separate units. For example, the marketing information collection device 400 may not include the imaging unit 101 and may receive image data from the imaging unit 101 of another unit. The frame memory 103, the frame memory 312 and the frame memory 107 may be configured as one memory. The product data and storage position storage unit 404 and the data recording unit 109 may be combined into one. Further, the product data and storage location storage unit 404, the data recording unit 109, the frame memory 103, the frame memory 312 and the frame memory 107 may be combined into one. Furthermore, the moving region estimation unit 311 and the movement direction calculation unit 306 may be combined into one.

  Since the marketing information collecting apparatus 400 according to the fourth embodiment is configured as described above, it does not directly lead to purchase without attaching an RF tag to the product, picks up or returns the product, etc. Can collect marketing information. Furthermore, since the product data and storage location storage unit 404 is provided, marketing information can be collected at a higher speed than in the third embodiment.

Embodiment 5. FIG.
FIG. 16 is a block diagram schematically showing a configuration of a marketing information collecting apparatus 500 according to the fifth embodiment.
The marketing information collecting apparatus 500 according to the fifth embodiment includes an imaging unit 101, an image capturing unit 102, a frame memory 103, a product data storage unit 104, product data and storage position storage unit 504, and a product estimation unit. 505, a motion direction calculation unit 506, a frame memory 107, a marketing information determination unit 508, a data recording unit 109, a motion region estimation unit 311, a frame memory 312, a person data storage unit 513, and a person estimation unit 514, a display operation detection unit 515, and a display product information registration unit 516.

The imaging unit 101, the image capture unit 102, the frame memory 103, the product data storage unit 104, and the data recording unit 109 in the marketing information collection device 500 according to the fifth embodiment are the same as the marketing information collection device 100 according to the first embodiment. It is constituted similarly.
The moving region estimation unit 311 and the frame memory 312 according to the fifth embodiment are configured in the same manner as the marketing information collection device 300 according to the third embodiment.
In the following, product data and storage position storage unit 504, product estimation unit 505, motion direction calculation unit 506, marketing information determination unit 508, person data storage unit 513, and person estimation unit, which are different from those in the first or third embodiment. The display operation detection unit 515 and the display product information registration unit 516 will be described.
The image capturing unit 102 reads image data from the frame memory 103 and supplies the image data to the product estimation unit 505, the moving region estimation unit 311, and the person estimation unit 514. The moving area estimation unit 311 supplies the moving area information to the product estimation unit 505 and the movement direction calculation unit 506.

The marketing information collection device 500 according to Embodiment 5 has a function of registering display merchandise information and a function of generating marketing information.
Note that the information processing method according to the fifth embodiment is a method executed by the marketing information collecting apparatus 500 according to the fifth embodiment.

  First, the structure regarding the function which registers display merchandise information among the functions of the marketing information collection apparatus 500 which concerns on Embodiment 5 is demonstrated.

  The product estimation unit 505 is a product specification unit that specifies the product name of the product included in the area specified by the dynamic region estimation unit 311. Specifically, product estimation section 505 receives the motion area information from motion area estimation section 311 and estimates the product name by the same method as product estimation section 305 in the third embodiment. The product estimation unit 505 supplies product information, product region information, and image data of the current frame to the display product information registration unit 516 in accordance with the estimation result.

  The movement direction calculation unit 506 is a movement direction identification unit that identifies the movement direction of the product included in this area from the area specified by the movement area estimation unit 311. Similar to the third embodiment, the movement direction calculation unit 506 calculates the product from the vertical and horizontal widths of the product area, the coordinate position, the current frame image data, and the past frame image data indicated by the motion area information. The motion vector of is calculated. Then, the motion direction calculation unit 506 supplies the motion vector, the motion area information, and the current frame image data to the marketing information determination unit 508 and the display motion detection unit 515.

  The person data storage unit 513 is a person data storage unit that stores information necessary for detecting a store-related person such as a store clerk from image data. For example, the person data storage unit 513 stores a texture feature amount of a uniform worn by store personnel. In general, since a store such as a convenience store often wears a specific uniform, it is possible to estimate whether it is a customer or a store-related person using these feature quantities.

The person estimation unit 514 is a person determination unit that determines whether or not there is a store related person such as a store clerk included in the image captured by the image capturing unit 101. For example, the person estimation unit 514 sequentially specifies the positions in the store-related person images included in the images sequentially captured by the imaging unit 101.
Specifically, the person estimation unit 514 receives the image data of the current frame from the image capturing unit 102, and detects a store related person in the image of the received image data. For example, the person estimation unit 514 calculates a feature amount in the image data using a person recognition technique. Then, the person estimation unit 514 calculates the similarity between the calculated feature quantity and the feature quantity stored in the person data storage unit 513, and if the similarity exceeds a predetermined threshold, the person related to the store It is judged that. Details of the person estimation unit 514 will be described later.

  The display operation detection unit 515 detects an operation of displaying a product on a product shelf by a store related person detected by the person estimation unit 514. Specifically, the display motion detection unit 515 determines whether or not the product is placed on the product shelf based on the motion vector information given from the motion direction calculation unit 506. When the display operation detection unit 515 determines that the display operation has been performed, the display operation detection unit 515 supplies the display position information and the current frame image data to the display product information registration unit 516. The display position information is represented by coordinate information (x, y) in an image captured by the imaging unit 101. Here, the details of the display motion detection based on the motion vector information from the motion direction calculation unit 506 will be described later.

When the display operation detection unit 515 detects a display operation by a store-related person, the display product information registration unit 516 determines which display information is provided based on the product information provided from the product estimation unit 505 and the display position information provided from the display operation detection unit 515. Displayed product information indicating where the product is displayed on the product shelf is generated and stored in the product data and storage position storage unit 504.
Specifically, the display product information registration unit 516 receives display position information and product information from the display operation detection unit 515. The display product information registration unit 516 generates display product information including the product name, the display position of the product, and the partial image data of the product extracted from the current frame image data, and stores the product data and storage position storage unit. Store in 504.

The product data and storage position storage unit 504 stores the feature amount of the product required when the product estimation unit 505 estimates the product, similarly to the product data and storage position storage unit 404 in the fourth embodiment. The product data storage unit stores storage position information indicating the storage position (display position) of the product.
The product data storage unit 104 is referred to as a first product data storage unit, and the product data and storage position storage unit 504 is also referred to as a second product data storage unit.

  Details of the operation in the function of registering the display merchandise information of the marketing information collecting apparatus 500 configured as described above will be described.

  The person estimation unit 514 estimates a store-related person such as a salesclerk included in the image data of the current frame using a sliding window method. Since the sliding window method is the same as the method in the product estimation unit 105 in the first embodiment, the description is omitted. The person estimation unit 514 calculates a feature amount for each sliding window to detect a person, and then calculates a similarity with the feature amount of the store-related person stored in the person data storage unit 513. When it exceeds a predetermined threshold, it is determined that the person is a store related person. This person detection method can be realized, for example, by combining the person detection method disclosed in Non-Patent Document 2 and the general object recognition method disclosed in Non-Patent Document 1.

  Here, the outline of the person detection method disclosed in Non-Patent Document 2 will be briefly described. For example, there is a method based on face detection. This is a rule-based method for determining whether the face is a face or a face other than the face from the luminance distribution or positional relationship of the eyes or mouth. In recent years, a person detection method using a statistical learning method has become mainstream, and this method can also be adopted. This consists of two phases: a learning phase in which a discriminator for identifying whether a person is a person or not and a detection phase in which a person is detected using the discriminator. In the learning phase, a large number of human images and other images are prepared, and local luminance gradient information or color information included in each of them is statistically learned. In the detection phase, a sliding window is set for the query image, local luminance information or color information is extracted for each sliding window, and compared with learning data to identify a person or not a person. .

The general object recognition method is the same as the method in the product estimation unit 105 in the above-described first embodiment, and a description thereof will be omitted. Finally, when a person related to the store is detected, the person estimation unit 514 displays the left end position (x _c , y _c ) of the rectangular area Rect _c in the sliding window and the rectangular size (w _c , h _c ). It supplies to the detection part 515.

Next, detection of display operation in the display operation detection unit 515 will be described. Here, the display motion detection unit 515 determines that the person related to the store is included in the image data by the person estimation unit 514, and the left end position (x _c , y _c ) of the rectangular area Rect _c from the person estimation unit 514 and the rectangle It works only when the size (w _c , h _c ) is sent. The display motion detection unit 515 includes a left end position (x _c , y _c ) of the rectangular area Rect _c indicating the person area given from the person estimation unit 514, a rectangular size (w _c , h _c ), and a motion direction calculation unit Based on the motion vector (v ^c _x , v ^c _y ) indicated by the vector information given from 506 and the coordinate position (x ′ _c , y ′ _c ), it is determined whether or not the merchandise has been displayed by the store personnel.

同様に、店舗関係者５と商品棚１が、図５とは異なる配置の場合でも、陳列動作検出部５１５は、動きベクトルＶの傾きにより動きを判定することができる。 For example, in the arrangement as shown in FIG. 1, as shown in FIG. 5, when the product shelf 1 is arranged at the upper part in the image Im <b> 3 of the current frame image data, When the y component v ^c _y of the vector V satisfies the following expression (15), the display operation detection unit 515 determines that the product has been displayed.

Similarly, even when the store-related person 5 and the merchandise shelf 1 are arranged differently from those in FIG. 5, the display operation detection unit 515 can determine the movement based on the inclination of the motion vector V.

この処理は、動き方向算出部５０６で算出された商品の動きが、店舗関係者５の動作によるものであることを担保するものである。店舗関係者５と座標が大きく異なるところで商品が動いていた場合、店舗関係者５による陳列動作ではないと考えられるため、（１６）式によってその判定が行われる。 However, the display motion detection unit 515 receives the left end position (x _c , y _c ) of the rectangular area Rect _c of the person received from the person estimation unit 514 and the coordinate position (x ′ _c , y received from the movement direction calculation unit 506). It is determined that the product is displayed only when the distance dist _hc to ' _c ) satisfies the following expression (16).

This process guarantees that the movement of the product calculated by the movement direction calculation unit 506 is due to the operation of the store-related person 5. If the product is moving in a place where the coordinates are significantly different from the store-related person 5, it is considered that it is not a display operation by the store-related person 5, and therefore the determination is made by equation (16).

  Next, the structure regarding the function which produces | generates marketing information among the functions of the marketing information collection apparatus 500 which concerns on Embodiment 5 is demonstrated.

  The product estimation unit 505 is a product specifying unit that specifies a product included in the image captured by the imaging unit 101. Specifically, the product estimation unit 505 calculates the feature amount of the product area in the image data using the general object recognition technology, and stores the feature amount and the product data and storage position storage unit 504. The product is estimated based on the similarity with the feature amount of the product being sold, but the feature amount of the product for which the similarity is determined is narrowed down based on the storage position (display position) of the product as in the fourth embodiment.

The person estimation unit 514 is a person detection unit that detects a customer included in the image captured by the imaging unit 101. For example, the person estimation unit 514 sequentially specifies the position in the customer image included in the images sequentially captured by the imaging unit 101. For example, as described above, the person estimation unit 514 may detect a store related person and detect a person other than the store related person as a customer.
Here, when the person estimation unit 514 detects a customer, the person estimation unit 514 supplies customer detection information indicating that the customer has been detected to the marketing information determination unit 508. In addition, the person estimation part 514 stops supply of customer detection information, when a customer is no longer detected.

  The marketing information determination unit 508 generates marketing information as in the third embodiment. However, the marketing information determination unit 508 records the marketing information in the data recording unit 109 only while the customer detection information is supplied from the person estimation unit 514.

  The image capturing unit 102, the product estimation unit 505, the motion direction calculation unit 506, the marketing information determination unit 508, the motion region estimation unit 311 and the person in the display product information generation unit 500a of the marketing information collection device 500 configured as described above. The estimation unit 514, the display operation detection unit 515, and the display product information registration unit 516 can be realized by the CPU executing a program stored in an external storage device such as a memory or an HDD. However, these are not limited to those realized as software on a computer system. For example, all or a part of these may be realized in hardware by an integrated logic IC such as ASIC or FPGA. Also, some or all of these may be realized in software by a DSP or the like. Alternatively, a component specialized for a specific application may be realized by hardware such as an expansion board mounted on the computer system, and other general-purpose components may be realized by software on the computer system. Good.

  In the fifth embodiment, as shown in FIG. 16, the marketing information collection device 500 is a single device, but the imaging unit 101 and other portions may be configured as separate units. For example, the marketing information collection apparatus 500 may not include the imaging unit 101 and may receive image data from the imaging unit 101 of another unit. Further, the frame memory 103, the frame memory 107, and the frame memory 312 may be configured as one memory. The person data storage unit 513, the product data storage unit 104, and the product data and storage position storage unit 504 may be combined into one, or may be combined with the frame memory 103, the frame memory 107, and the frame memory 312 described above.

  Since the marketing information collecting apparatus 500 according to the fifth embodiment is configured as described above, marketing such as picking up or returning a product that does not directly lead to purchase without attaching an RF tag to the product. Information can be collected. Furthermore, since it is possible to automatically register display product data in the product data and storage location storage unit 504, it is possible to collect marketing information without requiring a burden on the store in comparison with the fourth embodiment. Can do.

Embodiment 6 FIG.
FIG. 17 is a block diagram schematically showing a configuration of a marketing information collecting apparatus 600 according to the sixth embodiment.
The marketing information collection device 600 according to the sixth embodiment includes an imaging unit 101, an image capture unit 102, a frame memory 103, product data and attribute data storage unit 604, a product estimation unit 605, and a motion direction calculation unit. 306, a frame memory 107, a marketing information determination unit 308, a data recording unit 109, a moving region estimation unit 311, a frame memory 312, and an attribute determination unit 617. The information processing method according to the sixth embodiment is a method executed by the marketing information collection device 600 according to the sixth embodiment.
The marketing information collection device 600 according to the sixth embodiment is the same as the marketing information collection device 300 according to the third embodiment, except for the product data and attribute data storage unit 604, the product estimation unit 605, and the attribute determination unit 617. It is configured. Hereinafter, product data and attribute data storage unit 604, product estimation unit 605, and attribute determination unit 617, which are different from those in the third embodiment, will be mainly described. The image capturing unit 102 reads image data from the frame memory 103 and supplies the image data to the product estimation unit 605 and the moving region estimation unit 311. In addition, the moving region estimation unit 311 supplies the moving region information and the image data of the threshold circumscribed rectangle indicated by the moving region information to the attribute determination unit 617.

The attribute determination unit 617 determines the attribute of the product included in the region from the region specified by the moving region estimation unit 311.
Specifically, the attribute determination unit 617 receives the moving area information and the image data of the threshold circumscribed rectangle indicated by the moving area information from the moving area estimation unit 311, and stores the product included in the image data of the threshold circumscribed rectangle. Determine the attribute. Here, the attribute of the product is a feature of the product that appears in the image, such as the shape or color of the product. The attribute determination unit 617 will be specifically described.

  FIG. 18 is a flowchart showing the flow of processing of the attribute determination unit 617.

First, the attribute determination unit 617 identifies an attribute in the shape of the product included in the image data of the threshold circumscribed rectangle indicated by the moving region information given from the moving region estimation unit 311 (S40). For example, the attribute determination unit 617 classifies the outer shape into three types of portrait, landscape, and square based on the aspect ratio of the threshold circumscribed rectangle. Assuming that the threshold circumscribed rectangle has coordinates (x _c , y _c ) and size (w _c , h _c ), the aspect ratio Aspect can be obtained by the following equation (17).

ここで、Ｒ_{ａｓｐｅｃｔ}は、商品の形状の判定結果（形状属性）である。Ｄ_ｔｈｉｎは、１よりも小さい閾値であり、Ｄ_ｗｉｄｅは、１よりも大きい閾値である。これらの閾値は、商品の傾き又は撮像部１０１の傾きによる商品の形状変化に対する頑健性を担保するものである。 In addition, the determination of vertically long, horizontally long, or square follows the following equation (18).

Here, R _aspect is a determination result (shape attribute) of the shape of the product. D _thin is a threshold value smaller than 1, and D _wide is a threshold value larger than 1. These threshold values ensure robustness against changes in the shape of the product due to the product tilt or the tilt of the imaging unit 101.

Next, the attribute determination unit 617 identifies an attribute in the color of the product included in the image data of the threshold circumscribed rectangle given from the moving region estimation unit 311 (S41). For example, the attribute determination unit 617 classifies the product color based on the color component of the threshold circumscribed rectangle image data. The image data of the threshold circumscribed rectangle is assumed to be in RGB format. The attribute determination unit 617 calculates an average value of R, G, and B values of the threshold circumscribed rectangle according to the following equations (19) to (21).

但し、Ｄ_{ｃｏｌｏｒ}は、０よりも大きい閾値である。 Next, the attribute determination unit 617 rearranges R _ave , G _ave, and B _ave in descending order. Here, for convenience of explanation, it is assumed that R _ave > G _ave > B _ave .
When the maximum value (here, R _ave ) and the second largest value (here, G _ave ) satisfy the following expression (22), the attribute determination unit 617 includes the image data. The product color is determined to be a product (here, a red product) included in the maximum color.

However, _Dcolor is a threshold value greater than zero.

As described above, the attribute determination unit 617 determines the color of the maximum value as a candidate color when the difference between the maximum value and the second value is larger than the threshold value _Dcolor . On the other hand, when this expression is not satisfied, the attribute determination unit 617 determines that there is no dominant color. This threshold value D _color guarantees robustness against influences such as the color of the light source of the store, sunlight, or color correction of the imaging unit 101.
Further, this color candidate is a determination result (color attribute) of the color of the product, and is denoted as R _color . In the following, for convenience, R _color takes a value of 1, 2, 3, or 4. When R _color = 1, a red product, when R _color = 2, a blue product, R _color = 3 In this case, the product is a green product, and when R _color = 4, the product has no dominant color.

Then, the attribute determination unit 617 supplies R _aspect and R _color to the product estimation unit 605 as determination attribute information.
Note that the order of steps S40 and S41 is as shown in FIG. 18 for the sake of convenience, but may be reversed, and may be parallel processing.

Returning to FIG. 17, the product data and attribute data storage unit 604 stores the feature amount of the product that is required when the product estimation unit 605 estimates the product, similarly to the product data storage unit 104 in the first embodiment. A product data storage unit that stores attribute data indicating product attributes. The attribute data is data indicating the shape attribute R ′ _aspect and the color attribute R ′ _color calculated by the above-described method for each product.

  The product estimation unit 605 receives the motion area information and the threshold circumscribing rectangle image data passed from the motion region estimation unit 311 and the determination attribute information passed from the attribute determination unit 617, and estimates a product included in the threshold circumscribing rectangle. . For example, the product estimation unit 605 performs product estimation as follows.

FIG. 19 is a flowchart showing the estimation process of the product estimation unit 605.
First, the product estimation unit 605 filters the shape attribute R ′ _aspect stored in the product data and attribute data storage unit 604 using the shape attribute R _aspect indicated by the determination attribute information given from the attribute determination unit 617. By doing so, the candidates for the products are narrowed down (S50). In other words, products satisfying R _aspect = R ′ _aspect are narrowed down as candidates.

Next, the product estimation unit 605 uses the color attribute R _color indicated by the determination attribute information given from the attribute determination unit 617 to obtain the color attribute R ′ _color stored in the product data and attribute data storage unit 604. The product candidates are narrowed down by filtering (S51). In other words, products satisfying R _color = R ′ _color are narrowed down as candidates. However, when the color attribute passed from the attribute determination unit 617 is R _color = 4, that is, there is no color attribute, the color attribute may be narrowed down by R ′ _color = 4, but the narrowing by the color attribute in step S51 may be performed. You may cancel.

Then, the product estimation unit 605 estimates the product names for the product candidates narrowed down in steps S50 and S51 by the same method as the product estimation unit 305 of the third embodiment (S52).
Note that the order of step S50 and step S51 may be interchanged, or either one may be performed.

In the sixth embodiment, as shown in FIG. 1, the marketing information collection device 600 is a single device, but the imaging unit 101 and other portions may be separate units. For example, the marketing information collection device 600 may not include the imaging unit 101 and may receive image data from the imaging unit 101 of another unit. The frame memory 103, the frame memory 312 and the frame memory 107 may be configured as one memory. The product data / attribute data storage unit 604 and the data recording unit 109 may be combined into one. The product data and attribute data storage unit 604, the data recording unit 109, the frame memory 312, and the frame memory 107 may be combined. Furthermore, the moving region estimation unit 311 and the movement direction calculation unit 306 may be combined into one.
Further, the function of narrowing down the product storage position by the product data and storage position storage unit 404 and the product estimation unit 405 in the fourth embodiment is added to the product data and attribute data storage unit 604 and the product estimation unit 605. May be.

  The product estimation unit 605 and the attribute determination unit 617 may be realized by the CPU executing a program. Also, some or all of these may be implemented in hardware by an integrated logic IC such as an ASIC or FPGA. Furthermore, some or all of these may be implemented by software such as a DSP.

  Since the marketing information collecting apparatus 600 according to the sixth embodiment is configured as described above, marketing such as picking up or returning a product that does not directly lead to purchase without attaching an RF tag to the product. Information can be collected. Furthermore, since the product data and attribute data storage unit 604 is provided, marketing information can be collected at a higher speed than in the third embodiment.

Embodiment 7 FIG.
FIG. 20 is a block diagram schematically showing a configuration of a marketing information collecting apparatus 700 according to the seventh embodiment.
The marketing information collection device 700 according to the seventh embodiment includes an imaging unit 101, an image capture unit 102, a frame memory 103, a product data storage unit 104, a product estimation unit 105, a motion direction calculation unit 106, A frame memory 107, a marketing information determination and distribution unit 708, and a data recording unit 109 are provided. Note that the information processing method according to the seventh embodiment is a method executed by the marketing information collecting apparatus 700 according to the seventh embodiment.
The marketing information collection device 700 according to the seventh embodiment is configured in the same manner as the marketing information collection device 100 according to the first embodiment, except for the marketing information determination and distribution unit 708. Hereinafter, the marketing information determination and distribution unit 708 that is different from the first embodiment will be mainly described.

  The marketing information determination and distribution unit 708 generates marketing information including the movement of the product based on the product information given from the product estimation unit 105 and the motion vector information given from the motion direction calculation unit 106, and the data recording unit It functions as a marketing information determination unit stored in 109 and also functions as a transmission unit that transmits this marketing information to an external device. For example, the marketing information determination and distribution unit 708 distributes to a POS system or a monitoring recorder via a network such as Ethernet (registered trademark) or WIFI (Wireless Fidelity), or a dedicated interface. Alternatively, the marketing information determination and distribution unit 708 may transmit to the parent company's server or the like via the Internet or the like, as in a convenience store chain.

  For example, the marketing information determination and distribution unit 708, according to the presence / absence of the product passed from the product estimation unit 105, if there is no product taken in the customer's hand, does not record and distribute anything and ends the process? Or, information that there is no product is recorded and distributed to the outside. When there is a product, the marketing information determination and distribution unit 708 records and distributes the marketing information including the product name, the product region information, and the product movement determination result to the outside. As for the data to be recorded and distributed, the product portion may be cut out from the current frame image data and the product area information and recorded or distributed together. Further, both recording and distribution may be performed, or either of them may be performed.

  In addition, about the function as a marketing information determination part among the marketing information determination and distribution parts 708, CPU may be implement | achieved by running a program. Further, part or all of the function as the marketing information determination unit may be realized in hardware by an integrated logic IC such as ASIC or FPGA. Furthermore, a part or all of the functions as the marketing information determination unit may be realized in software by a DSP or the like. Furthermore, the function as a distribution unit in the marketing information determination and distribution unit 708 can be realized by the CPU controlling a communication device such as a NIC (Network Interface Card).

  Since the marketing information collecting apparatus 700 according to the seventh embodiment is configured as described above, it does not directly lead to purchase without attaching an RF tag to the product, picks up or returns the product, etc. Can collect marketing information. Furthermore, by providing the marketing information determination and distribution unit 708, the marketing information can be distributed to an external device.

  Further, the marketing information determination unit according to the second to sixth embodiments can be replaced with the marketing information determination and distribution unit 708 according to the seventh embodiment.

  100, 200, 300, 400, 500, 600, 700 Marketing information collection device, 101 imaging unit, 102 image capture unit, 103 frame memory, 104 product data storage unit, 404, 504 product data and storage location storage unit, 604 Product data and attribute data storage unit, 105, 305, 405, 505, 605 Product estimation unit, 106, 306, 506 Motion direction calculation unit, 107 Frame memory, 108, 208, 308, 508 Marketing information determination unit, 708 Marketing information Determination and distribution unit, 109 data recording unit, 210 clock unit, 311 moving region estimation unit, 312 frame memory, 513 person data storage unit, 514 person estimation unit, 515 display operation detection unit, 516 display product information registration unit, 617 Attribute determination unit.

Claims (13)

An imaging unit for sequentially capturing images including a part of the product shelf;
A product identification unit for identifying a product included in the image captured by the imaging unit;
A movement direction identification unit that identifies a movement direction of the product identified by the product identification unit;
Marketing that determines whether the product has been picked up from the product shelf or returned to the product shelf from the movement direction specified by the movement direction specifying unit, and generates marketing information including a determination result An information determination unit;
An information processing apparatus comprising: a data storage unit that stores marketing information generated by the marketing information determination unit. The product specifying unit sequentially specifies the product name and position of the product included in the images sequentially captured by the imaging unit,
The movement direction specifying unit specifies the movement direction of the product from the positions sequentially specified by the product specifying unit,
The information processing apparatus according to claim 1, wherein the marketing information determination unit includes the product name specified by the product specifying unit in the marketing information. A product data storage unit for storing the product name of the product displayed on the product shelf and the feature amount of the product;
The product specifying unit calculates a similarity between a feature amount of a plurality of regions included in an image captured by the imaging unit and a feature amount of a product stored in the product data storage unit. 3. The information according to claim 2, wherein the product name and area position of the product having the highest similarity are specified as the product name and position of the product included in the image captured by the imaging unit. Processing equipment. It further includes a clock unit that measures the date and time,
The information processing apparatus according to any one of claims 1 to 3, wherein the marketing information determination unit includes the date and time measured by the clock unit in the marketing information. In the images sequentially picked up by the image pickup unit, further comprising a moving region specifying unit for sequentially specifying a region having movement,
The product specifying unit specifies a product name of a product included in the area specified by the dynamic region specifying unit,
The movement direction identification unit identifies the movement direction of the product from the areas sequentially identified by the movement area identification unit,
The information processing apparatus according to claim 1, wherein the marketing information determination unit includes the product name specified by the product specifying unit in the marketing information. A product data storage unit for storing the product name and feature amount of the product displayed on the product shelf;
The product specifying unit calculates a similarity between the feature amount of the region specified by the moving region specifying unit and the feature amount of the product stored in the product data storage unit, and the calculated similarity is The information processing apparatus according to claim 5, wherein the product name of the highest product is specified as the product name of the product included in the area specified by the moving area specifying unit. The product data storage unit further stores a display position of the product on the product shelf,
The product specifying unit is characterized by the product stored in the product data storage unit based on the position of the area specified by the moving region specifying unit and the display position stored in the product data storage unit. The information processing apparatus according to claim 6, wherein the information is narrowed down and a similarity between the feature amount of the area specified by the moving area specifying unit and the reduced feature quantity is calculated. A person data storage unit for storing feature quantities of store-related persons;
A person determination unit that determines whether or not there is a store related person included in the image captured by the imaging unit,
The person determination unit calculates a similarity between a feature amount of a plurality of regions included in an image captured by the imaging unit and a feature amount representing a store related person stored in the person data storage unit, and calculates the calculation The information processing apparatus according to claim 7, wherein a store-related person is determined to exist if the similarity degree exceeds a predetermined threshold. A display operation detection unit that detects a product display operation on the product shelf when the person determination unit determines that a store-related person exists;
When the person determination unit determines that there is a store related person, the display position of the product specified by the product specifying unit is stored in the product data storage based on the product display operation detected by the display operation detection unit. The information processing apparatus according to claim 8, further comprising a display merchandise information registration unit stored in the unit. An attribute determining unit that determines an attribute of a product included in the region from the region specified by the moving region specifying unit;
The product data storage unit further stores an attribute of a product displayed on the product shelf,
The product specifying unit narrows down the feature amount of the product stored in the product data storage unit with the attribute determined by the attribute determination unit, the feature amount of the region specified by the moving region specifying unit, and The information processing apparatus according to claim 6, wherein the similarity with the narrowed-down feature amount is calculated. The information processing apparatus according to claim 10, wherein the attribute is at least one of a color and a shape. The information processing apparatus according to any one of claims 1 to 11, further comprising a transmission unit that transmits the marketing information. Take images sequentially including part of the product shelf,
Identify the product included in the captured image,
Identify the direction of movement of the identified product;
From the identified direction of movement, determine whether the product has been picked up from the product shelf or whether the product has been returned to the product shelf, and generate marketing information including a determination result,
An information processing method characterized by storing the generated marketing information.

Images