JP2012173903A - Rack monitoring device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce the labor and costs of device installation.SOLUTION: A movement calculation part 103 divides an image photographed by a rack peripheral camera 101 into a plurality of blocks, and calculates the movement of the photographic image of each block. A rack area estimation part 108 integrates the size of the movement of the photographic image of each block in a predetermined direction to calculate an integrated value, and estimates a rack area on the basis of a difference between the averages of the respective integrated values of those two divided areas and the integrated value. A rack area entry determination part 106 determines whether or not the portion of a person has entered the rack area on the basis of person area information and rack area information. An area difference calculation part 109 calculates an area difference between time before and after a portion of the person enters the rack area. A customer interest determination part 110 determines that the person is interested in the merchandise of the rack when the area difference is large.

Description

  The present invention relates to a shelf monitoring device, and more particularly to a shelf monitoring device that monitors whether a customer is interested in a product displayed on a product display shelf such as a retail store.

  In order to check whether a customer is interested in a product, especially when a product advertisement generally called POP (Point Of Purchase advertising) is installed on a product display shelf in a retail store or the like, This is important for judging the effectiveness and quality of product display. Conventionally, a technique for detecting such customer interest is known.

  For example, in Patent Literature 1, based on a video image of a product shelf and a customer located in the vicinity of the product shelf, the time when the customer stays in front of the product shelf and a part of the customer's body are in the product shelf. There has been proposed a customer motion analysis device that detects that a customer has picked up a product by detecting entry. According to this customer behavior analysis apparatus, since the purchase behavior of the product by the customer can be detected, it can be determined that the customer is interested in the product.

  For example, Patent Literature 2 proposes a product display state monitoring system that detects a product displacement in a product shelf by using a displacement detection sensor attached to the product and determines that a customer has picked up the product. . According to this merchandise display state monitoring system, the purchase behavior of merchandise by the customer can be detected, so it can be determined that the customer is interested in the merchandise.

JP 2009-048430 A JP 2009-237696 A

  In the customer motion analysis device described in Patent Document 1, in a video image of a product shelf and a customer located around the product shelf, an area corresponding to the product shelf is set in advance, Is detected to have entered the product shelf. Therefore, when installing the customer behavior analysis apparatus in a retail store or the like, it is a problem that it takes time and effort to set the product shelf area in the video.

  Moreover, in the product display state monitoring system described in Patent Document 2, a displacement detection sensor is attached to each product to detect product displacement. For this reason, the problem is that it takes time and cost to install the displacement detection sensor.

  The present invention has been made in view of the above points, and an object of the present invention is to provide a shelf monitoring device that can reduce the labor and cost of installing the device.

  In order to achieve the above object, the shelf monitoring apparatus of the present invention divides a captured image area of a predetermined range including a monitored shelf into a plurality of blocks, and calculates the movement of the captured image for each block. One of two blocks adjacent to each other in a direction orthogonal to the predetermined direction among the plurality of blocks is calculated by integrating the motion calculation means and the magnitude of the motion of the captured image of each block in the predetermined direction. An area consisting of the block having the largest difference between the average of the integrated values of the first area and the average of the second area consisting of the other block and the average of the integrated values being smaller is the shelf area. A shelf area estimation means for estimating a person from a captured image, a shelf area entry determination means for determining that a part of the person has entered the shelf area, and a shelf area entry determination means. Part of the shelf area An area difference calculation unit that calculates a difference between the captured images of the shelf area at a time before and after the time when it is determined that the vehicle has entered, and a threshold that is set in advance by a difference between the captured images of the shelf area calculated by the area difference calculation unit It is characterized by comprising customer interest determination means for determining that the person is interested in the product on the shelf when the size is larger.

  In order to achieve the above object, the shelf monitoring apparatus of the present invention divides a captured image area of a predetermined range including a monitored shelf into a plurality of blocks, and moves the captured image for each block. The motion calculating means for calculating and the integrated value of the motion of the captured image for each block are integrated to calculate an integrated value, and one of two blocks adjacent to the direction orthogonal to the predetermined direction among the plurality of blocks An area consisting of the block having the largest difference between the average of the integrated values of the first area consisting of the blocks and the average of the integrated values of the second area consisting of the other block and having the smaller average of the integrated values A shelf area estimation means for estimating a shelf area, a person detection means for detecting a person from a photographed image, a shelf area entry determination means for determining that a part of the person has entered the shelf area, and a peripheral position of the shelf Detect human gaze When it is determined by the line-of-sight detection means and the shelf area entry determination means that a part of the person has entered the shelf area, and the line-of-sight detection means detects that the person's line of sight is facing the shelf direction, Customer interest determination means for determining that the product on the shelf is interested is provided.

  ADVANTAGE OF THE INVENTION According to this invention, the effort and cost for installing an apparatus in the shelf monitoring apparatus which detects the customer's interest with respect to goods in a retail store etc. can be made low.

It is a block diagram of 1st Embodiment of the shelf monitoring apparatus of this invention. It is a figure which shows the example of installation of the shelf monitoring apparatus of FIG. It is a figure which shows an example of the shelf periphery image image | photographed with the shelf periphery camera of the shelf monitoring apparatus of FIG. FIG. 1 is a flowchart for explaining a main process for one cycle in a shelf area estimation process that is periodically operated at a predetermined time interval in the shelf area estimation process in the shelf area estimation unit of the shelf monitoring apparatus of FIG. 1. It is. It is a figure explaining the block division | segmentation of the screen in the shelf area estimation process of the shelf area estimation part of the shelf monitoring apparatus of FIG. It is a flowchart explaining the shelf area approach determination process in the shelf area approach determination part of the shelf monitoring apparatus of FIG. It is a flowchart explaining the customer interest determination process in the customer interest determination part of the shelf monitoring apparatus of FIG. It is a block diagram of 2nd Embodiment of the shelf monitoring apparatus of this invention. It is a figure which shows the example of installation of the shelf monitoring apparatus of FIG. It is a flowchart explaining the customer interest determination process in the customer interest determination part of the shelf monitoring apparatus of FIG.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
<Configuration of shelf monitoring device>
FIG. 1 shows a block diagram of a first embodiment of a shelf monitoring apparatus according to the present invention. As shown in the figure, the shelf monitoring apparatus 1 of the present embodiment includes a shelf peripheral camera 101, an image storage unit 102, a motion calculation unit 103, a motion information storage unit 104, a person detection unit 105, a shelf Area entry determination unit 106, shelf area information storage unit 107, shelf area estimation unit 108, area difference calculation unit 109, customer interest determination unit 110, customer interest information storage unit 111, and pre-entry time storage unit 112 It consists of.

  The shelf peripheral camera 101 photographs the product shelf and the surrounding installation area. Then, the shelf peripheral camera 101 stores the photographed image in the image storage unit 102 in association with the photographing time at a frequency of, for example, 30 fps, that is, 30 frames per second.

  The image storage unit 102 is configured by a storage device such as a hard disk, for example, and stores a captured image supplied from the shelf peripheral camera 101 and a shooting time in association with each other.

  The motion calculation unit 103 operates at a predetermined time interval such as 1 second, reads an image from the image storage unit 102, and calculates a motion for each block when the image is divided into appropriate blocks, so-called optical flow. To do. Specifically, the last image stored in the image storage unit 102, that is, the latest frame, and the image stored only a predetermined time before the last stored image, that is, the latest frame. Frames that are a predetermined number of frames before are read out, the image is divided into blocks of a predetermined size, and motion information is calculated by a known technique of optical flow calculation such as block matching. Then, the motion calculation unit 103 causes the motion information storage unit 104 to store the calculated motion information, that is, a vector value representing the direction and magnitude of movement for each block of the image. The predetermined time or the predetermined number of frames is appropriately determined in consideration of motion calculation accuracy. The predetermined size of the block is appropriately determined in consideration of the accuracy of motion calculation. The motion information storage unit 104 is configured by a storage device such as a hard disk, for example, and stores the motion information supplied from the motion calculation unit 103.

  The person detection unit 105 operates at a predetermined time interval such as 1/30 second, for example, reads the image and the shooting time of the read image from the image storage unit 102, and detects a person from the read image. Specifically, for example, the moving object detection process described in Patent Document 1 is performed, and the detected moving object is a person. Then, for the detected person, the person area information in the image and the shooting time of the image are supplied to the shelf area entry determination unit 106.

  When the person area information and the image capturing time are supplied from the person detection unit 105, the shelf area entry determination unit 106 performs a shelf area entry determination. Specifically, the shelf area entry determination unit 106 reads the shelf area information from the shelf area information storage unit 107 and determines whether a part of the person has entered the shelf area based on the person area information and the shelf area information. Determine. Then, the image capturing time of the image, information indicating whether or not the image has entered, and information indicating the area in which the image has entered when the image has entered are supplied to the customer interest determination unit 110. The shelf area information storage unit 107 is configured by a storage device such as a hard disk, and stores shelf area information indicating the area of the shelf in the image.

  The shelf area estimation unit 108 operates at a predetermined time interval such as one day, for example, and performs a shelf area estimation process. Then, the shelf area estimation unit 108 causes the shelf area information storage unit 107 to store shelf area information indicating the shelf area in the estimated image. Note that the shelf monitoring device 1 may be configured to include operation means such as a button for operating the shelf area estimation process, and the shelf area estimation process may be operated by a user operation.

  The area difference calculation unit 109 calculates the difference between the captured images of the shelf area at times before and after the time when the shelf area entry determination unit 106 determines that a part of the person has entered the shelf area. Specifically, the area difference calculation unit 109 receives the supplied first area when the area of the image whose difference is to be calculated, the first imaging time, and the second imaging time are supplied from the customer interest determination unit 110. The first image and the second image photographed at the respective photographing time and the second photographing time are read from the image storage unit 102, and the customer interest determination unit for the read first image and second image The difference in the area supplied from 110 is calculated, and the calculated area difference is supplied to the customer interest determination unit 110. The calculation of the area difference is performed, for example, by obtaining an average of the difference values of the pixel values in the area in the first image and the second image.

  The customer interest determination unit 110 performs a customer interest determination process when a signal indicating that there is an entry into the shelf area or no entry and a shooting time of the image are supplied from the shelf area entry determination unit 106. Specifically, when the information indicating whether or not the customer interest determination unit 110 has entered the shelf area indicates that the shelf area has been entered, the customer interest determination unit 110 sets the shooting time of the image before entering the image before entry. The time is stored in the pre-entry time storage unit 112 as time.

  Further, the customer interest determination unit 110 supplies the region difference calculation unit 109 with the region of the image whose difference is to be calculated, the shooting time of the first image, and the shooting time of the second image, and instructs the calculation of the region difference. To do. Then, the image difference supplied from the region difference calculation unit 109 is compared with a preset threshold value, the customer's interest in the product displayed on the product shelf is determined according to the comparison result, and the determined customer's interest Customer interest information is stored in the customer interest information storage unit 111.

  The customer interest information storage unit 111 is configured by a storage device such as a hard disk, for example, and stores customer interest information supplied from the customer interest determination unit 110. The pre-entry time storage unit 112 is configured by a storage device such as a hard disk, for example, and stores the pre-entry time supplied from the customer interest determination unit 110.

The shelf monitoring apparatus 1 of the present embodiment includes an image storage unit 102, a motion information storage unit 104, a shelf area information storage unit 107, a customer interest information storage unit 111, and a pre-entry time storage unit 112. Although configured as separate storage devices, each storage unit may be partially shared, or all storage units may be shared by a single storage device.
<Installation example of shelf monitoring device>
Next, how to install the shelf monitoring device 1 in a retail store or the like will be described with a specific example.

  FIG. 2 shows an installation example of the first embodiment of the shelf monitoring device according to the present invention. In the figure, the installation example 2 assumes that the products for which the customer interest is detected by the shelf monitoring device 1 are displayed on the product shelf 203 at an angle close to the horizontal.

  In such an environment, the shelf peripheral camera 101 provided in the shelf monitoring device 1 is installed above the product shelf 203 that can simultaneously photograph the customer 202 and the product shelf 203. Thereby, the customer 202, the passage through which the customer 202 passes, and the product shelf 203 are simultaneously photographed in the photographed image by the shelf peripheral camera 101.

FIG. 3 shows an example of a shelf peripheral image photographed by the shelf peripheral camera 101 of the shelf monitoring device 1. By installing the shelf peripheral camera 101 as shown in FIG. 2, the shelf peripheral image 3 becomes the shelf region image 302 at the lower portion and the passage region image 301 through which the customer 202 passes as shown in FIG. . Note that a part of the image of the customer 202 does not enter the shelf area image 302 unless the customer 202 performs an operation such as reaching for the product shelf.
<Operation of the shelf monitoring device>
The shelf monitoring device 1 mainly performs shelf region estimation processing, shelf region entry determination processing, and customer interest determination processing. Therefore, these processes will be described in detail below.
≪Shelf area estimation process≫
First, the shelf area estimation process in the shelf area estimation unit 108 of the shelf monitoring apparatus 1 will be described with reference to FIG.

  FIG. 4 shows a main process for one cycle of the shelf area estimation process that is periodically operated at a predetermined time interval in the shelf area estimation process in the shelf area estimation unit 108 of the shelf monitoring apparatus 1. It is the shown flowchart.

  When the shelf area estimation process is started, the shelf area estimation unit 108 reads motion information from the motion information storage unit 104 (step S101). The motion information storage unit 104 stores and stores motion information periodically calculated by the motion calculation unit 103 at predetermined time intervals. Here, all the motion information is read out.

  Next, the shelf area estimation unit 108 calculates the magnitude of the movement in the horizontal direction (step S102). A specific calculation method will be described in detail below with reference to the drawings.

FIG. 5 shows an explanatory diagram of block division of the screen in the shelf area estimation process of the shelf area estimation unit 108 of the shelf monitoring device 1. FIG. 5 is a diagram in which the entire image area of the shelf peripheral image 3 shown in FIG. 3 is divided into blocks, where M is the number of blocks in the horizontal direction x and N is the number of blocks in the vertical direction y when divided into blocks. Numbers from 1 to M and 1 to N are assigned. Here, in FIG. 5, the motion of the image of the i-th block in the horizontal direction and the j-th block in the vertical direction at time t is
[X _t (i, j), y _t (i, j)]
As shown in FIG. 4, the absolute value | x _t (i, j) | of the x-axis value is represented as the horizontal size when represented by a two-dimensional vector in the horizontal direction x and the vertical direction y. Here, i is a natural number from 1 to M, and j is a natural number from 1 to N.

  Next, the shelf area estimation unit 108 integrates the horizontal size of the image motion for each block of the screen (step S103). Specifically, the horizontal size of the image movement for each block obtained in step S102 is expressed by the following equation:

に示すようにブロック毎の画像を全ての時刻で足し合わせることにより、横方向の大きさの積算値Ａ（ｉ,ｊ）を求める。

As shown in FIG. 5, the horizontal value of the integrated value A (i, j) is obtained by adding the images for each block at all times.

  Next, the shelf area estimation unit 108 calculates the average difference between the integrated values of the areas when the screen is divided vertically between the vertical blocks (step S104). Specifically, the screen is vertically divided between the n-th block in the vertical direction shown in FIG. 5 and the n + 1-th block (the upper area composed of the n-th block and the bottom composed of the n + 1-th block). Based on the integrated value of the horizontal magnitude of the motion obtained in step S103 of each region, the average difference of the integrated values is obtained by the equation (2).

そして、１からＮ−１までのそれぞれのｎの値について、上下それぞれの領域の積算値の平均の差を計算する。

Then, for each value of n from 1 to N−1, the average difference between the integrated values of the upper and lower regions is calculated.

  Next, the shelf area estimation unit 108 determines that the lower area of the division position where the average difference between the integrated values of the upper and lower areas obtained in step S104 is the maximum is the shelf area (step S105). That is, n is obtained in which the value of the expression (2) is maximized when the value of n is changed from 1 to N−1 in step S104, and the area below the n + 1th block in the vertical direction is determined as the shelf area. To do. Since there is no movement in the shelf area, the average of the integrated values in the shelf area is the smallest, and the integrated value of the horizontal magnitude of the movement between the nth block in the vertical direction and the n + 1th block is This is because the average difference is maximized. FIG. 5 shows an example of n = J.

  Next, the shelf area estimation unit 108 causes the shelf area information storage unit 107 to store shelf area information indicating the shelf area on the screen determined as the shelf area in Step S105 (Step S106).

  By the above shelf area estimation process, it is assumed that the passage area on the screen tends to increase the horizontal movement due to the passage of the customer, whereas the shelf area tends to decrease the horizontal movement, A shelf area can be determined by determining an area with small lateral movement.

Note that the method of dividing the screen based on the movement of each block, so-called optical flow, is not limited to the method of step S104 and step S105 described above. For example, an area in which the integrated value of the horizontal direction magnitude of the motion obtained in step S103 is smaller than the threshold may be determined as a shelf area based on a predetermined threshold.
≪Shelf area entry judgment process≫
Next, the shelf area entry determination process in the shelf area entry determination unit 106 of the shelf monitoring device 1 will be described with reference to FIG. FIG. 6 is a flowchart of the shelf area entry determination process in the shelf area entry determination unit 106 of the shelf monitoring device 1.

  The shelf area entry determination unit 106 starts the shelf area entry determination process when the person area information and the image capturing time are supplied from the person detection unit 105, and reads the shelf area information from the shelf area information storage unit 107 (step S110). S201).

  Next, the shelf area entry determination unit 106 calculates the size of the overlap between the person area supplied from the person detection unit 105 and the shelf area indicated by the shelf area information read in step S201 (step S202). Specifically, the number of pixels that overlap in the person area and the shelf area is counted, and the number of pixels is defined as the size.

  Next, the shelf area entry determination unit 106 determines whether or not the size of the overlap between the person area and the shelf area obtained in step S202 is equal to or greater than a predetermined threshold (step S203). If it is determined that the size of the overlap is greater than or equal to the threshold, the process proceeds to step S204. If it is determined that the size of the overlap is not greater than or equal to the threshold, that is, smaller than the threshold, the process proceeds to step S205. . The threshold used here is appropriately determined in consideration of the accuracy of the shelf area entry determination.

  When it is determined in step S203 that the size of the overlap is greater than or equal to the threshold value, the shelf area entry determination unit 106 uses the area where the person area and the shelf area on the screen overlap as entry area information, and from the person detection unit 105 The customer's interest determination unit 110 is supplied with a photographing time of the supplied image and a signal indicating that there is an approach together with the approach area information (step S204).

On the other hand, if it is determined in step S203 that the size of the overlap is smaller than the threshold value, the shelf area entry determination unit 106 obtains a shooting time of the image supplied from the person detection unit 105 and a signal indicating that there is no entry. It supplies to the customer interest determination part 110 (step S205).
≪Customer interest judgment process≫
Next, the customer interest determination process in the customer interest determination unit 110 of the shelf monitoring device 1 will be described with reference to FIG. FIG. 7 shows a flowchart of the customer interest determination process in the customer interest determination unit 110 of the shelf monitoring device 1.

  The customer interest determination unit 110 starts the customer interest determination process when information indicating that there is an entry into the shelf area or no entry and the image capturing time are supplied from the shelf area entry determination unit 106. It is determined whether there is an entry in the area, that is, whether the supplied signal is a signal indicating that there is an entry or a signal indicating that there is no entry (step S301). If it is determined that there is an entry in the shelf area, the process proceeds to step S302. If it is determined that there is no entry in the shelf area, the process proceeds to step S304.

  When it is determined in step S301 that there is an entry in the shelf area, the customer interest determination unit 110 determines whether or not it is determined in step S301 in the previous customer interest determination process that there is an entry in the shelf area (step S302). ). If it is determined that there is an entry in the shelf area in the previous customer interest determination process, the customer interest determination process is terminated, and if it is determined in the previous customer interest determination process that there is no entry in the shelf area, the process proceeds to step S303. To migrate.

  If it is determined in step S302 that there is no entry into the shelf area in the previous customer interest determination process, the customer interest determination unit 110 determines that there is an entry in the shelf area this time, and the shelf area in the previous customer interest determination process. The photographing time of the image supplied from the entry determination unit 106 is stored in the pre-entry time storage unit 112 as the pre-entry time (step S303).

  On the other hand, when it is determined in step S301 that there is no entry into the shelf area, the customer interest determination unit 110 determines whether or not it is determined that there is an entry in the shelf area in step S301 in the previous customer interest determination process ( Step S304). If it is determined that there is an entry in the shelf area in the previous customer interest determination process, the process proceeds to step S305, and if it is determined in the previous customer interest determination process that there is no entry in the shelf area, the customer interest determination process Exit.

  If it is determined in step S304 that the previous customer interest determination process has entered the shelf area, the customer interest determination unit 110 reads the pre-entry time from the pre-entry time storage unit 112 (step S305). Subsequently, the customer interest determination unit 110 executes a shelf area difference calculation process (step S306). Specifically, the entry area information supplied from the shelf area entry determination unit 106 is used as information on an area in which a shelf area difference is to be calculated, and the time before entry read out in step S305 is used as the imaging time of the first image. The shooting time of the image supplied from the shelf area entry determination unit 106 is set as the shooting time of the second image, the information on the area where the shelf area difference should be calculated, the shooting time of the first image, and the second image The shooting time is supplied to the area difference calculation unit 109 to instruct the calculation of the area difference. And the difference of the approach area supplied from the area | region difference calculation part 109 is received.

  Next, the customer interest determination unit 110 determines whether or not the difference in the entry area supplied from the area difference calculation unit 109 in step S306 is greater than or equal to a predetermined threshold (step S307). If it is determined that the difference in the entry area is equal to or greater than the threshold value, the process proceeds to step S308. If the difference in the entry area is not equal to or greater than the threshold value, that is, it is determined that the difference is smaller than the threshold value, the customer interest determination process is terminated. . Note that the threshold used here is appropriately determined in consideration of the difference between the product shelf image before and after the customer picks up the product from the product shelf.

  If it is determined in step S307 that the difference between the entry areas is equal to or greater than the threshold, the customer interest determination unit 110 determines that the customer has picked up the product, and provides customer interest information indicating that the customer is interested in the product. Store in the customer interest information storage unit 111 (step S308).

  Since the customer interest information indicating that the customer is interested in the product is accumulated and stored in the customer interest information storage unit 111 by the above-described customer interest determination process, the customer interest information reading unit (not shown in FIG. 1) is used to store the customer interest information. Customer interest information can be read from the interest information storage unit 111 and used for marketing or the like.

  As described above, according to the shelf monitoring apparatus 1 of the present embodiment, as shown in FIG. 2, the shelf peripheral camera 101 is simply installed above the product shelf 203 that can photograph the customer 202 and the product shelf 203 at the same time. Therefore, when the customer motion analysis device described in Patent Document 1 is installed in a retail store or the like, a product shelf area in a video is set, or a product display state monitoring system described in Patent Document 2 is used. Compared with the case where a displacement detection sensor is attached to each of these, the labor and cost for installing the device can be reduced.

  Note that the customer interest determination process in the customer interest determination unit 110 of the shelf monitoring device 1 is based on the size of the image difference in the entry area before and after a part of the person enters the shelf area. However, it may be configured to determine that the product is picked up without using the image difference. In that case, in the configuration of the shelf monitoring apparatus 1 in FIG. 1, the area difference calculation unit 109 and the pre-entry time storage unit 112 are unnecessary. Further, in the customer interest determination process of FIG. 7, step S302, step S303, step S305, step S306, and step S307 are not required.

  The shelf monitoring device 1 according to the present embodiment is configured to determine customer interest in the product through customer interest determination processing and store customer interest information, but also includes other information used for marketing. You may make it memorize | store. For example, by analyzing the image taken by the shelf peripheral camera 101, the residence time around the customer's shelf may be measured, or the attributes such as the customer's age and sex may be estimated and stored together with the customer interest information. .

  Note that the above-described shelf monitoring apparatus 1 of the present embodiment can also be realized by executing a shelf monitoring program installed in a computer. For example, the shelf monitoring program may be read from a recording medium in which the shelf monitoring program is stored and executed by a central processing unit (CPU) to configure the shelf monitoring device 1 or a communication network. The shelf monitoring apparatus 1 may be configured by being transmitted and installed via the CPU and executed by the CPU.

(Second Embodiment)
Next, a second embodiment of the shelf monitoring apparatus according to the present invention will be described.

The shelf monitoring apparatus according to the present embodiment relates to a shelf monitoring apparatus that detects customer interest in products at retail stores or the like, and is an example in which the installation environment of the shelf monitoring apparatus at retail stores or the like is different from that of the first embodiment. .
<Configuration of shelf monitoring device>
FIG. 8 shows a block diagram of a second embodiment of the shelf monitoring apparatus according to the present invention. As shown in the figure, the shelf monitoring apparatus 8 of the present embodiment includes a shelf peripheral camera 101, an image storage unit 102, a motion calculation unit 103, a motion information storage unit 104, a person detection unit 105, a shelf The area entry determination unit 106, the shelf area information storage unit 107, the shelf area estimation unit 108, the face camera 801, the customer interest determination unit 803, the customer interest information storage unit 111, and the line-of-sight detection unit 802 are configured. The In the figure, the same components as those in FIG.

  In FIG. 8, the face camera 801 images the periphery of the customer's face and supplies the captured image to the line-of-sight detection unit 802. When instructed by the customer interest determination unit 803 to detect the customer's gaze, the gaze detection unit 802 analyzes the image supplied from the face camera 801, detects the customer's gaze, and detects the detected gaze angle, that is, the face camera. The angle toward the front of 801 is set to 0 degree, and the vertical and horizontal gaze misalignment angles from the front are supplied to the customer interest determination unit 803. In addition, for the customer's line-of-sight detection, for example, the line-of-sight direction measurement processing method in the line-of-sight direction measurement unit described in JP 2007-28695 A can be used.

  The customer interest determination unit 803 performs a customer interest determination process when the shelf area entry determination unit 106 is supplied with a signal indicating that there is an entry into the shelf area or when there is no entry and an image capturing time. Specifically, the customer interest determination unit 803 instructs the line-of-sight detection unit 802 to detect the line of sight of the customer when a signal indicating that there is an entry into the shelf area is supplied from the shelf area entry determination unit 106. Based on the customer's line-of-sight angle supplied from the line-of-sight detection unit 802, the customer's interest in the product is determined. Then, the customer interest information storage unit 111 stores customer interest information related to the determined customer interest.

The shelf monitoring device 8 of the present embodiment is configured by configuring the image storage unit 102, the motion information storage unit 104, the shelf region information storage unit 107, and the customer interest information storage unit 111 as separate storage devices. However, each storage unit may be partially shared, or all storage units may be shared by one storage device.
<Installation example of shelf monitoring device>
Next, how to install the shelf monitoring device 8 in a retail store or the like will be described with a specific example.

  FIG. 9 shows an installation example of the second embodiment of the shelf monitoring apparatus according to the present invention. In the figure, the installation example 9 is a part of a vertical product shelf 901 in which the products targeted for customer interest detection by the shelf monitoring device 8 of the present embodiment can display products on a plurality of levels. It is assumed that they are on display.

  In such an environment, the shelf peripheral camera 101 provided in the shelf monitoring device 8 is installed on the ceiling above the product shelf 901 where the customer 902 and the product shelf 901 can be photographed simultaneously. As a result, the customer 902, the passage through which the customer 902 passes, and the product shelf 901 are simultaneously photographed in the photographed image by the shelf peripheral camera 101 from above. The installation direction of the shelf peripheral camera 101 is set so that the upper part of the image is a passage area and the lower part is a shelf area. That is, in FIG. 9, it installs so that the upper part of the shelf periphery camera 101 may be located in the left side of FIG.

Further, the face camera 801 provided in the shelf monitoring device 8 is installed in the vicinity of the stage where the products for which the customer interest is detected in the product shelf 901 are displayed. As a result, the face of the customer 902 who is looking at the product that is the target of the customer interest detection is photographed almost from the front in the photographed image by the face camera 801.
<Operation of the shelf monitoring device>
The shelf monitoring device 8 of the present embodiment mainly performs a shelf area estimation process, a shelf area entry determination process, and a customer interest determination process. Among these, the shelf area estimation process and the shelf area entry determination process are the same as the shelf area estimation process and the shelf area entry determination process in the shelf monitoring apparatus 1 according to the first embodiment, and thus description thereof is omitted. . Therefore, the customer interest determination process unique to this embodiment will be described in detail below.
≪Customer interest judgment process≫
The customer interest determination process of the present embodiment is performed by the customer interest determination unit 803 of the shelf monitoring device 8. FIG. 10 shows a flowchart of the customer interest determination process in the customer interest determination unit 803 of the shelf monitoring device 8 of the present embodiment.

  The customer interest determination unit 803 determines the customer interest when the signal supplied from the shelf area entry determination unit 106 is supplied with information indicating that there is an entry or no entry into the shelf area and the image capturing time. The process is started, and it is determined whether or not there is an entry in the shelf area, that is, whether the supplied signal is a signal indicating that there is an entry or a signal indicating that there is no entry (step S401). If it is determined that there is an entry in the shelf area, the process proceeds to step S402. If it is determined that there is no entry in the shelf area, the customer interest determination process is terminated.

  If it is determined in step S401 that there is an entry into the shelf area, the customer interest determination unit 803 performs a line-of-sight detection process (step S402). Specifically, the customer interest determination unit 803 instructs the line-of-sight detection unit 802 to detect the line of sight of the customer, and receives the customer's line-of-sight angle supplied from the line-of-sight detection unit 802.

  Next, the customer interest determination unit 803 determines whether or not the customer's line-of-sight angle supplied from the line-of-sight detection unit 802 in step S402 is the front (step S403). Specifically, the customer interest determination unit 803 determines whether or not the absolute value of the viewing angle in the vertical direction and the horizontal direction is equal to or less than a predetermined threshold for each of the vertical direction and the horizontal direction. If the customer interest determination unit 803 determines that the line-of-sight angle is the front, that is, the absolute values of the vertical and horizontal line-of-sight angles are equal to or less than the respective threshold values, the process proceeds to step S404, and the line-of-sight angle is the front In other words, when it is determined that the absolute values of the viewing angles in the vertical direction and the horizontal direction are not less than the respective threshold values, that is, larger than the threshold values, the customer interest determination process is terminated.

  If it is determined in step S403 that the line-of-sight angle is the front, the customer interest determination unit 803 determines that the customer has picked up the product, and sets the customer interest information indicating that the customer is interested in the product as customer interest information. It memorize | stores in the memory | storage part 111 (step S404).

  Since the customer interest information indicating that the customer is interested in the product is accumulated and stored in the customer interest information storage unit 111 by the above-described customer interest determination process, the customer interest information reading unit (not shown in FIG. 8) Customer interest information can be read from the interest information storage unit 111 and used for marketing or the like.

  Thus, according to the shelf monitoring device 8 of the present embodiment, as shown in FIG. 9, the shelf peripheral camera 101 is installed above the product shelf 901 capable of photographing the customer 902 and the product shelf 901 at the same time, and Since it is only necessary to install the face camera 801 on the product shelf 801, when installing the customer motion analysis apparatus described in Patent Document 1 in a retail store or the like, Compared to the case where a displacement detection sensor is attached to each product as in the product display state monitoring system described, the labor and cost for installing the device can be reduced.

  The shelf monitoring device 8 of the present embodiment is configured to determine the customer's interest in the product by the customer interest determination process and store the customer interest information, but also includes other information used for marketing. You may make it memorize | store. For example, by analyzing the captured image of the shelf peripheral camera 101, the residence time around the shelf of the customer is measured, or by analyzing the captured image of the face camera 801, attributes such as the customer's age and sex are estimated. It may be stored together with customer interest information.

  Note that the above-described shelf monitoring apparatus 8 of the present embodiment can also be realized by executing a shelf monitoring program installed in a computer. For example, the shelf monitoring program may be read from a recording medium in which the shelf monitoring program is stored and executed by the CPU to configure the shelf monitoring device 8, or may be transmitted via a communication network. The shelf monitoring device 8 may be configured by being installed and executed by the CPU.

DESCRIPTION OF SYMBOLS 1, 8 Shelf monitoring apparatus 101 Shelf peripheral camera 102 Image storage part 103 Motion calculation part 104 Motion information storage part 105 Person detection part 106 Shelf area approach determination part 107 Shelf area information storage part 108 Shelf area estimation part 109 Area difference calculation part 110 Customer interest determination unit 111 Customer interest information storage unit 112 Pre-entry time storage unit 801 Face camera 802 Gaze detection unit 803 Customer interest determination unit

Claims (2)

A motion calculation unit that divides a region of a captured image in a predetermined range including a shelf to be monitored into a plurality of blocks, and calculates a motion of the captured image for each block;
The integrated value is calculated by integrating the magnitude of the movement of the captured image for each block, and one of the two blocks adjacent in the direction orthogonal to the predetermined direction is selected from the plurality of blocks. An area consisting of a block having the largest difference between the average of the integrated values of the first area and the average of the integrated values of the second area consisting of the other block, and having the smaller average of the integrated values. A shelf area estimation means for estimating a shelf area;
Person detecting means for detecting a person from the captured image;
Shelf area entry determining means for determining that a part of the person has entered the shelf area;
An area difference calculating means for calculating a difference between the captured images of the shelf area at a time before and after a time at which a part of the person is determined to have entered the shelf area by the shelf area entry determining means;
Customer interest determination means for determining that the person is interested in the product on the shelf when the difference between the captured images of the shelf area calculated by the area difference calculation means is greater than a preset threshold value. Shelf monitoring device featuring. A motion calculation unit that divides a region of a captured image in a predetermined range including a shelf to be monitored into a plurality of blocks, and calculates a motion of the captured image for each block;
The integrated value is calculated by integrating the magnitude of the movement of the captured image for each block, and one of the two blocks adjacent in the direction orthogonal to the predetermined direction is selected from the plurality of blocks. An area consisting of a block having the largest difference between the average of the integrated values of the first area and the average of the integrated values of the second area consisting of the other block, and having the smaller average of the integrated values. A shelf area estimation means for estimating a shelf area;
Person detecting means for detecting a person from the captured image;
Shelf area entry determining means for determining that a part of the person has entered the shelf area;
Line-of-sight detection means for detecting the line of sight of the person at the peripheral position of the shelf;
When it is determined by the shelf area entry determination means that a part of the person has entered the shelf area, and the line-of-sight detection means detects that the person's line of sight is facing the shelf direction, A shelf monitoring device comprising: customer interest determination means for determining that a person is interested in the product on the shelf.

Images