JP2018097412A - Information processing device, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To correlate behavior of a person to be imaged to the characteristic appropriately, without complicating device configuration.SOLUTION: An information processing device 30 comprises: identifying means for identifying behavior of a person to be imaged from a first image obtained by imaging a monitoring area; identification means for identifying the person whose behavior is identified by the identifying means, from a second image obtained by imaging a specific area which is a part within the monitoring area; extraction means for extracting characteristic of the person identified by the identification means from the second image; and correlation means for correlating the behavior of the person identified by the identifying means to the characteristic of the person extracted by the extraction means.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program.

 In recent years, a technique using analysis of an image obtained from a network camera (hereinafter referred to as “NW camera”) has been used other than in-store monitoring. For example, an NW camera is installed in front of a cash register at a retail store, and customer characteristics (face, age, gender, etc.) are extracted from the image obtained from the NW camera, and the customer characteristics and purchase information at the cash register There is a technology that associates and uses for marketing analysis.

  Patent Document 1 discloses a customer behavior recording device that analyzes customer behavior in a store and collects data useful for marketing. This technology includes an entrance camera that images a person entering the facility, an in-facility camera that images a person moving through the facility, and an accounting area camera that images a person who pays for the purchase of goods. .

JP 2010-113692 A

In the technique disclosed in Patent Document 1, the person imaged in each of the entrance camera, the in-facility camera, and the accounting area camera is associated by collating the image of the captured human face portion. Therefore, each camera needs to be installed at a position where a person's face can be captured from the front.
If a security camera is installed near the ceiling of the cash register space or diagonally behind the store clerk standing at the cash register, the security camera may be used for marketing analysis purposes. However, with security cameras installed in such places, the customer's face may not be captured from the front, or the customer's face may not be reflected by the store clerk. The accuracy at the time of extracting a feature will fall.

Therefore, in order to capture an image of a customer standing in front of a cash register from the front, a marketing camera must be installed separately from the security camera. And the camera for marketing must be installed for each cash register. The reason is that, when there are a plurality of cash registers, simply installing a camera on one cash register body cannot capture images of customers existing before all cash registers due to insufficient angle of view. That is, in the technique described in Patent Document 1, marketing cameras corresponding to the number of cash registers are required as accounting area cameras, and the cost increases when the number of cash registers is large.
Therefore, an object of the present invention is to appropriately associate the behavior and characteristics of a person to be imaged without complicating the apparatus configuration.

  In order to solve the above problem, an aspect of the information processing apparatus according to the present invention includes a specifying unit that specifies a behavior of a person to be imaged from a first image obtained by imaging a monitoring area, and a part of the monitoring area. The identification means for identifying the person whose action is specified by the specifying means is extracted from the second image obtained by imaging the specific area, and the characteristics of the person identified by the identification means are extracted from the second image An extraction unit; and an association unit that associates the behavior of the person specified by the specifying unit with the characteristics of the person extracted by the extraction unit.

  According to the present invention, it is possible to appropriately associate the behavior and characteristics of the person to be imaged without complicating the apparatus configuration.

It is a network block diagram of the system in 1st embodiment. It is a figure which shows an example of the hardware constitutions of information processing apparatus. It is a flowchart which shows the flow of a marketing analysis process. It is a conceptual diagram which shows the flow of a preset. It is a conceptual diagram of a video analysis process. It is a flowchart which shows the flow of a video analysis process. It is an example of the data structure used by video analysis processing. It is a flowchart which shows the flow of a marketing report creation process. It is an example of the data structure used by marketing report creation processing. It is an example of a marketing report. It is a network block diagram of the system in 2nd embodiment. It is a conceptual diagram of the video analysis process of 2nd embodiment.

DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings.
The embodiment described below is an example as means for realizing the present invention, and should be appropriately modified or changed depending on the configuration and various conditions of the apparatus to which the present invention is applied. It is not limited to the embodiment.

(First embodiment)
FIG. 1 is a network configuration diagram showing an example of the operating environment of the system in the present embodiment. In this embodiment, a marketing analysis system that performs marketing analysis using a network camera installed in a facility (store) and provides a marketing report service will be described.
The marketing analysis system 10 includes a register camera 20A, an exit camera 20B, an information processing device 30, a POS (Point Of Sales) register 40, and client devices 50A and 50B. The cash register camera 20A, the exit camera 20B, the information processing device 30, the POS cash register 40, and the client devices 50A and 50B are connected by a network 60. The network 60 may be a LAN (Local Area Network), the Internet, a WAN (Wide Area Network), or the like. The physical connection form to the network 60 may be wired or wireless.

In the present embodiment, the information processing apparatus 30 is described as being built on a cloud computing platform. However, the present invention is not limited to the above, and the information processing apparatus 30 may be constructed on an on-premises server.
The register camera 20A is installed in the vicinity of the ceiling of the register space where a plurality of POS registers 40 are installed in the store, and includes a pre-register region of each POS register 40 and an exit region set in the vicinity of the store entrance. Image. As the registration camera 20A, a wide-angle camera or an omnidirectional camera can be used. The image picked up by the register camera 20A is used for the purpose of specifying the action of the person (customer) to be picked up moving within the monitoring area. In addition, the image imaged by the register camera 20A can be used for a purpose for crime prevention.

The exit camera 20B is a camera that captures an image of a specific area in the monitoring area. Here, a part of the specific area in the monitoring area is an exit area of the store. The exit camera 20B is a camera installed from the store entrance to the store, and is installed at a position where the face of the customer who goes out of the store can be imaged from the front in the store exit area. The image picked up by the exit camera 20B is used for the purpose of extracting customer personal features.
The information processing apparatus 30 acquires an image (video) captured by the cashier camera 20A and the exit camera 20B and performs a marketing analysis process. Specifically, the information processing apparatus 30 identifies the action of the person to be imaged from the image of the registration camera 20A, identifies the person who has identified the action from the image of the exit camera 20B, and determines the characteristics of the person. Extract. Then, the information processing apparatus 30 associates the person's action with the feature and outputs the marketing data as marketing data. Furthermore, the information processing apparatus 30 creates and outputs a marketing report that analyzes the behavior of the person based on the marketing data.

  In the present embodiment, the information processing apparatus 30 specifies the purchase behavior of the purchaser and outputs marketing data in which the purchase behavior of the purchaser is associated with the personal characteristics. Here, the purchase behavior is behavior in which the purchaser purchases the product, and can be specified by purchase information transmitted by the POS register 40 that the purchaser has purchased. That is, it can be said that the marketing data in the present embodiment is data for analyzing the purchasing tendency of the purchaser having a specific person characteristic. The information processing apparatus 30 performs video analysis processing on the image of the registration camera 20A, and specifies purchase behavior of the purchaser using the result of the video analysis processing and purchase information transmitted by the POS register 40.

Further, the information processing apparatus 30 performs video analysis processing on the image of the register camera 20A, tracks the purchaser who has made the purchase from the pre-register area to the exit area, and based on the tracking result, the exit camera 20B Identify the tracked buyer from the image. Then, the information processing apparatus 30 performs video analysis processing on the image of the exit camera 20B, detects the tracked purchaser's face, and extracts a person feature based on the detected face image.
One or more POS cash registers 40 are installed in the store and transmit purchase information to the information processing apparatus 30. In the present embodiment, a case where three POS cash registers 40a to 40c (FIG. 4) are installed in the store will be described. The purchase information transmitted by the POS register 40 includes a register ID for identifying the register, a purchase time, a product ID for identifying the purchased product, and the like.

  The client device 50A is a client PC for video analysis setting. The client device 50 </ b> A receives an operation from the operator, and transmits a video analysis processing preset to the information processing device 30. The client device 50A sets an area corresponding to the pre-registration area and the exit area in the image captured by the registration camera 20A from the operator and an area corresponding to the exit area in the image captured by the exit camera 20B. Accept information about.

The client device 50B is a service use client PC. The client device 50B receives an instruction from the operator, and transmits a marketing analysis instruction to the information processing device 30. For example, the client device 50B receives information related to selection of an analysis target store and an analysis target cash register and setting of an analysis target period from the operator. Further, the client device 50B has a function of receiving a marketing report generated by the information processing device 30 and presenting it to the operator.
Note that the client devices 50A and 50B are not limited to PCs, and may be replaced by tablet terminals or the like. Further, the client devices 50A and 50B are not limited to separate terminals, and may be a single terminal.

The information processing apparatus 30 includes a camera control unit 301, a video analysis unit 302, a POS data processing unit 303, and an analysis processing unit 304.
The camera control unit 301 controls the operations of the registration camera 20A and the exit camera 20B. For example, the camera control unit 301 transmits an image transmission instruction to the cashier camera 20A and the exit camera 20B, and receives an image transmitted from each of the cameras 20A and 20B in response to the image transmission instruction. The camera control unit 301 stores the images received from the cameras 20A and 20B as the recorded video 71. Further, the camera control unit 301 transmits the recorded video 71 to the video analysis unit 302 in accordance with an instruction from the video analysis unit 302 described later.

The video analysis unit 302 analyzes images captured by the registration camera 20A and the exit camera 20B. Specifically, when receiving a video analysis instruction from an analysis processing unit 306 described later, the video analysis unit 302 receives an analysis target image among the images stored as the recorded video 71 via the camera control unit 301. Perform video analysis processing. At this time, the video analysis unit 302 performs video analysis processing according to the preset setting received from the client device 50A. Details of the video analysis processing will be described later.
The POS data processing unit 303 has a function as a POS server. The POS data processing unit 303 receives purchase information from the POS register 40 and stores the received purchase information as POS data 72.

The analysis processing unit 304 has a function as a marketing server. Upon receiving the marketing analysis instruction from the client device 50B, the analysis processing unit 304 outputs a video analysis instruction to the video analysis unit 302. Then, the analysis processing unit 304 combines the result of the video analysis processing received from the video analysis unit 302 and the POS data 72 received via the POS data processing unit 303 to create and store the marketing data 73. The analysis processing unit 304 performs a marketing report creation process for creating a marketing report based on the marketing data 73, and outputs the created marketing report to the client device 50B.
Note that the recorded video 71, the POS data 72, and the marketing data 73 may be stored in a storage area in the information processing apparatus 30, or may be stored in a storage device (not shown) connected to the network 60.

FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus 30.
The information processing device 30 includes a CPU 31, a RAM 32, a ROM 33, an HDD 34, an output device 35, an input device 36, a communication device 37, and a system bus 38.
The CPU 31 is a processor that comprehensively controls the operation of the information processing apparatus 30, and controls each component (32 to 37) via the system bus 38. The RAM 32 has an area for temporarily storing programs and data loaded from the ROM 33 and the HDD 34. Further, the RAM 32 has an area for temporarily storing data received from an external device via the communication device 37. The RAM 32 also has a work area used when the CPU 31 executes each process.

  The ROM 33 is a non-volatile memory that stores a program necessary for the CPU 31 to realize the function of the marketing report service. The program may be an application program that runs on an OS (Operating System). The program may be recorded on the HDD 34 or a recording medium (not shown). The CPU 31 can implement various functional operations by reading the OS and application programs from the ROM 33 or the HDD 34, loading them into the RAM 32, and executing them.

  The output device 35 includes a monitor that displays various processing results and a user interface screen (UI screen). The input device 36 can be configured by a pointing device such as a keyboard and a mouse, a touch panel, and the like. A user of the information processing apparatus 30 (for example, a system setting person) can input an instruction to the information processing apparatus 30 by operating the input device 36. The communication device 37 is an interface for communicating with an external device connected to the network 60. The system bus 38 connects the CPU 31, RAM 32, ROM 33, HDD 34, output device 35, input device 36, and communication device 37 so that they can communicate with each other.

  Functions of each unit of the information processing apparatus 30 illustrated in FIG. 1 can be realized by the CPU 31 executing a program. However, at least some of the elements shown in FIG. 1 may operate as dedicated hardware. In this case, the dedicated hardware operates based on the control of the CPU 31. Note that the client devices 50A and 50B can also have the same hardware configuration as in FIG.

  FIG. 3 is a flowchart showing a flow of marketing analysis processing executed by the information processing apparatus 30. The information processing apparatus 30 can implement each process illustrated in FIG. 3 by the CPU 31 reading and executing a necessary program. However, as described above, at least a part of each element shown in FIG. 1 may operate as dedicated hardware so that the processing in FIG. 3 may be realized. In this case, the dedicated hardware operates based on the control of the CPU 31. Hereinafter, the alphabet S means a step in the flowchart.

First, in S1, the video analysis unit 302 receives a preset from the client device 50A. Here, as described above, the pre-setting corresponds to the setting of the area corresponding to the pre-registration area and the exit area in the image captured by the registration camera 20A, and the exit area in the image captured by the exit camera 20B. At least the setting of the area is included. Details of the presetting will be described later with reference to FIG.
In S2, the analysis processing unit 304 receives a marketing analysis instruction from the client device 50B. Here, as described above, the marketing analysis instruction includes the identification number of the analysis target store, the cash register number of the analysis target cash register, and the analysis target period. Upon receiving the marketing analysis instruction, the analysis processing unit 304 outputs the video analysis instruction to the video analysis unit 302.

In S <b> 3, the video analysis unit 302 receives a video analysis instruction from the analysis processing unit 304 and executes video analysis processing. The video analysis unit 302 acquires images to be analyzed from the registration camera 20A and the exit camera 20B via the camera control unit 301 during the video analysis process. Here, the analysis target image is an image captured at a time corresponding to the analysis target period. Then, the video analysis unit 302 performs video analysis processing on the acquired image. Details of the video analysis processing will be described later with reference to FIGS.
In S4, the analysis processing unit 304 creates marketing data based on the analysis result of the video analysis processing in S3 and the POS data 72. The analysis processing unit 304 creates a marketing report based on the marketing data. Details of this marketing report creation processing will be described later with reference to FIGS.

(Pre-setting)
FIG. 4 is a diagram for explaining a flow of processing in which the video analysis unit 302 receives a preset from the client device 50A. FIG. 4A is a diagram showing a store to be marketed and a camera installation position. FIGS. 4B and 4C are examples of a preset screen displayed on the client device 50A. It is.
In the present embodiment, it is necessary to detect a person who has entered the pre-register area and a person who has entered the exit area in the video analysis process. Therefore, in the pre-setting, it is set which area in the image captured by each camera 20A, 20B corresponds to the pre-registration area or the exit area.

  As shown in FIG. 4A, a cash register space 81 and a passage space 82 are provided in the store 80. The registration camera 20A is installed near the ceiling of the registration space 81, and the exit camera 20B is installed near the entrance / exit. Purchasers P1 and P2 enter the store 80 from the entrance, move through the aisle space 82 to take out the product from the product shelf, move to the pre-register area and pay the purchase price, and then move to the exit area and enter the entrance To go outside the store 80. In the present embodiment, the operator of the client device 50A operates the area setting screen and designates areas corresponding to the pre-registration area and the exit area, respectively.

FIG. 4B is an example of an area setting screen 51 for the registration camera 20A displayed on the client device 50A. The operator of the client device 50A sets areas corresponding to the pre-registration area and the exit area in the image captured by the registration camera 20A while looking at the area setting screen 51 shown in FIG. 4B.
The area setting screen 51 includes an image display unit 52 and a designation button 53. The image display unit 52 is an area for displaying an image captured by the registration camera 20A. The designation button 53 is a button for designating a register to be set in the pre-register area. The designation button 53 displays information for identifying each cash register, such as a cash register name acquired from the information processing apparatus 30 in advance.

The operator of the client device 50A designates a pre-registration region in the image displayed on the image display unit 52 using a mouse or the like. The operator designates an area where it can be determined that the purchaser has entered before the checkout to pay the purchase price as the pre-checkout area. Subsequently, the operator of the client device 50 </ b> A uses the designation button 53 to instruct to which area the designated pre-registration area corresponds.
For example, in FIG. 4B, if the operator designates the pre-registration area 83a and then designates the designation button 53 of “Register 1”, the pre-registration area 83a is set as the area in front of the POS register 40a. . Similarly, the pre-registration area 83b can be set as an area before the POS register 40b, and the pre-registration area 83c can be set as an area before the POS register 40c. In this manner, the POS registers 40a to 40c and the pre-register areas 83a to 83c can be associated with each other.

Further, the operator of the client device 50A designates an area corresponding to the exit area in the image displayed on the image display unit 52 using a mouse or the like. The operator designates, as the exit area 84, an area in which it can be determined that the customer (including the customer) is going out of the store.
When designating the pre-registration area, the operator of the client device 50A may first designate the registration name using the designation button 53 and then designate the corresponding pre-registration area. Further, the image display unit 52 may display a still image captured by the registration camera 20A at a specific time or a moving image. When a moving image is displayed, it is possible to designate a pre-registration area while looking at where the purchaser who is paying stands before the checkout.

FIG. 4C is an example of the area setting screen 54 of the exit camera 20B displayed on the client device 50A. The operator of the client device 50A sets an area corresponding to the exit area in the image captured by the exit camera 20B while viewing the area setting screen 54 shown in FIG.
The area setting screen 54 includes an image display unit 55. The image display unit 55 is an area for displaying an image captured by the exit camera 20B. Similar to the image display unit 52 of FIG. 4B, the image display unit 55 may display a still image captured by the exit camera 20B at a specific time or a moving image. May be. The operator of the client device 50A designates the exit area 84 in the image displayed on the image display unit 55 using a mouse or the like. At this time, the client device 50A associates the exit area 84 set for the image of the registration camera 20A with the exit area 84 set for the image of the exit camera 20B.
When the presetting is completed, the client device 50A transmits information related to the presetting to the video analysis unit 302.

(Video analysis processing)
Hereinafter, the video analysis processing will be specifically described with reference to FIGS.
FIG. 5 is a conceptual diagram of video analysis processing executed by the video analysis unit 302. 5A is an image 21A captured by the registration camera 20A, and FIG. 5B is an image 21B captured by the exit camera 20B. FIG. 5A shows a state where the purchaser P3 who has entered the pre-register area 83b has moved from the pre-register area 83b to the exit area 84. Reference sign P3 ′ indicates the purchaser P3 in the exit area 84.

  The video analysis unit 302 first detects a purchaser who has entered one of the pre-registration areas 83a to 83c in order to specify the behavior of the person to be imaged (the purchase behavior of the purchaser) from the image of the registration camera 20A. . Then, the video analysis unit 302 determines the time when the purchaser who has entered the pre-register area enters the pre-register area (the register intrusion time range) and which POS register 40 corresponds to the pre-register area. To detect. Further, the video analysis unit 302 tracks the purchaser who has entered the pre-registration area based on the image of the registration camera 20A, and based on the tracking result, the time when the purchaser has entered the exit area 84. (Exit entry time range) is detected.

Next, the video analysis unit 302 identifies the purchase customer tracked based on the image of the cashier camera 20A from the image of the exit camera 20B. In the present embodiment, the video analysis unit 302 tracks a person existing in the area corresponding to the exit area in the image captured by the exit camera 20 in the exit entry time range described above based on the image of the registration camera 20A. Identify as a buyer. Then, the video analysis unit 302 detects the identified purchaser's face from the image of the exit camera 20B, and extracts the personal features of the purchaser based on the face image.
Through the above video analysis processing, it is possible to analyze when and at what cash register a purchaser who has a person characteristic.

FIG. 6 is a flowchart showing the flow of the video analysis process. The process of FIG. 6 is executed in S3 of FIG. 6A is a video analysis process for which the image of the registration camera 20A is an analysis target, and FIG. 6B is a video analysis process for which the image of the exit camera 20B is an analysis target.
First, in S31 of FIG. 6A, the video analysis unit 302 enters a person who has entered one of the pre-registration areas 83a to 83c shown in FIG. 4B from the analysis target image captured by the registration camera 20A. (Purchaser) is detected. At this time, the video analysis unit 302 records the purchase customer ID 91a, the cash register ID 91b, and the cash register entry time range 91c as the customer information 91 as shown in FIG. Here, the purchase customer ID 91a is an ID for uniquely identifying the purchase customer, and the video analysis unit 302 issues an ID every time a purchase customer is detected in S31. The cash register ID 91b is an ID for specifying a cash register corresponding to the pre-register area where the purchaser has invaded. The cash register intrusion time range 91c indicates from the time when the purchaser enters the area before the cash register to the time when the purchaser leaves the area before the cash register.

  In S32, the video analysis unit 302 tracks the purchaser detected in S31 based on the analysis target image captured by the cashier camera 20A. As a tracking method, for example, the face area (or head area) of a person is detected from the analysis target image by comparing the analysis target image with a template stored in advance, and the person is tracked. The method can be used. Note that the method of realizing the person tracking process is not limited to the above, and any method can be applied.

  In S <b> 33, the video analysis unit 302 detects that the purchase customer detected in S <b> 31 has entered the exit area 84 based on the analysis target image captured by the registration camera 20 </ b> A. At this time, the video analysis unit 302 records a corresponding purchase customer exit intrusion time range 91d as the purchase customer information 91 shown in FIG. The exit intrusion time range 91d indicates from the time when the purchaser enters the exit area 84 to the time when the customer leaves the exit area 84.

  Next, in S34 of FIG. 6B, the video analysis unit 302 acquires, as the analysis target image, the image of the exit camera 20B captured in the vicinity of the exit entry time range 91d for each purchaser ID 91a. In S35, the video analysis unit 302 performs face detection for each purchaser ID 91a. Specifically, in S35, the video analysis unit 302 identifies a person existing in the exit area 84 in the analysis target image acquired in S34 as a purchase customer who is a target of face detection, and the face Perform detection. By performing face detection for a person existing in the exit area 84, it is possible to avoid accidentally performing face detection for another purchaser who appears on the exit camera 20B at that time. Can do.

  In S36, the video analysis unit 302 extracts a person feature for each purchaser ID 91a. Specifically, in S36, the video analysis unit 302 extracts person features such as the purchaser's age and sex based on the purchaser's face image detected in S35. As a method for extracting a person feature, for example, a method of extracting a feature amount from an image to be analyzed and estimating age and sex using the extracted feature amount and pre-learned criterion data can be used. The method for realizing the human feature estimation process is not limited to the above, and any method can be applied.

The video analysis unit 302 records the person characteristic (gender, age) estimated in S36 as the corresponding customer characteristic 91e of the purchaser in the purchaser information 91 shown in FIG. In the present embodiment, it is assumed that the age of the purchaser is classified and recorded as “10's”, “20's”, “30's”.
Through the video analysis process described above, it is possible to obtain information about what personal characteristics 91e the purchaser who has made purchases at (in cash register ID 91b) and when (in cash register entry time range 91c) has. .

(Marketing report creation process)
Hereinafter, the marketing report creation process will be specifically described with reference to FIGS.
FIG. 8 is a flowchart showing a flow of marketing report creation processing. The process of FIG. 8 is executed in S4 of FIG. Based on the result of the video analysis processing performed by the video analysis unit 302 and the POS data 72, the analysis processing unit 304 creates marketing data 73 by associating the purchaser's personal characteristics with purchase information. Then, the analysis processing unit 304 creates a marketing report using the marketing data 73.

  First, in S41, the analysis processing unit 304 acquires the corresponding POS data 72 for each purchaser ID 91a in FIG. 7 via the POS data processing unit 303. FIG. 9A is a diagram showing a data structure of purchase information 92 constituting the POS data 72. The purchase information 92 includes a purchase ID 92a, a cash register ID 92b, a purchase time 92c, and a product ID 92d. The analysis processing unit 304 refers to the cash register ID 91b and the cash register entry time range 91c included in the customer information 91 for each customer ID 91a. Purchase information 92 included in the.

  In other words, the analysis processing unit 304 is transmitted from the POS register 40 where the purchaser has purchased based on the purchaser information including information on the POS register 40 where the purchaser purchased and the time when the purchase was made. In addition, purchase information including information related to the purchase product of the purchaser is acquired. For example, the purchaser with purchaser ID = “0013” in FIG. 7 was purchased at “Register 2” at “11: 31: 24-11: 31: 53 on June 10, 2015”. Is obtained. In the example shown in FIG. 9A, purchase ID = “00041” meets the above condition. Therefore, it can be determined that the purchase information of purchase ID = “00041” is the purchase information of the purchaser of purchaser ID = “0013”.

  In S <b> 42, the analysis processing unit 304 stores, as marketing data 73, marketing information that associates the purchaser's personal characteristics with purchase information. FIG. 9B is a diagram showing a data structure of marketing information 93 constituting the marketing data 73. The marketing information 93 includes a purchaser ID 93a, a person feature 93b, and a purchase ID 93c. The marketing information 93 is information indicating what purchase behavior (purchase ID 93c) the purchase customer having the person characteristic 93b has performed.

In S43, the analysis processing unit 304 creates a marketing report based on the marketing data created in S42. Then, the analysis processing unit 304 transmits the created marketing report to the client device 50B.
Examples of marketing reports are shown in FIGS. 10 (A) to 10 (C). FIG. 10A is a graph showing the purchase amount for each age group. FIG. 10B is a graph showing sales by product category of women in their 20s. FIG. 10C is a graph of sales by product in the sandals category of women in their 20s. In addition, the marketing report shown in FIG. 10 is an example, and the report content and the display method are not limited to the above.

  As described above, the information processing apparatus 30 includes an image in which a monitoring area including a pre-register area and an exit area of a store is captured by a registration camera 20A such as a wide angle camera, and a part of the monitoring area in the monitoring area by an exit camera 20B. An image obtained by capturing an exit area as a specific area is acquired. Then, the information processing apparatus 30 specifies the behavior of the person to be imaged (the purchase behavior of the purchaser) from the image of the register camera 20A. In addition, the information processing apparatus 30 identifies the person whose purchase behavior is specified from the image captured by the exit camera 20B, extracts the personal feature of the person, and associates the purchase behavior with the personal feature.

  In this way, the information processing apparatus 30 identifies only the purchase behavior of the purchaser from the wide area image, and associates the purchase behavior with the personal feature of the purchaser extracted from the image captured in the specific area. Therefore, in order to link the purchase behavior and the person feature, it is not necessary to image the purchase customer from the front and extract the person feature in the pre-registration region where the purchase customer performs the purchase behavior. That is, it is not necessary to install a camera for extracting person features in each of the plurality of cash register bodies. This eliminates the need for cameras corresponding to the number of cash registers, thereby reducing the cost.

  Further, when the purchase behavior of the purchaser is specified, the information processing apparatus 30 first purchases with the POS register purchased by the purchaser from the image of the register camera 20A that images the area before the cash register of the facility (store). Acquire buyer information including time and information. At this time, the information processing apparatus 30 detects a person who has entered the pre-registration area set for each POS register from the image of the registration camera 20A, and stores the POS register corresponding to the pre-registration area and the pre-registration area. Detect the time of intrusion. Next, the information processing apparatus 30 acquires purchase information including information related to the purchased product from the POS register corresponding to the pre-register area where the purchaser has invaded. Then, the information processing device 30 associates the acquired purchaser information with the purchase information.

Thereby, the information processing apparatus 30 can specify the purchase behavior indicating when and at which cash register the purchaser has made a purchase. Therefore, by associating the purchase behavior with the personal feature, it is possible to obtain marketing data such as when and at what cash register the person who made the purchase is a personal feature.
Furthermore, the information processing device 30 associates the purchase behavior specified from the image of the cashier camera 20A with the person feature extracted from the image of the exit camera 20B and the person reflected in the image of the cashier camera 20A and the exit camera 20B. Link with the person in the image. At this time, when the information processing apparatus 30 detects that the purchaser has entered the pre-register area based on the image of the register camera 20A, the information processing apparatus 30 tracks the purchase customer from the pre-register area to the exit area. Further, the information processing device 30 detects the time when the purchaser has entered the specific area (exit area) based on the tracking result.

  Then, the information processing apparatus 30 acquires the image of the exit camera 20B captured at the time when the purchase customer has entered the exit area, and identifies the person existing in the area corresponding to the exit area in the image as described above. Identify the customer who made the purchase. That is, the time when the purchaser has entered the area corresponding to the exit area in the image of the cashier camera 20A is equal to the time when the purchaser has entered the area corresponding to the exit area in the image of the exit camera 20B. Use. Thereby, it is possible to appropriately associate the person shown in the image of the registration camera 20A and the person shown in the image of the exit camera 20B.

Further, the information processing apparatus 30 detects the purchaser's face from the image of the exit camera 20B, and extracts at least one of gender and age as a person feature based on the detected face image. Therefore, based on the marketing data in which the purchase behavior and the personal features are associated, it is possible to analyze the purchase amount for each age group, sales for each product category of women in their 20s, or sales for each product.
Furthermore, the information processing device 30 creates a marketing report based on the marketing data and outputs it to the client device 50B. Here, the marketing report can be a graph showing the result of analyzing the behavior of the purchaser. As a result, the operator of the client device 50B can easily grasp the customer's age group, best-selling products, and the like based on the marketing report.

As described above, the information processing apparatus 30 identifies the purchaser and purchase information from the image of the cashier camera 20A, and extracts the personal features of the purchaser from the image of the exit camera 20B. Therefore, even when the number of cashiers is large, it is possible to appropriately associate the purchase behavior of the purchaser with the personal characteristics without complicating the device configuration. That is, it is possible to collect information useful for marketing analysis without increasing costs.
In the present embodiment, the case where the image of the exit camera 20B is used for extracting the human feature has been described. However, the image of the registration camera 20A may be used for detecting the face and extracting the human feature. In this case, as a result of performing face detection and person feature extraction using the image of the registration camera 20A, if the processing fails, the tracking process shown in this embodiment and the image of the exit camera 20B are used. Face detection and person feature estimation may be performed. By doing so, it is possible to reduce the calculation cost for the tracking process.

  In the present embodiment, a specific area in the monitoring area is used as an exit area of the facility (store), and the face of the purchaser is imaged from the front in the exit area, and the person feature is extracted. The specific area is not limited to the exit area. The specific area is an area in a monitoring area through which a purchaser who purchases at each POS register 40 always passes, and may be an area where the face of the purchaser can be imaged from the front.

(Second embodiment)
Next, a second embodiment of the present invention will be described.
In the first embodiment described above, the information processing apparatus 30 has been described with respect to the case where the purchase information and the person feature are associated with each other and stored as marketing data. In the second embodiment, a case will be described in which purchaser flow line data is stored as marketing data in association with purchase information and person characteristics.
FIG. 11 is a network configuration diagram showing an example of the operating environment of the marketing analysis system 10 in the present embodiment. In FIG. 11, parts having the same configurations as those of the first embodiment described above are denoted by the same reference numerals as those in FIG. 1, and hereinafter, the parts having different configurations will be mainly described.

  The marketing analysis system 10 further includes an in-store camera 20C. The in-store camera 20C is desirably an omnidirectional camera or a wide-angle camera so that the inside of the store can be covered widely. As shown in FIG. 12A, the in-store camera 20C is installed, for example, in the vicinity of the ceiling in the center of the store, and can capture an image of a purchaser buying around. A plurality of in-store cameras 20C can be installed in accordance with the size of the store. When a plurality of in-store cameras 20C are installed, it is assumed that the correspondence between the in-store cameras 20C is known.

The camera control unit 301 stores an image captured by the in-store camera 20C as the recorded video 71 in addition to the image captured by the cashier camera 20A and the exit camera 20B.
The video analysis unit 302 performs video analysis processing on the images of the in-store camera 20C in addition to video analysis processing on the images of the registration camera 20A and the exit camera 20B. Specifically, the video analysis unit 302 performs a process of extracting a flow line of a person (customer) moving in the store based on an image captured by the store camera 20C. Thereby, for example, the flow line F1 of the purchaser P1 as shown in FIG. 12A is extracted. When the flow line of the purchaser is extracted by the video analysis process, the video analysis unit 302 stores data indicating the extracted flow line as the flow line data 74.

  The analysis processing unit 304 combines the purchase customer information including the flow line data obtained as a result of the video analysis processing by the video analysis unit 302 and the POS data 72 obtained through the POS data processing unit 303 to obtain the marketing data 73. create. FIG. 12B is an example of the marketing information 94 constituting the marketing data 73 in the present embodiment. The marketing information 94 includes a purchaser ID 94a, a person feature 94b, a purchase ID 94c, and a flow line data ID 94d. The purchase customer ID 94a, the person feature 94b, and the purchase ID 94c are the same as the purchase customer ID 93a, the person feature 93b, and the purchase ID 93c in the marketing information 93 of FIG. 9B.

As described above, the information processing apparatus 30 according to the present embodiment identifies the flow line of the person as the action of the person to be imaged from the image captured by the in-store camera 20C, and associates the flow line data with the person feature. Therefore, the user who has received the marketing report can grasp the tendency that a person having a specific person characteristic visits a specific sales floor, and can reflect the tendency to a layout change of the sales floor.
In this embodiment, the case where the register camera 20A and the in-store camera 20C are separately installed has been described. However, when the in-store camera 20C can cover the imaging range of the register camera 20A, the register camera 20A is not necessarily installed. do not have to.

(Modification)
In each of the embodiments described above, the information processing apparatus 30 has been described with respect to a case where at least purchase information of a purchaser is associated with a person feature. However, only customer flow line data may be associated with a person feature. Also in this case, it is not necessary to install a camera for imaging the purchaser from the front in each product shelf, and information necessary for marketing analysis can be collected without increasing costs. Furthermore, the information associated with the person feature is not limited to purchase information or flow line data. The information associated with the person feature may be information indicating a person's action, and may be information such as what product is picked up.
Moreover, in each said embodiment, although the case where the monitoring area | region was made into a shop was demonstrated, it is not limited to a shop. For example, the present invention can be applied to facilities other than stores such as amusement facilities such as an amusement park and exhibition facilities such as a museum. In this case, it is possible to obtain marketing data that can analyze popular attractions and popular exhibition areas for each age group.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a recording medium, and one or more processors in the computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  DESCRIPTION OF SYMBOLS 10 ... Marketing analysis system, 20A ... Cash register camera, 20B ... Exit camera, 30 ... Information processing device, 40 ... POS cash register, 50A, 50B ... Client device, 301 ... Camera control part, 302 ... Video analysis part, 303 ... POS data Processing unit, 304 ... analysis processing unit

Claims (12)

A specifying means for specifying the action of the person to be imaged from the first image obtained by imaging the monitoring area;
Identification means for identifying the person whose action is specified by the specifying means, from a second image obtained by imaging a part of the specific area in the monitoring area;
Extraction means for extracting features of the person identified by the identification means from the second image;
An information processing apparatus comprising: an association unit that associates the behavior of the person specified by the specifying unit with the feature of the person extracted by the extraction unit.   The information processing apparatus according to claim 1, wherein the specifying unit specifies a purchase behavior of the person.   The information processing apparatus according to claim 1, wherein the specifying unit specifies a flow line of the person. The first image is an image obtained by capturing an area including a pre-registration area of a facility,
The specifying means is:
First acquisition means for acquiring customer information including information on the POS cash register where the person has purchased and the time of purchase from the first image;
Second acquisition means for acquiring purchase information including information related to purchased products from the POS cash register,
3. The information according to claim 2, wherein the purchase behavior is specified based on purchaser information acquired by the first acquisition unit and purchase information acquired by the second acquisition unit. Processing equipment. The identification means includes
Tracking means for tracking the person based on the first image;
Detecting means for detecting a time at which the person has entered the specific area based on a tracking result by the tracking means; and
In the second image captured at the time detected by the detecting means, a person existing in an area corresponding to the specific area is identified as a person whose action is specified by the specifying means. The information processing apparatus according to any one of claims 1 to 4.   The information processing apparatus according to claim 1, wherein the specific area is an exit area of a facility. The extraction means includes
The face of the person identified by the identification means is detected from the second image, and the feature is extracted based on the detected face image. The information processing apparatus described.   The information processing apparatus according to claim 1, wherein the extraction unit extracts at least one of the sex and age of the person as the feature.   The information processing according to any one of claims 1 to 8, further comprising a creation unit that creates a report analyzing the behavior of the person based on the information associated by the association unit. apparatus. Identifying the action of the person to be imaged from the first image obtained by imaging the monitoring area;
Identifying the person identified as the action from a second image obtained by imaging a part of the specific area within the monitoring area;
Extracting features of the person identified from the second image;
An information processing method comprising: associating the identified action of the person with the extracted feature of the person.   The program for functioning a computer as each means of the information processing apparatus of any one of Claim 1 to 9.   A computer-readable recording medium on which the program according to claim 11 is recorded.

Images