JP2017027529A - Information processing device, information processing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To connect a purchase customer appearing on an image taken with a network camera with purchase information in a case where a tablet terminal is used as a POS register terminal.SOLUTION: An information processing device comprises: detection means for detecting a person from the image acquired from photographing means; identification means for identifying a clerk from among persons detected by the detection means on the basis of the previously acquired feature of the clerk; distinction means for distinguishing a person other than the clerk identified by the identification means as a candidate of the purchase customer; and transmission means for transmitting the image of the candidate of the purchase customer distinguished by the distinction means to a terminal device for a check-out.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, an information processing method, and a program that provide marketing analysis using a network camera.

  In recent years, network cameras (hereinafter referred to as NW cameras) are being used not only for in-store crime prevention but also for purposes other than monitoring utilizing analysis of images obtained from NW cameras.

  For example, an NW camera is installed in front of a cash register at a retail store, and the customer's characteristics (face, age, sex, etc.) are extracted from the image obtained from the NW camera, and the customer's characteristics and purchase information at the register are obtained. Systems that are linked and used for marketing analysis are being studied.

  For example, Patent Literature 1 discloses a customer behavior recording apparatus and method for collecting data useful for marketing from customer behavior in a store.

  Patent Document 2 discloses a customer data creation system that can effectively use information obtained from a point-of-sale (POS) cash register terminal and image information obtained from a camera for accounting for merchandise. Has been.

  On the other hand, in recent years, a system that uses a portable tablet terminal as a POS cash register terminal instead of a fixed POS cash register terminal has appeared. When a tablet terminal is used as a POS cash register terminal, a store clerk can carry a tablet terminal to serve a customer and directly perform a product accounting process. By introducing a tablet terminal as a POS cash register terminal, it is possible to use POS at low cost, and it is possible to connect smoothly to accounting from customer service. Even when such a tablet terminal is used as a POS cash register terminal, it is expected that a need for marketing analysis using an NW camera will arise.

JP 2010-113692 A JP 2006-126926 A

  When a tablet terminal is used as a POS cash register terminal, a store clerk walks in the store holding the tablet terminal and accounts for the purchaser at an arbitrary place. Therefore, it is assumed that the NW camera is installed at a position where the inside of the store can be photographed widely.

  When the clerk accounts for the purchaser, the NW camera takes a picture of the situation in order to perform customer analysis, but the NW camera image shows the clerk himself and the customers other than the actual purchaser. May also be included. For this reason, in order to perform accurate marketing analysis, it is necessary to appropriately identify the purchaser from the image of the NW camera and to link the purchase information input at the tablet terminal.

Purchaser identifying means (S404, S405) for identifying the purchaser from the image obtained from the imaging device (102);
Purchaser feature extraction means (S606) for extracting purchaser features by performing image analysis on the image corresponding to the purchaser;
There is provided an information processing apparatus having the purchase customer feature and storage means (S411) for storing purchase information related to the purchase customer in association with each other and a control method therefor.

  According to the present invention, it is possible to appropriately identify the purchaser from the image of the NW camera, link the purchase information input at the cash register terminal, and perform accurate marketing analysis.

System configuration example Hardware configuration diagram of server 103 and terminals 101 and 108 Installation example of NW camera Flow of processing in the first embodiment Screen example of tablet cash register terminal (accounting start instruction) Flow of identification process for shop assistants and potential customers An example of data structure in clerk identification processing An example of data structure in purchase candidate identification process Screen example of tablet-type cash register terminal (accounting execution) Flow of processing in the second embodiment Screen example of tablet-type cash register terminal (alert notification) Flow of processing in the third embodiment An example of data structure in tablet-type cash register terminal identification processing

(First embodiment)
<System configuration>
FIG. 1 is an example of a system configuration according to the first embodiment. Each device described below is connected via a network.

  In FIG. 1, the system includes a tablet cash register terminal 101, an NW camera (network camera) 102, a server 103, and a browsing terminal 108. The server 103 is connected to a store clerk face feature database (store clerk face feature DB) 105, a customer face feature database (customer face feature DB) 106, and a POS database (Point Of Sales DB) 107.

  The tablet-type cash register terminal 101 is a terminal device for inputting customer purchase information in accordance with a store clerk's operation. The NW camera 102 is installed in the store as an imaging device for shooting the faces of store clerk and customers. In addition, image analysis described later is performed on the captured image. In the present embodiment, it is assumed that the NW camera 102 includes a storage, and captured images are sequentially written to the storage. The server 103 is connected to the tablet-type cash register terminal 101 and the NW camera 102 via a network, and serves as an information processing apparatus that stores information in which the identified customer information and purchase information are associated with each other in the POS database. It is a server.

  The clerk face feature DB 105 is a storage device as a database in which store clerk face features are stored. The facial features include, for example, the shape of each part of the face, the positional relationship between the facial parts, the outline of the face, and the like, and the face of the store clerk is identified using these pieces of information.

  The customer face feature DB 106 is a storage device as a database in which customer face features are stored. Similar to the clerk face feature DB 105, the face feature stores information such as the shape of each part of the face, the positional relationship of the face parts, and the contour of the face. In addition, age and gender information estimated from facial features is also stored.

  The POS data DB 107 is a database in which purchase information of products purchased at a store is stored. The purchase information is stored in association with the purchaser's ID (identification information) so that the customer can be associated with the purchase information later.

  The server 103 generates a marketing report based on information stored in the customer face feature DB 106 and the POS data DB 107. For example, a report such as “Sale of men for men in their 30s” “Women who are in their 40s are high in purchase price” is generated. Detailed description of the processing contents of report generation will be omitted.

  The browsing terminal 108 is a terminal device used by a store manager, an area manager, or the like to browse a marketing report. The browsing terminal 108 receives the marketing report generated by the server 103 and displays the marketing report on the display.

  In the present embodiment, the control server 103, the clerk face feature DB 105, the customer face feature DB 106, and the POS data DB 107 are constructed on a cloud platform as a network, but are constructed on an on-premises server. Also good.

  In the present embodiment, as the information stored in the clerk face feature DB 105 and the customer face feature DB 106, in addition to the face feature, other features such as the whole body feature and the clothing feature may be stored. In this case, for example, it can be used to generate a marketing report such as “A customer who has a characteristic of OO is selling △ Δ well”.

<Hardware configuration>
FIG. 2 is a diagram illustrating a hardware configuration of the server 103. An HDD (hard disk drive) 202 or ROM 203 stores application programs that operate on an OS (operating system). The CPU 205 reads the OS and application programs from the HDD 202 or the ROM 203, loads them into the RAM 204, and executes them, thereby realizing the operation processing on the server described below. The processing result is stored in the HDD 202 as a file or stored in the RAM 204 as data. The input device 207 acquires input information from the server user and reading values of various sensors. The output device 206 is a display device that displays the display processing result on the display screen. Furthermore, it communicates with the cashier terminal 101, NW camera 102, DB 105-107, etc. connected to the network via the communication device 208. These are connected to each other by a bus 201. The tablet-type cash register terminal 101 and the marketing report browsing terminal 108 have the same configuration as that shown in FIG.

<Example of store installation of NW camera 102>
FIG. 3 shows an installation example of the NW camera 102 of FIG. 1 in a store assumed in the present embodiment. Hereinafter, a customer who visits a store is referred to as a “customer” and a customer who purchases a product is referred to as a “customer”.

  A layout 301 is a sketch in the store. The NW camera 302 is installed toward the store entrance. The NW camera 302 is basically assumed to be used for crime prevention. The NW camera 303 has the NW camera 102 installed in the back of the store. The NW camera 303 is used for identifying the purchaser and estimating the age and sex from the captured image.

  An area 304 indicates a range in which a person photographed by the NW camera 303 can be identified. FIG. 3 shows a state in which the store clerk A responds to the purchase customer and operates the tablet cash register terminal 101 to perform accounting. At this time, the NW camera 303 may show customers other than the purchaser, so the customer who has not actually purchased the product is mistakenly linked with the POS data, and a correct marketing report cannot be obtained. there is a possibility. In the present embodiment, the point is that the POS data and the information of the purchaser can be correctly linked by correctly identifying the purchaser.

<Process flow>
The flow of processing from when the clerk starts accounting to when the purchase information and the customer's facial features are stored in association with each other will be described with reference to FIG.

  In the present embodiment, the face of the store clerk at the time of accounting is identified from the image acquired from the NW camera 303, and the purchase customer is identified based on the relationship between the store clerk and the store visitor. For example, based on the spatial proximity between the store clerk and the store visitor, the closest store visitor is selected as a purchase customer candidate.

  Note that as pre-preparation for this processing, it is assumed that the facial features of all the clerks in the store targeted in the present embodiment are stored in advance in the clerk facial feature DB 105. Specifically, the face of the clerk is photographed with the NW camera 303, and the facial features of the clerk are extracted. Then, the server 103 stores the clerk face feature in the clerk face feature DB 105. In the example illustrated in FIG. 4, it is assumed that the facial features of the salesclerk A and the salesclerk B are stored in the salesclerk face feature DB 105.

  In S401 of FIG. 4, the cashier terminal 101 accepts an accounting start instruction in accordance with the operation of the store clerk. At this time, the cashier terminal 101 also receives information on “a clerk who has instructed to start accounting”, that is, “a clerk in charge of accounting”. FIG. 5 shows a screen example of the cash register terminal 101 at this time. The information of “the clerk who has given the accounting start instruction” is received when the clerk selects the button 501 or the button 502. FIG. 5 shows an example in which the buttons of the clerk A and B are displayed on the cash register terminal 101, but buttons corresponding to the number of clerks stored in the clerk face feature DB 105 are displayed. In the following description, it is assumed that the store clerk A has pressed the button 501. When the cashier terminal 101 accepts the accounting start instruction, the cashier terminal 101 transmits the accounting start instruction to the server 103.

  In step S <b> 402, the server 103 receives an accounting start instruction from the cashier terminal 101. The server 103 receives from the cashier terminal 101 information on “the clerk who issued the accounting start instruction” and “the time when the accounting start instruction was issued”. When the server 103 receives the transaction start instruction, it issues an “account ID” for identifying each transaction.

  In S403, the server 103 instructs the NW camera 303 to identify the purchaser to the NW camera 303 (102). Since the NW camera 303 identifies the purchaser based on the clerk's face, the server 103 needs to transmit the clerk's facial features to the NW camera 303 as well. Since the clerk A is accounting this time, the server 103 acquires the facial feature of the clerk A from the clerk facial feature DB 105 and transmits it to the NW camera 303. In addition, the server 103 also transmits “the time when the accounting start instruction is given” to the NW camera 303.

  In step S404, the NW camera 303 executes processing for identifying a store clerk. The NW camera 303 uses the facial feature of the clerk A received for an image taken in a predetermined time range such as a predetermined time before and after the “time when the accounting start instruction is given” (for example, 1 minute before and after). The clerk is identified. By performing this process, the NW camera 303 calculates the coordinates of the clerk A in each frame image. Details of the processing will be described later with reference to FIGS.

  In S405, the NW camera 303 identifies the purchaser candidate based on the store clerk's coordinates obtained in S404. Details of the processing will be described later with reference to FIGS. Here, the reason why a plurality of candidates are identified instead of determining a single customer is to consider a case where there are a plurality of customers who can be purchased customers as a result of S405. Here, for example, three purchase customer candidates are identified and ranked, and each facial feature is extracted from the image of the NW camera 303. The purchase customer candidate is presented to the store clerk in S408 described later, and it is assumed that the store clerk uniquely determines the purchase customer. When there is only one purchase customer candidate, the purchase customer may be confirmed.

  In S <b> 406, the server 103 receives from the NW camera 303 the facial images and facial features of each of the purchaser candidates obtained in S <b> 405. Information regarding the ranking of the purchaser candidates may also be received from the NW camera 303, but may be determined on the server 103 side. The server 103 temporarily stores the facial features of the purchase customer candidate together with an ID (purchase customer candidate ID) for identifying the purchase customer candidate in its HDD 202 or RAM 204. Then, the face image of the purchase customer candidate is transmitted to the cash register terminal 101 together with the purchase customer candidate ID. It should be noted that the ranking information of the purchaser candidates may also be transmitted at the same time.

  In step S <b> 407, the cash register terminal 101 also receives information regarding the purchase customer candidate's face image and the order of the purchase customer candidate. In S <b> 408, the cash register terminal 101 is a display process for displaying the purchaser candidate's face image on the display. The display contents on the cash register terminal 101 will be described later with reference to FIG.

  In S409, the cash register terminal 101 receives the purchase information and the customer candidate ID selected according to the operation of the store clerk. The purchase information includes, for example, an ID, a name, a price, and a total price of the purchased product selected by the store clerk from product information stored in advance in the HDD 202 of the cash register terminal 101. As the purchaser ID, a purchaser candidate ID selected by the store clerk as at least one purchaser from a plurality of existing purchaser candidates is accepted as the purchaser ID. The cash register terminal 101 transmits the received information to the server 103.

  In S <b> 410, the server 103 receives purchase information and a purchase customer ID corresponding to the purchase customer from the cash register terminal 101. The server 103 acquires the facial features stored in S406 based on the purchased customer ID received here.

  In S411, the server 103 associates the purchase information with the facial features of the purchaser and stores them in its own HDD 202. Further, the server 103 stores the purchase information and the purchaser ID in the POS data DB 107. In addition, the server 103 stores the facial feature and the purchaser ID associated with the facial feature in the customer facial feature DB 106.

Through the above processing, the purchase information and the facial features of the purchaser can be stored in association with each other.
The server 103 generates a marketing report based on the customer face feature DB 106 and the POS data DB 107 accumulated for a predetermined period, but detailed description is omitted.

  In this embodiment, in S401, as shown in FIG. 5, “the clerk who issued the accounting start instruction (the clerk in charge of accounting)” was received on the screen of the cash register terminal 101. This is because it is assumed that the cashier terminal 101 is shared by the clerk A and the clerk B. If each of the clerk A and the clerk B has a dedicated tablet cash register terminal, the clerk may be linked to each tablet cash register terminal in advance, and only the accounting start button may be provided in FIG. .

<Clerk and purchaser candidate identification processing>
With reference to FIG. 6 to FIG. 8, a process for identifying a store clerk and a purchase customer candidate will be described.

  FIG. 6A is a flowchart showing the clerk identification process. 4 corresponds to the processing by the NW camera 303 in S404.

  FIG. 6B is a flowchart showing the analysis process of the purchaser candidate. 4 corresponds to the processing by the NW camera 303 in S405.

  FIG. 6C is a conceptual diagram showing a store clerk and two purchase customer candidates reflected in each frame. As shown in FIG. 6C, for an image (hereinafter referred to as “target image”) of a predetermined time before and after the time when the accounting start instruction is accepted (hereinafter referred to as “target image”), first, the coordinates of the clerk in the image are used. calculate. Next, a purchase customer candidate is identified based on the coordinates of the store clerk. The reason for processing before and after the time at which the accounting start instruction is received is that there is a possibility that the purchase customer cannot be identified because, for example, the purchase customer faces down at the moment when the accounting start instruction is issued.

  FIG. 7 shows a data structure regarding the position of the coordinates indicating the position where the face is detected in the store. FIG. 8 shows a data structure related to the purchaser candidate.

  In S601 of FIG. 6A, the NW camera 303 performs face detection by analyzing the captured image. For the face detection, for example, a technique disclosed in JP-A-7-31833 is used. Furthermore, the NW camera 303 estimates the coordinate position of the detected face. The estimation of the coordinate position of the face is estimated based on the shooting direction of the NW camera 303, the position of the face in the frame of the shot image, the size of the face in the frame, and the like.

  In FIG. 7A, the face coordinate ID 701 is a number for identifying the detected face. The frame number 702 is information indicating the time when the face is detected. The face coordinate 703 is position information of the detected face, that is, position information of a person corresponding to the detected face. The face coordinate data shown in FIG. 7A includes all face coordinates without distinguishing between the store clerk and the store visitor, and these data are associated with the detected face feature data.

  In step S602, the NW camera 303 executes a clerk face identification process. Each face image corresponding to each face coordinate ID in FIG. 7A is compared with the face feature of the clerk received from the server 103 to identify the face. For example, a technique disclosed in Japanese Patent Application Laid-Open No. 2003-150963 may be used for face identification. By this processing, as shown in FIG. 7B, identification information 704 indicating whether or not a store clerk is newly added to FIG. 7A is added.

  In step S603, the NW camera 303 tracks the clerk. Since the clerk's face is not reflected in every frame, it is necessary to interpolate the position coordinates between the frames in which the clerk can be identified. Specifically, the position of the face of the store clerk in the frame in which the store clerk could not be identified based on the face coordinates 703 of the frame in which the identification information 704 in FIG. An estimation process is performed. This processing uses a known technique (for example, JP-A-2003-346158). By this processing, as shown in FIG. 7C, the salesperson's face coordinate data of the frame in which the presence of the salesclerk is estimated is obtained. Item 705 is a frame number, and item 706 is a store clerk's face coordinate existing in each frame.

  In step S <b> 604, the NW camera 303 executes processing for identifying a purchase customer candidate. In the identification information 704 in FIG. 7B, the face (person) corresponding to the face coordinate ID having the value of False is a face (person) that is not a store clerk, that is, the face coordinates of the purchaser candidate.

  Similar to the processing in S603, the purchaser candidate is tracked (tracked) for the face coordinate ID having the value of False. By this processing, as shown in FIG. 8A, data in which the facial coordinates of the purchaser candidate are stored is obtained. An item 801 is a face coordinate ID of the purchase customer candidate. An item 802 is a frame number indicating time information, and an item 803 is a facial coordinate of a purchase customer candidate. An item 804 is a purchase customer candidate ID. As a result of tracking, items having close coordinate positions between frames are regarded as the same purchase customer candidate, and the same ID is assigned. An item 805 is a distance from the store clerk in the image. The distance to the store clerk is calculated from the store clerk's coordinates and the purchase customer candidate's coordinates.

  In step S605, the NW camera 303 ranks the purchaser candidates. For example, the NW camera 303 ranks the data shown in FIG. 8A according to the index of “the distance from the store clerk is as close as possible”. In the example of FIG. 8A, the distance from the purchaser candidate clerk with purchaser candidate ID: 002 is closer than the distance from the purchaser candidate clerk with purchaser candidate ID: 001. Therefore, the purchase customer candidate with purchase customer candidate ID: 002 is identified as a higher rank. As other indexes for ranking the purchaser candidates, indexes such as “present before and after the accounting start instruction frame” and “the distance from the store clerk is not separated” may be incorporated. In the case of incorporating a plurality of indices, the respective indices are weighted and the final ranking is executed.

In step S <b> 606, the NW camera 303 executes a facial feature analysis process for the purchaser candidate. The facial feature analysis process uses a known technique (for example, Japanese Patent Application Laid-Open No. 2009-86901).
By this processing, data shown in FIG. 8B is obtained. An item 806 is a purchase customer candidate ID, and items 807, 808, and 809 indicate different types of facial features. The facial features include, for example, information on the shape of each part of the face, the positional relationship between the facial parts, the contour of the face, the age estimated from them, and sex.

  Through the above processing, the purchase customer candidate and each facial feature are obtained.

<UI on the screen of a tablet cash register terminal>
In S408 of FIG. 4, the display processing in which the cashier terminal 101 displays the purchaser candidate's face image has been described. Here, an example of the display screen of the cashier terminal 101 displayed at that time is shown in FIG. 9. explain.

  In FIG. 9, a barcode reader 901 exchanges data with the tablet cash register terminal 101 by short-range wireless communication. The store clerk reads the barcode attached to the product using the barcode reader 901, and the product ID is transmitted to the cash register terminal 101. A list 902 is a list of accounting target products. As in the list 902, the content read by the barcode reader 901 is displayed as a product list on the display screen of the cash register terminal. The amount 903 is the total amount of the accounting target product. The face image 904 is a face image of a purchase customer candidate. The cash register terminal 101 displays a face image of a high order based on the face image of the purchase customer candidate and the order of the purchase customer candidate. The button 905 is a button for switching the face image 904 to the next purchase customer candidate. The store clerk confirms whether or not the face image displayed on the face image 904 is the face of the purchaser who is responding to the face image.

  If the face image displayed on the face image 904 is not the face of the purchase customer who is responding to the face image 904, the button 905 is operated until the face of the purchase customer is displayed. If the purchaser's face is not finally displayed, the NW camera 303 is inquired from the cashier terminal 101 via the server 103. For example, in the process of FIG. You may make it identify again. A button 906 is a button for instructing accounting. When the cashier terminal 101 accepts a transaction instruction by selecting the button 906 by the user, the cashier terminal 101 transmits the purchase customer ID corresponding to the face image 904 and purchase information obtained from the contents of the list 902 to the server 103. In the server 103, the purchase customer candidate ID having the same ID as the received purchase customer ID and the facial features 807 to 809 associated with the purchase customer candidate ID are associated with the received purchase information.

  As described above, it is possible to associate the correct purchase customer with the purchase information by determining the purchase customer based on the face of the store clerk. As a result, a highly accurate marketing report can be obtained.

  In this embodiment, a case has been described in which one NW camera 303 for photographing a purchaser is installed, but a plurality of NW cameras 303 may be installed. In that case, the server 103 instructs the purchaser to identify all connected NW cameras in S403. At that time, the server 103 transmits the facial features of the clerk and “the time when the accounting start instruction is given” to each NW camera. Each NW camera performs the processes of S404 and S405, and the server 103 receives the face image and the facial feature of the purchaser candidate from one of the NW cameras in S405.

  In this embodiment, the NW camera 303 performs the clerk identification process (S404) and the purchase customer candidate identification process (S405), but this process may be performed on the server 103 side. In that case, the NW camera 303 always transmits the captured image to the server 103, and the server 103 stores the image. The server 103 may perform a store clerk identification process (S404) and a purchase customer candidate identification process (S405) on the stored image.

(Second Embodiment)
In the first embodiment, it is assumed that the store clerk who responds to the purchase customer is always photographed by the NW camera 303. There is a possibility of appearing. Even within the shooting angle of view, there is a possibility that the face of the clerk may fail to be identified because it is too far from the NW camera 303. In this case, there is a possibility that the purchase customer cannot be identified and cannot be associated with purchase information.

  In the present embodiment, an alert is displayed on the cash register terminal 101 in the above situation so that the store clerk is aware. As a result, the purchaser can be more accurately associated with the purchase information.

<Process flow>
Also in this embodiment, the overall flow is basically the same as in FIG. 4, but the internal processing in S404 is different from that in the first embodiment.

  The flow of processing in the present embodiment will be described using FIG.

  FIG. 10A is a flowchart showing the internal processing of S404 executed by the NW camera 303 in the present embodiment. Here, S601, S602, and S603 have already been described with reference to FIG.

  S1001 is processing in which the NW camera 303 determines whether or not the face of the store clerk has been identified. If no store clerk's face is identified in S602, or if the store clerk's face is identified with a number smaller than a certain number of frames, the process proceeds to S1002. If a clerk is identified, the process proceeds to S603.

  In step S <b> 1002, the NW camera 303 transmits warning information to the server 103.

  FIG. 10B is a diagram illustrating a flow when the NW camera 303 transmits warning information (alert). In S1003, the server 103 receives the warning information. Then, the server 103 transmits warning information to the cashier terminal 101 that has accepted the accounting instruction.

  In S1004, when the cashier terminal 101 receives the alert notification, in S1005, the cashier terminal 101 executes processing for displaying a warning message.

  FIG. 11 is a display example of a warning message on the cash register terminal 101. A message 1101 is a warning message, and a message to the effect of approaching the camera is displayed to the store clerk who has accepted the start of accounting. Note that an icon or the like may be displayed instead of the message to warn.

  A map 1102 is a sketch of a store. On the map 1102, an index indicating which direction the store clerk should approach is displayed.

  When the map 1102 is displayed, it is necessary to register the floor plan itself and the camera position 1102 in the floor plan in advance with the tablet-type cash register terminal 101 or the server 103. A button 1103 is a button for instructing start of accounting. The store clerk approaches the camera and presses the button 1103 again to start accounting.

  In this embodiment, the map 1102 is displayed to notify the store clerk which direction should be approached. Instead of displaying the map 1102, when the NW camera 303 notifies the alert in S1102 of FIG. 10 (A), the lamp provided in the NW camera 303 is turned on to notify the store clerk in which direction to approach. You may make it do. Further, the NW camera 303 may make a sound to notify the store clerk which direction should be approached.

  In the present embodiment, the NW camera 303 performs a process of identifying the store clerk after the cashier terminal 101 receives the transaction start instruction. Before accepting an accounting start instruction, a clerk may be photographed with an NW camera, and if the clerk is not shown, an alert may be sent to the clerk. In this case, for example, the cashier terminal 101 detects that the cashier terminal 101 has been taken out from the charger (cradle), and notifies the NW camera 303 via the server 103. If the clerk is tracked from that timing and the clerk is not shown, an alert is displayed in the same flow as in FIG.

(Third embodiment)
In the first and second embodiments, the purchaser is identified based on the face of the store clerk. In the third embodiment, the cash register terminal 101 includes a device that emits infrared light, and the purchaser is identified by detecting the infrared light with the NW camera 303.

  In the first and second embodiments, it is necessary to register the clerk face feature DB 105 in the clerk face feature DB 105 as advance preparation, but in this embodiment, it is not necessary to have the clerk face feature DB 105. On the other hand, in this embodiment, it is necessary to prepare a cash register terminal 101 having a device that emits infrared light and an NW camera that can receive infrared light. If the store clerk is fixed to some extent, the first and second embodiments may be used, and if the store clerk is frequently replaced, the third embodiment may be used.

<Process flow>
FIG. 12 is a diagram for explaining the flow of processing in the third embodiment.

  Also in this embodiment, the overall flow is basically the same as in FIG. 4, but S1201 is executed instead of S404. S401 to S411 are the same as those in FIG.

  In step S1201, the NW camera 303 executes processing for identifying the cashier terminal 101. The NW camera 303 identifies infrared rays from the target image, and records the infrared coordinates in the image as the coordinates of the cash register terminal 101. In S1201, the face of the clerk is also detected as in S601 of FIG.

  FIG. 13 shows a data structure indicating the coordinates of the cashier terminal 101. An item 1301 is a frame number. An item 1302 is the coordinates of the cashier terminal 101 in the captured image. In subsequent S <b> 1202, the NW camera 303 identifies the purchase customer candidate based on the image coordinates of the cash register terminal 101. The processing content is the same as in FIG. 6B, and the purchaser candidate identification is executed using the coordinates of the cashier terminal 101 instead of using the store clerk's face coordinates.

  In the present embodiment, it is assumed that the NW camera 303 does not have a function of identifying the salesclerk's face. Therefore, there is a possibility that a store clerk's face is also displayed on the cash register terminal 101 shown in FIG. 9 as a purchase customer candidate. Actually, the store clerk makes a judgment by himself and selects the purchasers who have actually purchased, so there is no problem.

  When the NW camera 303 has a function of identifying the clerk's face, as a method of not displaying the clerk's face, the NW camera 303 identifies the person closest to the cashier terminal 101 as the clerk and removes it from the purchaser candidate. Processing can be performed.

  As another embodiment, the tablet-type cash register terminal 101 may include a location information transmission device, and the location information in the store of the cashier terminal 101 may be notified to the NW camera 303 via the server 103. As a position information transmission device, a GPS (Global Positioning System) information transmission device or the like can be considered.

  The NW camera 303 calculates the position in the image where the tablet cash register terminal 101 exists from the received location information in the store. Then, the purchaser candidate is identified by a process similar to the process described in S1202 of FIG. In this case, it is necessary that the three-dimensional position and orientation of the NW camera 303 with respect to the store are measured in advance.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

101 Tablet-type Cash Register Terminal 102 Network Camera 103 Server 105 Clerk Face Feature Database 106 Customer Face Feature Database

Claims (15)

Detecting means for detecting a person from an image obtained from the imaging means;
Identification means for identifying the clerk from the person detected by the detection means based on the characteristics of the clerk acquired in advance;
A discriminating means for discriminating a person who is not a clerk identified by the discriminating means as a candidate for a purchaser;
A transmission means for transmitting the image of the candidate of the purchaser determined by the determination means to a terminal device for accounting;
An information processing apparatus comprising:   The information processing apparatus according to claim 1, wherein the determination unit determines a purchaser candidate based on a relationship with a salesclerk identified by the identification unit.   The information processing apparatus according to claim 2, wherein the determination unit determines a candidate for a purchaser based on a spatial proximity with a clerk identified by the identification unit.   The determination means determines a person detected from an image captured by the imaging means as a purchase customer candidate within a range corresponding to the time when the transaction is started at a terminal where the transaction processing is performed. The information processing apparatus according to claim 1.   The information processing apparatus according to claim 1, wherein when the identification unit fails to identify a store clerk, the transmission unit transmits warning information to a terminal device that performs the transaction. Detecting means for detecting a person from an image obtained from the imaging means;
Terminal identification means for identifying a terminal device that performs accounting from an image obtained from the imaging means;
A discriminating means for discriminating a purchaser candidate from the person detected by the detecting means based on the relevance with the terminal identified by the terminal identifying means;
An information processing apparatus comprising: a transmission unit configured to transmit an image of a purchase customer candidate determined by the determination unit to a terminal device that performs the transaction. Display processing means for displaying an image of a purchaser candidate transmitted by the transmission means of the information processing apparatus according to claim 1;
A terminal device for performing accounting, comprising: means for associating at least one image among purchase candidate images displayed by the display means with purchase information. A detection step of detecting a person from an image obtained from an imaging means;
An identification step of identifying a clerk from the person detected in the detection step based on the characteristics of the clerk acquired in advance;
A determination step of determining a person who is not a clerk identified in the identification step as a candidate for a purchaser;
A transmission step of transmitting the image of the candidate of the purchaser determined in the determination step to a terminal device for accounting;
A processing method for an information processing apparatus, comprising:   9. The processing method of the information processing apparatus according to claim 8, wherein, in the determination step, a purchase customer candidate is determined based on the relevance with the clerk identified in the identification step.   The processing method of the information processing apparatus according to claim 9, wherein in the determination step, a purchaser candidate is determined based on a spatial proximity with the clerk identified in the identification step.   In the determination step, determining a person detected from the image captured by the imaging means as a purchaser candidate within a range corresponding to the time when the transaction is started at the terminal where the transaction is performed. The processing method of the information processing apparatus according to claim 8, wherein:   9. The processing method of the information processing apparatus according to claim 8, wherein, when a store clerk is not identified in the identification step, warning information is transmitted to the terminal device that performs the transaction in the transmission step. A detection step of detecting a person from an image obtained from an imaging means;
A terminal identifying step for identifying a terminal device that performs accounting from an image obtained from the imaging means;
A determination step of determining a candidate of a purchaser from a person detected in the detection step based on a relationship with the terminal identified in the identification step of the terminal;
A processing method for an information processing apparatus, comprising: a transmission step of transmitting an image of a candidate for a purchaser determined in the determination step to a terminal device that performs the transaction.   7. A display processing step for displaying on the display means of the terminal device for accounting the image of the purchase customer candidate transmitted in the transmission step according to claim 1; and a purchase customer candidate displayed in the display processing step. A processing method for a terminal device for accounting, comprising a step of associating at least one of images with purchase information.   A program for causing a computer to execute the processing method according to claim 8.

Images