JP7631442B2 - Surveillance system - Google Patents

Description

FIELD OF THE DISCLOSURE The present invention relates to a surveillance system .

Traditionally, in stores such as supermarkets, store clerks operate Point Of Sale (POS) terminals to register products and process payments. Stores also use a checkout system known as semi-self-service, in which the clerk registers products and the customer processes the payment themselves, and a checkout system known as self-service, in which the customer registers products and processes the payment themselves.

However, in the semi-self-service and self-service mentioned above, the payment for the products is left to the customer, so there is a possibility of fraudulent behavior such as "basket skipping," where the customer leaves the store without paying. A technology has been known in the past in which a semi-self-service store is provided with a first imaging device that captures an image of the customer who has requested to register the products, and a second imaging device that captures an image of the vicinity of the exit, etc., and the images of the customer captured by these imaging devices are compared to determine whether or not a customer who has not yet paid is near the exit.

However, in the conventional technology described above, the determination of whether or not a transaction has been completed is made based on a completion notification sent from the accounting device, rather than using captured images. Therefore, in a surveillance system based on comparing captured images, the conventional technology described above cannot be said to be efficient, and there is room for further improvement.

The problem to be solved by the present invention is to provide a surveillance system capable of efficiently monitoring for fraudulent acts such as taking unpaid merchandise outside a store in a store where customers pay for their own purchases.

The surveillance system of the embodiment comprises a first imaging means for imaging customers in a first area where products are registered, a second imaging means for imaging customers in a second area where the customers make payment for the registered products, a third imaging means for imaging a third area through which customers pass after completing their payment, a first selection means for identifying the customer who appears most frequently from the images for each transaction captured by the first imaging means and selecting the image in which the area of the customer's face region is the largest, a second selection means for identifying the customer who appears most frequently from the images for each transaction captured by the second imaging means and selecting the image in which the area of the customer's face region is the largest, a tracking means for tracking the movement path of the same customer from the first area to the third area based on the similarity between feature information extracted from the images selected by the first selection means and the second selection means and feature information extracted from the images captured by the third imaging means, and a notification means for issuing an alert when the movement path indicates that the customer has reached the third area from the first area by skipping the second area.

FIG. 1 is a diagram illustrating an example of a store layout according to the first embodiment. FIG. 2 is a diagram illustrating an example of the configuration of the monitoring system according to the first embodiment. FIG. 3 is a diagram illustrating an example of the configuration of a registration device according to the first embodiment. FIG. 4 is a flowchart illustrating an example of a transmission image selection process executed by the registration device according to the first embodiment. FIG. 5 is a diagram illustrating an example of the configuration of the accounting device according to the first embodiment. FIG. 6 is a diagram illustrating an example of the configuration of the monitoring server according to the first embodiment. FIG. 7 is a diagram illustrating an example of the data structure of the tracing table. FIG. 8 is a flowchart illustrating an example of a monitoring process executed by the monitoring server of the first embodiment. FIG. 9 is a diagram illustrating an example of a store layout according to the second embodiment. FIG. 10 is a diagram illustrating an example of the configuration of a monitoring system according to the second embodiment. FIG. 11 is a diagram illustrating an example of the configuration of a monitoring server according to the second embodiment.

Embodiments of a monitoring device and a program according to the present invention will be described in detail below with reference to the drawings. In the embodiment shown below, an example in which the present invention is applied to a store such as a supermarket will be described, but the present invention is not limited to this embodiment.

[First embodiment]
Fig. 1 is a diagram showing a schematic example of a store layout according to the first embodiment. As shown in Fig. 1, the store has a checkout area A1 for registering products and making payments. The checkout area A1 includes a registration area A11, a payment area A12, and an exit area A13.

The registration area A11 is a first area located at the entrance to the checkout area A1, and is provided with a registration device 10. The accounting area A12 is a second area where customers can pay for merchandise, and is provided with an accounting device 20. The exit area A13 is a third area located at the exit of the checkout area A1, and is provided with the store's exit B11. Although not shown, it is assumed that there is an area within the store where merchandise for sale is displayed (for example, above the checkout area A1).

The registration device 10 is a product data processing device that is operated by a store clerk and performs registration processing of products purchased by customers. The accounting device 20 is a product data processing device that is operated by a customer and performs accounting processing of products registered by the registration device 10. In other words, the registration device 10 and the accounting device 20 realize a checkout method known as semi-self-service or the like.

In a store with the layout shown in Figure 1, the actions taken when a customer purchases an item are as follows. First, the customer places the items to be purchased in a shopping cart or similar and carries them to the registration area A11 (registration device 10). In the registration device 10, a store clerk registers the items by having the barcode attached to the item read by a reading unit 14 (see Figure 3), which will be described later. Once registration of all items related to one transaction has been completed, the store clerk selects (designates) one accounting device 20 to be used for payment, and notifies the customer of that accounting device 20 to guide them to the accounting device. The registration device 10 also sends accounting information, including information such as the unit price and quantity of each registered item, to the designated accounting device 20.

When the customer has completed registering the products, they move to the accounting area A12 with the products (shopping basket) and complete the transaction at the accounting device 20 designated by the store clerk. At that time, the accounting device 20 performs the transaction process based on the transaction information transmitted in advance from the registration device 10. Then, when the customer has completed the transaction, they place the purchased products in a shopping bag or the like and move to the exit area A13 (exit B11).

In this way, the semi-self-service system allows the registration and payment tasks to be shared, which can improve the efficiency of the process, such as shortening waiting times at the register. However, because the semi-self-service system leaves payment up to the customer, there is a risk of fraudulent behavior, such as basket skipping, in which items are taken out of the store without being paid for.

Therefore, in the surveillance system of this embodiment, images of the registration area A11, the accounting area A12, and the exit area A13 are captured, and customers who may be engaging in fraudulent activities are detected based on the captured images. The configuration of the surveillance system according to this embodiment is described below.

Figure 2 is a diagram showing a schematic example of the configuration of a monitoring system according to the first embodiment. As shown in the figure, the monitoring system includes the above-mentioned registration device 10 and accounting device 20, as well as a first camera 31, a second camera 32, a third camera 33, and a monitoring server 40. The registration device 10, accounting device 20, third camera 33, and monitoring server 40 are connected to a network N1 such as a LAN (Local Area network).

The first camera 31 is an imaging device that captures images of customers present in the registration area A11 (registration device 10). The first camera 31 is provided in each registration device 10 (see FIG. 1). More preferably, the first camera 31 is provided in a position in each registration device 10 where it can capture an image of the customer's face. The registration device 10 and the first camera 31 are connected by a connection line L1. The installation position of the first camera 31 is not particularly important, and it may be provided integrally with the registration device 10, or may be provided on the ceiling of the store in the form of a surveillance camera or the like.

The registration device 10 cooperates with the first camera 31 to transmit a captured image (hereinafter referred to as a first captured image) captured during the registration process to the monitoring server 40. Note that the concept of a captured image includes still images and videos.

Fig. 3 is a diagram showing an example of the configuration of the registration device 10 according to the first embodiment. As shown in Fig. 3, the registration device 10 includes a control unit 11. The control unit 11 has a computer configuration including a CPU (Central Processing Unit), a SoC (System-on-a-Chip), a ROM (Read Only Memory), and a RAM (Random Access Memory).

The control unit 11 is connected to a display unit 12, an operation unit 13, a reading unit 14, a memory unit 15, a connection interface 16, a communication interface 17, etc. via a bus or the like.

The display unit 12 has a display device such as a liquid crystal display. Under the control of the control unit 11, the display unit 12 displays various information such as the product names and product codes of registered products. The operation unit 13 has various operation keys and a pointing device, and accepts operations by an operator (store clerk). For example, the operation unit 13 has operation keys for instructing the start and completion of the registration process, and number keys for inputting product codes, the number of products, etc. The operation unit 13 may be a touch panel provided on the display surface of the display unit 12.

The reading unit 14 is a code reader capable of reading code symbols such as barcodes and two-dimensional codes. For example, the reading unit 14 reads a product code stored in a code symbol attached to a product. Here, the product code is identification information for identifying the product, and unique identification information is assigned to each type of product.

The memory unit 15 is a storage device such as a HDD (Hard Disk Drive) or flash memory. The memory unit 15 stores programs executed by the control unit 11 and various data used to execute the programs. For example, the memory unit 15 stores in advance a product master (not shown) that associates the product code of each product with product information including the product name and unit price of the product. The memory unit 15 may also store in advance captured images of the faces of each store clerk and feature information representing facial features extracted from the captured images in order to distinguish between the faces of store clerks and the faces of customers.

The connection interface 16 is an interface that can be connected to the first camera 31. The connection interface 16 accepts the first captured image input from the first camera 31. The communication interface 17 is a communication interface that can be connected to the network N1. The communication interface 17 transmits and receives various information to and from external devices (e.g., the accounting device 20, the monitoring server 40) connected to the network N1.

In the registration device 10 configured as described above, when a store clerk operates to read the code symbol attached to the product to be purchased, the control unit 11 registers the product by storing the product code read by the reading unit 14 in a RAM or the like. When the number of items (quantity) of the product is input, the control unit 11 registers the number of items in association with the product code. The control unit 11 also acquires the first captured image taken by the first camera 31 while the product registration process is being performed.

When registration of products for one transaction is complete, the control unit 11 generates accounting information based on the product codes and points registered up to that point. Here, accounting information includes, for example, the product name and price of each product (product code), the total payment amount, etc. Note that the product name and price are derived based on the product information registered in the product master.

When a specific accounting device 20 is selected (specified) by a store clerk via the operation unit 13 or the like, the control unit 11 sends product information to that accounting device 20. The control unit 11 then sends the first captured images acquired during the registration process to the monitoring server 40. Note that there is no particular restriction on the number of first captured images (still images) or the number of hours (video) sent to the monitoring server 40. The selection of the accounting device 20 may also be configured so that the control unit 11 automatically selects the accounting device 20 based on the availability of the accounting device 20, etc.

The first captured image serves as a reference image for identifying (recognizing) the same customer in the monitoring process described below. For this reason, it is preferable for the control unit 11 to transmit to the monitoring server 40 a first captured image in which characteristic parts of the customer, such as the customer's face, are captured. Therefore, for example, the control unit 11 may be configured to select a first captured image to be transmitted to the monitoring server 40 from among the first captured images captured during the registration process, based on the face area, the frequency with which the face of the same person appears, and the like. The process for selecting the first captured image will be described below.

Figure 4 is a flowchart showing an example of a transmission image selection process executed by the registration device 10 of the first embodiment. Note that this process is executed in the background of the registration process. Also, in this process, known technologies are used for face area detection, face recognition, etc.

First, when the control unit 11 receives an instruction to start registration via the operation unit 13 or the like (step S11), it causes the first camera 31 to start capturing images (step S12).

Next, the control unit 11 determines whether or not a face area has been detected from the first captured image input from the first camera 31 (step S13). If the control unit 11 cannot detect a face area (step S13; No), the control unit 11 proceeds to step S17. Furthermore, when the control unit 11 detects a face area from the first captured image (step S13; Yes), the control unit 11 compares the features of the face area with the features of the face areas of each store clerk stored in the memory unit 15 to determine whether the captured person is a store clerk (step S14).

If the captured person is a store clerk (step S14; Yes), the control unit 11 returns the process to step S13. If the captured person is not a store clerk, i.e., a customer (step S14; No), the control unit 11 derives the area of the face region (step S15) and temporarily stores the first captured image in descending (or ascending) order of area in a RAM or the like (step S16).

The control unit 11 then determines whether or not an instruction to complete registration has been given via the operation unit 13 or the like (step S17). If an instruction to complete registration has not been given (step S17; No), the control unit 11 returns the process to step S13. As a result, while the product is being registered, the first captured image including the customer's facial area captured by the first camera 31 is temporarily stored in order of the area of the facial area.

When the control unit 11 is instructed to complete registration (step S17; Yes), it compares the features of the facial areas between the temporarily stored first captured images to identify the same person (customer). Next, the control unit 11 identifies the face of the customer that appears most frequently among the identified customers (step S18). For example, the first captured images captured by the first camera 31 may include customers other than the customer involved in the transaction (e.g., customers waiting in line). Therefore, in step S18, the control unit 11 recognizes the same customer included in each of the first captured images and identifies the customer that appears most frequently as the customer involved in the transaction.

Next, the control unit 11 selects the first captured image in which the area of the face area of the customer identified in step S18 is the largest from the first captured images that include the face area of the customer identified in step S18 (step S19). The control unit 11 then transmits the first captured image selected in step S19 to the monitoring server 40 (step S20), and ends this process.

By performing the above process, the registration device 10 can transmit the first captured image showing the customer's features (face) to the monitoring server 40. This can improve the customer recognition accuracy in the monitoring process described below.

Returning to FIG. 2, the second camera 32 is an imaging device that captures images of customers present in the accounting area A12. A second camera 32 is provided for each accounting device 20. More preferably, the second camera 32 is provided in a position in each accounting device 20 where it is possible to capture an image of the face of the customer. The accounting device 20 and the second camera 32 are connected by a connection line L2. There is no particular restriction on the installation location of the second camera 32, and it may be provided integrally with the accounting device 20, or may be provided on the ceiling of the store in the form of a surveillance camera or the like.

The accounting device 20 cooperates with the second camera 32 to transmit images captured during the accounting process (hereinafter referred to as second images) to the monitoring server 40.

Figure 5 is a diagram showing an example of the configuration of the accounting device 20 according to the first embodiment. As shown in Figure 5, the accounting device 20 includes a control unit 21. The control unit 21 has a computer configuration including a CPU, SoC, ROM, RAM, etc.

The control unit 21 is connected to a display unit 22, an operation unit 23, a printing unit 24, a memory unit 25, a connection interface 26, a communication interface 27, etc. via a bus or the like.

The display unit 22 has a display device such as a liquid crystal display. Under the control of the control unit 21, the display unit 22 displays various information such as accounting information. The operation unit 23 has various operation keys and a pointing device, and accepts operations by the operator (customer). For example, the operation unit 23 has operation keys for instructing the start and completion of accounting processing. The operation unit 23 may be a touch panel provided on the display surface of the display unit 22.

The printing unit 24 is a printing device such as a thermal printer. Under the control of the control unit 21, the printing unit 24 prints details of the transaction (accounting information) on paper media such as receipt paper. The memory unit 25 is a storage device such as a HDD or flash memory. The memory unit 25 stores the programs executed by the control unit 21 and various data used to execute the programs.

The connection interface 26 is an interface that can be connected to the second camera 32. The connection interface 26 accepts the second captured image input from the second camera 32. The communication interface 27 is a communication interface that can be connected to the network N1. The communication interface 27 transmits and receives various information to and from external devices (e.g., the registration device 10, the monitoring server 40) connected to the network N1.

The control unit 21 is also connected via a bus or the like to a money deposit device that accepts coins and bills, a money dispensing device that dispenses change, etc. (neither shown).

In the accounting device 20 configured as described above, the control unit 21 temporarily stores the accounting information sent from the registration device 10 in RAM or the like, and waits for the start of accounting processing related to that accounting information. When a customer moves to the accounting device 20 instructed by the clerk at the registration device 10, the customer instructs the start of accounting processing via the operation unit 23. When the start of accounting processing is instructed, the control unit 21 accepts payment (deposit) for the product price based on the temporarily stored accounting information. Then, when the payment is completed, the control unit 21 causes the printing unit 24 to output receipt paper on which the details are printed, and ends the accounting processing.

The control unit 21 also acquires second captured images captured by the second camera 32 while the transaction is being processed. Then, when the transaction is completed, the control unit 21 transmits the second captured images acquired during the transaction to the monitoring server 40. Here, the number of second captured images (still images) or the number of hours (video) transmitted to the monitoring server 40 is not particularly important. The control unit 21 may select the second captured images to transmit to the monitoring server 40 based on the area and appearance frequency of the face region, etc., by performing a transmission image selection process similar to that of the registration device 10 described above.

Returning to FIG. 2, the third camera 33 is an imaging device that captures images of customers present in the exit area A13. The third camera 33 is installed in a position that allows it to capture images of the faces of customers passing through the exit B11. The third camera 33 transmits the captured image (hereinafter referred to as the third captured image) to the surveillance server 40. There is no particular restriction on the installation location of the third camera 33, and it may be installed on the ceiling of the store in the form of a surveillance camera or the like.

The monitoring server 40 is a server device that monitors the behavior of customers in the checkout area A1 based on images captured by the first camera 31, the second camera 32, and the third camera 33. The monitoring server 40 will be described below.

Fig. 6 is a diagram showing an example of the configuration of the monitoring server 40 according to the first embodiment. As shown in Fig. 6, the monitoring server 40 includes a control unit 41. The control unit 41 has a computer configuration including a CPU, SoC, ROM, RAM, etc.

The control unit 41 is connected to the memory unit 42 via a bus or the like. The memory unit 42 is a storage device such as a HDD or flash memory. The memory unit 42 stores the programs executed by the control unit 41 and various data used to execute the programs. The memory unit 42 also stores a tracking table T1 (see Figure 7) for tracking the movement path of a customer in the checkout area A1.

The control unit 41 is also connected to a communication interface (I/F) 43 via a bus or the like. The communication interface 43 is connected to the network N1 and transmits and receives various types of information to and from other devices (the registration device 10, the accounting device 20, the third camera 33, etc.).

As shown in FIG. 6, the control unit 41 includes functional units such as an image acquisition unit 411, a feature extraction unit 412, a registration processing unit 413, a tracking processing unit 414, a fraud detection unit 415, and a notification processing unit 416. These functional units are realized in software by executing a program stored in the memory unit 42, or in hardware by a dedicated processor or the like included in the control unit 41.

The image acquisition unit 411 acquires the captured images captured by each of the first camera 31, the second camera 32, and the third camera 33 via the communication interface 43. More specifically, the image acquisition unit 411 acquires the first captured image transmitted from each of the registration devices 10. The image acquisition unit 411 also acquires the second captured image transmitted from each of the accounting devices 20. The image acquisition unit 411 also acquires the third captured image transmitted from the third camera 33.

The feature extraction unit 412 extracts feature information representing the features of the person (customer) included in each of the captured images (first captured image, second captured image, third captured image) acquired by the image acquisition unit 411. Here, the feature information is, for example, a feature amount representing the features of the face area included in the captured image. In addition, there are cases where the face area cannot be detected because the person in the captured image is wearing sunglasses, a mask, or the like. In such cases, the feature extraction unit 412 extracts the features of the element from other elements such as the person's clothing, hairstyle, and body type as feature information. Hereinafter, the feature information extracted from each of the first captured image, second captured image, and third captured image will be referred to as the first feature information, the second feature information, and the third feature information, respectively. Note that there is no particular limit to the method of extracting the feature information, and known techniques such as face recognition and image recognition may be used.

The registration processing unit 413 is a functional unit that sets the customers to be tracked. More specifically, the registration processing unit 413 registers the first feature information extracted from the first captured image by the feature extraction unit 412 in the tracking table T1 of the memory unit 42.

Here, FIG. 7 is a diagram showing an example of the data structure of the tracking table T1. As shown in FIG. 7, the tracking table T1 has a tracking target column, a first check column, a second check column, and a third check column as items. Each of these items is prepared for each customer and constitutes one data entry.

The tracking target column is an item in which the first characteristic information of the customer to be tracked is registered. The registration processing unit 413 registers the first characteristic information extracted from the first captured image in the tracking target column. The first check column is an item for the tracking target customer to check that product registration has been completed. In this embodiment, since the first captured image is captured by the registration device 10, the registration processing unit 413 registers the first characteristic information in the tracking target column and registers check information indicating that product registration has been completed in the first check column. The second check column is a column for checking that the tracking target customer has completed payment at the payment device 20. The third check column is a column for checking that the tracking target customer is heading to the exit B11. Hereinafter, registering check information in the first check column, second check column, and third check column is also simply referred to as checking.

Returning to FIG. 6, the tracking processing unit 414 is a functional unit that tracks the movement path of customers in the accounting area A12 and the exit area A13 by comparing the captured images (characteristic information) of each area. Note that the tracking processing unit 414 uses known technologies such as face authentication and image recognition when comparing the captured images (characteristic information).

Specifically, the tracking processing unit 414 compares the second feature information extracted from the second captured image with each of the first feature information registered in the tracking table T1, and determines whether there is any first feature information whose similarity is equal to or greater than a threshold. If there is any first feature information whose similarity is equal to or greater than the threshold, the tracking processing unit 414 recognizes that the customer corresponding to the first feature information and the customer corresponding to the second feature information are the same person, and determines that the customer has completed the transaction at the accounting device 20. The tracking processing unit 414 then registers check information indicating that the transaction has been completed in the second check field of the data entry in which the first feature information is registered. Note that if there are multiple pieces of first feature information whose similarity is equal to or greater than the threshold, the tracking processing unit 414 selects the first feature information with the highest similarity.

The tracking processing unit 414 also compares the third feature information extracted from the third captured image with each of the first feature information registered in the tracking table T1, and determines whether or not there is first feature information whose similarity is equal to or greater than a threshold. If there is first feature information whose similarity is equal to or greater than a threshold, the tracking processing unit 414 recognizes that the customer corresponding to the first feature information and the customer corresponding to the third feature information are the same person, and determines that the customer is heading to exit B11. Then, the tracking processing unit 414 registers check information indicating that the customer is heading to exit B11 in the third check box of the data entry in which the first feature information is registered. Note that if there are multiple pieces of first feature information whose similarity is equal to or greater than the threshold, the tracking processing unit 414 selects the first feature information with the highest similarity, and checks the third check box of the data entry in which the first feature information is registered.

The fraud detection unit 415 monitors the tracking table T1 and detects customers who may be committing fraud. More specifically, the fraud detection unit 415 determines whether or not there is a possibility of fraud based on the state of the check information registered in the first check box, the second check box, and the third check box, that is, the customer's movement route.

For example, if the first and third check boxes have been checked but the second check box has not been checked, there is a possibility that a customer who has registered an item is attempting to take the item out of the store without paying for it. Therefore, if the state of the check information registered in the tracking table T1 indicates that the item has left the registration area A11, skipped the payment area A12, and reached the exit area A13, the fraud detection unit 415 detects the possibility of fraudulent activity.

Furthermore, if all three check boxes are checked, the fraud detection unit 415 determines that registration and transaction have been performed normally, and excludes the data entry from tracking. Here, "excluding from tracking" means adding flag information indicating that the data entry is not to be tracked, deleting the data entry itself from tracking table T1, or moving the data entry to another data table. Note that it is preferable for the fraud detection unit 415 to determine whether or not there is a possibility of fraudulent activity at the time check information is registered in the third check box.

The notification processor 416 is a functional unit that issues a notification according to the detection result of the fraud detection unit 415. More specifically, the notification processor 416 issues a notification to the store clerk on the condition that the fraud detection unit 415 detects the possibility of fraud. There is no particular restriction on the notification method, and various methods can be adopted. For example, the notification processor 416 may send notification information including a message notifying the occurrence of fraudulent activity and an image (characteristic information) of the relevant customer to each of the registration devices 10. In addition, if each store clerk has a portable communication device connected to the network N1, the notification processor 416 may send the notification information to the communication device. In addition, if a notification device such as an alarm buzzer or alarm light is provided in the exit area A13, etc., the notification may be issued by activating the notification device.

Next, the operation of the monitoring server 40 described above will be described with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the monitoring process executed by the monitoring server 40 of the first embodiment.

First, the image acquisition unit 411 acquires the captured images transmitted from each device (step S31). Next, the feature extraction unit 412 extracts feature information from the captured images acquired in step S31 (step S32).

On the condition that the first feature information is extracted from the first captured image in step S32 (step S33; Yes), the registration processing unit 413 registers the extracted first feature information in the tracking target column of the tracking table T1 (step S34). Next, the registration processing unit 413 checks the first check column of the data entry in which the first feature information is registered in step S34 (step S35), and returns the process to step S31.

In addition, the tracking processing unit 414 proceeds to step S37 on the condition that second feature information has been extracted from the second captured image in step S32 (step S33; No → step S36; Yes). The tracking processing unit 414 compares the extracted second feature information with each of the first feature information registered in the tracking table T1, and calculates the similarity between the two pieces of feature information (step S37). Next, the tracking processing unit 414 determines whether there is first feature information whose similarity with the second feature information exceeds a predetermined threshold, that is, whether there is a customer corresponding to the customer of the second feature information (step S38).

If there is no first characteristic information that exceeds the threshold, the tracking processing unit 414 determines that there is no match (step S38; No) and returns the process to step S31. On the other hand, if there is first characteristic information that exceeds the threshold, the tracking processing unit 414 determines that there is a match (step S38; Yes) and checks the second check box of the data entry in which the first characteristic information is registered (step S39). Then, the tracking processing unit 414 returns the process to step S31.

In addition, the tracking processing unit 414 proceeds to step S40 on the condition that the third feature information has been extracted from the third captured image in step S32 (step S33; No -> step S36; No). The tracking processing unit 414 compares the extracted third feature information with each of the first feature information registered in the tracking table T1, and calculates the similarity between the two pieces of feature information (step S40). Next, the tracking processing unit 414 determines whether there is first feature information whose similarity with the third feature information exceeds a predetermined threshold, that is, whether there is a customer corresponding to the customer of the third feature information (step S41).

If there is no first characteristic information that exceeds the threshold, the tracking processing unit 414 determines that there is no match (step S41; No) and returns the process to step S31. On the other hand, if there is first characteristic information that exceeds the threshold, the tracking processing unit 414 determines that there is a match (step S41; Yes) and checks the third check box of the data entry in which the first characteristic information is registered (step S42).

Then, the fraud detection unit 415 refers to the state (path of movement) of each check box of the data entry in which the third check box was checked in step S42, and determines whether or not there is a possibility of fraudulent activity (step S43). Here, if the first check box, the second check box, and the third check box are all checked, the fraud detection unit 415 determines that the checkout procedure for the product has been performed normally (step S43; No), and returns to step S31.

On the other hand, if the first and third check boxes have been checked and the second check box has not been checked, the fraud detection unit 415 determines that there is a possibility of fraud (step S43; Yes). The notification processing unit 416 then notifies the store clerk by sending notification information or the like (step S44), and the process returns to step S31.

As described above, the monitoring server 40 of this embodiment tracks the movement path of customers within the checkout area A1 based on captured images of the registration area A11, the checkout area A12, and the exit area A13. The monitoring server 40 then detects customers who may be committing fraudulent acts based on the tracked movement path, and notifies store staff. This allows the monitoring server 40 to efficiently detect fraudulent acts such as taking unchecked merchandise out of the store in a monitoring system based on matching captured images.

In this embodiment, the registration device 10 is configured to transmit the first captured image to the monitoring server 40 when the registration process is completed, but this is not limited, and the image may be transmitted during the registration process. In addition, the accounting device 20 is also configured to transmit the second captured image to the monitoring server 40 when the accounting process is completed, but this is not limited, and the image may be transmitted during the accounting process. In addition, when the accounting device 20 detects a face area of a person (customer) from the second captured image, it may be configured to assume that the customer has made a transaction and transmit the second captured image to the monitoring server 40. In addition, when these configurations are adopted, the criteria for determining whether or not a transaction has been made are looser than the configurations of the above-mentioned embodiments. For this reason, these configurations are preferably used in cases where the customer's movement route is simply tracked.

In addition, in this embodiment, the customer performs the accounting process using the accounting device 20 specified by the store clerk, but this is not limited to the above, and the accounting process may be performed using any accounting device 20. When this configuration is adopted, for example, the registration device 10, accounting device 20, and monitoring server 40 can be operated as follows to perform the accounting process using any accounting device 20.

First, when the control unit 11 of the registration device 10 receives an instruction to end the registration process, it sends to the monitoring server 40 a pair of the accounting information generated in this registration process and the first captured image captured during the registration process. In the monitoring server 40, the feature extraction unit 412 extracts the first feature information from the first captured image sent from the registration device 10. The registration processing unit 413 registers the extracted first feature information in the tracking table T1 and checks the first check box. The feature extraction unit 412 also registers (stores) the accounting information paired with the first captured image from which the first feature information was extracted in the tracking table T1, etc., in association with the registered first feature information.

Meanwhile, once a customer has completed registering the product at the registration device 10, they move to any accounting device 20. When the control unit 21 of the accounting device 20 receives an instruction from the customer to start the accounting process, it sends the second captured image taken by the second camera 32 to the monitoring server 40. In the monitoring server 40, the feature extraction unit 412 extracts second feature information from the second captured image sent from the accounting device 20. Next, based on the second feature information, the tracking processing unit 414 identifies the first feature information of the customer captured by the accounting device 20 (second camera 32) from the first feature information registered in the tracking table T1. Next, the control unit 41 of the monitoring server 40 sends accounting information associated with the identified first feature information to the accounting device 20 that sent the second captured image.

Next, when the control unit 21 of the accounting device 20 receives accounting information from the monitoring server 40, it executes accounting processing based on the accounting information. After the accounting processing is completed, the control unit 21 of the accounting device 20 transmits the second captured image captured during the accounting processing to the monitoring server 40. Subsequent processing is the same as in the embodiment described above.

Second Embodiment
Next, a second embodiment will be described. In the second embodiment, a store that employs a checkout system in which customers themselves register and pay will be described as an example. Note that elements similar to those in the first embodiment will be given the same reference numerals and descriptions thereof will be omitted.

Figure 9 is a diagram showing a schematic example of a store layout according to the second embodiment. As shown in Figure 9, the store has a checkout area A2 for registering and paying for products. The checkout area A2 includes an entrance area A21, a registration and payment area A22, and an exit area A23.

The entrance area A21 is a first area located at the entrance to the checkout area A2, and has an entrance B21 to the checkout area A2. The registration and accounting area A22 is a second area where customers register and pay for products, and has a registration and accounting device 50. The exit area A23 is a third area located at the exit of the checkout area A2, and has the store's exit B22. Although not shown, it is assumed that there is an area within the store where products for sale are displayed (for example, above the checkout area A2).

The registration and accounting device 50 is a product data processing device that is operated by a customer and performs registration and accounting processes for products purchased by the customer. In other words, the registration and accounting device 50 realizes a checkout method known as a self-service checkout method.

In a store with the layout shown in Figure 9, the actions taken when a customer purchases an item are as follows. First, the customer places the items they wish to purchase in a shopping cart or similar and moves from the entrance area A21 (entrance B21) to the registration accounting area A22 (registration accounting device 50). At the registration accounting device 50, the customer registers the item by having a reading unit read the barcode attached to the item. Then, once the registration of the item is complete, the customer pays at the registration accounting device 50. Then, once the payment is complete, the customer places the purchased items in a shopping bag or similar and moves to the exit area A23 (exit B22).

In this way, with a self-service system, customers themselves register and pay, which can improve processing efficiency, such as shortening waiting times at the register. However, with a self-service system, because registration and payment are left to the customers themselves, there is a risk of fraudulent behavior, such as cart skipping, just like with a semi-self-service system.

Therefore, in the surveillance system of this embodiment, images of the entrance area A21, the registration and accounting area A22, and the exit area A23 are captured, and customers who may be engaging in fraudulent activities are detected based on the captured images. The configuration of the surveillance system according to this embodiment is described below.

Figure 10 is a diagram showing a schematic example of the configuration of a monitoring system according to the second embodiment. As shown in Figure 10, the monitoring system includes a first camera 61, a second camera 62, a third camera 63, and a monitoring server 40a, in addition to the above-mentioned registered accounting device 50. The registered accounting device 50, the first camera 61, the third camera 63, and the monitoring server 40a are connected to a network N2 such as a LAN.

The first camera 61 is an imaging device that captures images of customers present in the entrance area A21. The first camera 61 is installed in a position that allows it to capture an image of the face of a customer entering the checkout area A2 from the entrance B21. The first camera 61 transmits the captured image (hereinafter referred to as the first captured image) to the surveillance server 40a. There is no particular restriction on the installation location of the first camera 61; for example, it may be installed at the gate of the entrance B21, or it may be installed on the ceiling of the store in the form of a surveillance camera or the like.

The second camera 62 is a camera that captures images of customers present in the registered accounting area A22. The second camera 62 is provided in each registered accounting device 50. More preferably, the second camera 62 is provided in a position in each registered accounting device 50 where it is possible to capture an image of the customer's face. Each pair of a registered accounting device 50 and a second camera 62 is connected by a connection line L3. The installation location of the second camera 62 is not particularly important; for example, it may be provided integrally with the registered accounting device 50, or it may be provided on the ceiling of the store in the form of a surveillance camera or the like.

The registration accounting device 50 cooperates with the second camera 62 to transmit the captured image (hereinafter referred to as the second captured image) captured during the accounting process to the monitoring server 40a. Note that the configuration of the registration accounting device 50 is similar to that of the combination of the registration device 10 and accounting device 20 described above, and therefore a detailed description is omitted.

The registration accounting device 50 acquires the second captured image captured by the second camera 62 while the accounting process is being performed. Then, when the accounting process is completed, the registration accounting device 50 transmits the second captured image acquired during the accounting process to the monitoring server 40a. Here, the number of second captured images (still images) and the number of times (video) to be transmitted to the monitoring server 40a are not particularly important. The registration accounting device 50 may acquire the second captured image while the registration process is being performed, and transmit the acquired second captured image to the monitoring server 40a at the timing when the registration process or the accounting process is completed. In addition, the registration accounting device 50 (control unit) may select the second captured image to be transmitted to the monitoring server 40a based on the area and appearance frequency of the face area by performing a transmission image selection process similar to that of the registration device 10 described above.

The third camera 63 is an imaging device that captures images of customers present in the exit area A23. The third camera 63 is installed in a position that allows it to capture images of the faces of customers passing through the exit B22. The third camera 63 transmits the captured image (hereinafter referred to as the third captured image) to the surveillance server 40a. There is no particular restriction on the location where the third camera 63 is installed, and it may be installed at the gate of the exit B22, or may be installed on the ceiling of the store in the form of a surveillance camera or the like.

The monitoring server 40a is a server device that monitors the behavior of customers in the checkout area A2 based on images captured by the first camera 61, the second camera 62, and the third camera 63. The monitoring server 40a will be described below. Note that the same reference numerals will be used to designate components similar to those in the first embodiment, and descriptions thereof will be omitted.

Fig. 11 is a diagram showing an example of the configuration of a monitoring server 40a according to the second embodiment. As shown in Fig. 11, the monitoring server 40a includes a control unit 41a. The control unit 41a has a computer configuration including a CPU, SoC, ROM, RAM, etc.

The control unit 41a is connected to a memory unit 42 and a communication interface (I/F) 43 via a bus or the like. The memory unit 42 stores a tracing table T1 (see FIG. 7) having the same data structure as in the first embodiment.

As shown in FIG. 11, the control unit 41a includes functional units such as an image acquisition unit 411a, a feature extraction unit 412, a registration processing unit 413, a tracking processing unit 414, a fraud detection unit 415, and a notification processing unit 416. These functional units are realized in software by executing a program stored in the storage unit 42, or in hardware by a dedicated processor or the like included in the control unit 41a.

The image acquisition unit 411a acquires the captured images captured by each of the first camera 61, the second camera 62, and the third camera 63 via the communication interface 43. More specifically, the image acquisition unit 411a acquires the first captured image transmitted from the first camera 61. The image acquisition unit 411a also acquires the second captured image transmitted from each of the registered accounting devices 50. The image acquisition unit 411a also acquires the third captured image transmitted from the third camera 63.

The control unit 41a executes the monitoring process described in FIG. 8 by working together with each of the functional units described above. As a result, the monitoring server 40a of this embodiment tracks the movement path of customers within the checkout area A2 based on captured images of each of the entrance area A21, the registration checkout area A22, and the exit area A23. The monitoring server 40a then detects customers who may be committing fraudulent acts based on the tracked movement path, and notifies store staff. As a result, the monitoring server 40a can efficiently detect fraudulent acts such as taking unchecked merchandise out of the store in a monitoring system based on comparing captured images.

Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

For example, in the above embodiment, the monitoring server 40 (40a) is configured to extract feature information from the captured image, but the present invention is not limited to this, and the feature information may be extracted by the device that transmits the captured image. For example, any or all of the registration device 10, the accounting device 20, and the registration accounting device 50 may be equipped with a feature extraction unit 412 and transmit the feature information extracted from the captured image to the monitoring server 40 (40a). Also, each camera (e.g., the third camera 33, the first camera 61, the third camera 63, etc.) may be equipped with a feature extraction unit 412 and transmit the feature information extracted from the captured image to the monitoring server 40 (40a). In this case, in the monitoring server 40 (40a), the registration processing unit 413 registers the first feature information transmitted from the registration device 10 or the first camera 61 in the tracking table T1. Also, the tracking processing unit 414 tracks the movement route of the same customer based on the similarity between each of the first feature information registered in the tracking table T1 and the second feature information and the third feature information transmitted from the other devices.

In the above embodiment, the monitoring server 40 (registration processing unit 413) is configured to register the first feature information extracted from the first captured image in the tracking table T1, but this is not limiting, and the first captured image may be registered in the tracking table T1. In this case, the feature extraction unit 412 cooperates with the tracking processing unit 414 to extract feature information from each captured image when comparing the similarity between the captured images.

In addition, in the above embodiment, the monitoring server 40 (40a) is configured to monitor for fraudulent activity, but this is not limited to the above, and another device may be configured to monitor for fraudulent activity. For example, one representative device among the registration device 10 and the accounting device 20 (or the registration accounting device 50) may be equipped with the functionality of the monitoring server 40 (40a) and may be configured to monitor for fraudulent activity.

In the above embodiment, the registration device 10, accounting device 20, and registration accounting device 50 are configured to transmit captured images, but this is not limited to the above, and each camera may transmit directly. Also, the first camera 31 and the third camera 33, 36 are configured to transmit captured images directly, but they may also be configured to transmit from an information processing device such as a PC (Personal Computer) by cooperating with the information processing device. Also, when transmitting from an information processing device, the captured image to be transmitted may be selected by performing a process similar to the transmission image selection process.

10 Registration device 20 Accounting device 31, 61 First camera 32, 62 Second camera 33, 63 Third camera 40, 40a Surveillance server 50 Registration accounting device 411, 411a Image acquisition unit 412 Feature extraction unit 413 Registration processing unit 414 Tracking processing unit 415 Fraud detection unit T1 Tracking table

JP 2013-186495 A

Claims (6)

a first imaging means for imaging a customer in a first area where a product is to be registered;
a second imaging means for imaging a customer in a second area who is making a payment for the registered product;
a third imaging means for imaging a third area through which customers who have completed their transaction can pass;
a first selection means for identifying a customer who appears most frequently from among the captured images for each transaction captured by the first imaging means, and selecting an image in which the area of the face region of the identified customer is the largest;
a second selection means for identifying the customer who appears most frequently from among the captured images for each transaction captured by the second imaging means, and selecting the captured image in which the area of the face region of the identified customer is the largest;
a tracking means for tracking a movement path of a same customer from the first area to the third area based on a similarity between feature information extracted from the captured images selected by the first selection means and the second selection means and feature information extracted from the captured image of the third imaging means; and
a notification means for notifying a user when the movement path indicates that the user has traveled from the first area to the third area by skipping the second area;
A surveillance system having: The surveillance system according to claim 1, wherein the third imaging means images the third area located at the exit of an area in which the first area and the second area are provided. The surveillance system according to claim 1 or 2, wherein the first imaging means images a customer who uses a registration device provided in the first area. 4. The monitoring system according to claim 3, wherein the first selection means, when a process relating to the registration of a commodity for one transaction is completed in the registration device, selects from among the images captured during the process. The surveillance system according to any one of claims 1 to 4, wherein the second imaging means images a customer using an accounting device provided in the second area. The monitoring system according to claim 5, wherein the second imaging means outputs an image captured during a transaction-related process after the transaction-related process is performed at the accounting device.

Images