JP2023162229A - Monitoring device and program - Google Patents

Abstract

To provide a monitoring device capable of efficiently monitoring fraudulent acts of taking out unaccounted commodities from a shop where a system of causing customers themselves to perform accounting is applied, and a program.SOLUTION: The monitoring device includes: imaging means for imaging each of a first area positioned at an entrance, a second area where customers themselves perform accounting of commodities, and a third area positioned at an exit in a checkout area related to registration and accounting of commodities; extraction means for extracting feature information expressing features of a customer from an image captured at each of the areas; tracing means for tracing a moving route through which the same customer reaches the third area from the first area on the basis of a similarity between feature information extracted from the captured image of the first area and feature information extracted from the captured image of each of the second and third areas; and notification means for making a notification when the moving route has reached the third area from the first area, skipping the second area.SELECTED DRAWING: Figure 2

Description

Embodiments of the present invention relate to a monitoring device and a program.

2. Description of the Related Art Conventionally, in stores such as supermarkets, store clerks register products and make payments by operating POS (Point Of Sale) terminals. In addition, in stores, there are also checkout methods called semi-self, where a store clerk registers the products and customers themselves pay for the products, and self-checkout, where the customers register and pay for the products themselves. ing.

By the way, in the above-mentioned semi-self-service and self-service stores, the customer is responsible for paying for the products, so there is a possibility that fraudulent acts such as "cart skipping", in which the customer leaves the store without paying, may occur. Conventionally, in semi-self-service shops, a first imaging device takes an image of a customer who has requested product registration, and a second imaging device takes an image of the vicinity of the exit, etc., and the images of the customer taken by these imaging devices are used. There is a known technique for determining whether or not a customer who has not yet paid is present near the exit by performing a check.

However, in the above-mentioned conventional technology, whether or not the transaction is completed is determined based on the notification of completion sent from the transaction device, without using the captured image. Therefore, in a monitoring system based on collation of captured images, the above-mentioned conventional techniques cannot be said to be efficient and there is room for further improvement.

The problem to be solved by the present invention is to provide a monitoring device and a program that can efficiently monitor fraud in which unchecked products are taken out of the store in a store where customers themselves pay the checkout. That's true.

The monitoring device of the embodiment includes an imaging means, an extraction means, a tracking means, and a notification means. The imaging means is configured to capture a first area located at the entrance, a second area where customers pay for their products, and a third area located at the exit in the checkout area related to product registration and accounting. Take images of each. The extracting means extracts feature information representing the customer's characteristics from the captured images captured in each of the regions. The tracking means identifies the same customer based on the similarity between the feature information extracted from the captured image of the first area and the feature information extracted from the captured images of each of the second area and the third area. A moving route from the first area to the third area is tracked. The notification means performs notification when it is indicated that the moving route skips the second area from the first area and reaches the third area.

FIG. 1 is a diagram schematically showing an example of a store layout according to the first embodiment. FIG. 2 is a diagram schematically showing an example of the configuration of the monitoring system according to the first embodiment. FIG. 3 is a diagram showing an example of the configuration of the registration device according to the first embodiment. FIG. 4 is a flowchart illustrating an example of a transmission image selection process executed by the registration device of the first embodiment. FIG. 5 is a diagram showing an example of the configuration of the accounting device according to the first embodiment. FIG. 6 is a diagram showing an example of the configuration of the monitoring server according to the first embodiment. FIG. 7 is a diagram showing an example of the data structure of the tracking table. FIG. 8 is a flowchart illustrating an example of a monitoring process executed by the monitoring server of the first embodiment. FIG. 9 is a diagram schematically showing an example of a store layout according to the second embodiment. FIG. 10 is a diagram schematically showing an example of the configuration of a monitoring system according to the second embodiment. FIG. 11 is a diagram illustrating an example of the configuration of a monitoring server according to the second embodiment.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of a monitoring device and a program according to the present invention will be described in detail below with reference to the drawings. In the embodiment shown below, an example in which the present invention is applied to a store such as a supermarket will be described, but the present invention is not limited to this embodiment.

[First embodiment]
FIG. 1 is a diagram schematically showing an example of a store layout according to the first embodiment. As shown in FIG. 1, the store has a checkout area A1 for product registration and checkout. Checkout area A1 includes a registration area A11, a checkout area A12, and an exit area A13.

The registration area A11 is a first area located at the entrance of the checkout area A1, and the registration device 10 is provided therein. The checkout area A12 is a second area where customers pay for products, and is provided with a checkout device 20. The exit area A13 is a third area located at the exit of the checkout area A1, and is provided with an exit B11 of the store. Although not shown, it is assumed that there is an area within the store where products to be sold are displayed (for example, above the checkout area A1).

The registration device 10 is a product data processing device that is operated by a store clerk and performs registration processing for products purchased by customers. The accounting device 20 is a product data processing device that is operated by a customer and performs accounting processing for products registered in the registration device 10. In other words, the registration device 10 and the accounting device 20 implement a checkout method called semi-self checkout.

In the store with the layout shown in FIG. 1, the operations when a customer purchases a product are as follows. First, a customer places the products to be purchased in a shopping cart or the like and carries them to the registration area A11 (registration device 10). In the registration device 10, a store clerk registers a product by having a reading unit 14 (see FIG. 3), which will be described later, read a barcode attached to the product. When the registration of all products related to one transaction is completed, the clerk selects (designates) one of the checkout devices 20 to be the destination of payment, and guides the customer to the checkout destination by notifying the customer of the checkout device 20. Further, the registration device 10 transmits accounting information including information such as the unit price and quantity of each registered product to the designated accounting device 20.

When the customer completes product registration, the customer moves to the checkout area A12 with the goods (shopping cart) and pays the checkout using the checkout device 20 specified by the store clerk. At this time, the accounting device 20 executes accounting processing based on accounting information sent in advance from the registration device 10. When the customer completes the transaction, the customer places the purchased product in a plastic bag or the like and moves to the exit area A13 (exit B11).

In this way, in the semi-self system, registration and accounting can be shared, so it is possible to improve processing efficiency, such as shortening the waiting time at the register. However, in the semi-self-service system, since the checkout is left to the customer himself/herself, there is a possibility that fraudulent acts such as cart removal, in which items are taken out of the store without being checked out, may occur.

Therefore, in the monitoring system of this embodiment, images are taken of each of the registration area A11, the checkout area A12, and the exit area A13, and based on the captured images, customers who may be committing fraudulent acts are detected. The configuration of the monitoring system according to this embodiment will be described below.

FIG. 2 is a diagram schematically showing an example of the configuration of the monitoring system according to the first embodiment. As shown in the figure, the monitoring system includes a first camera 31, a second camera 32, a third camera 33, and a monitoring server 40 in addition to the registration device 10 and accounting device 20 described above. The registration device 10, the accounting device 20, the third camera 33, and the monitoring server 40 are connected to a network N1 such as a LAN (Local Area Network).

The first camera 31 is an image capturing device that captures images of customers present in the registration area A11 (registration device 10). The first camera 31 is provided in each of the registration devices 10 (see FIG. 1). More preferably, the first camera 31 is provided in each registration device 10 at a position where it can image the customer's face. The registration device 10 and the first camera 31 are connected by a connection line L1. Note that the installation position of the first camera 31 is not particularly limited, and it may be installed integrally with the registration device 10 or may be installed on the ceiling of a store in the form of a surveillance camera or the like.

The registration device 10 cooperates with the first camera 31 and transmits a captured image (hereinafter referred to as a first captured image) captured during the registration process to the monitoring server 40 . Note that the captured image is a concept that includes still images and moving images.

FIG. 3 is a diagram showing an example of the configuration of the registration device 10 according to the first embodiment. As shown in FIG. 3, the registration device 10 includes a control section 11. The control unit 11 has a computer configuration such as a CPU (Central Processing Unit), an SoC (System-on-a-Chip), a ROM (Read Only Memory), and a RAM (Random Access Memory).

A display section 12, an operation section 13, a reading section 14, a storage section 15, a connection interface 16, a communication interface 17, etc. are connected to the control section 11 via a bus or the like.

The display unit 12 includes a display device such as a liquid crystal display. The display unit 12 displays various information such as the product name and product code of registered products under the control of the control unit 11. The operation unit 13 has various operation keys and pointing devices, and receives operations from an operator (store clerk). For example, the operation unit 13 includes operation keys for instructing the start and completion of registration processing, numeric keys for inputting a product code, the number of products, and the like. Note that the operation unit 13 may be a touch panel provided on the display surface of the display unit 12.

The reading unit 14 is a code reader that can read code symbols such as barcodes and two-dimensional codes. For example, the reading unit 14 reads the product code held in the code symbol from the code symbol attached to the product. Here, the product code is identification information for identifying a product, and unique identification information is given to each type of product.

The storage unit 15 is a storage device such as an HDD (Hard Disk Drive) or a flash memory. The storage unit 15 stores programs executed by the control unit 11 and various data used to execute the programs. For example, the storage unit 15 stores in advance a product master (not shown) in which the product code of each product is associated with product information including the product name, unit price, etc. of the product. Further, in order to identify the faces of the store staff and the faces of the customers, the storage unit 15 may store in advance captured images of each store staff's face and feature information representing facial features extracted from the captured images. good.

The connection interface 16 is an interface that can be connected to the first camera 31. The connection interface 16 receives a first captured image input from the first camera 31. The communication interface 17 is a communication interface connectable to the network N1. The communication interface 17 transmits and receives various information to and from external devices (for example, the accounting device 20 and the monitoring server 40) connected to the network N1.

In the registration device 10 having the above configuration, when a code symbol attached to a product to be purchased is read by a store clerk's operation, the control unit 11 stores the product code read by the reading unit 14 in a RAM or the like. Register your product with . Further, when the number of products (number of products) is input, the control unit 11 registers the number of products in association with the product code. Further, the control unit 11 acquires a first captured image captured by the first camera 31 while the product registration process is being performed.

When the registration of products for one transaction is completed, the control unit 11 generates accounting information based on the product codes and points registered so far. Here, the accounting information includes, for example, the product name and price of each product (product code), the total payment amount, and the like. Note that the product name and price are derived based on product information registered in the product master.

Further, when a specific checkout device 20 is selected (designated) by a store clerk via the operation unit 13 or the like, the control unit 11 transmits product information to the checkout device 20. Then, the control unit 11 transmits the first captured image acquired during the registration process to the monitoring server 40. Note that the number of first captured images (still images) and the number of hours (videos) to be transmitted to the monitoring server 40 are not particularly limited. Further, the selection of the accounting device 20 may be configured such that the control unit 11 automatically selects the accounting device 20 based on the availability of the accounting device 20 or the like.

By the way, the first captured image becomes a reference image when identifying (recognizing) the same customer in the monitoring process described later. Therefore, it is preferable that the control unit 11 transmits to the monitoring server 40 a first captured image in which a characteristic part such as the customer's face is captured. Therefore, for example, the control unit 11 selects the first captured image to be transmitted to the monitoring server 40 based on the face area, the frequency of appearances of the same person's face, etc. from the first captured images captured during the registration process. It is also possible to have a configuration in which Processing related to selection of the first captured image will be described below.

FIG. 4 is a flowchart illustrating an example of a transmission image selection process executed by the registration device 10 of the first embodiment. Note that this process is executed in the background of the registration process. Furthermore, in this process, known techniques are used for face area detection, face recognition, and the like.

First, when the control unit 11 receives an instruction to start registration via the operation unit 13 or the like (step S11), the control unit 11 causes the first camera 31 to start capturing an image (step S12).

Subsequently, the control unit 11 determines whether a face area has been detected from the first captured image input from the first camera 31 (step S13). If the face area cannot be detected (step S13; No), the control unit 11 moves to step S17. Further, when the control unit 11 detects a face area from the first captured image (step S13; Yes), the control unit 11 compares the characteristics of the face area with the characteristics of the face area of each store clerk stored in the storage unit 15. Then, it is determined whether the photographed person is a store clerk (step S14).

If the photographed person is a store clerk (step S14; Yes), the control unit 11 returns the process to step S13. Further, if the person imaged is other than a store clerk, that is, a customer (step S14; No), the control unit 11 derives the area of the face area (step S15), and arranges the area in the first image capturing order in descending order (or ascending order). The image is temporarily stored in a RAM or the like (step S16).

Subsequently, the control unit 11 determines whether or not completion of registration has been instructed via the operation unit 13 or the like (step S17). If there is no instruction to complete the registration (step S17; No), the control unit 11 returns the process to step S13. As a result, while the product is being registered, the first captured images including the customer's face area captured by the first camera 31 are temporarily stored in the order of the area of the face area.

When the control unit 11 is instructed to complete the registration (step S17; Yes), the control unit 11 identifies the same person (customer) by comparing the characteristics of the face area between the temporarily stored first captured images. . Next, the control unit 11 identifies the face of the customer that appears most frequently from among the identified customers (step S18). For example, the first captured image captured by the first camera 31 may include customers other than the customer involved in the transaction (for example, customers waiting in line). Therefore, in step S18, the control unit 11 recognizes the same customer included in each of the first captured images, and specifies the customer who appears the most as the customer involved in the transaction.

Subsequently, the control unit 11 selects the first captured image with the largest area of the face area from among the first captured images that include the customer's face area identified in step S18 (step S19). Then, the control unit 11 transmits the first captured image selected in step S19 to the monitoring server 40 (step S20), and ends this process.

Through the above processing, the registration device 10 can transmit the first captured image showing the customer's features (face) to the monitoring server 40. Thereby, customer recognition accuracy can be improved in the monitoring process described later.

Returning to FIG. 2, the second camera 32 is an image capturing device that captures images of customers present in the checkout area A12. The second camera 32 is provided for each accounting device 20. More preferably, the second camera 32 is provided in each of the checkout devices 20 at a position where it can capture an image of the customer's face. The accounting device 20 and the second camera 32 are connected by a connection line L2. Note that the installation position of the second camera 32 is not particularly limited; it may be provided integrally with the checkout device 20, or may be provided in the form of a surveillance camera or the like on the ceiling of the store.

The accounting device 20 cooperates with the second camera 32 and transmits a captured image (hereinafter referred to as a second captured image) captured during the accounting process to the monitoring server 40 .

FIG. 5 is a diagram showing an example of the configuration of the accounting device 20 according to the first embodiment. As shown in FIG. 5, the accounting device 20 includes a control section 21. As shown in FIG. The control unit 21 has a computer configuration such as a CPU, SoC, ROM, and RAM.

A display section 22, an operation section 23, a printing section 24, a storage section 25, a connection interface 26, a communication interface 27, and the like are connected to the control section 21 via a bus or the like.

The display section 22 includes a display device such as a liquid crystal display. The display unit 22 displays various information such as accounting information under the control of the control unit 21. The operation unit 23 has various operation keys and pointing devices, and receives operations from an operator (customer). For example, the operation unit 23 includes operation keys for instructing the start and completion of accounting processing. Note that the operation unit 23 may be a touch panel provided on the display surface of the display unit 22.

The printing unit 24 is a printing device such as a thermal printer. The printing unit 24 prints the details of the accounting (accounting information) on a paper medium such as a receipt paper under the control of the control unit 21 . The storage unit 25 is a storage device such as an HDD or a flash memory. The storage unit 25 stores programs executed by the control unit 21 and various data used to execute the programs.

The connection interface 26 is an interface that can be connected to the second camera 32. The connection interface 26 receives the second captured image input from the second camera 32. The communication interface 27 is a communication interface connectable to the network N1. The communication interface 27 transmits and receives various information to and from external devices (eg, the registration device 10 and the monitoring server 40) connected to the network N1.

Further, the control unit 21 is connected, via a bus or the like, to a money depositing device that accepts deposits of coins and banknotes, a money dispensing device that dispenses change, etc. (none of which are shown).

In the accounting device 20 configured as described above, the control unit 21 temporarily stores the accounting information transmitted from the registration device 10 in the RAM or the like, and waits for the start of accounting processing related to the accounting information. When the customer moves to the checkout device 20 designated by the clerk of the registration device 10, the customer instructs the start of checkout processing via the operation section 23. When the control unit 21 is instructed to start accounting processing, it accepts payment (deposit) of the product price based on the temporarily stored accounting information. When the payment is completed, the control section 21 causes the printing section 24 to output a receipt paper with the details printed thereon, and ends the accounting process.

Further, the control unit 21 acquires a second captured image captured by the second camera 32 while the accounting process is being performed. Then, when the accounting process is completed, the control unit 21 transmits the second captured image acquired during the accounting process to the monitoring server 40. Here, the number of second captured images (still images) and the number of hours (videos) to be transmitted to the monitoring server 40 are not particularly limited. Note that the control unit 21 selects the second captured image to be transmitted to the monitoring server 40 based on the area of the face area, frequency of appearance, etc. by executing the same transmission image selection process as that of the registration device 10 described above. You can.

Returning to FIG. 2, the third camera 33 is an image capturing device that captures images of customers present in the exit area A13. The third camera 33 is provided at a position where it can capture images of the faces of customers passing through the exit B11. The third camera 33 transmits the captured image (hereinafter referred to as the third captured image) to the monitoring server 40. Note that the installation position of the third camera 33 is not particularly limited, and it may be installed on the ceiling of a store or the like in the form of a surveillance camera or the like.

The monitoring server 40 is a server device that monitors the behavior of customers within the checkout area A1 based on captured images captured by the first camera 31, the second camera 32, and the third camera 33. The monitoring server 40 will be explained below.

FIG. 6 is a diagram showing an example of the configuration of the monitoring server 40 according to the first embodiment. As shown in FIG. 6, the monitoring server 40 includes a control unit 41. The control unit 41 has a computer configuration such as a CPU, SoC, ROM, and RAM.

A storage unit 42 is connected to the control unit 41 via a bus or the like. The storage unit 42 is a storage device such as an HDD or a flash memory. The storage unit 42 stores programs executed by the control unit 41 and various data used to execute the programs. The storage unit 42 also stores a tracking table T1 (see FIG. 7) for tracking the travel route of the customer in the checkout area A1.

Further, a communication interface (I/F) 43 is connected to the control unit 41 via a bus or the like. The communication interface 43 is connected to the network N1 and exchanges various information with other devices (registration device 10, accounting device 20, third camera 33, etc.).

Further, as shown in FIG. 6, the control unit 41 includes functional units such as an image acquisition unit 411, a feature extraction unit 412, a registration processing unit 413, a tracking processing unit 414, a fraud detection unit 415, and a notification processing unit 416. These functional units are realized in software by executing a program stored in the storage unit 42, or in hardware by a dedicated processor included in the control unit 41.

The image acquisition unit 411 acquires images captured by each of the first camera 31 , the second camera 32 , and the third camera 33 via the communication interface 43 . More specifically, the image acquisition unit 411 acquires the first captured image transmitted from each of the registration devices 10. Further, the image acquisition unit 411 acquires the second captured image transmitted from each of the accounting devices 20. The image acquisition unit 411 also acquires a third captured image transmitted from the third camera 33.

The feature extraction unit 412 represents the characteristics of the person (customer) included in each of the captured images (the first captured image, the second captured image, and the third captured image) acquired by the image acquisition unit 411. Extract feature information. Here, the feature information is, for example, a feature amount representing a feature of a face area included in a captured image. Furthermore, the face area may not be detected because the person in the captured image is wearing sunglasses, a mask, or the like. In such a case, the feature extraction unit 412 extracts the feature of the element from other elements such as the person's clothes, hairstyle, body shape, etc. as feature information. Hereinafter, the feature information extracted from each of the first captured image, the second captured image, and the third captured image will be referred to as first feature information, second feature information, and third feature information, respectively. Note that the method for extracting the feature information is not particularly limited, and known techniques such as face recognition and image recognition may be used.

The registration processing unit 413 is a functional unit that sets customers to be tracked. More specifically, the registration processing unit 413 registers the first feature information extracted from the first captured image by the feature extraction unit 412 in the tracking table T1 of the storage unit 42.

Here, FIG. 7 is a diagram showing an example of the data structure of the tracking table T1. As shown in FIG. 7, the tracking table T1 has a tracking target column, a first check column, a second check column, and a third check column as items. Each of these items is prepared for each customer and constitutes one data entry.

The tracked person column is an item in which first characteristic information of a customer to be tracked is registered. The registration processing unit 413 registers the first feature information extracted from the first captured image in the tracking target person column. Further, the first check column is an item for checking that the customer to be tracked has completed registration of the product. In this embodiment, since the first captured image is captured by the registration device 10, the registration processing unit 413 registers the first characteristic information in the tracked person column and registers the product in the first check column. Register check information indicating that. Further, the second check column is a column for checking that the customer to be tracked has completed the transaction using the transaction device 20. Further, the third check column is a column for checking that the customer to be tracked is heading toward exit B11. Hereinafter, registering check information in the first check column, second check column, and third check column will also be simply referred to as check.

Returning to FIG. 6, the tracking processing unit 414 is a functional unit that tracks the travel route of the customer in the checkout area A12 and the exit area A13 by comparing captured images (feature information) of each area. Note that the tracking processing unit 414 uses known techniques such as face authentication and image recognition in comparing captured images (feature information).

Specifically, the tracking processing unit 414 compares the second feature information extracted from the second captured image with each of the first feature information registered in the tracking table T1, and determines that the degree of similarity is equal to or higher than the threshold. It is determined whether or not the first feature information exists. If there is first feature information with a degree of similarity equal to or higher than the threshold, the tracking processing unit 414 recognizes that the customer corresponding to the first feature information and the customer corresponding to the second feature information are the same person, and It is determined that the user has completed the transaction using the accounting device 20. Then, the tracking processing unit 414 registers check information indicating that the transaction has been completed in the second check column of the data entry in which the first characteristic information is registered. Note that if there is a plurality of pieces of first feature information with a degree of similarity equal to or higher than the threshold value, the tracking processing unit 414 selects the piece of first feature information with the highest degree of similarity.

The tracking processing unit 414 also compares the third feature information extracted from the third captured image with each of the first feature information registered in the tracking table T1, and the tracking processing unit 414 matches the third feature information extracted from the third captured image with each of the first feature information registered in the tracking table T1, and the tracking processing unit 414 matches the third feature information extracted from the third captured image with each of the first feature information registered in the tracking table T1, and Determine whether feature information exists. If there is first feature information with a degree of similarity equal to or higher than the threshold, the tracking processing unit 414 recognizes that the customer corresponding to the first feature information and the customer corresponding to the third feature information are the same person, and It is determined that the vehicle is heading toward exit B11. Then, the tracking processing unit 414 registers check information indicating that the vehicle is heading for exit B11 in the third check column of the data entry in which the first characteristic information is registered. If there are multiple first feature information with a threshold or higher, the tracking processing unit 414 selects the first feature information with the highest similarity, and the first feature information is registered. Check the third check box.

The fraud detection unit 415 monitors the tracking table T1 and detects customers who may be committing fraud. More specifically, the fraud detection unit 415 detects fraud based on the state of the check information registered in the first check field, the second check field, and the third check field, that is, the travel route of the customer. Determine whether there is a possibility.

For example, if the first check column and the third check column are checked, but the second check column is unchecked, a customer who has registered the product may attempt to take the product out of the store without paying the checkout. There is a possibility that there are. Therefore, if the state of the check information registered in the tracking table T1 indicates that the check information has skipped the accounting area A12 from the registration area A11 and reached the exit area A13, the fraud detection unit 415 detects the possibility of fraud. do.

Furthermore, if all three check boxes have been checked, the fraud detection unit 415 determines that registration and accounting have been performed normally, and excludes the data entry from the tracking target. Here, "excluding from tracking targets" means adding flag information indicating non-tracking targets, deleting the data entry itself from the tracking table T1, or moving the data entry to another data table. It means that. Note that it is preferable that the fraud detection unit 415 determines whether there is a possibility of fraud at the timing when check information is registered in the third check column.

The notification processing unit 416 is a functional unit that performs notification according to the detection result of the fraud detection unit 415. More specifically, the notification processing unit 416 notifies the store clerk on the condition that the fraud detection unit 415 detects that there is a possibility of fraudulent activity. The notification method is not particularly limited, and various methods can be employed. For example, the notification processing unit 416 may transmit to each of the registration devices 10 notification information including a message notifying the occurrence of a fraudulent act, a captured image (characteristic information) of the customer in question, and the like. Further, if each store clerk has a portable communication device connected to the network N1, the notification processing unit 416 may transmit notification information to the communication device. Further, if a notification device such as a warning buzzer or a warning light is provided in the exit area A13 or the like, notification may be made by operating the notification device.

Next, with reference to FIG. 8, the operation of the above-mentioned monitoring server 40 will be explained. FIG. 8 is a flowchart illustrating an example of a monitoring process executed by the monitoring server 40 of the first embodiment.

First, the image acquisition unit 411 acquires captured images transmitted from each device (step S31). Next, the feature extraction unit 412 extracts feature information from the captured image acquired in step S31 (step S32).

On the condition that the first feature information has been extracted from the first captured image in step S32 (step S33; Yes), the registration processing unit 413 uses the extracted first feature information as the tracking target person in the tracking table T1. column (step S34). Next, the registration processing unit 413 checks the first check column of the data entry in which the first feature information was registered in step S34 (step S35), and returns the process to step S31.

Further, the tracking processing unit 414 moves to step S37 on the condition that the second feature information is extracted from the second captured image in step S32 (step S33; No → step S36; Yes). The tracking processing unit 414 compares the extracted second feature information with each piece of first feature information registered in the tracking table T1, and calculates the degree of similarity between both pieces of feature information (step S37). Next, the tracking processing unit 414 determines whether there is first feature information whose degree of similarity with the second feature information exceeds a predetermined threshold, that is, whether there is a customer corresponding to the customer of the second feature information. (Step S38).

If there is no first feature information exceeding the threshold, the tracking processing unit 414 determines that there is no matching person (step S38; No), and returns the process to step S31. Further, if there is first feature information exceeding the threshold, the tracking processing unit 414 determines that there is a corresponding person (step S38; Yes), and checks the second check column of the data entry in which the first feature information is registered. Check (step S39). The tracking processing unit 414 then returns the process to step S31.

Further, the tracking processing unit 414 moves to step S40 on the condition that the third feature information has been extracted from the third captured image in step S32 (step S33; No → step S36; No). The tracking processing unit 414 compares the extracted third feature information with each piece of first feature information registered in the tracking table T1, and calculates the degree of similarity between both pieces of feature information (step S40). Next, the tracking processing unit 414 determines whether there is first feature information whose degree of similarity with the third feature information exceeds a predetermined threshold, that is, whether there is a customer corresponding to the customer of the third feature information. (Step S41).

If there is no first feature information exceeding the threshold, the tracking processing unit 414 determines that there is no matching person (step S41; No), and returns the process to step S31. Further, if there is first feature information exceeding the threshold, the tracking processing unit 414 determines that there is a corresponding person (step S41; Yes), and checks the third check column of the data entry in which the first feature information is registered. Check (step S42).

Subsequently, the fraud detection unit 415 refers to the status of each check column (travel route) of the data entry whose third check column was checked in step S42, and determines whether there is a possibility of fraud ( Step S43). Here, if the first check column, the second check column, and the third check column are all checked, the fraud detection unit 415 determines that the product accounting procedure has been performed normally (step S43; No), Return to step S31.

On the other hand, if the first check column and the third check column are checked and the second check column is unchecked, the fraud detection unit 415 determines that there is a possibility of fraud (step S43; Yes). Then, the notification processing unit 416 notifies the store staff by transmitting notification information (step S44), and returns to step S31.

As described above, the monitoring server 40 of this embodiment tracks the movement route of the customer within the checkout area A1 based on the captured images of each of the registration area A11, the checkout area A12, and the exit area A13. Then, the monitoring server 40 detects a customer who may be committing fraudulent acts based on the tracked movement route, and notifies the store staff. As a result, the monitoring server 40 can efficiently detect fraudulent acts of taking unpaid items out of the store in a monitoring system based on collation of captured images.

In the present embodiment, the registration device 10 is configured to transmit the first captured image to the monitoring server 40 upon completion of the registration process, but the first captured image is not limited to this, and may be transmitted during the registration process. Furthermore, although the accounting device 20 is configured to transmit the second captured image to the monitoring server 40 when the accounting process is completed, the present invention is not limited to this, and the second captured image may be transmitted during the accounting process. Furthermore, when the accounting device 20 detects the face area of a person (customer) from the second captured image, it may be configured to assume that the customer has paid the bill and send the second captured image to the monitoring server 40. good. Note that when these configurations are adopted, the criteria for determining whether or not a transaction has been made is less strict than in the configuration of the embodiment described above. Therefore, these configurations are preferably used when the travel route of a customer is simply tracked.

Further, in the present embodiment, the customer is configured to perform the checkout process using the checkout device 20 specified by the clerk, but the configuration is not limited to this, and the configuration may be such that the customer can perform the checkout process using any checkout device 20. good. When adopting this configuration, for example, by operating the registration device 10, accounting device 20, and monitoring server 40 as follows, accounting processing can be performed with any accounting device 20.

First, upon receiving an instruction to end the registration process, the control unit 11 of the registration device 10 sends a set of the accounting information generated in the registration process and the first captured image captured during the registration process to the monitoring server 40. Send. In the monitoring server 40, the feature extraction unit 412 extracts first feature information from the first captured image transmitted from the registration device 10. The registration processing unit 413 registers the extracted first feature information in the tracking table T1 and checks the first check column. Further, the feature extraction unit 412 registers (stores) accounting information that is paired with the first captured image from which the first feature information is extracted in the tracking table T1 or the like in association with the registered first feature information. .

On the other hand, the customer who has completed registration of the product with the registration device 10 moves to an arbitrary checkout device 20. When the control unit 21 of the checkout device 20 receives an instruction to start checkout processing from the customer, it transmits the second image taken by the second camera 32 to the monitoring server 40 . In the monitoring server 40, the feature extraction unit 412 extracts second feature information from the second captured image transmitted from the accounting device 20. Next, the tracking processing unit 414 selects first feature information of the customer imaged by the checkout device 20 (second camera 32) from among the first feature information registered in the tracking table T1 based on the second feature information. Identify. Next, the control unit 41 of the monitoring server 40 transmits accounting information associated with the identified first feature information to the accounting device 20 that transmitted the second captured image.

Subsequently, upon receiving accounting information from the monitoring server 40, the control unit 21 of the accounting device 20 executes accounting processing based on the accounting information. After completing the transaction, the control unit 21 of the accounting device 20 transmits the second captured image captured during the transaction to the monitoring server 40. The subsequent processing is similar to the embodiment described above.

[Second embodiment]
Next, a second embodiment will be described. In the second embodiment, a store that employs a checkout method in which customers themselves register and pay will be described as an example. Note that the same elements as those in the first embodiment are given the same reference numerals and the description thereof will be omitted.

FIG. 9 is a diagram schematically showing an example of a store layout according to the second embodiment. As shown in FIG. 9, the store has a checkout area A2 for product registration and checkout. The checkout area A2 includes an entrance area A21, a registration accounting area A22, and an exit area A23.

The entrance area A21 is a first area located at the entrance of the checkout area A2, and is provided with an entrance B21 to the checkout area A2. The registration/accounting area A22 is a second area where customers register products and make accounting, and is provided with a registration/accounting device 50. Exit area A23 is a third area located at the exit of checkout area A2, and is provided with store exit B22. Although not shown, it is assumed that there is an area in the store where products to be sold are displayed (for example, above the checkout area A2).

The registration accounting device 50 is a product data processing device that is operated by a customer and performs registration processing and accounting processing for products purchased by the customer. In other words, the registered accounting device 50 implements a checkout method called self-checkout or the like.

In the store with the layout shown in FIG. 9, the operations when a customer purchases a product are as follows. First, the customer puts the products to be purchased into a shopping cart or the like and moves from the entrance area A21 (entrance B21) to the registration accounting area A22 (registration accounting device 50). In the registration/accounting device 50, a customer registers a product by having a reader read a barcode attached to the product. When the customer completes the product registration, the customer pays the bill using the registration and billing device 50. When the customer completes the transaction, the customer places the purchased product in a plastic bag or the like and moves to exit area A23 (exit B22).

In this way, in the self-service system, the customer registers and pays the bill himself, so it is possible to improve processing efficiency, such as shortening the waiting time at the register. However, in the self-service system, registration and accounting are left to the customers themselves, so similar to the semi-self-service system, there is a possibility that fraudulent activities such as cart omissions may occur.

Therefore, in the monitoring system of this embodiment, images are taken of the entrance area A21, registered accounting area A22, and exit area A23, respectively, and based on the captured images, customers who may be committing fraudulent acts are detected. The configuration of the monitoring system according to this embodiment will be described below.

FIG. 10 is a diagram schematically showing an example of the configuration of a monitoring system according to the second embodiment. As shown in FIG. 10, the monitoring system includes the above-described registration accounting device 50, a first camera 61, a second camera 62, a third camera 63, and a monitoring server 40a. The registration accounting device 50, the first camera 61, the third camera 63, and the monitoring server 40a are connected to a network N2 such as a LAN.

The first camera 61 is an image capturing device that captures images of customers present in the entrance area A21. The first camera 61 is provided at a position where it can image the face of a customer entering the checkout area A2 from the entrance B21. The first camera 61 transmits a captured image (hereinafter referred to as a first captured image) to the monitoring server 40a. The installation position of the first camera 61 is not particularly limited; for example, it may be installed at the gate of the entrance B21, or it may be installed on the ceiling of the store in the form of a surveillance camera or the like.

The second camera 62 is a camera that captures images of customers present in the registered accounting area A22. A second camera 62 is provided in each registered accounting device 50. More preferably, the second camera 62 is provided in each registered accounting device 50 at a position where it can capture an image of the customer's face. Each set of the registration accounting device 50 and the second camera 62 is connected by a connection line L3. The installation position of the second camera 62 is not particularly limited; for example, it may be installed integrally with the registration accounting device 50, or it may be installed on the ceiling of a store in the form of a surveillance camera or the like.

The registered accounting device 50 cooperates with the second camera 62 and transmits a captured image (hereinafter referred to as a second captured image) captured during the accounting process to the monitoring server 40a. Note that the configuration of the registration accounting device 50 is similar to the combination of the registration device 10 and the accounting device 20 described above, so a detailed explanation will be omitted.

The registered accounting device 50 acquires a second captured image captured by the second camera 62 while the accounting process is being performed. Then, when the accounting process is completed, the registered accounting device 50 transmits the second captured image acquired during the accounting process to the monitoring server 40a. Here, the number of second captured images (still images) and the number of hours (videos) to be transmitted to the monitoring server 40a are not particularly limited. Note that the registration accounting device 50 acquires the second captured image while the registration process is being performed, and transmits the acquired second captured image to the monitoring server 40a at the timing when the registration process or accounting process is completed. It's okay. In addition, the registration accounting device 50 (control unit) executes the same transmission image selection process as the registration device 10 described above, so that the second A captured image may also be selected.

The third camera 63 is an image capturing device that captures images of customers present in the exit area A23. The third camera 63 is provided at a position where it can capture images of the faces of customers passing through exit B22. The third camera 63 transmits the captured image (hereinafter referred to as the third captured image) to the monitoring server 40a. Note that the third camera 63 may be installed at any particular location; it may be installed at the gate of the exit B22, or it may be installed on the ceiling of the store in the form of a surveillance camera or the like.

The monitoring server 40a is a server device that monitors the movements of customers within the checkout area A2 based on captured images captured by the first camera 61, the second camera 62, and the third camera 63. The monitoring server 40a will be explained below. Note that the same configurations as those in the first embodiment are given the same reference numerals and explanations are omitted.

FIG. 11 is a diagram showing an example of the configuration of a monitoring server 40a according to the second embodiment. As shown in FIG. 11, the monitoring server 40a includes a control unit 41a. The control unit 41a has a computer configuration such as a CPU, SoC, ROM, and RAM.

A storage unit 42 and a communication interface (I/F) 43 are connected to the control unit 41a via a bus or the like. The storage unit 42 stores a tracking table T1 (see FIG. 7) having the same data structure as the first embodiment.

Further, as shown in FIG. 11, the control unit 41a includes functional units such as an image acquisition unit 411a, a feature extraction unit 412, a registration processing unit 413, a tracking processing unit 414, a fraud detection unit 415, and a notification processing unit 416. These functional units are realized in software by executing a program stored in the storage unit 42, or in hardware by a dedicated processor included in the control unit 41a.

The image acquisition unit 411a acquires images captured by each of the first camera 61, the second camera 62, and the third camera 63 via the communication interface 43. More specifically, the image acquisition unit 411a acquires the first captured image transmitted from the first camera 61. Further, the image acquisition unit 411a acquires the second captured image transmitted from each of the registered accounting devices 50. The image acquisition unit 411a also acquires a third captured image transmitted from the third camera 63.

The control unit 41a executes the monitoring process described in FIG. 8 by cooperating with each of the functional units described above. Thereby, the monitoring server 40a of the present embodiment tracks the movement route of the customer within the checkout area A2 based on the captured images of each of the entrance area A21, registration area A22, and exit area A23. Then, the monitoring server 40a detects a customer who may be committing fraudulent acts based on the traced movement route, and notifies the store staff. As a result, the monitoring server 40a can efficiently detect fraudulent acts of taking unpaid items out of the store in a monitoring system based on collation of captured images.

Although the embodiments of the present invention have been described above, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included within the scope and gist of the invention, as well as within the scope of the invention described in the claims and its equivalents.

For example, in the above embodiment, the monitoring server 40 (40a) has a configuration in which the feature information is extracted from the captured image, but the configuration is not limited to this. good. For example, any or all of the registration device 10, the accounting device 20, and the registered accounting device 50 may include the feature extraction unit 412, and may transmit feature information extracted from the captured image to the monitoring server 40 (40a). Further, each camera (for example, the third camera 33, the first camera 61, the third camera 63, etc.) includes a feature extraction unit 412, and transmits feature information extracted from the captured image to the monitoring server 40 (40a). Good too. In this case, in the monitoring server 40 (40a), the registration processing unit 413 registers the first characteristic information transmitted from the registration device 10 and the first camera 61 in the tracking table T1. In addition, the tracking processing unit 414 determines whether the same customer track the route of travel.

Further, in the above embodiment, the monitoring server 40 (registration processing unit 413) is configured to register the first feature information extracted from the first captured image in the tracking table T1, but the present invention is not limited to this. The captured image may be registered in the tracking table T1. In this case, the feature extraction unit 412 cooperates with the tracking processing unit 414 to extract feature information from each captured image when comparing the degree of similarity between captured images.

Further, in the above embodiment, the configuration is such that the monitoring server 40 (40a) monitors for fraudulent activities, but the configuration is not limited to this, and another device may perform the monitoring. For example, one representative device among the registration device 10 and the accounting device 20 (or the registered accounting device 50) may be configured to have the function of the monitoring server 40 (40a) to monitor fraudulent activities.

Further, in the above embodiment, the registration device 10, the accounting device 20, and the registered accounting device 50 transmit captured images, but the present invention is not limited to this, and each camera may directly transmit the captured images. In addition, although the first camera 31 and the third cameras 33 and 36 are configured to directly transmit captured images, they are configured to transmit captured images from the information processing device by cooperating with an information processing device such as a PC (Personal Computer). You can also use it as Furthermore, when transmitting from the information processing device, a captured image to be transmitted may be selected by performing a process similar to the transmission image selection process.

10 Registration device 20 Accounting device 31, 61 First camera 32, 62 Second camera 33, 63 Third camera 40, 40a Monitoring server 50 Registration accounting device 411, 411a Image acquisition unit 412 Feature extraction unit 413 Registration processing unit 414 Tracking process Section 415 Fraud detection section T1 Tracking table

Japanese Patent Application Publication No. 2013-186495

Claims (6)

In a checkout area related to product registration and accounting, imaging is performed to image each of the following areas: a first area located at the entrance, a second area where customers themselves check out their products, and a third area located at the exit. means and
Extracting means for extracting feature information representing the customer's characteristics from the captured images captured in each of the regions;
Based on the similarity between the feature information extracted from the captured image of the first area and the feature information extracted from the captured images of each of the second area and the third area, the same customer tracking means for tracking a movement route from the area to the third area;
Notifying means for notifying when the moving route skips the second area from the first area and reaches the third area;
A monitoring device equipped with. A registration device is installed in the first area for a store clerk to register products purchased by a customer,
An accounting device is installed in the second area for customers to pay for products registered with the registration device,
2. The monitoring device according to claim 1, wherein the imaging means for imaging the second area is provided in each of the registration device and the accounting device at a position where it can take an image of the customer. 3. The monitoring device according to claim 2, wherein the imaging means provided in the accounting device transmits a captured image taken during the transaction to the monitoring device after the transaction related processing is performed in the accounting device. A registration and accounting device is installed in the second area for customers to register and pay for products;
2. The monitoring device according to claim 1, wherein the imaging means for imaging the second area is provided in each of the registered accounting devices at a position where it can take an image of the customer. 5. The image capturing means provided in the registration accounting device transmits a captured image taken during the processing to the monitoring device after the registration and accounting processing is performed in the registration accounting device. Monitoring equipment. computer,
A first area located at the entrance, a second area where customers check out their products, and a third area located at the exit in the checkout area for product registration and accounting. Extracting means for extracting characteristic information representing characteristics of the customer from the image;
Based on the similarity between the feature information extracted from the captured image of the first area and the feature information extracted from the captured images of each of the second area and the third area, the same customer tracking means for tracking a movement route from the area to the third area;
Notifying means for notifying when the moving route skips the second area from the first area and reaches the third area;
A program to make it work.

Images