JP2023031766A - Self-service checkout machine monitoring system and self-service checkout machine monitoring method - Google Patents

Abstract

To provide a "self-service checkout machine monitoring system" and the like that perform monitoring for preventing an illegal operation in a self-service checkout machine, in which false detection of an illegal operation is reduced to improve reliability as monitoring means.SOLUTION: Provided is a self-service checkout machine monitoring system 1 including: a monitoring and imaging unit 10; a motion detection unit 31 that analyzes an image captured by the monitoring and imaging unit and detects a specific motion of a user of the self-service checkout machine to move a commodity during the settlement work from an unsettled commodity placed position to another position; a suspicious motion determination unit 32; and an attention-calling unit 34 that outputs attention-calling information to a management information processing terminal. The suspicious motion determination unit 32 compares a detection result of the specific motion detected by the motion detection unit 31 and a reading result of commodity information obtained by a commodity information reading unit 110, and determines suspiciousness of an operation by the user of the self-service checkout machine.SELECTED DRAWING: Figure 1

Description

The present invention relates to a "self-checkout monitoring system" and a "self-checkout monitoring method" that perform monitoring to prevent unauthorized operations at self-checkouts.

2. Description of the Related Art In recent years, mainly in various retail stores, a self-checkout system has been spreading, in which a purchaser himself/herself causes a product information reading unit such as a bar code reader to read the product information of the product to be purchased and performs checkout.

In a self-checkout, a purchaser performs checkout by having a bar code reader or the like read product information such as a bar code attached to the product. At this time, the self-checkout manager needs to prevent the purchaser from taking out the product intentionally or by mistake without reading the product information correctly.

In self-checkout, as a means for preventing the above-mentioned unauthorized operation, Patent Document 1 discloses that "a purchaser's action to cause a bar code reader to read product information" is detected, and even if the "action" is detected, Nevertheless, there has been disclosed a self-checkout system intended to prevent unauthorized operation by displaying a message prompting rereading on an operation assisting monitor or the like when product information is not actually read.

In the self-checkout disclosed in Patent Document 1, the detection of "a purchaser's action of having a bar code reader read product information" is to determine whether or not an object to be monitored crosses and passes through the target area of the surveillance camera. It is realized by image processing software that can However, for example, when the operator's arm passes over the bar code reader without holding the product, an erroneous detection of "the purchaser's operation to read the product information with the bar code reader" inevitably occurs. It is clear that this is also mentioned in the same document (see paragraph [0037] of the same document).

Further, in Patent Document 2, in a self-checkout, "a purchaser's action of having a bar code reader read product information" is detected, and although the action is detected, the product information is not actually read. A self-checkout system is disclosed in which image data recording the operation when there is no security is efficiently extracted from a huge amount of monitoring images, and unauthorized operation can be easily confirmed.

In the self-checkout disclosed in Patent Document 2, the detection of "the operation of the purchaser reading the product information with a barcode reader" is performed by detecting the movement of the purchaser's arm with a photoelectric sensor or the like. . Even with such a detection method, the accuracy of detecting "the action of the purchaser reading the product information with a barcode reader" is insufficient, and as a result, more than a certain amount of false positives occur. Sex wasn't enough.

JP-A-10-208149 JP 2010-39545 A

As described above, conventionally known means for monitoring fraudulent operations at self-checkout counters have many false detections, and do not have sufficient reliability as monitoring means.

In view of the above situation, the present invention improves the detection accuracy of "a purchaser's operation to read product information with a barcode reader" in a "self-checkout monitoring system" that monitors to prevent unauthorized operations at self-checkouts. , to reduce erroneous detection of unauthorized operations and improve reliability as a monitoring means.

The present invention is a "self-checkout monitoring system" or the like that monitors to prevent unauthorized operations at self-checkouts. It was conceived that the above problem can be solved by performing processing to detect "specific actions", which are detected as information that includes not only actions but also the movements of products accompanying the actions, by means of deep learning type image analysis means. As a result, the present invention has been completed.

(1) A monitoring system for a self-checkout in which a purchaser himself/herself causes a product information reading unit to read product information and performs checkout work, wherein the monitoring and photographing unit photographs the self-checkout user, and the surveillance photographing unit captures the images. a motion detection unit that analyzes an image and detects a specific motion of the self-checkout user holding the unsettled product and moving it from a position where the unsettled product is placed to another position; A suspicious action, wherein the degree of suspiciousness of the self-checkout user's operation is determined by comparing the detection result of the detected specific action by the self-checkout user and the reading result of the product information by the product information reading unit. A self-checkout monitoring system, comprising: a determining unit; and an alerting unit that outputs alerting information according to a result of determination of a degree of suspiciousness by the suspicious action determining unit.

According to the self-checkout monitoring system of (1), in the "self-checkout monitoring system" that monitors to prevent unauthorized operations at the self-checkout, the user of the self-checkout holds the unpaid product and places the unpaid product Accuracy of detection of "buyer's action to read product information with a barcode reader" by detecting "specific action" defined as action to move from one position to another position and comparing it with the reading result of product information can be increased, erroneous detection of unauthorized operations can be reduced, and reliability as a monitoring means can be improved.

(2) The motion detection unit analyzes the movement of the self-checkout user's hand and the movement of the product in the monitoring image by various machine learning type image analysis devices having a neural network, thereby detecting the The self-checkout monitoring system according to (1), which detects specific actions.

According to the self-checkout monitoring system of (2), in the self-checkout monitoring system of (1), the above-mentioned "specific action" detected as information including not only human actions but also product movements accompanying the actions is extremely high. can be detected with precision. As a result, erroneous detection of unauthorized operations can be further reduced, and the reliability of the monitoring means can be more reliably improved to a higher level.

(3) An ID information acquisition unit that acquires the ID information of the self-checkout user, and the combination of the ID information of the individual self-checkout user and the past determination result of the degree of suspicion of the self-checkout user is obtained by the individual a determination history storage unit for storing an action history of the self-checkout user, wherein the suspicious action determination unit also refers to the action history of the self-checkout user stored in the determination history storage unit, The self-checkout monitoring system according to (1) or (2), which determines the degree of suspiciousness of the self-checkout user's operation.

The self-checkout monitoring system of (3) further acquires and manages ID information of the self-checkout user in determining the degree of suspiciousness of the operation of the self-checkout user in the self-checkout monitoring system of (1) or (2), also made available for reference. As a result, users who have performed unauthorized operations in the past can be monitored according to stricter monitoring standards, and the recurrence of unauthorized operations by the same purchaser can be prevented with high accuracy.

(4) The self-checkout monitoring system according to (3), wherein the ID information acquisition unit is a face authentication information acquisition unit that acquires face authentication information of the self-checkout user as ID information.

The self-checkout monitoring system of (4) enables the acquisition of face authentication information as the ID information acquired in the self-checkout monitoring system of (3). It is possible to acquire the ID information of the person and refer to the past action history. As a result, it is possible to enjoy the above effects of the self-checkout monitoring system in (3) without requiring separate ID acquisition and management work for an unspecified number of self-checkout users.

(5) The action detection unit detects a suspicious action other than the specific action of the self-checkout user from the monitoring image as an unauthorized preliminary action, and the suspicious action determination unit also detects the detection result of the unauthorized preliminary action. The self-checkout monitoring system according to any one of (1) to (4), which determines the degree of suspiciousness of the self-checkout user's operation.

In the self-checkout monitoring system of (5), in determining the degree of suspicion of the operation of the self-checkout user in any of the self-checkout monitoring systems of (1) to (4), unauthorized operations other than "specific actions" related to checkout work By detecting an "unauthorized preliminary movement" that can be regarded as a preliminary movement of the player, this "unauthorized preliminary movement" can also be referred to. As a result, users who are likely to perform unauthorized operations can be monitored according to a stricter monitoring standard, so that unauthorized operations can be prevented with higher accuracy.

(6) The self-checkout monitoring system according to (5), wherein as the unauthorized preliminary operation, a change in the line-of-sight direction of the self-checkout user is detected.

In the self-checkout monitoring system of (6), in the self-checkout monitoring system of (5), it is essential to detect the change in the line-of-sight direction of the self-checkout user as the "unauthorized preliminary action". As a result, it is possible to reliably detect "unauthorized preliminary actions" related to line-of-sight fluctuations that are likely to be overlooked by the supervisor's visual monitoring, thereby improving the reliability of the self-checkout monitoring system (5) to a higher level.

(7) In the self-checkout having an operation assistance information display unit that displays operation assistance information to the self-checkout user, the attention calling unit displays the attention calling information on the operation assistance information display unit when the degree of suspicion exceeds a threshold. The self-checkout monitoring system according to any one of (1) to (6), which outputs.

According to the self-checkout monitoring system of (7), necessary warning information can be effectively conveyed to the self-checkout user promptly and reliably.

(8) A self-checkout comprising the self-checkout monitoring system according to any one of (1) to (7).

According to the self-checkout of (8), by enjoying each of the above-mentioned effects of the self-checkout monitoring system of any one of (1) to (7), the loss of goods due to fraudulent operations can be prevented in stores where the self-checkout is installed. It is also possible to reduce the burden of monitoring work by monitoring personnel.

(9) A monitoring method for a self-checkout in which a purchaser performs a checkout operation by a product information reading step in which a purchaser himself/herself causes a product information reading unit to read product information, wherein the monitoring photographing unit photographs a self-checkout user. Then, the operation detection unit, from the monitoring image captured by the monitoring imaging unit, holds the unpaid product while the self-checkout user is performing the payment operation, and moves the unpaid product from the unpaid product placement position to another position. A specific action detection step of detecting a specific action, and a suspicious action determination unit compares the detection result of the specific action and the read result of the product information to determine the degree of suspiciousness of the self-checkout user's operation. A self-checkout monitoring method, comprising: a suspicious action determination step; and an alerting step in which an alerting unit outputs alerting information when the degree of suspicion exceeds a threshold.

According to the self-checkout monitoring method of (9), in the "self-checkout monitoring method" that monitors to prevent unauthorized operations at the self-checkout, the detection accuracy of "the operation of the purchaser reading product information with a barcode reader" is improved. By doing so, it is possible to reduce erroneous detections of unauthorized operations and improve the reliability of monitoring work.

(10) In the specific action detection step, various machine learning type image analysis devices having a neural network analyze the movement of the hand of the self-checkout user and the movement of the product in the monitoring image. , the self-checkout monitoring method according to (9), wherein the specific action is detected.

According to the self-checkout monitoring method (10), it is possible to further reduce erroneous detection of unauthorized operations in the self-checkout monitoring method (9), and improve the reliability of the monitoring work to a higher level.

(11) Before the product information reading step, the ID information acquisition unit performs an ID information acquisition step of acquiring the ID information of the self-checkout user. a determination history storing step of storing a combination of ID information of each self-checkout user and the degree of suspiciousness of the self-checkout user as an action history of each self-checkout user; The self-checkout monitoring method according to (9) or (10), wherein, in the step, the degree of suspiciousness of the self-checkout user's operation is determined by also referring to the action history of the self-checkout user to be determined.

According to the self-checkout monitoring method of (11), in determining the degree of suspiciousness of the operation of the self-checkout user in the self-checkout monitoring method of (9) or (10), the ID information of the self-checkout user is acquired and managed, and the past action history is obtained. also made available for reference. As a result, it is possible to detect with higher accuracy a second unauthorized operation by a user who has performed an unauthorized operation in the past.

(12) The self-checkout monitoring method according to (11), wherein in the ID information acquisition step, face authentication information acquisition processing is performed to acquire face authentication information of the self-checkout user as the ID information.

According to the self-checkout monitoring method of (12), face authentication information can be acquired as the ID information acquired in the self-checkout monitoring method of (11). By acquiring the ID information of the self-checkout user, it is possible to refer to the past action history. As a result, it is possible to enjoy the above effect of the self-checkout monitoring method (11) without requiring separate ID acquisition and management work for an unspecified number of self-checkout users.

(13) In the specific action detection step, a suspicious action other than the specific action of the self-checkout user is detected from the monitoring image as an unauthorized preliminary action, and in the suspicious action determination step, the unauthorized preliminary action is detected. The self-checkout monitoring method according to any one of (9) to (12), wherein the degree of suspiciousness of the self-checkout user's operation is determined by also referring to detection results.

In the self-checkout monitoring method of (13), in determining the degree of suspicion of the self-checkout user's operation in any of the self-checkout monitoring methods of (9) to (12), unauthorized operations other than "specific actions" related to checkout work By detecting an "unauthorized preliminary movement" that can be regarded as a preliminary movement of the player, this "unauthorized preliminary movement" can also be referred to. As a result, users who are likely to perform unauthorized operations can be monitored according to a stricter monitoring standard, so that unauthorized operations can be prevented with higher accuracy.

(14) The self-checkout monitoring method according to (13), wherein as the fraudulent preliminary action, a change in the line-of-sight direction of the self-checkout user is detected.

In the self-checkout monitoring system of (14), in the self-checkout monitoring method of (13), it is essential to detect the change in the line-of-sight direction of the self-checkout user as the "unauthorized preliminary action". As a result, it is possible to reliably detect "unauthorized preliminary actions" related to line-of-sight fluctuations that are likely to be overlooked by visual monitoring by the supervisor, and to improve the reliability of the self-checkout monitoring method (13) to a higher level.

(15) In the self-checkout monitoring method according to any one of (9) to (14), the specific motion detection step and the suspicious motion determination step include the motion detection section and the suspicious motion determination section. A program for monitoring illegal operations that is executed by the arithmetic processing unit.

According to the unauthorized operation monitoring program of (15), in the "self-checkout monitoring method" for monitoring to prevent unauthorized operation at the self-checkout, detection of "the operation of the purchaser reading the product information with the barcode reader" Accuracy can be improved, false detections of unauthorized operations can be reduced, and the reliability of monitoring work can be improved.

According to the present invention, it is possible to reduce erroneous detection of unauthorized operations and improve reliability as a monitoring means in a "self-checkout monitoring system" or the like that performs monitoring to prevent unauthorized operations in a self-checkout.

1 is a block diagram showing the configuration of a self-checkout monitoring system of the present invention; FIG. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic diagram showing an example of a self-checkout configuration provided with the self-checkout monitoring system of the present invention; 4 is a flow chart showing the operation flow of the self-checkout monitoring method of the present invention; It is a schematic diagram which shows an example of embodiment of this invention in a real store.

BEST MODE FOR CARRYING OUT THE INVENTION The best mode for carrying out the present invention will be described below with reference to the drawings as appropriate.

<Self-checkout>
FIG. 2 is a diagram schematically showing the configuration of a self-checkout 100, which is an example of a self-checkout that can perform monitoring work using the self-checkout monitoring system or self-checkout monitoring method of the present invention. The self-checkout to which the present invention is applied means that, like the self-checkout 100, the self-checkout user B, who is the purchaser, reads the product information of the product P by the product information reading unit 110 such as the bar code reader 111. It is a device that performs settlement work by The self-checkout 100 preferably includes, in addition to the product reading unit 110, an operation auxiliary information display unit 120 that conveys auxiliary information related to the self-checkout operation to the self-checkout user B by means of images and sounds. In addition, an unpaid product placement position 131 where unpaid products are placed and a paid product placement position 132 where paid products are placed are arranged with the product information reading unit 110 interposed therebetween. It is desirable that the left and right sides are designated as prescribed positions for the user.

<Self-checkout monitoring system>
FIG. 1 is a block diagram showing the configuration of a self-checkout monitoring system 1, which is an example of the self-checkout monitoring system of the present invention. The self-checkout monitoring system 1 is, for example, a system that monitors the self-checkout 100 configured as shown in FIG. 2 to prevent unauthorized operation by the self-checkout user B. In the following description, specific configurations are exemplified and described, but these can be changed as appropriate.

[overall structure]
The self-checkout monitoring system 1 includes at least a monitoring photographing unit 10 and an arithmetic processing unit 30 . In addition, it is preferable that the self-checkout monitoring system 1 further include an ID information acquisition unit 20 . More specifically, the ID information acquisition section 20 is more preferably the face authentication information acquisition section 21 .

The arithmetic processing unit 30 includes at least a motion detection unit 31 , a suspicious motion determination unit 32 , and a warning unit 34 . Moreover, it is preferable that the arithmetic processing unit 30 further includes a determination history storage unit 33 . Moreover, it is also possible to adopt a configuration in which a coordinate setting section that assists the performance of the function of the surveillance photographing section 10 is provided as necessary.

[Surveillance shooting unit]
The monitoring photographing unit 10 is a device that obtains a monitoring image for detecting unauthorized operations by photographing the actions of a self-checkout user (a purchaser who uses a self-checkout to check out merchandise). Specifically, for example, as shown in FIG. 2, the surveillance imaging unit 10 captures a range including the whole body or at least the upper body of the self-checkout user B and a range that both hands of the self-checkout user B can reach. The range including the area is configured by the monitoring camera 11 installed at a position where continuous shooting can be performed as a shooting target area.

The monitoring photographing unit 10 preferably has a function of outputting the surveillance image photographed by the surveillance camera 11 as digital image data so that the arithmetic processing unit 30 can perform arithmetic processing. For example, various existing digital cameras can be used without particular limitations.

A 3D camera capable of directly acquiring three-dimensional positional information in a space to be photographed may be used as the surveillance camera 11. A typical monocular camera can also be configured for this. In the self-checkout monitoring system 1, when the monitoring photographing unit 10 is a monocular camera capable of capturing a monitoring image having only two-dimensional information, the self-checkout monitoring system 1 captures a position in the monitoring image in a three-dimensional space as an area to be captured. It is more preferable to further include a coordinate setting unit (illustration in the block diagram is omitted) that sets identifiable coordinates in association with the actual position in the interior. In the case where the monitoring photographing unit 10 is a 3D camera capable of directly obtaining three-dimensional positional information in the space to be photographed, the coordinate setting unit of the above mode is not essential. The coordinate setting unit may be built in the surveillance photographing unit 10 or may be incorporated in the arithmetic processing unit 30 . By providing a coordinate setting unit capable of generating 3D coordinates from 2D image information, only 2D information acquired only by a monocular camera that can be obtained at low cost without introducing a distance measuring device or 3D camera can be obtained. Human motion can be detected with high accuracy only by automatic processing, even from images possessed by the system.

[ID information acquisition part]
More preferably, the self-checkout monitoring system 1 includes an ID information acquiring section 20 in addition to the monitoring photographing section 10 and the arithmetic processing section 30 . The ID information acquisition unit 20 acquires, in some form, ID information of each self-checkout user B, that is, personal data that enables individual self-checkout user B to be identified from other users and managed individually. Any means can be used as long as it can be obtained. For example, the self-checkout user is asked to register some ID information such as name and address in advance, and the self-checkout user is asked to read the membership card, etc., on which the ID information is recorded, and the ID information of the card is requested. The self-checkout monitoring system 1 including the ID information acquisition unit 20 can also be configured by installing a device such as a card reader for reading as the ID information acquisition unit 20 .

However, it is more preferable that the face authentication information acquisition unit 21 be used as the ID information acquisition unit 20 installed in the self-checkout monitoring system 1 . As the face authentication information acquisition unit 21 capable of acquiring the face authentication information of the self-checkout user as ID information, various conventional known face authentication information acquisition devices can be used. Moreover, as a specific face authentication information acquisition device, it is preferable to use various face authentication devices to which deep learning technology, whose recognition accuracy has been dramatically improved in recent years, is applied. When the self-checkout monitoring system 1 of the present invention is operated, for example, as shown in FIG. It is preferable to incorporate it into . As a result, it is possible to reliably acquire the unique biometric identification information of the self-checkout user when the checkout process is started.

The face authentication information, which is the ID information of each self-checkout user acquired by the face authentication information acquiring section 21, is stored in the storage device in the arithmetic processing section 30 and referred to by other components as necessary. In addition, this face authentication information is stored in one of the above storage devices as combination information linked with other incidental information useful for monitoring related to the self-checkout user (determination history of suspiciousness of past operations, etc.) It is preferable to be registered in a certain determination history storage unit 33 .

[Arithmetic processing part]
The arithmetic processing unit 30 performs various kinds of arithmetic processing necessary for judging the degree of suspiciousness of the operation of the self-checkout user on the image data output from the surveillance photographing unit 10 and transmitted. The arithmetic processing unit 30 can be configured using, for example, a personal computer, a tablet terminal, a smart phone, or the like. Alternatively, the arithmetic processing unit 30 can be configured as a dedicated device specialized for the operation according to the implementation of the present invention. Further, the arithmetic processing unit 30 may be configured such that an information processing device having the above functions is configured as a server on the cloud and shared by many clients.

In any of the configurations described above, the arithmetic processing unit 30 includes hardware such as a CPU, a memory, and a communication unit. The arithmetic processing unit 30 configured as described above executes the "unauthorized operation monitoring program", which is a computer program according to the present invention, to perform the various operations and the self-checkout monitoring method described below. can be executed effectively.

The arithmetic processing unit 30 is connected to the surveillance photographing unit 10 so as to be able to receive image data from the surveillance photographing unit 10 . This connection can be a wired connection using a dedicated communication cable or a wired LAN connection. Also, connection using various types of wireless communication such as wireless LAN, short-range wireless communication, mobile phone line, etc. may be used instead of wired connection. Note that the arithmetic processing unit 30 is not placed near the monitoring/photographing unit 10, but is located away from the monitoring/photographing unit 10, in the work area of the supervisor of the monitoring work, or in another remote location as necessary. can also be placed in

(motion detector)
The action detection unit 31 analyzes the monitoring image captured by the monitoring image capturing unit 10 and detects the "specific action" of the self-checkout user B. FIG. The "specific action" of the self-checkout user in the present invention means "the action of the self-checkout user moving the product from the unsettled product placement position to another position". That is, the detection result of the "specific action" includes not only information on the position and movement of the "hand" but also information on the position and movement of the "goods" moved by the "hand".

For example, in the self-checkout 100 shown in FIG. 2, the self-checkout user B holds an unsettled product P with either hand, and moves the unsettled product P from the unsettled product placement position 131 at the start of checkout. The operation of moving to another position apart from the space above the information reading unit 110 (barcode reader 111) or the space above the product information reading unit 110 (barcode reader 111) away from the unsettled product placement position 131 is performed. It is a “specific motion” to be detected by the motion detection unit 31 in the detection process.

In order to detect the above-described “specific motion”, the motion detection unit 31 uses, for example, various machine learning type image analysis devices having a neural network (so-called “image recognition device using deep learning technology”). can be configured A specific example of image recognition technology using deep learning is disclosed below.
"Deep Learning and Image Recognition, Operations Research"
(http://www.orsj.o.jp/archive2/or60-4/or60_4_198.pdf)

Further, in order to mechanically recognize the details of the self-checkout user's motion from the monitoring image in the motion detection unit 31, for example, a known image analysis technology called "OpenPose" can be used. As disclosed in Japanese Patent No. 6534499 by the inventor of the present application, by using "OpenPose", the skeleton information of the "self-checkout user" is extracted from the monitoring image, and the position and position of each feature point that constitutes the skeleton. By analyzing the speed, it is possible to recognize the actions of the "self-checkout user".

Further, by using an image analysis technique such as "OpenPose", it is possible to extract from the monitoring image the movement of the self-checkout user B's hand and the movement of the product P as independent information. By using such a function, by analyzing the movement of the hand of the self-checkout user B and the movement of the product P, the difference between the movements of the two, etc., the self-checkout user B can identify the products P The movement of the hand holding the product P and the movement of the hand not holding the product P can be mechanically detected separately. Detection of fraudulent operations can be greatly reduced by excluding "the movement of self-checkout user B's hand that does not hold product P" from the "specific movement" by distinguishing it from the movement of the product. be able to.

More preferably, the motion detection unit 31 can also detect suspicious motions other than the above-described “specific motion” of the self-checkout user B as “unauthorized preliminary motions” from the monitoring image. Here, the above-mentioned "fraudulent preliminary operation" in this specification means that the self-checkout monitoring system 1 is operated as an operation that is often performed prior to the fraudulent act when the self-checkout user B commits fraudulent act at the time of payment. It refers to a "specific action" that is preset by the monitoring entity that uses it. As an example, determination of such a "specific action" is performed by comparing a person in a monitoring image with a plurality of regions (for example, upper region: head region), as disclosed in Japanese Patent No. 6534499 by the inventor of the present application. part, middle region: torso, and lower region: legs), and extracting the feature amount related to the positional variation of each part of the person for each of these regions.

As an example of a typical "unauthorized preliminary action" by self-checkout user B, "the action of looking around multiple times prior to checkout" can be cited. It is possible to detect such actions accompanied by line-of-sight changes by various well-known image analysis means. The line-of-sight direction can also be detected with high accuracy by detecting means for detecting the position of both ears and nose.

(Suspicious behavior determination unit)
The suspicious action determination unit 32 reads the above-mentioned "specific action" by the self-checkout user B detected by the action detection unit 31, and reads the product information by the product information reading unit 110 (barcode reader 111) of the self-checkout 100. Calculation processing is performed to determine the degree of suspiciousness of self-checkout user B's operation by comparing the result with the result. For example, this calculation process takes as input values a combination of the above-mentioned "specific action" by the self-checkout user B and the "reading result of product information", and a unique suspiciousness degree is defined for each combination. This is an operation that outputs degrees.

An output of the degree of suspicion is performed as follows, for example. For example, as a specific operation, "Self-checkout user B grabs an unsettled product P by hand and moves the item from the unsettled product placement position 131 to the space above the product information reading unit 110 (barcode reader 111). If the "moving action" is detected and the "product information reading result" corresponding to the action is not detected (when "no reading result" is detected), the "product information Since a situation of "reading error" is presumed, the suspicious action determination unit 32 outputs a specific suspiciousness level (for example, a suspiciousness level of "medium") to deal with the situation of "reading error of product information".

In addition, as another example of outputting the degree of suspicion, as a specific action, "Self-checkout user B grabs unsettled product P by hand, picks up the unsettled product P from unsettled product placement position 131, and reads product information reading unit 110. (bar code reader 111) is detected to move it to a place other than the upper space", and if the "product information reading result" corresponding to the action is not detected, the "product information Since it is presumed that the situation is "suspicion of unauthorized operation (shoplifting) to avoid reading", the suspicious action determination unit 32 determines the situation "suspicion of unauthorized operation (shoplifting) to avoid reading product information". A higher inherent suspiciousness level (for example, "high" suspiciousness level) is output.

On the other hand, as a specific action, "the self-checkout user B's action of grabbing the unpaid product P and moving it from the unpaid product placement position 131 to a place other than the paid-up product placement position 132" is detected. and the "product information reading result" corresponding to the action is detected, the "suspect of unauthorized operation (shoplifting) is low because the product information has been read". Since it can be presumed that the product for which payment has been made is simply handed over to a companion without putting it in a shopping bag, the suspicious action determination unit 32 determines the “lowest degree of suspiciousness” (for example, “low” degree of suspiciousness). ) will be output.

Further, in determining the degree of suspiciousness of operations by self-checkout user B by the suspicious action determination unit 32, the ID information (preferably face authentication information) of individual self-checkout users and the degree of suspiciousness of past operations of the self-checkout user are used. As a combination with the determination result, the action history of the self-checkout user B stored in the determination history storage unit 33 may also be referenced to adjust the determination result of the degree of suspiciousness. As a result of referring to the action history, for the self-checkout user who is determined to be a customer requiring attention, adjustment may be made to lower the threshold, which is the criterion for sending the alert information, or the self-checkout user Separately, alert information indicating that "the customer is a customer requiring attention" may be transmitted to a monitoring worker such as the accounting assistant A, regardless of the judgment result of the degree of suspiciousness of the current action.

Furthermore, the action detection unit 31 of the self-checkout monitoring system 1 detects suspicious actions other than the above-mentioned "specific action" of the self-checkout user from the monitoring image as "unauthorized preliminary action", and the detection result of this "unauthorized preliminary action" is detected. may also be referenced to adjust the determination result of the degree of suspicion. Here, the "fraudulent preliminary action" in this specification is an action that is often performed prior to the fraudulent act of the self-checkout user B, and is monitored using the self-checkout monitoring system 1. Refers to a "specific action" preset by the subject. As an example, determination of such a "specific action" is performed by comparing a person in a monitoring image with a plurality of regions (for example, upper region: head region), as disclosed in Japanese Patent No. 6534499 by the inventor of the present application. part, middle region: torso, and lower region: legs), and extracting the feature amount related to the positional variation of each part of the person for each of these regions.

As an example of a typical "unauthorized preliminary action" by self-checkout user B, "the action of looking around multiple times prior to checkout" can be cited. An example of such an operation accompanied by a change in line of sight is, as disclosed in Japanese Patent No. 6562437 by the inventor of the present application, detection means for detecting the direction of line of sight of the self-checkout user from the positions of both ears and nose. can be done.

(Judgment history storage unit)
As described above, the determination history storage unit 33 stores the combination of the ID information of each self-checkout user B and the past determination result of the degree of suspicion of the self-checkout user B as the action history of each self-checkout user B. . The determination history storage unit 33 is a database composed of various storage devices capable of registering such information.

(Warning part)
The alerting unit 34 outputs alerting information according to the result of the determination of the degree of suspiciousness of the self-checkout user B by the suspicious action determination unit 32 . The alerting information is transmitted to self-checkout user B and supervisory workers such as accounting assistant A as information effective in deterring fraudulent operations.

As a device for receiving the alerting information, the self-checkout user B or the supervisory worker such as the accounting assistant A uses video, text, or the like to allow the user to perceive the information sent by the alerting unit 34. It can be configured by various information processing terminals capable of appropriately displaying information using voice or the like, or various information display devices including a monitor capable of responding to such requests. There is no particular limitation as long as it is such a device or device, and various information receiving terminals possessed or lent by the user can be used.

In order to prevent fraudulent operations, the alert information transmitted from the self-checkout monitoring system 1 may is transmitted to the information processing terminal for management possessed by the supervisor.

<Self-checkout monitoring method>
The "self-checkout monitoring method" of the present invention can be suitably executed by the operation of the above-described "self-checkout monitoring system" of the present invention. This "self-checkout monitoring method" includes, as shown in FIG. A product information reading step S30 executable in the product information reading unit 110 constituting Step S50, attention calling step S60 that can be performed by the attention calling unit 34, and determination history storage step S70 that can be performed by the determination history storage unit 33 are sequentially performed, thereby executing the entire process. However, among the above steps, the ID information acquisition step S20 and the determination history storage step S70 are not essential components, and the "self-checkout monitoring method" of the present invention is implemented as a minimum configuration in which these steps are omitted. can also In the following, a detailed description will be given of an embodiment of the "self-checkout monitoring method" of the present invention in which all the steps of S10 to S70 are performed.

(Surveillance shooting step)
In the surveillance photographing step S10, the surveillance photographing unit 10 photographs the actions of the self-checkout user B (the product purchaser who uses the self-checkout to pay for the product) (FIG. 4(1)). The monitoring image captured in the monitoring photographing step S10 is output as digital image data, input to the arithmetic processing unit 30, and subjected to predetermined arithmetic processing.

(ID information acquisition step)
In the ID information obtaining step S20, the ID information about the self-checkout user B is obtained by the ID information obtaining unit 20. FIG. This ID information acquisition step S20 is performed before the product information reading step S30 is started. As mentioned above, this ID information is preferably face authentication information. The ID information (preferably face authentication information) acquired in the ID information acquisition step S20 is also input to the arithmetic processing section 30 as digital data and subjected to predetermined arithmetic processing.

In the "self-checkout monitoring method", when the ID information of the self-checkout user B is specified in the ID information acquisition step S20, if the self-checkout user B has acted with a high degree of suspicion in the past, may output preliminary alert information to the effect that the self-checkout user B is a customer requiring special attention to the information processing terminal owned by the accounting assistant A' prior to subsequent processing (Fig. 4(2)). Whether or not the self-checkout user B is a customer requiring special attention is determined based on the past actions stored in the database (determination history storage unit 33) in the arithmetic processing unit 30 based on the ID information acquired in the ID information acquisition step S20. It can be determined mechanically by referring to the history.

(Product information reading step)
In the product information reading step S30, the self-checkout user B, who is the purchaser, causes the product information reading unit 110 (bar code reader 111) of the self-checkout 100 to read the product information of the product P, and the checkout operation is performed. . At this time, the product information actually read normally by the product information reading unit 110 of the self-checkout 100 is also sent to the system (arithmetic processing unit 30) for executing the "self-checkout monitoring method" in accordance with the operation of the self-checkout user B. The combination is input as information for determining suspicious behavior.

(Specific action detection step)
In the specific motion detection step S40, the hand of the self-checkout user B and the product are extracted from the monitoring image obtained in the monitoring image capturing step S10, and the movement of the hand and the product are analyzed. A "specific action" related to the settlement work as described above is detected (FIGS. 4(3) and 4(4)).

Further, in the specific action detection step S40, a suspicious action other than the "specific action" of the self-checkout user B (for example, frequent changes in the line-of-sight direction, etc.) is detected as an unauthorized preliminary action from the monitoring image, and the suspicious action is detected. This detection result may contribute to the determination of .

(Suspicious behavior determination step)
In the suspicious action determination step S50, the detection result of the "specific action" of the self-checkout user B detected in the specific action detection step S40 is compared with the product information reading result to determine if the operation of the self-checkout user B is suspicious. The degree is determined (FIG. 4(5)). In determining the degree of suspicion, as described above, the detection result of the "unauthorized preliminary motion" may also be referred to, and the criteria for determining the degree of suspicion may be adjusted in consideration of this. As a result of the determination, if the degree of suspiciousness of self-checkout user B exceeds a predetermined threshold value (suspicious action? YES), a warning step S60 is performed. If the degree of suspiciousness of the self-checkout user B does not exceed the predetermined threshold value (suspicious action? NO), the process returns to the monitoring photographing step S10 to continue the monitoring work.

(Attention step)
In the alerting step S60, if the suspicious behavior of the self-checkout user B exceeds a predetermined threshold as a result of the determination in the suspicious motion determination step S50 (suspicious motion? YES), the suspicious motion determination unit 32 Appropriate warning information is output according to the degree of suspicion. Specifically, in order to prevent unauthorized operations, the sent alert information is sent as an alert notification to the management information processing terminal owned by the accounting assistant A'' of the self-checkout register 100 (see FIG. 4). (5)).

Further, in the attention calling step S60, the attention calling information may be output to the operation auxiliary information display section 120 (see FIG. 2) of the self-checkout 100. FIG. In this way, the alerting information can be transmitted to the self-checkout user B and the monitoring operator such as the accounting assistant A as information effective in deterring fraudulent operations. For example, in addition to detecting unauthorized operations by self-checkout users after the fact, the possibility of unauthorized operations occurring by self-checkout users is predicted from ID information and "unauthorized preliminary actions", In such a way that it is notified as alert information, Accounting Assistant A who receives this notification observes the self-checkout user and, if necessary, asks, "Is there something wrong?" It is also effective to prevent fraudulent operations by calling out to the user such as "Are you sure?" In this case, an announcement such as "If you do not understand how to use it, our staff will help you" from the operation auxiliary information display unit 120 to the user B of the self-checkout register is also considered as illegal operation. It is an effective countermeasure to suppress the occurrence of "

(Judgment history storage step)
In the determination history storage step S70, the combination of the ID information of each self-checkout user and the degree of suspicion of the action related to the past settlement work of the self-checkout user is stored as the action history of the individual self-checkout user in the determination history storage unit 33. (database) ((6) in FIG. 4). When the ID information is face authentication information, it is possible to manage the action history without registering the name of the individual and use it to prevent unauthorized operations.

After executing the judgment history storage step S70, the arithmetic processing unit 30 judges whether or not to end the monitoring. The end of monitoring is determined to end, for example, when a command to end monitoring is input. If it ends, the monitoring operation is terminated. Otherwise, the process returns to the monitoring photographing step S10 to continue monitoring.

1 self-checkout monitoring system 10 surveillance photography unit 11 surveillance camera 20 ID information acquisition unit 21 face authentication information acquisition unit 30 arithmetic processing unit 31 motion detection unit 32 suspicious motion determination unit 33 determination history storage unit 34 alerting unit 100 self-checkout 110 product information reading Section 111 Barcode Reader 120 Operation Auxiliary Information Display Section 131 Placement Position of Unsettled Goods 132 Placement Position of Settled Goods B Self-Checkout User A Accounting Assistant (Monitor)
P Merchandise S10 Surveillance photographing step S20 ID information acquisition step S30 Product information reading step S40 Specific motion detection step S50 Suspicious motion determination step S60 Alert step S70 Determination history storage step

Claims (15)

A self-checkout monitoring system in which a purchaser himself/herself causes a product information reading unit to read product information and performs checkout,
a surveillance photography unit that photographs self-checkout users;
Detecting a specific action of the self-checkout user holding the unsettled item and moving it from the unsettled item placement position to another position by analyzing the monitoring image captured by the monitoring imaging unit. Department and
The detection result of the specific action by the self-checkout user detected by the action detection unit is compared with the result of reading the product information by the product information reading unit to determine the degree of suspiciousness of the operation of the self-checkout user. a suspicious action determination unit for determining;
an alerting unit that outputs alerting information according to the result of determination of the degree of suspiciousness by the suspicious action determination unit;
comprising
Self-checkout monitoring system. The motion detection unit detects the specific motion by analyzing the movement of the self-checkout user's hand and the movement of the product in the monitoring image using various machine learning type image analysis devices having a neural network. To detect,
The self-checkout monitoring system according to claim 1. an ID information acquisition unit that acquires the ID information of the self-checkout user;
a judgment history storage unit for storing, as an action history of each self-checkout user, a combination of the ID information of each self-checkout user and a judgment result of the degree of suspicion of past operations of the self-checkout user;
further comprising
The suspicious action determination unit also refers to the action history of the self-checkout user stored in the determination history storage unit to determine the degree of suspiciousness of the operation of the self-checkout user.
The self-checkout monitoring system according to claim 1 or 2. The ID information acquisition unit is a face authentication information acquisition unit that acquires the face authentication information of the self-checkout user as ID information,
The self-checkout monitoring system according to claim 3. The motion detection unit detects a suspicious motion other than the specific motion of the self-checkout user from the monitoring image as an unauthorized preliminary motion,
The suspicious action determination unit also refers to the detection result of the unauthorized preliminary action to determine the degree of suspiciousness of the self-checkout user's operation.
The self-checkout monitoring system according to any one of claims 1 to 4. Detecting a change in the line-of-sight direction of the self-checkout user as the unauthorized preliminary operation;
The self-checkout monitoring system according to claim 5. In the self-checkout including an operation assistance information display unit that displays operation assistance information to the self-checkout user,
The alerting unit outputs alerting information to the operation auxiliary information display unit when the suspiciousness degree of the operation exceeds a threshold.
The self-checkout monitoring system according to any one of claims 1 to 6. A self-checkout, comprising the self-checkout monitoring system according to any one of claims 1 to 7. A monitoring method for a self-checkout in which a purchaser performs checkout by a product information reading step in which a purchaser himself/herself causes a product information reading unit to read product information,
a monitoring photography step in which the monitoring photography unit photographs the user of the self-checkout;
A motion detection unit performs a specific operation in which the self-checkout user holds an unpaid item during the checkout operation and moves the unsettled item from the unsettled item placement position to another position from the monitoring image captured by the monitoring imaging unit. a specific motion detection step of detecting
a suspicious action determination step in which a suspicious action determination unit compares the detection result of the specific action and the read result of the product information to determine the degree of suspiciousness of the self-checkout user's operation;
A warning step in which the warning unit outputs warning information when the degree of suspicion exceeds a threshold;
Self-checkout monitoring method. In the specific motion detection step, the motion of the hand of the self-checkout user and the motion of the product in the monitoring image are analyzed by various machine-learning image analysis devices having a neural network. detect motion,
The self-checkout monitoring method according to claim 9. Before starting the product information reading step, an ID information acquisition unit acquires the ID information of the self-checkout user, performing an ID information acquisition step;
After execution of the suspicious action determination step, a determination history storage unit stores a combination of the ID information of the individual self-checkout user and the degree of suspiciousness of the self-checkout user as an action history of the individual self-checkout user. A determination history storage step is performed,
In the suspicious action determination step, the degree of suspiciousness of the operation of the self-checkout user is determined by also referring to the action history of the self-checkout user to be determined.
The self-checkout monitoring method according to claim 9 or 10. In the ID information obtaining step, face authentication information obtaining processing is performed to obtain the face authentication information of the self-checkout user as the ID information,
The self-checkout monitoring method according to claim 11. In the specific action detection step, a suspicious action other than the specific action of the self-checkout user is detected from the monitoring image as an unauthorized preliminary action,
In the suspicious action determination step, the degree of suspiciousness of the self-checkout user's operation is determined by also referring to the detection result of the unauthorized preliminary action.
The self-checkout monitoring method according to any one of claims 9 to 12. Detecting a change in the line-of-sight direction of the self-checkout user as the unauthorized preliminary operation;
The self-checkout monitoring method according to claim 13. In the self-checkout monitoring method according to any one of claims 9 to 14,
the specific action detection step and the suspicious action determination step,
A program for monitoring unauthorized operation, which is executed by an arithmetic processing unit including the operation detection unit and the suspicious operation determination unit.

Images