JP7039084B1 - Self-registration monitoring system and self-registration monitoring method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce false detection of unauthorized operation and improve reliability as a monitoring means in a "self-registration monitoring system" or the like which monitors for preventing unauthorized operation in self-registration. SOLUTION: A monitoring and photographing unit 10 and an image taken by the monitoring and photographing unit are analyzed, and the self-registration user performs a specific operation of moving a product from an unsettled product placement position to another position during a settlement operation. The motion detection unit 31 for detecting, the suspicious motion determination unit 32, and the alerting unit 34 for outputting the alerting information to the management information processing terminal are provided, and the suspicious motion determining unit 32 is detected by the motion detecting unit 31. The self-registration monitoring system 1 determines the degree of suspicion of the operation of the self-registration user by comparing the detection result of the specific operation and the reading result of the product information by the product information reading unit 110. [Selection diagram] Fig. 1

Description

The present invention relates to a "self-registration monitoring system" and a "self-registration monitoring method" for monitoring to prevent unauthorized operations in self-registration.

In recent years, self-registration, in which the purchaser himself / herself reads the product information of the product to be purchased by a product information reading unit such as a barcode reader and performs the settlement work, is becoming widespread mainly in various retail stores.

In the self-checkout, the purchaser performs the settlement work by having a barcode reader or the like read the product information such as the barcode attached to the product. At this time, the self-registration manager needs to prevent an unauthorized operation in which the purchaser intentionally or accidentally takes out the product without reading the product information correctly.

In the self-registration, as a means for preventing the above-mentioned unauthorized operation, Patent Document 1 detects "an operation in which a purchaser causes a barcode reader to read product information", and the "operation" is detected. Nevertheless, when the product information is not actually read, a self-registration intended to prevent unauthorized operation by displaying a message prompting the reading again on a monitor or the like for operation assistance is disclosed.

In the self-registration disclosed in Patent Document 1, the detection of "the action of the purchaser having the bar code reader read the product information" is to determine whether or not the monitored object has passed the target area of the surveillance camera. It is realized by the video processing software that can be used. However, for example, when the operator's arm passes over the barcode reader without holding the product, an erroneous detection of "the operation of the purchaser having the barcode reader read the product information" occurs inevitably. It is clear that this is also mentioned in the same document (see paragraph [0037] of the same document).

Further, in Patent Document 2, in the self-registration, "the operation of causing the purchaser to read the product information by the barcode reader" is detected, and the product information is actually read even though the operation is detected. A self-registration is disclosed that enables efficient confirmation of unauthorized operations by efficiently extracting image data recording the operation when there is no such operation from a huge amount of surveillance images.

In the self-registration disclosed in Patent Document 2, the detection of "the operation of the purchaser to read the product information by the barcode reader" is executed by detecting the movement of the purchaser's arm with a photoelectric sensor or the like. .. Even with such a detection means, the detection accuracy of the "action of the purchaser to read the product information by the barcode reader" is still insufficient, which causes false detection above a certain level, and is reliable as a monitoring means. The sex was not enough.

Japanese Unexamined Patent Publication No. 10-208149 Japanese Unexamined Patent Publication No. 2010-39545

As described above, all of the conventionally known self-registration monitoring means for unauthorized operations have many false positives, and are not sufficiently reliable as the monitoring means.

In view of the above situation, the present invention has improved the detection accuracy of "the operation of the purchaser to have the bar code reader read the product information" in the "self-registration monitoring system" or the like that monitors to prevent unauthorized operations in the self-registration. The purpose is to reduce false positives of unauthorized operations and improve reliability as a monitoring means.

The present invention relates to a self-registration user when detecting "an operation in which a purchaser causes a barcode reader to read product information" in a "self-registration monitoring system" or the like that monitors to prevent unauthorized operations in self-registration. I came up with the idea that the above problem can be solved by performing a process of detecting "specific motion", which is detected as information including not only the motion but also the motion of the product accompanying the motion, by a deep learning type image analysis means or the like. However, the present invention has been completed.

(1) This is a self-registration monitoring system in which the purchaser reads the product information by himself / herself into the product information reading unit and performs the settlement work. By the motion detection unit and the motion detection unit, which analyzes the image and detects a specific motion in which the self-registration user holds the unsettled product and moves it from the unsettled product placement position to another position. A suspicious operation for determining the suspicious degree of the operation of the self-registration user by comparing the detected detection result of the specific operation by the self-registration user with the reading result of the product information by the product information reading unit. A self-registration monitoring system including a determination unit and an alert unit that outputs alert information according to a determination result of a suspicious degree by the suspicious motion determination unit.

According to the self-registration monitoring system (1), in the "self-registration monitoring system" that monitors to prevent unauthorized operations in the self-registration, the self-registration user holds the unpaid products and places the unpaid products. Detection accuracy of "action that causes the purchaser to read the product information by the barcode reader" by detecting "specific action" defined as the action of moving from to another position and comparing it with the reading result of the product information. It is possible to reduce false positives of unauthorized operations and improve reliability as a monitoring means.

(2) The motion detection unit analyzes the movement of the hand of the self-registration user and the movement of the product in the monitoring image by various machine learning type image analysis devices having a neural network. The self-registration monitoring system according to (1), which detects a specific operation.

According to the self-registration monitoring system of (2), in the self-registration monitoring system of (1), the above-mentioned "specific operation" detected as information including not only the movement of a person but also the movement of a product accompanying the movement is extremely high. It can be detected with accuracy. As a result, the false detection of unauthorized operation can be further reduced, and the reliability as a monitoring means can be improved to a higher level more reliably.

(3) The combination of the ID information acquisition unit that acquires the ID information of the self-registration user, the ID information of each self-registration user, and the past determination result of the suspicious degree of the self-registration user is individually described. The suspicious motion determination unit further includes a determination history storage unit that stores the behavior history of the self-registration user, and the suspicious motion determination unit also refers to the behavior history of the self-registration user stored in the determination history storage unit. The self-registration monitoring system according to (1) or (2), which determines the degree of suspicion of the operation of the self-registration user.

The self-registration monitoring system of (3) further acquires and manages the ID information of the self-registration user in determining the suspiciousness of the operation of the self-registration user in the self-registration monitoring system of (1) or (2), and has a history of past actions. Can also be referred to. As a result, it is possible to monitor a user who has performed an unauthorized operation in the past according to a stricter monitoring standard and prevent the recurrence of the unauthorized operation by the same purchaser with high accuracy.

(4) The self-registration monitoring system according to (3), wherein the ID information acquisition unit is a face authentication information acquisition unit that acquires the face authentication information of the self-registration user as ID information.

The self-registration monitoring system of (4) can acquire face authentication information as ID information acquired by the self-registration monitoring system of (3), so that an unspecified number of self-registrations can be used without going through procedures such as member registration work. It is possible to acquire the ID information of the person so that the past action history can be referred to. As a result, the above-mentioned effect of the self-registration monitoring system of (3) can be enjoyed without requiring a separate ID acquisition and management work for an unspecified number of self-registration users.

(5) The motion detection unit detects a suspicious motion other than the specific motion of the self-registration user from the monitoring image as an illegal preliminary motion, and the suspicious motion determination unit also detects the illegal preliminary motion. The self-registration monitoring system according to any one of (1) to (4), which is referred to and determines the degree of suspicion of the operation of the self-registration user.

The self-registration monitoring system of (5) is an unauthorized operation other than the "specific operation" related to the settlement work in the determination of the suspiciousness of the operation of the self-registration user in any of the self-registration monitoring systems (1) to (4). The "illegal preliminary operation" that can be regarded as the preliminary operation of is detected, and this "illegal preliminary operation" can also be referred to. As a result, it is possible to monitor a user who has a high probability of performing an unauthorized operation according to a stricter monitoring standard, and thus it is possible to prevent the occurrence of an unauthorized operation with higher accuracy.

(6) The self-registration monitoring system according to (5), which detects a change in the line-of-sight direction of the self-registration user as the illegal preliminary operation.

In the self-registration monitoring system of (6), it is essential to detect the change in the line-of-sight direction of the self-registration user as an "illegal preliminary operation" in the self-registration monitoring system of (5). As a result, it is possible to reliably detect "illegal preliminary movements" related to line-of-sight fluctuations that are easily overlooked by the visual monitoring of the observer, and improve the reliability of the self-registration monitoring system of (5) to a higher level.

(7) In the self-registration including the operation assistance information display unit for displaying the operation assistance information to the self-registration user, the attention-calling unit sends the attention-calling information to the operation assistance information display unit when the suspiciousness exceeds the threshold value. The self-registration monitoring system according to any one of (1) to (6), which outputs.

According to the self-registration monitoring system of (7), necessary alert information can be promptly and surely and effectively transmitted to the self-registration user.

(8) A self-registration equipped with the self-registration monitoring system according to any one of (1) to (7).

According to the self-registration of (8), by enjoying each of the above-mentioned effects of the self-registration monitoring system according to any one of (1) to (7), product loss due to unauthorized operation can be caused at the store where the self-registration is located. It can be reduced and the burden of monitoring work by monitoring personnel can be reduced.

(9) This is a monitoring method for self-registration in which the purchaser makes the purchaser read the product information by the product information reading unit by the product information reading step, and the monitoring and photographing unit photographs the self-registration user. Then, the motion detection unit holds the unsettled product during the settlement operation from the surveillance image taken by the monitoring and photographing unit, and moves the unsettled product from the unsettled product placement position to another position. The specific motion detection step for detecting the specific motion and the suspicious motion determination unit compare the detection result of the specific motion with the reading result of the product information to determine the suspicious degree of the operation of the self-registration user. A self-registration monitoring method comprising a suspicious operation determination step and a warning step in which the alerting unit outputs alerting information when the suspiciousness exceeds a threshold value.

According to the self-registration monitoring method (9), in the "self-registration monitoring method" that monitors to prevent unauthorized operations in the self-registration, the detection accuracy of "the operation of the purchaser to read the product information by the barcode reader" is improved. Therefore, it is possible to reduce false detection of unauthorized operations and improve the reliability of monitoring work.

(10) In the specific motion detection step, the movement of the hand of the self-registration user and the motion of the product in the monitoring image are analyzed by various machine learning type image analysis devices having a neural network. , The self-registration monitoring method according to (9), which detects the specific operation.

According to the self-registration monitoring method of (10), it is possible to further reduce the false detection of unauthorized operations in the self-registration monitoring method of (9) and improve the reliability of the monitoring work to a higher level.

(11) Before the start of the product information reading step, the ID information acquisition unit performs an ID information acquisition step of acquiring the ID information of the self-registration user, and after the execution of the suspicious operation determination step, the determination history storage unit. However, a determination history storage step is performed in which the combination of the ID information of the individual self-registration user and the suspicious degree of the self-registration user is stored as the action history of the individual self-registration user, and the suspicious operation determination is performed. The self-registration monitoring method according to (9) or (10), wherein in the step, the degree of suspicion of the operation of the self-registration user is determined with reference to the behavior history of the self-registration user to be determined.

According to the self-registration monitoring method of (11), in the determination of the suspiciousness of the operation of the self-registration user in the self-registration monitoring method of (9) or (10), the ID information of the self-registration user is acquired and managed, and the past action history is recorded. Can also be referred to. As a result, it is possible to detect the unauthorized operation again by the user who has performed the unauthorized operation in the past with higher accuracy.

(12) The self-registration monitoring method according to (11), wherein in the ID information acquisition step, a face authentication information acquisition process for acquiring the face authentication information of the self-registration user as the ID information is performed.

According to the self-registration monitoring method of (12), the face authentication information can be acquired as the ID information acquired by the self-registration monitoring method of (11), so that an unspecified number of people do not go through procedures such as member registration work. It is possible to acquire the ID information of the self-registration user so that the past action history can be referred to. As a result, it is possible to enjoy the above-mentioned effect of the self-registration monitoring method of (11) without requiring a separate ID acquisition and management work for an unspecified number of self-registration users.

(13) In the specific operation detection step, a suspicious operation other than the specific operation of the self-registration user is detected from the monitoring image as an illegal preliminary operation, and in the suspicious operation determination step, the illegal preliminary operation is performed. The self-registration monitoring method according to any one of (9) to (12), wherein the degree of suspicion of the operation of the self-registration user is determined with reference to the detection result.

The self-registration monitoring method of (13) is an unauthorized operation other than the "specific operation" related to the settlement work in the determination of the suspiciousness of the operation of the self-registration user in any of the self-registration monitoring methods (9) to (12). "Illegal preliminary operation" that can be regarded as a preliminary operation of is detected, and this "illegal preliminary operation" can also be referred to. As a result, it is possible to monitor a user who has a high probability of performing an unauthorized operation according to a stricter monitoring standard, and thus it is possible to prevent the occurrence of an unauthorized operation with higher accuracy.

(14) The self-registration monitoring method according to (13), which detects a change in the line-of-sight direction of the self-registration user as the illegal preliminary operation.

The self-registration monitoring system of (14) requires that the self-registration monitoring method of (13) detect a change in the line-of-sight direction of the self-registration user as an "illegal preliminary operation". As a result, it is possible to reliably detect "illegal preliminary movements" related to line-of-sight fluctuations that are easily overlooked by the visual monitoring of the observer, and improve the reliability of the self-registration monitoring method of (13) to a higher level.

(15) In the self-registration monitoring method according to any one of (9) to (14), the specific operation detection step and the suspicious operation determination step are configured to include the operation detection unit and the suspicious operation determination unit. A program for monitoring fraudulent operations that is executed by the arithmetic processing unit.

According to the program for monitoring unauthorized operations in (15), in the "self-registration monitoring method" that monitors to prevent unauthorized operations in self-registration, "the operation of the purchaser having the barcode reader read the product information" is detected. It is possible to improve the accuracy, reduce the false detection of unauthorized operations, and improve the reliability of monitoring work.

According to the present invention, in a "self-registration monitoring system" or the like that monitors for preventing unauthorized operations in self-registration, it is possible to reduce erroneous detection of unauthorized operations and improve reliability as a monitoring means.

It is a block diagram which shows the structure of the self-registration monitoring system of this invention. It is a schematic diagram which shows an example of the structure of the self-registration provided with the self-registration monitoring system of this invention. It is a flowchart which shows the operation flow of the self-registration monitoring method of this invention. It is a schematic diagram which shows an example of embodiment of this invention in a physical store.

Hereinafter, the best mode for carrying out the present invention will be described with reference to the drawings as appropriate.

<Self-registration>
FIG. 2 is a diagram schematically showing a configuration of a self-registration 100, which is an example of a self-registration capable of performing monitoring work by using the self-registration monitoring system or the self-registration monitoring method of the present invention. The self-registration that is the object of the present invention is that, like the self-registration 100, the self-registration user B, who is the purchaser, reads the product information of the product P by the product information reading unit 110 such as the barcode reader 111. It is a device that performs settlement work. In addition to the product reading unit 110, the self-registration 100 preferably includes an operation auxiliary information display unit 120 that conveys auxiliary information related to the self-registration operation to the self-registration user B by an image or voice. Further, the unsettled product placement position 131, which is the placement place of the unsettled product, and the settled product placement position 132, in which the settled product is placed, sandwich the product information reading unit 110 in between. It is desirable that the configuration is specified as a specified position for the user on the left and right.

<Self-registration monitoring system>
FIG. 1 is a block diagram showing a configuration of a self-registration monitoring system 1 which is an example of the self-registration monitoring system of the present invention. As an example, the self-registration monitoring system 1 is a system that executes monitoring for preventing unauthorized operation of the self-registration user B in the self-registration 100 having the configuration shown in FIG. In the following description, specific configurations will be illustrated and described, but these can be changed as appropriate.

[overall structure]
The self-registration monitoring system 1 includes at least a monitoring and photographing unit 10 and an arithmetic processing unit 30. Further, the self-registration monitoring system 1 is preferably configured to further include an ID information acquisition unit 20. Specifically, the ID information acquisition unit 20 is more preferably the face authentication information acquisition unit 21.

The arithmetic processing unit 30 includes at least an operation detection unit 31, a suspicious operation determination unit 32, and a warning unit 34. Further, it is preferable that the arithmetic processing unit 30 is further provided with the determination history storage unit 33. Further, if necessary, it may be configured to include a coordinate setting unit that assists the function of the monitoring and photographing unit 10.

[Surveillance shooting department]
The monitoring and photographing unit 10 is a device that obtains a monitoring image for detecting an unauthorized operation by photographing the movement of a self-registration user (a product purchaser who performs a product settlement operation using the self-registration). Specifically, for example, as shown in FIG. 2, the surveillance camera unit 10 covers the whole body of the self-registration user B, or at least the range including the upper body, and in addition, the range within the reach of both hands of the self-registration user B. It is composed of a surveillance camera 11 installed at a position where a range including the range can be continuously photographed as a photographing target area.

It is preferable that the surveillance imaging unit 10 has a function of outputting the surveillance image captured by the surveillance camera 11 as digital image data so that the arithmetic processing unit 30 can perform arithmetic processing. For example, various existing digital cameras can be used without particular limitation.

Further, the surveillance camera 11 may use a 3D camera that can directly acquire the three-dimensional position information in the space to be the shooting target area, but it is general to shoot the three-dimensional space to be the shooting target area as a two-dimensional image. This can also be configured with a typical monocular camera. In the self-registration monitoring system 1, when the surveillance imaging unit 10 is a monocular camera capable of capturing a surveillance image having only two-dimensional information, the self-registration monitoring system 1 sets the position in the surveillance image as a three-dimensional space which is a shooting target area. It is more preferable to further include a coordinate setting unit (not shown in the block diagram) for setting identifiable coordinates in association with the actual position in the block diagram. If the surveillance imaging unit 10 is a 3D camera capable of directly acquiring three-dimensional position information in the space targeted for imaging, the coordinate setting unit of the above aspect is not essential. The coordinate setting unit may be built in the monitoring and photographing unit 10, or may be built in the arithmetic processing unit 30. By providing a coordinate setting unit that can generate 3D coordinates from 2D image information, only 2D information acquired only by a monocular camera that can be acquired at low cost without introducing a distance measurement device or 3D camera, etc. can be obtained. Even from the image it has, it is possible to detect human movements with high accuracy only by automatic processing.

[ID information acquisition unit]
It is more preferable that the self-registration monitoring system 1 includes an ID information acquisition unit 20 in addition to the monitoring and photographing unit 10 and the arithmetic processing unit 30. The ID information acquisition unit 20 has some form of ID information of each self-registration user B, that is, personal data that can identify each self-registration user B from other users and manage them individually. Any means that can be obtained will do. For example, the self-registration user is requested to register some ID information such as name and address in advance, and the self-registration user is requested to read the membership card or the like in which the ID information is recorded. The self-registration monitoring system 1 including the ID information acquisition unit 20 can also be configured by installing a device such as a reading card reader as the ID information acquisition unit 20.

However, it is more preferable that the ID information acquisition unit 20 installed in the self-registration monitoring system 1 is the face authentication information acquisition unit 21. As the face authentication information acquisition unit 21 capable of acquiring the face authentication information of the self-registration user as ID information, various conventional known face authentication information acquisition devices can be used. Further, as a specific face recognition information acquisition device, it is preferable to use various face recognition devices to which deep learning technology whose recognition accuracy has been dramatically improved in recent years is applied. When operating the self-registration monitoring system 1 of the present invention, for example, as shown in FIG. 2, a touch panel for settlement operation is installed in front of the self-registration user B who performs settlement work by using the photographing device of the face authentication information acquisition unit 21. It is preferable to incorporate it in the above. As a result, it is possible to reliably obtain the biometric identification information unique to the self-registration user at the time of disclosure of the settlement work.

The face recognition information, which is the ID information of each self-registration user acquired by the face recognition information acquisition unit 21, is stored in the storage device in the arithmetic processing unit 30 and is referred to by other components as necessary. In addition, this face recognition information is one of the above-mentioned storage devices as combination information associated with other incidental information useful for monitoring the self-registration user (such as a history of determining the suspicious degree of past operations). It is preferable that it is registered in a certain determination history storage unit 33.

[Arithmetic processing unit]
The arithmetic processing unit 30 performs various arithmetic processing necessary for determining the suspicious degree of the operation of the self-registration user on the image data output and transmitted from the monitoring and photographing unit 10. The arithmetic processing unit 30 can be configured by using, for example, a personal computer, a tablet terminal, a smartphone, or the like. Alternatively, the arithmetic processing unit 30 can be configured as a dedicated device specialized for the operation according to the implementation of the present invention. Further, the arithmetic processing unit 30 may configure an information processing device having the above functions as a server on the cloud and share it with a large number of clients.

In any of the above configurations, the arithmetic processing unit 30 includes hardware such as a CPU, a memory, and a communication unit. Then, the arithmetic processing unit 30 having such a configuration concretely implements various operations described below and a self-registration monitoring method by executing the "program for monitoring unauthorized operations" which is the computer program according to the present invention. Can be executed.

The arithmetic processing unit 30 is connected to the monitoring and photographing unit 10 so that image data can be received from the monitoring and photographing unit 10. This connection can be a wired connection using a dedicated communication cable or a wired LAN connection. Further, the connection is not limited to a wired connection, and may be a connection using various wireless communications such as a wireless LAN, short-range wireless communication, and a mobile phone line. It should be noted that the arithmetic processing unit 30 is not arranged in the vicinity of the monitoring and photographing unit 10, but is located in the work area of the supervisor of the monitoring work, which is away from the monitoring and photographing unit 10, or, if necessary, another remote location. It can also be placed in.

(Motion detection unit)
The motion detection unit 31 analyzes the surveillance image captured by the surveillance imaging unit 10 and detects the “specific motion” of the self-registration user B. The "specific operation" of the self-registration user in the present invention means "the operation of the self-registration user to move the product from the unpaid product placement position to another position". That is, the detection result of the "specific motion" includes not only the information related to the position and movement of the "hand" but also the information related to the position and movement of the "commodity" moved by the "hand".

For example, in the self-registration 100 shown in FIG. 2, the self-registration user B holds the unsettled product P with any hand and obtains the product from the unsettled product placement position 131, which is the storage location at the start of settlement. The operation of moving to a position other than the space on the information reading unit 110 (bar code reader 111) or the space on the product information reading unit 110 (bar code reader 111) is separated from the unsettled product placement position 131. This is a "specific operation" to be detected by the operation detection unit 31 in the detection process.

In order to detect the above-mentioned "specific motion", the motion detection unit 31 uses, for example, various machine learning type image analysis devices having a neural network (so-called "image recognition device using deep learning technology"). Can be configured. Specific examples of image recognition technology using deep running are disclosed below.
"Deep Learning and Image Recognition, Operations Research"
(Http: //www.orsj.o.jp/archive2/or60-4/or60_4_198.pdf)

Further, in the motion detection unit 31, in order to mechanically recognize the details of the motion of the self-registration user from the monitoring image, a known image analysis technique called "OpenPose" can be used as an example. As disclosed in Japanese Patent No. 6534499 by the inventor of the present application, by using "OpenPose", the skeleton information of the "self-registration user" can be extracted from the monitoring image, and the position of each feature point constituting the skeleton and the position of each feature point can be obtained. By analyzing the speed, the movement of the "self-registration user" can be recognized.

Further, by using an image analysis technique such as "OpenPose", the movement of the hand of the self-registration user B and the movement of the product P can be extracted from the surveillance image as independent information. By using such a function, the self-registration user B analyzes the movement of each of the hand movements of the self-registration user B and the movements of the product P, the difference between the movements of the two, and the like, so that the self-registration user B can use the unpaid product P. The movement of the hand in the state of holding the product P and the movement of the hand in the state of not holding the product P can be distinguished and mechanically detected. By excluding "the action of self-registration user B moving the hand that does not hold the product P" from the "specific action" separately from the action of moving the product, the detection of unauthorized operations is greatly reduced. be able to.

Further, it is more preferable that the motion detection unit 31 can detect a suspicious motion other than the above-mentioned "specific motion" of the self-registration user B as an "illegal preliminary motion" from the monitoring image. Here, the above-mentioned "illegal preliminary operation" in the present specification refers to the self-registration monitoring system 1 as an operation that is often performed prior to the fraudulent act when the self-registration user B commits the fraudulent act at the time of settlement. It refers to a "specific operation" that is preset by the monitoring entity used. As an example, the determination of such a "specific operation" is performed by, as disclosed by the inventor of the present application in Japanese Patent No. 6534499, a person in a surveillance image in a plurality of regions (for example, an upper region: head). It can be performed by dividing into a part, an intermediate region: a body part, and a lower part: a leg part), and extracting a feature amount related to a position change of each part of a person for each of those areas.

As an example of a typical "illegal preliminary operation" by the self-registration user B, "an operation of looking around the surroundings a plurality of times prior to the settlement work" can be mentioned. Such movements accompanied by line-of-sight fluctuations can be detected by various well-known image analysis means, but as an example, as disclosed by the inventor of the present application in Japanese Patent No. 6562437, a self-registration user The line-of-sight direction can also be detected with high accuracy by a detection means that detects the positions of both ears and nose.

(Suspicious motion judgment unit)
The suspicious motion determination unit 32 reads the detection result of the above-mentioned "specific motion" by the self-registration user B detected by the motion detection unit 31 and the product information by the product information reading unit 110 (bar code reader 111) of the self-registration 100. By comparing the results with the results, an arithmetic process for determining the suspiciousness of the operation of the self-registration user B is performed. In this arithmetic processing, for example, a combination of the above-mentioned "specific operation" and "reading result of product information" by the self-registration user B is used as an input value, and suspiciousness is defined as a unique suspicious degree corresponding to each combination. It is an operation to output the degree.

The output of the suspicious degree is performed as follows, for example. For example, as a specific operation, "Self-registration user B grabs the unsettled product P by hand and moves the product from the unsettled product placement position 131 into the space on the product information reading unit 110 (bar code reader 111). If "moving action" is detected and "reading result of product information" corresponding to the action is not detected (when "no reading result" is detected), "product information" Since the situation of "reading error" is presumed, the suspicious operation determination unit 32 outputs a unique suspicious degree (for example, suspicious degree "medium") for dealing with the situation of "reading error of product information".

Further, as another example of the output of the suspicious degree, as a specific operation, "the self-registration user B grabs the unsettled product P by hand and reads the product from the unsettled product placement position 131 to the product information reading unit 110. If the "movement to move to a place other than the space on the (bar code reader 111)" is detected and the "reading result of the product information" corresponding to the movement is not detected, the "product information" is detected. Since the situation of "suspicion of illicit operation (shoplifting) to avoid reading" is presumed, the suspicious operation determination unit 32 responds to the situation of "suspicion of illicit operation (shoplifting) to avoid reading of product information". Try to output a higher specific suspiciousness (eg, suspiciousness "high").

On the other hand, as a specific operation, "an operation in which the self-registration user B grabs the unsettled product P by hand and moves it from the unsettled product placement position 131 to a place other than the settled product placement position 132" is detected. If the product information has been read and the "product information reading result" corresponding to the operation is detected, the "suspicion of unauthorized operation (shoplifting) because the product information has been read" is low, for example. Since it is presumed that the product that has just been settled is simply handed over to a companion or the like without being stored in a plastic shopping bag or the like, the suspicious motion determination unit 32 has a “lowest suspicious degree” (for example, a suspicious degree “low””. ) Is output.

Further, in the determination of the suspicious degree of the operation of the self-registration user B by the suspicious operation determination unit 32, the ID information (preferably face recognition information) of each self-registration user and the suspiciousness of the past operation of the self-registration user are determined. As a combination with the determination result, the determination result of the suspicious degree may be adjusted by referring to the action history of the self-registration user B stored in the determination history storage unit 33. As a result of referring to the behavior history, the self-registration user who is determined to be a customer requiring attention may be adjusted to lower the threshold value which is the standard for transmitting the alert information, or the self-registration user. Regardless of the determination result of the suspicious degree of the current operation of the above, the alert information "customer requiring attention" may be separately transmitted to the monitoring worker such as the accounting assistant A.

Further, the motion detection unit 31 of the self-registration monitoring system 1 detects a suspicious motion other than the above-mentioned "specific motion" of the self-registration user from the monitoring image as an "illegal preliminary motion", and the detection result of this "illegal preliminary motion". Also, the determination result of the suspicious degree may be adjusted by referring to. Here, the "illegal preliminary operation" in the present specification is an operation that is often performed prior to the fraudulent act when the self-registration user B commits the fraudulent act, and is monitored by using the self-registration monitoring system 1. It refers to a "specific action" that is preset by the subject. As an example, the determination of such a "specific operation" is performed by, as disclosed by the inventor of the present application in Japanese Patent No. 6534499, a person in a surveillance image in a plurality of regions (for example, an upper region: head). It can be performed by dividing into a part, an intermediate region: a body part, and a lower part: a leg part), and extracting a feature amount related to a position change of each part of a person for each of those areas.

As an example of a typical "illegal preliminary operation" by the self-registration user B, "an operation of looking around the surroundings a plurality of times prior to the settlement work" can be mentioned. As an example, such an operation accompanied by a line-of-sight fluctuation is based on a detection means for detecting the line-of-sight direction of a self-registration user from the positions of both ears and nose, as disclosed by the inventor of the present application in Japanese Patent No. 6562437. Can be done.

(Judgment history storage unit)
As described above, the determination history storage unit 33 stores the combination of the ID information of the individual self-registration user B and the determination result of the past suspiciousness of the self-registration user as the action history of the individual self-registration user B. .. The determination history storage unit 33 is a database configured by various storage devices and the like capable of registering such information.

(Caution section)
The alerting unit 34 outputs the alerting information according to the determination result of the suspicious degree of the self-registration user B by the suspicious operation determination unit 32. As necessary, the alert information is transmitted to monitoring workers such as self-registration user B and accounting assistant A as information effective in deterring unauthorized operations.

As a device for receiving the alert information, a self-registration user B, a monitoring worker such as an accounting assistant A, or the like can recognize the information transmitted by the alert unit 34. It can be configured by various information processing terminals capable of appropriately displaying by voice or the like, or various information display devices including a monitor or the like capable of responding to such a request. Such devices and devices are not particularly limited, and various information receiving terminals owned or rented by the user can be used.

Specifically, the alert information transmitted from the self-registration monitoring system 1 in order to prevent unauthorized operation includes, for example, the operation assistance information display unit 120 constituting the self-registration 100, the accounting assistant A of the self-registration 100, and others. It is sent to the management information processing terminal owned by the observer of.

<Self-registration monitoring method>
The "self-registration monitoring method" of the present invention can be suitably executed by the operation of the "self-registration monitoring system" of the present invention described above. As shown in FIG. 3, this "self-registration monitoring method" includes a monitoring and photographing step S10 that can be executed by the monitoring and photographing unit 10 of the "self-registration monitoring system", an ID information acquisition step S20 that can be executed by the ID information acquisition unit, and a self-registration 100. The product information reading step S30 that can be executed by the product information reading unit 110 constituting the above, the specific operation detecting step S40 that can be executed by the operation detecting unit 31 of the "self-registration monitoring system", and the suspicious operation determination that can be executed by the suspicious operation determining unit 32. Step S50, the alerting step S60 that can be executed by the alerting unit 34, and the determination history storage step S70 that can be executed by the determination history storage unit 33 are sequentially performed, so that the process is executed as an entire process. However, among the above steps, the ID information acquisition step S20 and the determination history storage step S70 are not essential constituent requirements, and the "self-registration monitoring method" of the present invention is carried out as a minimum configuration omitting these steps. You can also do it. In the following, the details of the "self-registration monitoring method" of the present invention will be described with respect to the embodiment in the case where all the steps of S10 to S70 are performed.

(Surveillance shooting step)
In the surveillance photography step S10, the operation of the self-registration user B (the product purchaser who performs the settlement work of the product using the self-registration) is photographed by the surveillance photography unit 10 (FIG. 4 (1)). The surveillance image captured in the surveillance imaging step S10 is output as digital image data, input to the arithmetic processing unit 30, and subjected to predetermined arithmetic processing.

(ID information acquisition step)
In the ID information acquisition step S20, the ID information acquisition unit 20 acquires the ID information of the self-registration user B. This ID information acquisition step S20 is performed before the start of the product information reading step S30. As described above, this ID information is preferably face recognition information. The ID information (preferably face authentication information) acquired in the ID information acquisition step S20 is also input to the arithmetic processing unit 30 as digital data and subjected to a predetermined arithmetic processing.

In the "self-registration monitoring method", when the ID information of the self-registration user B is specified in the ID information acquisition step S20, the self-registration user B has performed a highly suspicious action in the past. May output the preliminary alert information indicating that the self-registration user B is a customer requiring attention to the information processing terminal possessed by the accounting assistant A'prior to the subsequent processing (Fig.). 4 (2)). Whether or not the self-registration user B is a customer requiring attention is determined by past actions stored in the database (determination history storage unit 33) in the arithmetic processing unit 30 based on the ID information acquired in the ID information acquisition step S20. It can be determined mechanically by referring to the history.

(Product information reading step)
In the product information reading step S30, the self-registration user B, who is the purchaser, causes the product information reading unit 110 (bar code reader 111) of the self-registration 100 to read the product information of the product P, and the settlement operation is performed. .. At this time, the product information actually read normally by the product information reading unit 110 of the self-registration 100 is also the operation of the self-registration user B in the system (calculation processing unit 30) for executing the "self-registration monitoring method". It is input as information for determining suspicious operation depending on the combination.

(Specific motion detection step)
In the specific motion detection step S40, the hand and the product of the self-registration user B are extracted from the surveillance image obtained in the surveillance shooting step S10, and the movement of the hand and the product are analyzed. The "specific operation" related to the settlement work as described above is detected (FIGS. 4 (3) and 4 (4)).

Further, in the specific motion detection step S40, a suspicious motion other than the "specific motion" of the self-registration user B (for example, frequent fluctuations in the line-of-sight direction) is detected as an illegal preliminary motion from the monitoring image, and the suspicious motion is performed. This detection result may be contributed to the determination of.

(Suspicious operation determination step)
In the suspicious operation determination step S50, the detection result of the "specific operation" of the self-registration user B detected in the specific operation detection step S40 and the reading result of the product information are compared with each other, and the operation of the self-registration user B is suspicious. The degree is determined (FIG. 4 (5)). In the determination of the suspicious degree, as described above, the detection result of the “illegal preliminary operation” may be referred to, and the determination standard of the suspicious degree may be adjusted in consideration of this. As a result of the determination, when the suspicious degree of the self-registration user B exceeds a predetermined threshold value (suspicious operation? YES), the attention alert step S60 is performed. If the degree of suspicion of the self-registration user B does not exceed a predetermined threshold value (suspicious operation? NO), the process returns to the monitoring shooting step S10 and the monitoring work is continued.

(Caution step)
In the alert step S60, when the suspicious degree of the self-registration user B exceeds a predetermined threshold value as a result of the determination in the suspicious operation determination step S50 (suspicious operation? YES), the self-registration user B by the suspicious operation determination unit 32 Appropriate alert information is output according to the degree of suspicion. In order to prevent tampering, the alert information to be sent is specifically sent as an alert notification to the management information processing terminal possessed by the accounting assistant A'' of the self-registration 100 (FIG. 4). (5)).

Further, in the alerting step S60, the alerting information may be output to the operation assist information display unit 120 (see FIG. 2) of the self-registration 100. In this way, the alert information can be transmitted to the monitoring workers such as the self-registration user B and the accounting assistant A as information effective for deterring unauthorized operations. For example, it not only detects the fraudulent operation of the self-registration user after the fact, but also predicts the possibility of the fraudulent operation by the self-registration user from the ID information and the "illegal preliminary operation", and makes a preliminary to the accounting assistant A. The accounting assistant A who received this notification observes the self-registration user by notifying it as alert information, and if necessary, at the stage before performing an unauthorized operation, "Something is wrong. It is also effective to prevent unauthorized operations by calling out such as "Is it?" In this case, it is also possible to make an announcement from the operation assistance information display unit 120 to the self-registration user B, such as "If you do not know how to use it, the employee will respond", etc. It will be an effective measure to suppress the situation. "

(Judgment history storage step)
In the determination history storage step S70, the combination of the ID information of each self-registration user and the suspicious degree of the operation related to the past settlement work of the self-registration user is used as the action history of the individual self-registration user in the determination history storage unit 33. It is stored in (database) (Fig. 4 (6)). When the ID information is face authentication information, it is possible to manage the action history without registering an individual's name or the like and utilize it for preventing unauthorized operations.

After the execution of the determination history storage step S70, the arithmetic processing unit 30 determines whether or not to end the monitoring. The end of monitoring is determined to be the end when, for example, an instruction to end monitoring is input. When it ends, the monitoring operation ends, and in other cases, the process returns to the monitoring shooting step S10 and the monitoring is continued.

1 Self-registration monitoring system 10 Surveillance shooting unit 11 Surveillance camera 20 ID information acquisition unit 21 Face recognition information acquisition unit 30 Calculation processing unit 31 Motion detection unit 32 Suspicious motion judgment unit 33 Judgment history storage unit 34 Attention calling unit 100 Self-registration 110 Product information reading Department 111 Barcode Reader 120 Operation Assistance Information Display Unit 131 Unsettled Product Placement Position 132 Settled Product Placement Position B Self-Registration User A Accounting Assistant (Monitorer)
P Product S10 Surveillance shooting step S20 ID information acquisition step S30 Product information reading step S40 Specific motion detection step S50 Suspicious motion determination step S60 Attention calling step S70 Judgment history storage step

Claims (13)

It is a self-registration monitoring system in which the purchaser reads the product information by himself / herself and performs the settlement work.
A surveillance photography department that photographs self-registration users,
Motion detection that analyzes the surveillance image taken by the surveillance imaging unit and detects a specific operation in which the self-registration user holds an unsettled product and moves it from the unsettled product placement position to another position. Department and
A suspicious operation determination unit that determines the degree of suspicion of the operation of the self-registration user,
A warning unit that outputs warning information according to the judgment result of the suspicious degree by the suspicious movement judgment unit, and a warning unit.
The ID information acquisition unit that acquires the ID information of the self-registration user,
A determination history storage unit that stores a combination of the ID information of each self-registration user and the determination result of the suspiciousness of the past operation of the self-registration user as the action history of the individual self-registration user.
Equipped with
The suspicious motion determination unit compares the detection result of the specific motion by the self-registration user detected by the motion detection unit with the reading result of the product information by the product information reading unit, and further, the said. The degree of suspicion of the operation of the self-registration user is determined with reference to the behavior history of the self-registration user stored in the determination history storage unit.
Self-registration monitoring system. The ID information acquisition unit is a face authentication information acquisition unit that acquires the face authentication information of the self-registration user as ID information.
The self-registration monitoring system according to claim 1. The motion detection unit performs the specific motion by analyzing the movement of the hand of the self-registration user and the motion of the product in the monitoring image by various machine learning type image analysis devices having a neural network. To detect,
The self-registration monitoring system according to claim 1 or 2 . The motion detection unit detects a suspicious motion other than the specific motion of the self-registration user from the monitoring image as an illegal preliminary motion.
The suspicious operation determination unit determines the degree of suspicion of the operation of the self-registration user with reference to the detection result of the illegal preliminary operation.
The self-registration monitoring system according to any one of claims 1 to 3 . As the fraudulent preliminary operation, a change in the line-of-sight direction of the self-registration user is detected.
The self-registration monitoring system according to claim 4 . In the self-registration provided with the operation assistance information display unit for displaying the operation assistance information to the self-registration user.
The alerting unit outputs the alerting information to the operation assist information display unit when the suspicious degree of the operation exceeds the threshold value.
The self-registration monitoring system according to any one of claims 1 to 5 . A self-registration provided with the self-registration monitoring system according to any one of claims 1 to 6 . It is a monitoring method for self-registration in which the purchaser performs the settlement work by the product information reading step that causes the purchaser to read the product information by himself / herself.
The surveillance shooting step, where the surveillance shooting department shoots the self-registration user,
An ID information acquisition step in which the ID information acquisition unit acquires the ID information of the self-registration user before the start of the product information reading step.
A specific operation in which the motion detection unit holds an unsettled product during the settlement operation and moves the unsettled product from the unsettled product placement position to another position from the surveillance image taken by the surveillance imaging unit. With a specific action detection step to detect
The suspicious motion determination unit determines the suspicious degree of the operation of the self-registration user, and the suspicious motion determination step.
After executing the suspicious operation determination step, the determination history storage unit stores the combination of the ID information of the individual self-registration user and the suspicious degree of the self-registration user as the action history of the individual self-registration user. Judgment history storage step and
The alerting unit includes an alerting step that outputs alerting information when the suspiciousness exceeds a threshold value .
In the suspicious motion determination step, the self-registration is performed by comparing the detection result of the specific motion with the reading result of the product information, and further referring to the behavior history of the self-registration user to be determined. The degree of suspicion of the user's operation is judged,
Self-registration monitoring method. In the ID information acquisition step, a face authentication information acquisition process for acquiring the face authentication information of the self-registration user as the ID information is performed.
The self-registration monitoring method according to claim 8 . In the specific motion detection step, the specific motion detection step is performed by analyzing the movement of the hand of the self-registration user and the motion of the product in the monitoring image by various machine learning type image analysis devices having a neural network. Detect motion,
The self-registration monitoring method according to claim 8 or 9 . In the specific motion detection step, a suspicious motion other than the specific motion of the self-registration user is detected as an illegal preliminary motion from the monitoring image.
In the suspicious operation determination step, the suspicious degree of the operation of the self-registration user is determined with reference to the detection result of the illegal preliminary operation.
The self-registration monitoring method according to any one of claims 8 to 10 . As the fraudulent preliminary operation, a change in the line-of-sight direction of the self-registration user is detected.
The self-registration monitoring method according to claim 11 . In the self-registration monitoring method according to any one of claims 8 to 12 ,
The specific motion detection step and the suspicious motion determination step
A program for monitoring unauthorized operations to be executed by an arithmetic processing unit including the operation detection unit and the suspicious operation determination unit.

Images