JP2022187215A - Operation determination program, operation determination method, and operation determination device - Google Patents

Abstract

To accurately determine whether an operation to an object from a person is normal or not.SOLUTION: An operation determination device 1 acquires a photographed image 2 obtained by photographing an operation of a person 3a. Then, the operation determination device 1 detects a position of an object 3b related to the person 3a from the acquired photographed image 2. Along with this operation, the operation determination device 1 detects skeleton information of the person 3a from the acquired photographed image 2. The operation determination device 1 determines whether the operation performed by the person 3a to the object 3b is normal or not on the basis of the detected position of the object 3b and the detected skeleton information. Consequently, whether the operation is normal or not can be accurately determined.SELECTED DRAWING: Figure 1

Description

The present invention relates to a motion discrimination program, a motion discrimination method, and a motion discrimination device.

Image recognition technology for recognizing a specific object from an image is widely used. In this technique, for example, a specific object region in an image is specified as a bounding box. There is also a technique for performing image recognition of an object using machine learning. Such image recognition technology is being considered to be applied to, for example, the monitoring of customers' purchasing behavior in stores and the work management of workers in factories.

Japanese Patent Application Laid-Open No. 2020-53019 JP 2014-132501 A

By the way, when judging from an image whether or not a person's movement toward an object is normal, the position and movement of the person along with the position of the object can be recognized from the image. It becomes possible to discriminate fine differences between However, the above image recognition technology for recognizing a specific object can only recognize the object from the image, and cannot recognize the person together with the object. As a technology for recognizing a person from an image, there is a pose estimation technology, but this technology cannot recognize an object.

In one aspect, an object of the present invention is to provide a motion discrimination program, a motion discrimination method, and a motion discrimination device capable of highly accurately discriminating whether or not a motion performed by a person on an object is normal.

In one proposal, a photographed image obtained by photographing a person's movement is acquired in a computer, the position of an object related to the person and the skeleton information of the person are detected from the acquired photographed image, and the position of the object and the skeleton information are detected. There is provided a motion determination program for executing a process for determining whether or not a motion performed by a person on an object is normal based on and.

Also, in one proposal, there is provided a motion discrimination method in which a computer executes processing similar to the processing based on the above motion discrimination program.
Furthermore, one proposal provides a motion discrimination device that executes the same processing as the processing based on the above motion discrimination program.

In one aspect, it can be determined with high accuracy whether or not the action performed by a person on an object is normal.

BRIEF DESCRIPTION OF THE DRAWINGS It is a figure which shows about the motion discrimination|determination apparatus which concerns on 1st Embodiment. It is a figure which shows the structural example of the customer monitoring system which concerns on 2nd Embodiment. It is a figure which shows the hardware structural example of a monitoring apparatus. FIG. 10 is a diagram showing a comparative example of recognition of a person and an object by HOID; FIG. 2 shows examples of beverage cans and packaged goods; It is a figure which shows the example of an image when the wrong purchasing action is performed. FIG. 10 is a diagram showing an example of recognition processing using first and second comparative examples and its application; It is a figure which shows the structural example of the processing function with which a monitoring apparatus is provided. FIG. 4 is a diagram showing an example of data generated by an image feature extraction unit and a skeleton information extraction unit; It is a figure which shows the internal structural example of a purchasing action extraction part. It is a figure for demonstrating the process of the determination part as a predictor. It is an example of the flowchart which shows the learning processing procedure of a predictor (determination part). It is an example of the flowchart which shows the determination processing procedure using a predictor. It is an example of the flowchart which shows the learning processing procedure of a discriminator (determination part). It is an example of the flowchart which shows the determination processing procedure using a discriminator. FIG. 13 is a diagram illustrating a configuration example of processing functions provided in a monitoring device according to a fourth embodiment; FIG. It is a figure for demonstrating the determination rule which determines whether normal purchasing action was performed. It is a figure which shows the correction processing example by an environment difference correction|amendment part. FIG. 4 is a diagram showing an example internal configuration of a person difference correcting unit; FIG. 5 is a diagram for explaining correction processing by a person difference correction unit; FIG. 11 is an example (part 1) of a flowchart showing a determination processing procedure based on a determination rule; FIG. FIG. 11 is an example (part 2) of a flowchart showing a determination processing procedure based on a determination rule; FIG. FIG. 11 is an example of a flowchart showing a procedure of counting processing of the number of scan points; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[First embodiment]
FIG. 1 is a diagram showing a motion discrimination device according to a first embodiment. A motion determination device 1 shown in FIG. 1 is a device that determines whether or not a person's motion with respect to an object is normal. The motion determination device 1 is realized as a computer such as a personal computer or a server device.

The action determination device 1 acquires a photographed image 2 of a person's action (step S1). In this photographed image 2, a state in which a person 3a is making an action with respect to an object 3b is shown. In the photographed image 2 shown in FIG. 1, as an example, a person 3a holds an object 3b in his/her hand and brings it close to the reading device 3c, and the action of causing the reading device 3c to read the identification information added to the surface of the object 3b is shown. It is assumed that there is

The motion determination device 1 detects the position of the object 3b related to the person 3a from the acquired photographed image 2 (step S2a). In this detection, for example, a person 3a is recognized from the photographed image 2, an object 3b interacting with the recognized person 3a is recognized, and the positions of these person 3a and object 3b are detected. . In FIG. 1, as an example, an image area 4a including a person 3a and an image area 4b including an object 3b are detected. Such detection can be realized using, for example, HOID (Human Object Interaction Detection).

HOID recognizes a person and an object from one given image, and detects the person, the object, and the type of interaction when the person interacts with the object. Therefore, when HOID is applied, for example, when there is an object with which the person interacts in the captured image 2, the person, the object, and the type of interaction are detected. An object (corresponding to object 3b) related to the person (corresponding to person 3a) is then identified based on the person, the object, and the type of interaction. For example, when a user picks up a book, the user, the book, and the holding of the book are detected. The interaction includes all interaction that can be recognized from the image regardless of whether the user is directly conscious, unconscious, contact or non-contact.

Along with this, the motion determination device 1 detects the skeleton information of the person 3a from the captured image 2 (step S2b). In this detection, for example, the positions of a plurality of predetermined joints of the person 3a are detected. In FIG. 1, as an example, the positions of left and right shoulders, elbows and wrists are detected as joints of the person 3a. 5b shows a line connecting the position of the right elbow and the position of the right wrist. A skeleton line 6a indicates a line connecting the position of the left shoulder and the position of the left elbow, and a skeleton line 6b indicates a line connecting the position of the left elbow and the position of the left wrist.

Based on the position and skeleton information of the object 3b detected in this manner, the motion determination device 1 determines whether or not the motion performed by the person 3a on the object 3b is normal (step S3). In the example of FIG. 1, it is determined whether or not the operation of causing the reading device 3c to read the identification information added to the object 3b by the person 3a was performed normally.

With the above processing, it is possible to accurately determine whether or not the action performed by the person 3a on the object 3b is normal.
For example, in the present embodiment, the position of an object related to a person is recognized instead of simply recognizing the position of the object from the captured image 2 . As a result, an object held by a person can be reliably recognized as in the example of FIG. 1, and even if an object unrelated to the person is captured, it is not recognized. For example, image recognition technology for recognizing a specific object from an image cannot extract and recognize only objects related to people in this way.

Furthermore, in the present embodiment, by using the skeleton information of the human body together with the position of the object, it is possible to discriminate fine differences in the movement of the person between the normal state and the abnormal state. For example, when the operation to read the identification information added to the object 3b by the reading device 3c is performed normally and when it is not, the transition of the position of the object 3b and the transition of the positions of each joint of the person 3a are different. different. As a case where this operation is not performed normally, for example, it is conceivable that the angle of the object 3b when the object 3b is brought closer to the reading device 3c is different from that in the normal state. In this case, the state of the joints of the hand holding the object 3b (for example, relative positions between the joints) is also considered to be different from the normal state. In the present embodiment, by using skeleton information, it is possible to discriminate such fine differences in the movement of a person.

[Second embodiment]
Next, as a second embodiment, a case in which the processing of the action determination device 1 is applied to a customer monitoring system in a store will be described.

FIG. 2 is a diagram showing a configuration example of a customer monitoring system according to the second embodiment. The customer monitoring system shown in FIG. 2 is a system for monitoring the purchasing behavior of customers in a store where merchandise is sold, and includes a monitoring device 100 and a camera 101 connected to the monitoring device 100 . Note that the monitoring device 100 is an example of the motion determination device 1 shown in FIG.

Camera 101 is installed in a store where cash register 50 is installed. The cash register 50 is a POS terminal included in a POS (Point Of Sale) system. In addition, the cash register 50 is a self-type cash register in which the customer himself/herself performs a checkout operation, and is sometimes called a "self-checkout".

The cash register 50 includes a barcode scanner 51 , a display 52 and a deposit/withdrawal unit 53 . The barcode scanner 51 reads the barcode indicating the product code attached to the product. The display 52 displays the price of the product whose barcode has been read, the total price of the product to be purchased, the amount of change, and the like. The deposit/withdrawal unit 53 receives deposits from customers and dispenses change.

For example, the customer approaches the cash register 50 with an in-store basket containing products to be purchased, and brings the products in the basket one by one to the barcode scanner 51 to read the barcode ("scanning operation"). I do. When the customer completes the scanning operation for all the products, the customer performs "settlement operation" to request settlement. For example, if the display 52 is a touch panel, the customer can perform a checkout operation by pressing a checkout button on the touch panel. The customer who has performed the settlement operation puts the purchase amount into the deposit/withdrawal unit 53 according to the information displayed on the display 52, and receives the change from the deposit/withdrawal unit 53, if any.

The camera 101 photographs the front surface of the cash register 50 (particularly, the area around the barcode scanner 51) so that the purchase behavior of the customer using the cash register 50 can be captured. The monitoring device 100 can determine whether or not the customer made a correct purchase action from the image captured by the camera 101, and issue a warning when it is determined that the customer made an abnormal purchase action.

FIG. 3 is a diagram illustrating a hardware configuration example of a monitoring device. The monitoring device 100 is implemented as a computer as shown in FIG. 3, for example. Monitoring device 100 shown in FIG. It has an interface (I/F) 117 and a communication interface (I/F) 118 .

The processor 111 centrally controls the monitoring device 100 as a whole. The processor 111 is, for example, a CPU (Central Processing Unit), MPU (Micro Processing Unit), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), or PLD (Programmable Logic Device). Also, the processor 111 may be a combination of two or more of CPU, MPU, DSP, ASIC, and PLD.

A RAM 112 is used as a main storage device of the monitoring device 100 . The RAM 112 temporarily stores at least part of an OS (Operating System) program and application programs to be executed by the processor 111 . Also, the RAM 112 stores various data necessary for processing by the processor 111 .

The HDD 113 is used as an auxiliary storage device for the monitoring device 100 . The HDD 113 stores an OS program, application programs, and various data. Other types of non-volatile storage devices such as SSDs (Solid State Drives) can also be used as auxiliary storage devices.

A display device 114 a is connected to the GPU 114 . The GPU 114 causes the display device 114a to display an image according to instructions from the processor 111 . Display devices include a liquid crystal display and an organic EL (ElectroLuminescence) display.

An input device 115 a is connected to the input interface 115 . The input interface 115 transmits signals output from the input device 115 a to the processor 111 . The input device 115a includes a keyboard, pointing device, and the like. Pointing devices include mice, touch panels, tablets, touch pads, trackballs, and the like.

A portable recording medium 116 a is attached to and detached from the reading device 116 . The reading device 116 reads data recorded on the portable recording medium 116 a and transmits the read data to the processor 111 . The portable recording medium 116a includes an optical disk, a magneto-optical disk, a semiconductor memory, and the like.

The network interface 117 transmits and receives data to and from another device via the network 117a.
A communication interface 118 transmits and receives data to and from the camera 101 .

The processing functions of the monitoring apparatus 100 can be realized by the hardware configuration as described above.
By the way, self-service cash registers are rapidly becoming popular for purposes such as coping with labor shortages due to population decline, relieving congestion, and preventing virus infection. A problem with self-service cash registers is that customers may make erroneous purchases, resulting in unpaid bills. Such erroneous purchasing actions may be negligent or intentional, and there are various types of actions as follows.

An erroneous purchasing operation caused by negligence includes, for example, a "scan failure" in which a customer forgets to scan an item and moves the item directly from the basket for use in the store to the customer's bag. There is also a "cart leak" where the customer forgets to scan the items in the lower basket when the in-store baskets can be placed on the top and bottom of the shopping cart.

On the other hand, as an intentional erroneous purchasing action, for example, there is "barcode hiding" in which a customer hides only the barcode with his or her finger while pretending to perform a scanning operation. In addition, when multiple items of the same product are set in a package, there is also a "barcode scan error" in which the barcode of one product exposed from the package is scanned instead of the barcode on the package. be.

In these erroneous purchase actions, the appearance of the product and the movement of the body differ depending on the type, so it can be said that it is technically difficult to automatically recognize all types of actions as erroneous purchase actions. Therefore, for example, a method of recognizing a purchase action using a plurality of types of sensors or a plurality of sensors of a single type is conceivable. However, since this method requires a high equipment cost, it is desired to be able to recognize using only a single camera.

Here, a technique for recognizing a specific product from an image captured by a camera is commonly used. In this technique, for example, the product area (eg, bounding box) in the image is specified. There is also a technique for performing image recognition of such products using machine learning. However, with these techniques, templates and learning data for each product must be prepared in advance. Therefore, it is not realistic to use these technologies in stores that have a large number of products and frequently introduce new products. In addition, this technique only identifies the image area of a specific product, and cannot recognize the relationship with a person.

On the other hand, people tend to have image data that is easy to collect and often have a large size in the image. Therefore, a person can be easily recognized by machine learning using a neural network (NN), for example, compared with objects other than the person. Therefore, it is possible to reduce the difficulty of recognizing objects related to people by first recognizing a person from an image and then recognizing an object by focusing on the position of the person's body part (for example, the position of the hand). can. HOID is a technique for recognizing a person and related objects based on such a concept.

FIG. 4 is a diagram showing a comparative example of recognition of a person and an object by HOID. In HOID, objects are recognized using information about people on images. Among the objects on the image, only objects that interact with the person are recognized, and objects that do not interact are ignored.

An image 200 illustrated in FIG. 4 shows a person 201 holding an object 202 in his or her hand. When this image 200 is input to the learning model learned by HOID, for example, the position information of the person, the position information of the object with which the person interacts, the class name of the object, and the class name of the interaction and the confidence score for that class name.

Positional information for people and objects is output as bounding boxes that indicate, for example, rectangular regions that circumscribe those regions. A bounding box 203 indicating the position of the person 201 and a bounding box 204 indicating the position of the object 202 are detected from the image 200a illustrated in FIG. In this example, a class name indicating "has" is output as the interaction class name. Confidence score is actually related to the object class name when there is a person area and an object area, the object belongs to a certain object class name, and the interaction class name between the object and the person. It may represent a confidence score (probability value) of the fact.

By applying this HOID, it is possible to recognize a product related to a person from the image without registering the product image in advance, as long as the product has an appearance that does not assimilate with the background or hand. That is, it is possible to recognize an arbitrary object from an image as something (any object other than a person whose existence is unknown whether it is an object registered in advance) and estimate a bounding box indicating the area of the object. Hereinafter, such person and object recognition processing will be referred to as a “first comparative example”.

On the other hand, there is posture estimation technology as a technology for recognizing a person from an image. Posture estimation technology is a technology for estimating the posture of a human body by detecting skeletal information of the human body. As the skeletal information, for example, the positions on the image of a plurality of predetermined joints included in the human body are detected. Hereinafter, the person recognition processing based on skeleton detection will be referred to as a “second comparative example”.

Here, an example of an erroneous purchase operation using the self-service cash register 50 will be shown, and the process of recognizing this purchase operation using the first and second comparative examples will be shown. Here, the aforementioned "barcode scanning error" is exemplified as an erroneous purchasing operation. In addition, beverage cans and their package products are exemplified as products to be purchased.

FIG. 5 is a diagram showing examples of beverage cans and packaged goods. A beverage can 211 shown in FIG. 5 is a can containing a beverage such as beer. A bar code 212 indicating a product code is attached to the side surface of the beverage can 211 . A package product 213 shown in FIG. 5 is a product in which a plurality of beverage cans 211 are sold as a set, and the plurality of beverage cans 211 are stored inside the exterior. A bar code 214 indicating the product code of the package product 213 is added to the exterior surface.

In many of such packaged products 213, a part of the exterior such as both ends is opened, and a part of the beverage can 211 inside is exposed from the opening. Therefore, as shown in FIG. 5, the customer may intentionally cause the barcode scanner 51 of the cash register 50 to scan the barcode 215 attached to the beverage can 211 exposed from the exterior of the packaged product 213 . In this case, multiple beverage cans are illegally purchased for the price of one.

For example, as a method of detecting fraudulent purchasing behavior, there is a method of collating the number of products that have been scanned and the number of products that have been taken out. However, when such a "barcode scan error" purchase operation is performed, the number of scanned products matches the number of products taken out, so the fraudulent operation cannot be easily detected.

FIG. 6 is a diagram showing an example of an image when an erroneous purchasing action is performed. Images 221 and 222 shown in FIG. 6 show the situation when the purchase operation of "barcode scanning error" is performed for the package product 213 as described above.

An image 221 shows a state in which the customer has taken out the package product 213 from the basket 216 for the store. An image 222 shows a state in which the customer then brings the package product 213 held in his/her hand close to the bar code scanner 51 and performs a scanning operation. However, in image 222 , the customer is trying to have barcode scanner 51 scan the barcode attached to the beverage can within package 213 , not the barcode attached to package 213 . When performing such a scanning operation, the customer rotates the packaged product 213 held in his or her hand and moves it to a position where the bar code of the beverage can can be scanned. conduct. For example, in order to rotate the package product 213, the arm, fingers (thumb), and the like move in a unique manner different from normal movements.

Therefore, if such an action by the customer can be recognized from the image at least as an action different from the normal operation, a warning can be issued when such an action is performed. However, as shown in the following FIG. 7, when the first comparative example using HOID and the second comparative example using skeleton detection are used, the above action is recognized as an erroneous purchase action. difficult to do.

FIG. 7 is a diagram showing an example of recognition processing using the first and second comparative examples and its application.
Images 221a and 222a shown on the upper side of FIG. 7 show the case where the position of the product is recognized by applying the first comparative example based on the images 221 and 222 of FIG. In this case, HOID first finds a bounding box (not shown) indicating the area of the person and then finds the bounding box 217 indicating the area of the item that is correlated with this person. In reality, the product is recognized as some kind of object (Something) held by a person, but since the person is holding it in front of the cash register 50, it can be identified as a product.

Thus, when the first comparative example using HOID is applied, the position of the product can be detected from the captured image. Therefore, it is possible to recognize that the customer brought the product closer to the barcode scanner 51 . However, since the movement of the customer's body cannot be detected, it is not possible to recognize the characteristic movement of the customer's body (for example, movement of the finger) when rotating the package product 213 . Therefore, the customer's purchasing behavior as described above cannot be distinguished from normal purchasing behavior.

On the other hand, images 221b and 222b shown on the lower side of FIG. 7 show the case where the movement of a person is recognized by applying the second comparative example based on the images 221 and 222 of FIG. In this case, the positions of the joints of the person are detected by skeleton detection. In the images 221b and 222b, thick lines 218a to 218c represent lines connecting the wrist and the joint of the thumb.

In this way, when the second comparative example using skeleton detection is applied, it is possible to detect, from the captured image, specific body movements (for example, finger movements) when rotating the product due to unauthorized scanning operations. is. However, since the position of the product cannot be detected, the detected body movement cannot be associated with the product and recognized as "movement while holding the product".

Therefore, the monitoring apparatus 100 of the present embodiment combines the technique of the first comparative example using HOID and the technique of the second comparative example using skeleton detection, so that the position of the product and the uniqueness of the body can be detected. motion can be detected. As a result, the monitoring device 100 can recognize the product held by the customer in front of the cash register 50 without registering images of many products in advance. The monitoring apparatus 100 can accurately recognize the movement trajectory of the product and how it appears in the image, as well as the body movement of the customer who carries the product and performs the scanning operation. As a result, the monitoring device 100 can distinguish various erroneous purchase actions including the above-mentioned "barcode scan error" from normal purchase actions.

FIG. 8 is a diagram illustrating a configuration example of processing functions included in the monitoring apparatus. As shown in FIG. 8, the monitoring device 100 includes an image acquisition unit 121, an image feature extraction unit 122, a skeleton information extraction unit 123, a purchasing behavior extraction unit 124, a learning unit 125, a determination unit 126, an image storage unit 131, and a learning model. A storage unit 132 is provided. In the monitoring device 100, the processes of the image acquiring unit 121, the image feature extracting unit 122, the skeleton information extracting unit 123, the purchasing motion extracting unit 124, the learning unit 125, and the determining unit 126 are performed by the processor 111 provided in the monitoring device 100, for example. It is realized by executing the program of The image storage unit 131 and the learning model storage unit 132 are implemented by a storage area of a storage device included in the monitoring apparatus 100, such as the RAM 112 and the HDD 113. FIG.

The image acquisition unit 121 acquires data of moving images captured by the camera 101 . At the time of learning, the data of the acquired moving image is stored in the image storage unit 131 as learning data, and then input to the image feature extraction unit 122 and the skeleton information extraction unit 123 read out from the image storage unit 131. be. On the other hand, at the time of determining a purchasing action, the acquired moving image data is sequentially input to the image feature extraction unit 122 and the skeleton information extraction unit 123 .

The image feature extraction unit 122 inputs the input moving image into a HOID learning model (NN here), and extracts information on a person and information on products interacting with the person. Specifically, the appearance information of the product and the position information of the person and the product are extracted.

The skeleton information extraction unit 123 performs skeleton detection from the input moving image and extracts the skeleton information of the person.
Purchasing action extracting unit 124 generates a feature amount indicating a purchasing action based on the information extracted from the moving image by image feature extracting unit 122 and skeleton information extracting unit 123 .

The learning unit 125 learns a discriminator for discriminating between a normal buying action and an abnormal buying action using the feature amount extracted by the purchasing action extracting unit 124 from many moving images as learning data. In this embodiment, it is assumed that deep learning using NN is executed. The learning unit 125 stores data indicating the learning model (NN) obtained by learning in the learning model storage unit 132 .

The determination unit 126 operates as a discriminator based on the learning model data stored in the learning model storage unit 132 . The determining unit 126 distinguishes between normal purchasing actions and abnormal purchasing actions using the feature amount extracted by the purchasing action extracting unit 124 from the moving image captured by the camera 101, and detects abnormal purchasing actions. issue a warning if necessary.

In this embodiment, moving images obtained by recording actual purchasing actions are used as learning data without being labeled. Among such moving images, most are moving images when normal purchasing actions are performed, and a very small number are moving images when abnormal buying actions are performed. However, ideally, only moving images of normal purchasing actions may be used. As an example of a classifier, the learning unit 125 learns a predictor that predicts the feature amount at the next time from the feature amount at a certain time based on such moving images. Therefore, the decision unit 126 operates as such a predictor.

FIG. 9 is a diagram showing an example of data generated by the image feature extraction section and the skeleton information extraction section. The image feature extraction unit 122 and the skeleton information extraction unit 123 generate the following data each time frame data is input.

The image feature extraction unit 122 outputs HOID information by inputting the frame to the HOID learning model (NN). The HOID information includes information indicating interaction between a person and an object recognized from the frame. The information indicating interaction includes, for example, a person ID that identifies a person, an object ID that identifies an object that interacts with the person, and an action ID that indicates the type of action of the person with respect to the object.

In the present embodiment, the image feature extraction unit 122 outputs HOID information including a motion ID indicating "holding an object" as the type of motion. For example, when a plurality of combinations of a person and an object interacting with each other are extracted from the captured image, only HOID information for the combination indicating that the action ID is "holding the object" is output.

The HOID information further includes information about a person and information about an object. The information about the person includes the person ID and the position information of the person area. The information about the object includes an object ID, position information of the object area, and an object type ID indicating the type of the object. As the position information of the human region and the object region, for example, information indicating the positions of bounding boxes respectively indicating the human region and the object region (for example, the coordinates of the four corners) is used.

The image feature extraction unit 122 generates appearance information and position information about the object from the HOID information, and outputs the information to the purchase action extraction unit 124 . Appearance information about an object includes an image of the object region cut out from the frame. This appearance information may include, for example, information indicating the color, shape, and size of the object. The positional information about the object includes the positional information of the object area (bounding box). This position information includes, for example, the coordinates of the four corners of the object area. Also, the appearance information may further include information indicating the relative positional relationship between the object area and the person area.

In practice, it is desirable that supplementary processing be executed so that the product can be accurately recognized as an object by the HOID. For example, an ROI (Region Of Interest) is set at a predetermined position in the captured image, and the product is accurately recognized as the object based on the positional relationship between the recognized object and the ROI. As an example, in front of the cash register 50, a region in which a product is taken out from a basket for use in the store and a region in which the product is temporarily placed are set as ROIs. The image feature extraction unit 122 recognizes the object as a product when the position of the object newly recognized by the HOID is within the ROI, and when the position of the newly recognized object is within the bounding box of the person. , that the object is not a commodity (eg, a customer's personal item such as a wallet). Further, the image feature extraction unit 122 may distinguish the product as the object based on the image change (background difference etc.) in the ROI before and after the object enters the ROI.

On the other hand, the skeleton information extraction unit 123 generates the skeleton information of the person from the frame and outputs it to the purchasing motion extraction unit 124 . The skeleton information includes, for each detected joint point, a combination of a joint point ID for identifying a joint point (center point of the joint) and position information (coordinates) of the joint point. In addition, the skeleton information may include a joint point reliability score as information for each joint point.

Note that the skeleton information of a person is obtained by, for example, comparing the position of each joint point with the position of the person region or object region detected by the image feature extraction unit 122, and the person or object region detected by the image feature extraction unit 122. Associated with an object. When the bounding box of the person detected by the image feature extraction unit 122 includes the joint points (for example, a predetermined number or more of joint points) detected by the skeleton information extraction unit 123, the person and joints corresponding to the bounding box are identified. The person corresponding to the point is determined to be the same person. In this case, the person ID of the person corresponding to the bounding box is added to the joint point data.

FIG. 10 is a diagram showing an example of the internal configuration of a purchasing action extraction unit. As shown in FIG. 10 , the purchasing behavior extraction unit 124 includes an environment difference correction unit 141 , a feature amount calculation unit 142 and a time series information processing unit 143 .

The environment difference correction unit 141 corrects the external appearance information and position information of the object input from the image feature extraction unit 122 and the skeleton information of the person input from the skeleton information extraction unit 123 according to the shooting environment of the camera 101. do. For example, based on the relative distance between the camera 101 and the cash register 50 and the number of pixels in the photographed image, the distance at a reference position in the photographing space (for example, a predetermined location on the surface of the cash register 50) and the number of pixels on the image. Appearance information and position information of an object and skeleton information of a person are corrected so that they are always constant.

The feature amount calculation unit 142 calculates feature amount vectors from each of the object appearance information, the object position information, and the person's skeleton information, and integrates these feature amount vectors into one feature amount vector. The feature amount calculation unit 142 includes an appearance feature extraction unit 151 , an object position feature extraction unit 152 , a skeleton feature extraction unit 153 and a feature amount vector integration unit 154 .

The appearance feature extraction unit 151 converts the corrected appearance information of the object into a feature amount vector to calculate an appearance feature amount vector. For example, a feature amount vector is generated from appearance information (a partial image cut out from a photographed image) by the ROI Align method.

The object position feature extraction unit 152 converts the corrected object position information into a feature amount vector to calculate an object position feature amount vector. For example, the center coordinates, width, height, size, aspect ratio of the object region (bounding box), the ratio of the size of the object region to the human body region, and the overlap ratio between the human body region and the object region (Intersection over Union: IoU) , the distance between the center of the human body region and the center of the object region, and the relative coordinates between the object region and a reference position in the shooting space (for example, a table on which goods are temporarily placed next to the cash register 50). one is vectorized. Then, by linearly transforming the generated vector, an object position feature amount vector is generated.

The skeletal feature extraction unit 153 converts the corrected human skeletal information into a feature amount vector to calculate a skeletal position feature amount vector. For example, the coordinates of each joint point, the reliability score of each joint point, the relative coordinates between one joint point and another joint point, the relative coordinates between each joint point and the reference position in the shooting space, each joint point and the object area. At least one of the coordinates relative to the center of is vectorized. Then, by linearly transforming the generated vector, a skeleton position feature vector is generated.

The feature amount vector integration unit 154 integrates the calculated appearance feature amount vector, object position feature amount vector, and skeleton position feature amount vector to calculate an integrated feature amount vector. In this vector integration, each feature vector is integrated for each joint point. That is, the skeletal position feature vector, the object position feature vector, and the appearance feature vector for one joint point are integrated to calculate an integrated feature vector for the joint point. is calculated for each joint point. Thereby, a feature amount vector is generated as learning data that enables accurate learning of the motion of each joint point.

The feature amount vector integration unit 154 integrates feature amount vectors of a plurality of dimensions into feature amount vectors of the total number of dimensions by, for example, concatenation, and performs linear transformation or nonlinear transformation to calculate an integrated feature amount vector. do. For example, when the appearance feature amount vector is x ₁ , the object position feature amount vector is x ₂ , and the skeletal position feature amount vector of joint point i is y _i , the integrated feature amount vector z _i of joint point i is expressed by the following equation ( 1).
z _i =W ₂ *σ(W ₁ *[x ₁ ,x ₂ ,y _i ]) (1)
In equation (1), W ₁ and W ₂ represent predetermined weighting coefficients, [ ] represents concatenation, σ represents nonlinear transformation, and * represents an inner product.

The integrated feature amount vector for each joint point generated by the above procedure may be directly input to the learning unit 125 and the determination unit 126 . On the other hand, in the present embodiment, these integrated feature amount vectors are processed by time-series information processing section 143 and then input to learning section 125 and determination section 126 . The time-series information processing unit 143 processes the generated integrated feature amount vector for each joint point according to temporal continuity, thereby generating a feature amount vector representing the entire purchase action. This processing is performed using, for example, ST-GCN (Spatial Temporal-Graph Convolutional Networks). In ST-GCN, both spatial patterns and temporal patterns based on joint point positions are learned from integrated feature vector for each joint point.

Next, learning section 125 and determination section 126 will be described. As described above, in the present embodiment, the determining unit 126 operates as a predictor that predicts the feature quantity at the next time from the feature quantity at a certain time. The learning unit 125 learns such a predictor.

FIG. 11 is a diagram for explaining the processing of the determination unit as a predictor. A feature amount space 231 shown in FIG. 11 is a simple representation of a coordinate space having each dimension of the integrated feature amount vector output from the purchasing action extraction unit 124 as a two-dimensional coordinate space. Here, considering the frame period as a unit time, the change in the feature amount between a certain time T and time (T+1) is considered. If the frame at time T is the current frame, the frame at time (T+1) is the next frame.

The learning unit 125 learns a predictor that predicts feature amounts from time T to time (T+1) based on integrated feature amount vectors generated from moving images obtained by recording actual purchasing actions. Such learning is performed using, for example, an RNN (Recurrent Neural Network) as a learning model.

Here, labels indicating normality/abnormality may not be added to the integrated feature vector as learning data input to the learning unit 125 . However, the majority of such moving images are moving images when normal purchasing actions are performed, and the moving images when abnormal purchasing actions are performed are very few. Therefore, the learning unit 125 learns a predictor that predicts the feature amount when a normal purchase action is performed.

Then, the predictor learned in this way predicts the feature amount at time (T+1) when the feature amount at time T is input, when a normal purchasing action is performed. For example, if the product or the knuckle is in a certain position at time T, the predictor can predict where the product or the knuckle will move at time (T+1) if the purchase is normal. Conversely, when the feature amount at time T is input, the predictor cannot predict the feature amount at time (T+1) when an abnormal purchasing action is performed.

For example, in the feature amount space 231 shown in FIG. 11, when the feature amount FT at time _T is input to the predictor, the predictor selects the feature amount F' Predict _T+1 . However, if the input feature amount FT is a feature amount when an abnormal purchasing behavior is performed, the predicted feature amount F' _{T +1} at time (T+1) and the actual feature amount at time (T+1) A distance D in the feature amount space 231 between the feature amount F _T+1 is increased. Therefore, if the spatial distance D between the predicted feature amount F' _T+1 and the actual feature amount F _T+1 is greater than a predetermined threshold, the determination unit 126 determines whether the purchase is abnormal. It can be determined that an action has been performed.

According to this process, it is determined that an abnormal purchase action has been performed when the customer makes an action that is different from when the movement of the product or joints is normal, such as the above-mentioned "barcode scan error". It becomes possible to

Next, FIG. 12 is an example of a flowchart showing a learning processing procedure of the predictor (determination unit).
[Step S<b>11 ] The image acquisition unit 121 collects moving images in which the camera 101 captures the purchase action. These moving images include a relatively large number of moving images when normal purchasing actions are performed, and a small number of moving images when abnormal purchasing actions are performed. However, it is also possible to collect only moving images when a normal purchasing action is performed. The image acquisition unit 121 stores the collected data of each moving image in the image storage unit 131 .

[Step S12] For each moving image stored in the image storage unit 131, the processing of the feature generation loop up to step S18 is executed.
[Step S13] A frame processing loop up to step S17 is executed for each frame included in the moving image.

[Step S14] The image feature extraction unit 122 inputs the moving image data to the HOID learning model (NN) to calculate HOID information. The image feature extraction unit 122 extracts appearance information and position information of an object as feature amounts based on the HOID information. Also, the skeleton information extraction unit 123 performs skeleton detection from the moving image, and extracts the skeleton information of the person as a feature amount.

[Step S15] The environment difference correction unit 141 of the purchase action extraction unit 124 corrects the extracted feature amounts (appearance information, position information, and skeleton information) according to the shooting environment.
[Step S16] A feature amount vector is calculated based on the corrected feature amount.

Specifically, the appearance feature extraction unit 151 calculates an appearance feature amount vector based on the appearance information of the object. Also, the object position feature extraction unit 152 calculates an object position feature amount vector based on the position information of the object. Furthermore, the skeletal feature extraction unit 153 calculates a skeletal position feature vector based on the skeletal information of the person. Then, the feature amount vector integration unit 154 integrates the appearance feature amount vector, the object position feature amount vector, and the skeleton position feature amount vector, and calculates an integrated feature amount vector for each joint point. The time-series information processing unit 143 generates a feature vector representing the entire purchase action by processing the integrated feature vector for each joint point according to temporal continuity.

[Step S17] When the processes of steps S14 to S16 are executed for all the frames included in the moving image, the frame processing loop ends, and the process proceeds to step S18.

[Step S18] When the processes of steps S13 to S17 are executed for all the moving images stored in the image storage unit 131, the process of the feature generation loop ends, and the process proceeds to step S19.

[Step S19] The learning unit 125 learns a predictor that predicts the feature amount at time (T+1) from the feature amount at time T based on the feature amount vector representing the entire purchase action. The learning model (NN) data generated by this learning is stored in the learning model storage unit 132 .

FIG. 13 is an example of a flowchart showing a determination processing procedure using a predictor.
[Step S<b>21 ] The image acquisition unit 121 acquires frames of moving images shot by the camera 101 .

[Step S22] The image feature extraction unit 122 inputs the frame data to the HOID learning model (NN) to calculate HOID information. This frame is the frame acquired in the previous step S21 or step S29. The image feature extraction unit 122 extracts appearance information and position information of an object as feature amounts based on the HOID information. Also, the skeleton information extraction unit 123 performs skeleton detection from the moving image, and extracts the skeleton information of the person as a feature amount.

[Step S23] The environment difference correction unit 141 of the purchase action extraction unit 124 corrects the extracted feature amounts (appearance information, position information, and skeleton information) according to the shooting environment.
[Step S24] A feature amount vector is calculated based on the corrected feature amount.

Specifically, the appearance feature extraction unit 151 calculates an appearance feature amount vector based on the appearance information of the object. Also, the object position feature extraction unit 152 calculates an object position feature amount vector based on the position information of the object. Furthermore, the skeletal feature extraction unit 153 calculates a skeletal position feature vector based on the skeletal information of the person. Then, the feature amount vector integration unit 154 integrates the appearance feature amount vector, the object position feature amount vector, and the skeleton position feature amount vector, and calculates an integrated feature amount vector for each joint point. The time-series information processing unit 143 generates a feature vector representing the entire purchase action by processing the integrated feature vector for each joint point according to temporal continuity.

[Step S25] The determination unit 126 inputs the generated feature amount vector to a predictor based on the data of the learning model stored in the learning model storage unit 132, and predicts the feature amount vector of the next frame. The determination unit 126 temporarily stores the prediction result of the feature amount vector of the next frame in the RAM 112 .

[Step S26] If the prediction result of the current frame feature amount vector predicted based on the previous frame feature amount vector is stored in the RAM 112, steps S26 to S28 are executed. This prediction result is saved in the RAM 112 by the process of step S25 for the previous frame. The determination unit 126 acquires this prediction result from the RAM 112 and calculates the distance between the feature amount vector indicated by this prediction result and the feature amount vector calculated from the current frame in step S24.

[Step S27] The determination unit 126 compares the calculated distance with a predetermined threshold. If the distance exceeds the threshold, the process proceeds to step S28, and if the distance is less than or equal to the threshold, the process proceeds to step S29.

[Step S28] If the distance exceeds the threshold, it is determined that an abnormal purchasing action has been performed. The determination unit 126 executes a process of warning that an abnormal purchasing action has occurred.
For example, the determination unit 126 causes the display device 114a to display image information indicating that an abnormal purchasing action has occurred. If the monitoring device 100 can communicate with the cash register 50 , the determination unit 126 may display such image information on the display 52 of the cash register 50 .

Also, an audible warning may be given. For example, the determination unit 126 causes the speaker connected to the monitoring device 100 to output a warning sound warning that an abnormal purchasing behavior has occurred. In addition, when the clerk is wearing an earphone capable of listening to voice via wireless communication, the determination unit 126 transmits voice information warning that an abnormal purchasing action has occurred, and causes the earphone to output voice. to notify the store clerk of the occurrence of the abnormality.

[Step S<b>29 ] The image acquisition unit 121 acquires the next frame captured by the camera 101 .
[Step S30] The image feature extraction unit 122 or the skeleton information extraction unit 123 determines whether the same person as in the previous frame is detected. If the same person is detected, the process proceeds to step S22. On the other hand, when the same person is not detected (when the person moves out of the shooting area), the determination processing in FIG. 13 ends.

In the above-described second embodiment, the determination unit is capable of determining with high accuracy whether or not a correct purchase operation has been performed by learning based on the product detection result and the person's skeleton information detection result by HOID. 126 can be realized. In particular, it becomes possible to determine that a fraudulent purchase action in which the bar code of the product is actually scanned is not a correct purchase action, such as the barcode scan error described above.

[Third Embodiment]
In the third embodiment, part of the processing of the monitoring device 100 in the second embodiment is modified. In the second embodiment, the determination unit 126 operates as a predictor that predicts the feature amount at time (T+1) from the feature amount at time T, and predicts the feature amount at time (T+1) and the actual feature amount. Based on the difference from the amount, it was determined whether or not an abnormal operation had occurred. On the other hand, in the third embodiment, a moving image with a label indicating whether it is a normal operation or an abnormal operation is used as learning data, and a discriminator that clearly distinguishes between normal operation and abnormal operation is used. is learned. Then, the processing of the determination unit 126 is modified to operate as such a discriminator.

FIG. 14 is an example of a flowchart showing a learning processing procedure of the discriminator (determination unit). In FIG. 14, processing steps having the same processing contents as in FIG. 12 are given the same step numbers. In the discriminator learning process shown in FIG. 14, step S41 is executed between steps S11 and S12 of FIG. 12, and step S42 is executed instead of step S19 of FIG.

[Step S41] Each moving image collected in step S11 and stored in the image storage unit 131 is annotated. For example, when the discriminator discriminates between normal operation and abnormal operation, either a normal label indicating normal operation or an abnormal label indicating abnormal operation is added to the moving image data. Preferably, the abnormal label is added only to the frame of the moving image in which the abnormal buying action is performed, and the normal label is added to the frame of the other period. For example, in the moving image showing the action of "barcode scanning error" for the package product 213 described with reference to FIGS. An abnormal label may be added to frames during the period from when the packaged product 213 is started to rotate until the bar code of the beverage can is scanned.

If the discriminator further identifies types of abnormal motions, labels for each type of abnormal motion are added to the moving image as abnormal labels. For example, four types of abnormal labels are used to identify the four types of abnormal operations of "scan omission", "cart omission", "barcode hiding", and "barcode scanning error". In this case, one of five labels, ie, a normal label and four types of abnormal labels, is attached to each moving image frame.

It should be noted that in step S11, unlike the case of FIG. 12, it is desirable to collect a certain number or more of moving images when an abnormal purchasing action is performed. In particular, when making it possible to identify the types of abnormal actions as described above, it is desirable to collect a certain number of moving images when the corresponding type of abnormal action is performed for each type of abnormal action.

[Step S42] To the learning section 125, the feature amount vector of each moving image calculated by the purchasing action extraction section 124 is input with the above label added. The learning unit 125 learns a classifier that identifies a purchasing action based on the input feature amount vector and label. The learning model (NN) data generated by this learning is stored in the learning model storage unit 132 .

FIG. 15 is an example of a flowchart showing a determination processing procedure using a discriminator. In FIG. 15, processing steps having the same processing contents as in FIG. 13 are given the same step numbers. In the determination process shown in FIG. 15, steps S51 to S53 are executed instead of steps S25 to S28 in FIG.

[Step S51] The determination unit 126 inputs the generated feature amount vector to a discriminator based on learning model data stored in the learning model storage unit 132, and determines whether the motion is normal or abnormal.

[Step S52] If an abnormal operation has been performed, the process proceeds to step S53, and if a normal operation has been performed, the process proceeds to step S29.
[Step S53] The determination unit 126 executes a process of warning that an abnormal purchasing behavior has occurred. The warning method can be the same as in step S28 of FIG.

Here, it is assumed that the discriminator has also discriminated the type of abnormal operation. In this case, it is also possible to issue a warning according to the type of abnormal operation. For example, when the warning is given by display information or voice as in step S28 of FIG. 13, the type of abnormal operation is notified by the display information or voice. It is also possible to change the warning method depending on the type of abnormal operation. For example, when an abnormal operation such as "scan omission" or "cart omission" occurs due to the customer's negligence, the customer is warned by the display 52 of the cash register 50 or by voice output, and the customer is asked to redo the scanning operation. prompt. On the other hand, when an abnormal operation is intentionally performed by the customer, such as "barcode hiding" or "barcode scan error", a warning is issued to the store clerk by means of display information or voice.

In addition, in order to prevent trouble with the customer, when an intentional abnormal operation is performed, the current time, the identification number of the cash register 50, the type of abnormal operation, etc. are not issued without giving a warning that would be noticed by the customer. A process such as storing the information in a storage device or storing data of a captured moving image in a storage device as evidence may be performed.

In the above-described third embodiment, by learning based on the product detection result and the person's skeleton information detection result by HOID, it is possible to identify an abnormal purchase behavior in which the product or hand joints move in a unique way. classifier can be generated. In particular, when the learning data is labeled for each type of abnormal motion, it is possible to generate a discriminator capable of explicitly identifying multiple types of abnormal motions. By using the results of product detection by HOID and the results of human skeletal information detection, it becomes possible to discriminate subtle differences in purchasing behavior, and a classifier that can identify multiple types of abnormal behavior with high accuracy. can be generated.

[Fourth Embodiment]
In the fourth embodiment, part of the processing of the monitoring device 100 in the second embodiment is modified. Specifically, it is determined whether or not the purchase operation was performed normally according to a predetermined determination rule based on the position on the image of the object (product) detected by the image feature extraction unit 122 . Further, based on the skeleton information detected by the skeleton information extraction unit 123, the contents of the determination rule are corrected according to the customer's physique and standing position.

FIG. 16 is a diagram illustrating a configuration example of processing functions included in a monitoring apparatus according to a fourth embodiment; In FIG. 16, processing functions that perform the same processing as in FIG. 8 are denoted by the same reference numerals. The monitoring apparatus 100a shown in FIG. 16 includes, in addition to the image acquiring unit 121, the image feature extracting unit 122, and the skeleton information extracting unit 123 shown in FIG. 163, a customer operation recognition unit 164, and a purchase operation determination unit 124a.

The determination rule storage unit 161 is implemented as a storage area of a storage device included in the monitoring device 100a. The judgment rule storage unit 161 stores information (judgment rule information) indicating a judgment rule for judging whether or not a purchase operation was performed normally. For example, as the determination rule information, information indicating a plurality of determination areas set on the image for recognizing from the position of the product that the customer has performed a specific action is stored.

The processes of the environment difference correction unit 162, the person difference correction unit 163, the customer operation recognition unit 164, and the purchase action determination unit 124a are realized, for example, by the processor included in the monitoring device 100a executing a predetermined program.

The environment difference correction unit 162 corrects the image acquired by the image acquisition unit 121 according to the shooting environment so that the determination rule can be applied correctly.
Person difference correction section 163 corrects the determination rule information stored in determination rule storage section 161 based on the skeleton information detected by skeleton information extraction section 123 . By this correction, the determination rule information is corrected according to the customer's physique and standing position.

The customer operation recognition unit 164 recognizes customer operations using the cash register 50 . For example, as the customer's operation, an operation of scanning a product and an operation of starting payment after completion of the scanning operation are recognized.

The customer operation recognition unit 164 recognizes these operations based on images captured by the camera 101, for example. For example, a lamp mounted on the cash register 50 may be lit when a scan operation or a settlement start operation is performed. A different lamp may be lit for each operation, or a lamp may be lit in a different color for each operation. In such a case, the customer operation recognition unit 164 can recognize that the above operation has been performed by detecting the lighting of the lamp and the color when the lamp is lit from the photographed image.

Further, the customer operation recognition unit 164 may recognize each operation by recognizing from the captured image that the display content of the display 52 of the cash register 50 changes according to each operation.

Further, for example, when different notification sounds are generated for each operation from the cash register 50, the customer operation recognition unit 164 detects the generation of these notification sounds via the microphone, thereby confirming that the above operation is performed. can recognize that Further, when the monitoring device 100a and the cash register 50 can communicate with each other, the customer operation recognition unit 164 may receive a notification from the cash register 50 to that effect when the above operation is performed.

Based on the position information of the object (product) output from the image feature extraction unit 122, the purchasing motion determination unit 124a determines whether a normal purchasing motion is performed according to the determination rule indicated by the determination rule information corrected by the person difference correction unit 163. Determine whether or not it has been done. Further, the purchasing operation determination unit 124a counts the number of scanning operations performed based on the determination result, and counts the number of scanning operations recognized by the customer operation recognition unit 164. FIG. The purchase action determination unit 124a collates the count values of both when the customer performs the checkout start operation, and if they do not match, executes a process of warning that the correct purchase action is not performed. .

FIG. 17 is a diagram for explaining a determination rule for determining whether or not a normal purchase operation has been performed.
An image 241 illustrated in FIG. 17 is an example of a photographed image acquired by the image acquisition unit 121 . This image 241 is an image of the vicinity of the front surface of the cash register 50 (the surface on which the barcode scanner 51 is mounted) taken from above the cash register 50 . The image 241 also illustrates a product area (bounding box) 242 for the product 213 indicated by the HOID information output from the image feature extraction unit 122 . Furthermore, the center position 243 of the product area 242 is also shown.

In the judgment rule, for example, a plurality of judgment areas are used for recognizing that each customer has performed a specific action. In this case, the determination rule information stored in the determination rule storage unit 161 includes information indicating the position of each of these determination areas. Here, as an example, it is assumed that two determination areas, ie, an extraction area R1 and a take-out area R2, are set. An image 241a shown in FIG. 17 is obtained by superimposing the extraction area R1 and the extraction area R2 on the image 241. As shown in FIG.

The take-out area R1 is an area for detecting the action of the customer picking up the product for the scanning operation. In many cases, the customer approaches the cash register 50 with the products to be purchased in the in-store basket 216, takes out the products one by one from the basket 216, and scans them. In this case, the takeout area R1 is an area for detecting that the customer has taken out the product from the basket 216 for scanning operation. Therefore, in the following description, the above operation to be detected based on the extraction area R1 will be referred to as "extraction operation". Here, when the center position 243 of the commodity area 242 newly enters the take-out area R1, it is determined that the take-out operation has started, and when the center position 243 moves outside the take-out area R1, the take-out operation ends. It shall be judged that

The retrieval area R1 is set at a position a certain distance from the front surface of the cash register 50 . In addition to this, in the example of FIG. 17, among the areas close to the front surface of the cash register 50, it is also set between the area where the barcode scanner 51 is arranged and the area where the basket 216 is placed. .

However, the take-out area R1 is set so as not to include the area where the basket 216 is placed. For example, in the basket 216, multiple products are often close together. Therefore, according to the HOID process, when the customer is carrying the basket 216, it may be detected that the customer has multiple items in the basket 216. FIG. Also, when taking out the product after placing the basket 216, if another product that is placed under another product is taken out, it may be erroneously detected that the customer has picked up the upper product that is not to be taken out. . Such erroneous detection can be prevented by setting the extraction area R1 so as not to include the area where the basket 216 is placed.

On the other hand, the take-out area R2 is an area for detecting the operation of scanning the product picked up by the customer, and is set at a position close to the bar code scanner 51 . This take-out area R2 can also be said to be an area for detecting that the customer has performed a normal operation to take out the product picked up by the customer. It should be noted that the take-out area R2 is set so that at least one of the boundary lines is in contact with the boundary line of the take-out area R1. Specifically, among the boundary lines of the carry-out area R2, the boundary line separating the customer and the cash register 50 (the boundary line on the cash register 50 side and parallel to the front surface of the cash register 50) is It is set so as to be in contact with the boundary line of the extraction area R1.

Based on the relationship between the take-out area R1 and take-out area R2, and the center position 243 of the product area 242, it is determined whether or not the purchase operation was performed normally according to the following determination rule, for example. The purchasing motion determination unit 124a recognizes that one product has been taken out (a taking-out motion has been performed) when the center position 243 moves into the taking-out region R1. When the center position 243 moves from the take-out region R1 to the take-out region R2 from this state, the purchase motion determination unit 124a recognizes that the scanning operation has been performed (the correct take-out motion has been performed). When such a series of actions are detected, it is determined that a normal purchase action has been performed.

On the other hand, if the central position 243 has not moved into the take-out area R2, it is determined that the purchase operation is not normal. For example, when the central position 243 moves from within the take-out region R1 to outside the take-out region R1 without moving to the carry-out region R2, it is assumed that the customer put the product in a pocket or bag without scanning it. be. In such a case, the purchasing action determination unit 124a can recognize that an abnormal buying action has been performed and issue a warning.

During the above determination, as a method of recognizing that the same product is being held in a plurality of consecutive frames, for example, a method of recognizing based on the trajectory of the position of the product, or a method of recognizing the product region A method of recognizing based on the image information of the image can be used. As a method using image information, for example, a method of recognition by comparing luminance or color histograms within a bounding box can be used.

In the above example, the purchase behavior is determined from the relationship between the set area on the image and the position of the product. may also be used to determine the purchasing behavior. For example, by detecting image changes in the take-out region R1, it is possible to accurately determine that the recognized object is a product, and trace the movement locus of the product.

FIG. 18 is a diagram illustrating an example of correction processing by the environment difference correction unit; FIG. 18A shows a first correction processing example, and FIG. 18B shows a second correction processing example.
The environment difference correction unit 162 corrects the image acquired by the image acquisition unit 121 according to the shooting environment so that the determination rule can be applied correctly and the determination process can be executed.

For example, the environment difference correction unit 162 corrects the captured image based on the shooting environment information in which information indicating the shooting environment of the camera 101 is recorded. The shooting environment information includes, for example, the number of pixels of an image shot by the camera 101, the shooting direction of the camera 101, the distance between the camera 101 and a reference position in the physical space (for example, a predetermined position in front of the cash register 50), and the like. be done. Reference values related to information recorded in the shooting environment information are registered in the determination rule storage unit 161, and the environment difference correction unit 162 compares the shooting environment information with the reference values to determine the number of pixels of the captured image. , the shooting direction, and the distance from the reference position to match the reference values.

Also, the environment difference correction unit 162 may perform such correction using a plurality of markers placed on the physical space instead of the shooting environment information. In this case, the determination rule storage unit 161 registers reference values for the positional relationship (inter-marker distance and relative position) between the markers on the captured image. The environment difference correction unit 162 detects the position of each marker captured in the captured image, and corrects the captured image so that the positional relationship between the markers on the captured image matches the reference value registered in the determination rule storage unit 161 . correct.

For example, FIG. 18A shows an example in which the inter-marker distance on the captured image 251 is smaller than the reference value. In this example, markers M1 to M4 are placed in the physical space, and these markers M1 to M4 are shown in the captured image 251. FIG.

Here, in the determination rule information, it is assumed that the boundary line L1 separating the customer and the cash register 50 among the boundary lines of the retrieval area R1 is set at a position 300 pixels from the upper end of the image. When the distance between the markers on the image 251 captured by the camera 101 is smaller than the reference value, if the position of the boundary line L1 set in the determination rule information is used as it is, it is determined whether or not the product is located in the take-out area R1. cannot be executed correctly. Therefore, the environmental difference correction unit 162 performs correction to enlarge the captured image 251 so that the inter-marker distance matches the reference value. By executing the determination process using the corrected (enlarged) captured image 251a, the determination process can be accurately performed.

In addition, when a marker is used, as shown in FIG. 18B, it is also possible to perform correction by rotating the photographed image. In the captured image 252 shown in FIG. 18B, the positional relationship between the markers M1 to M4 is compared with the reference values registered in the determination rule storage unit 161, and the camera 101 rotates with respect to the lens optical axis. I know there is. In such a case, the environmental difference correction unit 162 rotates the captured image 252 so that the angle relationship between the markers matches the reference value, and then selects the image region 253 such that the inter-marker distance matches the reference value. , is cut out from the rotated photographed image 252 . By such correction, a photographed image 252a after correction is obtained.

FIG. 19 is a diagram illustrating an example of an internal configuration of a person difference correction unit; As shown in FIG. 19 , the person difference correction section 163 includes an individual difference correction section 171 and a person position correction section 172 .
The individual difference correction unit 171 estimates the physique of the person from the skeletal information detected by the skeletal information extraction unit 123, and stores the physique of the person in the judgment rule storage unit 161 so that an accurate judgment can be executed regardless of the physique difference of the person. Correct the judgment rule information.

The person position correction unit 172 recognizes the position where the person stands (standing position) based on the position information of the person detected by the image feature extraction unit 122 or the skeleton information detected by the skeleton information extraction unit 123. . The person position correction unit 172 corrects the determination rule information stored in the determination rule storage unit 161 based on the recognized standing position so that accurate determination can be performed regardless of the standing position of the person.

FIG. 20 is a diagram for explaining correction processing by the person difference correction unit. As described above, the purchasing motion determination unit 124a detects a take-out motion when the position of the product enters the take-out region R1, and then detects execution of the scanning operation when the position of the product enters the take-out region R2. do. For example, the length between the joint points of the person (especially the length of the arm from the elbow to the wrist) and the position where the normal picking operation is performed and the position where the normal scanning operation is performed depending on the standing position of the person is approximately determined. Therefore, the boundary line between the extraction region R1 and the extraction region R2, which is practically used for distinguishing between the extraction operation and the scanning operation, is appropriately set according to the length between predetermined joint points of the person in the photographed image and the standing position of the person. By doing so, it becomes possible to perform accurate motion determination.

A captured image 261 shown in FIG. 20 illustrates an extraction area R1 and a take-out area R2. A boundary line L2 between the take-out area R1 and the take-out area R2 is set horizontally on the captured image 261 so as to separate the customer and the cash register 50 from each other. The person difference correction unit 163 corrects the position of the boundary line L2 in the vertical direction according to the length between predetermined joint points of the person and the standing position of the person.

For example, the individual difference correction unit 171 calculates the arm length of the person based on the skeleton information detected by the skeleton information extraction unit 123 . If the person's arm is long, the vertical range of the correct take-out action may be widened downward (in the direction of the cash register 50). Therefore, the individual difference correction unit 171 corrects the boundary line L2 downward on the image when the person's arm is longer than a predetermined threshold, and moves the boundary line L2 upward on the image when the person's arm is equal to or less than the threshold. Correct the direction. The correction amount may be determined, for example, according to the difference between the arm length and the threshold.

Also, the person position correction unit 172 detects the standing position of the person in the captured image 261 . For example, a person's standing position can be detected from the person's position information detected by the image feature extraction unit 122 . Alternatively, if the skeleton information extraction unit 123 has detected the joint points of the person's feet (for example, ankles, knees, etc.), the positions of the joint points can be detected as the person's standing position.

The lower the person's standing position on the image, the wider the vertical range in the downward direction (in the direction of the cash register 50) when the product is taken out correctly. Therefore, when the y-coordinate (vertical coordinate) indicating the standing position of the person is larger than the predetermined reference vertical coordinate (when positioned downward), the human position correction unit 172 moves the boundary line L2 downward on the image. , and if the y-coordinate indicating the standing position of the person is equal to or less than the reference vertical coordinate, the boundary line L2 is corrected upward on the image. The correction amount may be determined, for example, according to the difference between the y-coordinate indicating the standing position of the person and the reference vertical coordinate.

Next, processing of the monitoring device 100a according to the fourth embodiment will be described using a flowchart.
21 and 22 are examples of flowcharts showing determination processing procedures based on determination rules. In FIGS. 21 and 22, it is determined whether or not a correct purchase operation has been performed. Along with this, the number of times the bar code scanning operation is recognized by the determination is compared with the actual number of times the scanning operation is performed in the cash register 50, and notification or warning processing is performed according to the comparison result. executed.

[Step S61] First, as preprocessing for the determination processing, processing for setting correction parameters used in the environment difference correction unit 162 is executed. At this time, a plurality of markers are placed at predetermined positions within the imaging range, and the imaging range including each marker is captured by the camera 101 . When the image acquisition unit 121 acquires the captured image, the environment difference correction unit 162 compares the positional relationship of each marker on the captured image with the reference value of the positional relationship of each marker registered in the determination rule storage unit 161. Thus, a correction parameter for correcting the captured image is determined. As correction parameters, for example, an image enlargement/reduction ratio, an image rotation angle, and the like are determined. Environmental difference correction unit 162 stores the determined correction parameters in a storage device such as HDD 113 .

Note that when the shooting environment information indicating the shooting environment of the camera 101 is given, the shooting environment information is compared with the reference value of each information included in the shooting environment information without the camera 101 shooting. , the correction parameter is determined.

After the correction parameters are set as described above, the camera 101 starts photographing for executing actual determination processing. At this time, the marker is removed from the imaging range.
[Step S62] The image captured by the camera 101 is corrected by the environmental difference correction unit 162 based on the correction parameters set in step S61, and is input to the image feature extraction unit 122 and the skeleton information extraction unit 123. . After that, every time a captured image is acquired, the environmental difference correction unit 162 corrects the captured image, and the corrected captured image is input to the image feature extraction unit 122 and the skeleton information extraction unit 123 .

[Step S63] The purchasing motion determination unit 124a determines whether the image feature extraction unit 122 has recognized a person and an object (product) interacting with the person. If the person is recognized but the object is not recognized, or if the person is not recognized, a waiting state is entered, and the process of step S63 is re-executed for the next frame. On the other hand, if the person and the object are recognized, the process proceeds to step S64.

The processing of the following steps S64 to S69 is executed for each frame.
[Step S<b>64 ] The person difference correction unit 163 corrects the determination rule information stored in the determination rule storage unit 161 based on the skeleton information detected by the skeleton information extraction unit 123 . For example, as described with reference to FIG. 20, the ranges of the extraction area R1 and the take-out area R2 are adjusted according to the length between predetermined joint points of the person and the standing position of the person.

[Step S65] The purchasing motion determination unit 124a uses the determination rule information corrected in step S64 to count the number of products. In this process, the number of times the purchase operation determination unit 124a determines that the scanning operation has been performed correctly is counted as the product score. Details of the processing in step S65 will be described in detail later with reference to FIG.

[Step S66] The purchase operation determination unit 124a counts the number of times the bar code of the product has been scanned at the cash register 50 as the number of scan points. In this process, when the customer operation recognition unit 164 recognizes that the barcode has been scanned, the number of scan points is counted up.

[Step S67] The purchasing motion determination unit 124a determines whether the product score is less than the scan score. If the product score is less than the scan score, the process proceeds to step S68, and if the product score and the scan score match, the process proceeds to step S69.

[Step S68] The purchasing motion determination unit 124a notifies the customer to perform a correction operation. For example, a voice is output indicating that there is an error. Alternatively, display information indicating that there is an error is displayed on the display 52 of the cash register 50 . Then, the correction operation by the customer is performed. For example, an input is made by the customer to the cash register 50 to redo the scan operation. In addition, the customer picks up again the product that was held in the hand immediately before. For example, when the customer operation recognition unit 164 detects an input operation for redoing the scanning operation, the purchasing operation determination unit 124a counts down (increments) the scan score. After that, the process proceeds to step S64.

[Step S69] The purchase action determination unit 124a determines whether the customer performs the settlement process. For example, when the customer operation recognition unit 164 detects that the customer has performed an operation for requesting settlement processing on the cash register 50, it is determined that the customer will perform the settlement processing. If it is not determined that the customer will perform the settlement process, the process proceeds to step S64. On the other hand, if it is determined that the customer will perform the settlement process, the process proceeds to step S71 in FIG.

The description will be continued below with reference to FIG.
[Step S71] The cash register 50 executes the settlement process. For example, the display 52 of the cash register 50 displays the total amount of the scanned item.

[Step S72] The purchasing motion determination unit 124a determines whether the product score is less than the scan score. If the product score is less than the scan score, the process proceeds to step S73. On the other hand, if the product score and the scan score match, the settlement process is continued, and when the settlement process is completed (that is, the payment is completed), the process proceeds to step S75.

[Step S73] The purchasing motion determination unit 124a executes a process of warning that the number of scanned products is insufficient. For example, warning information may be displayed on display 52 of cash register 50 . Alternatively, a sound of warning information is output to the earphone worn by the store clerk.

[Step S74] Correction operation is performed by the customer. For example, the customer makes an input to the cash register 50 to redo the scanning operation, and the settlement process is requested again. When the settlement process is completed, the process proceeds to step S75.

[Step S75] It is determined whether or not to end the determination process. When continuing the determination process, the process proceeds to step S63 in FIG. 21, and a new person and object recognition process is executed. On the other hand, when ending the determination process, the process ends.

FIG. 23 is an example of a flow chart showing the procedure for counting the number of scan points. The processing of FIG. 23 corresponds to the processing of step S65 of FIG. In addition, in the processing of FIG. 23, information (object information) indicating a newly recognized object is sequentially recorded in the RAM 112 . Flag information indicating whether or not scanning has been completed is added to this object information.

[Step S81] The purchasing motion determination unit 124a determines whether the center position of the object region (bounding box) is within the extraction region R1. If the center position is within the extraction region R1, the process proceeds to step S82, and if the center position is not within the extraction region R1, the process proceeds to step S86.

[Step S82] The purchasing motion determination unit 124a determines whether the object determined to be in the take-out region R1 in step S81 has already been recorded. A recorded object indicates that the object information about the object is recorded in the RAM 112 as the latest object information. If the object has already been recorded, the state in which the object is located within the extraction area R1 has continued from the previous frame, and the processing in FIG. 23 ends. On the other hand, if the object is not recorded, the process proceeds to step S83.

[Step S83] The purchasing motion determination unit 124a newly registers in the RAM 112 the object information of the object determined to be in the removal area R1 in step S81. At this time, flag information indicating that scanning has not been executed is added to the object information.

[Step S84] The purchasing motion determination unit 124a selects the object information registered immediately before the object information newly registered in step S83 (that is, the object information immediately preceding the object information that was newly registered in step S83). object information). Based on the flag information added to the object information, the purchasing operation determination unit 124a determines whether the object corresponding to the object information has not yet been scanned. If the object has not been scanned, the process proceeds to step S85, and if the object has been scanned, the process of FIG. 23 ends.

[Step S85] The case where it is judged as Yes in step S84 is the case where the previous object has moved out of the take-out area R1 without moving from the take-out area R1 to the take-out area R2. In this case, there is a possibility that the product was taken out without being scanned. Therefore, the purchasing motion determination unit 124a executes, for example, a process for warning the customer that scanning was not performed correctly. For example, a sound is output indicating that the scan was not performed correctly. Alternatively, display information is displayed on the display 52 of the cash register 50 indicating that the scan was not performed correctly. After that, the processing in FIG. 23 ends.

It should be noted that in step S85, for example, information indicating that an abnormal purchase operation has been performed may be recorded in the RAM 112 instead of issuing a warning. In this case, based on the recorded information, if it is determined in step S67 of FIG. 21 or step S72 of FIG. good.

[Step S86] The purchasing motion determination unit 124a determines whether the center position of the object region (bounding box) is within the bring-out region R2. In this process, if the center position is within the take-out area R2 and the corresponding object is already recorded in the RAM 112, the process proceeds to step S87. In this case, it means that the object has moved from the take-out area R1 to the take-out area R2. On the other hand, if the center position is outside the take-out area R2, or if the center position is inside the take-out area R2 but the object information of the corresponding object is not recorded in the RAM 112, the process of FIG. 23 ends. do.

[Step S87] The purchasing motion determination unit 124a counts up the number of products.
[Step S88] The purchasing motion determination unit 124a records the object as a scanned object. That is, the purchasing action determination unit 124a updates the flag information added to the object information of the object to indicate that the object has been scanned.

According to the fourth embodiment described above, a product related to a person is detected, and based on the position of this product, it is determined whether or not the correct purchasing action was performed. Along with this, the determination rule information used for determining the purchasing behavior is optimized based on the skeleton information detection result. As a result, it is possible to accurately determine whether or not a person has made a correct purchase operation without using a template or learning data for each product, and regardless of the person's physique or the position of the person in the shooting range.

It should be noted that the processing functions of the devices (for example, the motion determination device 1 and the monitoring device 100) described in each of the above embodiments can be realized by a computer. In that case, a program describing the processing contents of the functions that each device should have is provided, and the above processing functions are realized on the computer by executing the program on the computer. A program describing the processing content can be recorded in a computer-readable recording medium. Computer-readable recording media include magnetic storage devices, optical disks, semiconductor memories, and the like. Magnetic storage devices include hard disk drives (HDD) and magnetic tapes. Optical discs include CDs (Compact Discs), DVDs (Digital Versatile Discs), Blu-ray Discs (BD, registered trademark), and the like.

When distributing a program, for example, portable recording media such as DVDs and CDs on which the program is recorded are sold. It is also possible to store the program in the storage device of the server computer and transfer the program from the server computer to another computer via the network.

A computer that executes a program stores, for example, a program recorded on a portable recording medium or a program transferred from a server computer in its own storage device. The computer then reads the program from its own storage device and executes processing according to the program. The computer can also read the program directly from the portable recording medium and execute processing according to the program. Also, the computer can execute processing according to the received program every time the program is transferred from a server computer connected via a network.

REFERENCE SIGNS LIST 1 motion determination device 2 photographed image 3a person 3b object 3c reader 4a, 4b image area 5a, 5b, 6a, 6b skeleton line

Claims (15)

to the computer,
Acquire a photographed image of a person's movement,
Detecting the position of an object related to the person and the skeleton information of the person from the acquired captured image,
Based on the position of the object and the skeleton information, it is determined whether or not the action performed by the person with respect to the object is normal.
Action discrimination program that causes processing to be executed. In the above determination,
calculating a first feature amount based on the position of the object detected from the first frame in the captured image and the skeleton information;
calculating a predicted value of the feature amount in the second frame by inputting the first feature amount into a predictor that predicts the feature amount in the second frame following the first frame;
calculating a second feature amount based on the position of the object detected from the second frame and the skeleton information;
Determining whether the action performed by the person on the object is normal based on the distance between the second feature amount and the predicted value;
2. The motion discrimination program according to claim 1. The predictor is generated by learning using the first feature amount detected from each of a plurality of past images taken in the past as learning data,
3. The motion discrimination program according to claim 2. each of the plurality of past images is an image obtained by photographing a state in which the action performed by the person on the object was performed normally;
4. The motion discrimination program according to claim 3. In the determination, whether or not the person holds the object in his/her hand and causes the scanner mounted on the cash register to read the product information attached to the object is determined normally.
5. The motion discrimination program according to any one of claims 1 to 4. In the above determination,
calculating a first feature value based on the position of the object detected from the captured image and the skeleton information;
By inputting the first feature amount to the classifier, identifying whether the action performed by the person on the object is normal action or one of a plurality of types of abnormal actions;
3. The motion discrimination program according to claim 2. The discriminator includes the first feature quantity detected from each of a plurality of past images taken in the past, and the normal motion and the plurality of types of abnormal motion added to each of the plurality of past images. Generated by learning using a label indicating one of as learning data,
7. The motion discrimination program according to claim 6. In the above determination,
calculating a first feature amount vector based on the position of the object and a second feature amount vector for each joint of the person based on the skeleton information;
calculating the first feature amount by integrating the first feature amount vector and the second feature amount vector for each joint;
8. The motion discrimination program according to any one of claims 2, 3, 4, 6 and 7. area information indicating a plurality of image areas to be set in the captured image is stored in a storage unit of the computer;
In the above determination,
correcting the set positions of the plurality of image regions in the captured image based on the skeleton information;
Determining whether or not the action performed by the person with respect to the object is abnormal based on the time-series relationship between the plurality of image regions whose set positions are corrected and the position of the object;
2. The motion discrimination program according to claim 1. In the determination, whether or not the person holds the object in his/her hand and scans the product information attached to the object with a scanner mounted on the cash register is performed normally.
10. The motion discrimination program according to claim 9. Two image areas arranged adjacent to each other in the direction of the cash register from an area in which the person can be positioned in the photographed image are set as the plurality of image areas,
In the correction of the setting position,
calculating a distance between two specific joints of the person in the captured image based on the skeleton information;
repositioning a border between the two image regions based on the distance;
11. The motion discrimination program according to claim 10. to the computer;
counting a first number of times that the scan operation was determined to have been performed normally by the determination and a second number of times that the scan operation was notified by the cash register;
outputting predetermined warning information when the first number of times and the second number of times do not match;
further processing,
12. The motion discrimination program according to claim 10 or 11. to the computer;
detecting a type of interaction between the person and each of the one or more objects when the photographed image includes one or more objects with which the person interacts;
identifying the object related to the person among the one or more objects based on the detected type;
2. The motion discrimination program according to claim 1, further causing a process to be executed. the computer
Acquire a photographed image of a person's movement,
Detecting the position of an object related to the person and the skeleton information of the person from the acquired captured image,
Based on the position of the object and the skeleton information, it is determined whether or not the action performed by the person with respect to the object is normal.
operation determination method. Acquire a photographed image of a person's movement,
Detecting the position of an object related to the person and the skeleton information of the person from the acquired captured image,
a processing unit that determines whether or not an action performed by the person with respect to the object is normal based on the position of the object and the skeleton information;
A motion discrimination device having

Images